azure-ps-test-runner-vm

Azure test runner VM

A plain Ubuntu VM in the same Azure region as the HorizonDB server. It runs the full all-providers suite from a clone of the repo. It is not a GitHub Actions runner; for that, see the Manage Azure CI runner workflow and deploy/providers/azure/ci/runner/README.md.

Why

The HorizonDB phase makes many short database calls in a row. Distance decides whether it passes:

Where the tests run	One SELECT 1	Result
A laptop on another continent	130–160 ms	Most HorizonDB SDK tests hit their time limits
A VM in the server's region	0–1 ms	The full suite passes

Measure it first (step 6) when in doubt.

What you need

• Azure CLI signed in, with rights to create a resource group and a VM.
• The repo's local test config files: .env, .env.horizondb and .model_providers.json. They hold credentials (a GitHub token, model keys, the HorizonDB connection). See "Credentials" below before copying them.

Set these once in your shell. Use your own values:

SUB=<subscription-id>
RG=<resource-group>            # a new, dedicated group, e.g. pilotswarm-testrunner-rg
VM=<vm-name>                   # e.g. ps-testrunner
LOC=<horizondb-region>         # the region of your HorizonDB server
KEY=~/.ssh/id_ed25519_ps_testrunner
MYIP=$(curl -s https://api.ipify.org)

1. SSH key and setup script

[ -f "$KEY" ] || ssh-keygen -q -t ed25519 -N '' -C 'pilotswarm test runner' -f "$KEY"

cat > cloud-init.yaml <<'EOF'
#cloud-config
package_update: true
packages: [git, build-essential, python3, jq, ca-certificates, curl, docker.io, unzip]
runcmd:
  - curl -fsSL https://deb.nodesource.com/setup_24.x | bash -
  - apt-get install -y nodejs
  - usermod -aG docker azureuser
  - systemctl enable --now docker
EOF

2. Create the VM

Use 8 cores. The suite runs 8 test files at once, each with its own workers, plus PostgreSQL in Docker. On 4 cores the tests compete for CPU and hit their time limits.

az group create --subscription "$SUB" -n "$RG" -l "$LOC" -o none
az vm create --subscription "$SUB" -g "$RG" -n "$VM" \
  --image Canonical:ubuntu-24_04-lts:server:latest \
  --size Standard_D8s_v5 --os-disk-size-gb 128 --storage-sku Premium_LRS \
  --admin-username azureuser --ssh-key-values "$KEY.pub" --authentication-type ssh \
  --public-ip-sku Standard --nsg-rule NONE --custom-data cloud-init.yaml \
  --query publicIpAddress -o tsv

Write down the IP it prints.

3. SSH from your IP only, and a nightly shutdown

NSG=$(az network nsg list --subscription "$SUB" -g "$RG" --query '[0].name' -o tsv)
az network nsg rule create --subscription "$SUB" -g "$RG" --nsg-name "$NSG" \
  -n ssh-from-me --priority 1000 --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes "$MYIP/32" --destination-port-ranges 22 -o none
az vm auto-shutdown --subscription "$SUB" -g "$RG" -n "$VM" --time 0200 -o none   # UTC

Add an alias to ~/.ssh/config:

Host ps-testrunner
    HostName <vm-ip>
    User azureuser
    IdentityFile ~/.ssh/id_ed25519_ps_testrunner
    IdentitiesOnly yes
    StrictHostKeyChecking accept-new
    ServerAliveInterval 30

Then wait for the setup script: ssh ps-testrunner 'cloud-init status --wait; node --version; docker --version'.

If SSH times out while the VM runs, check the security rules: az network nsg rule list -g "$RG" --nsg-name "$NSG" -o table. Some organizations add their own deny rule for port 22 from the internet, with a higher priority than yours. Do not edit those rules. Connect from the network they allow, or drive the VM with Run Command (see "Without SSH").

4. The repo

ssh ps-testrunner 'git clone -q https://github.com/microsoft/PilotSwarm.git pilotswarm
  cd pilotswarm && git checkout -q -B <branch> origin/<branch> && npm ci --no-audit --no-fund'

To test another commit later: git fetch origin <branch> && git checkout -B <branch> origin/<branch> && npm ci.

5. PostgreSQL

The container's password must match DATABASE_URL in the .env you copy. run-tests.sh refuses to run with fewer than 1500 connections.

ssh ps-testrunner 'docker run -d --name pilotswarm-pg --restart unless-stopped \
    -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=<password-from-DATABASE_URL> \
    -e POSTGRES_DB=pilotswarm -p 127.0.0.1:5432:5432 postgres:16 \
  && sleep 5 \
  && docker exec pilotswarm-pg psql -U postgres -qc "ALTER SYSTEM SET max_connections = 1500;" \
  && docker restart pilotswarm-pg'

6. Credentials: copy for one run, delete after

The three config files hold real credentials. Keep them on the VM only while a run needs them:

1. Copy them right before the run.
2. In the same step, start a watcher that deletes them when the run ends.
3. Never leave them between runs. Ask the person before copying them to a machine that did not hold them before.

With SSH:

scp .env .env.horizondb .model_providers.json ps-testrunner:pilotswarm/
ssh ps-testrunner 'cd pilotswarm && chmod 600 .env .env.horizondb .model_providers.json
  grep -q "^HORIZON_DATABASE_URL=." .env.horizondb && echo "hdb config: ok"'

Check the HorizonDB latency once:

ssh ps-testrunner 'cd pilotswarm && node --env-file=.env.horizondb -e "
const pg = require(\"pg\");
(async () => {
  const u = new URL(process.env.HORIZON_DATABASE_URL);
  u.searchParams.delete(\"sslmode\"); u.searchParams.delete(\"uselibpqcompat\");
  const c = new pg.Client({ connectionString: u.toString(), ssl: { rejectUnauthorized: false } });
  await c.connect();
  const t = []; for (let i = 0; i < 5; i++) { const s = Date.now(); await c.query(\"SELECT 1\"); t.push(Date.now() - s); }
  console.log(\"SELECT 1 ms:\", t.join(\", \")); await c.end();
})().catch((e) => console.log(\"error:\", e.code || e.message));"'

7. Run the suite

Start it detached, so it keeps going when your laptop sleeps. Then start the watcher that deletes the credentials when the run ends. PS_TEST_SKIP_STALE_CLEANUP=1 skips the global sweep of old test schemas, so the run cannot drop schemas another run on the same HorizonDB still uses.

ssh ps-testrunner 'cd pilotswarm
  PS_TEST_SKIP_STALE_CLEANUP=1 setsid nohup bash -c "./scripts/run-tests.sh --all-providers > ~/all-providers.log 2>&1; echo EXIT=\$? >> ~/all-providers.log" > /dev/null 2>&1 < /dev/null &
  sleep 3
  setsid nohup bash -c "while pgrep -f \"[r]un-tests.sh --all-providers\" > /dev/null; do sleep 30; done; shred -u ~/pilotswarm/.env ~/pilotswarm/.env.horizondb ~/pilotswarm/.model_providers.json; echo \"credentials removed \$(date -u)\" >> ~/secrets-cleanup.log" > /dev/null 2>&1 < /dev/null &
  echo started'

Progress and result:

ssh ps-testrunner 'sed "s/\x1b\[[0-9;]*m//g" ~/all-providers.log | grep -aE "Provider phase|Test Files|Tests  |Combined provider result|EXIT="'

A full run takes about 35–40 minutes: the stock PostgreSQL pass (about 22), then the HorizonDB phase. To run only some files on HorizonDB: ./scripts/run-tests.sh --with-horizondb --suite=<name> --suite=<name>.

Without SSH: Run Command

Use az vm run-command invoke for commands that carry no secret (status, logs, pkill). It returns the output, but Azure keeps the script, so never put credentials in it.

To copy the credential files, use Run Command v2 with protected parameters. Azure stores them encrypted and never returns them:

az vm run-command create --subscription "$SUB" -g "$RG" --vm-name "$VM" \
  --name copy-and-run-$(date +%H%M%S) --location "$LOC" --timeout-in-seconds 600 \
  --script "$(cat vm-run.sh)" \
  --protected-parameters "PSENV=$(base64 < .env | tr -d '\n')" \
                         "PSHDB=$(base64 < .env.horizondb | tr -d '\n')" \
                         "PSMP=$(base64 < .model_providers.json | tr -d '\n')"
az vm run-command delete --subscription "$SUB" -g "$RG" --vm-name "$VM" --name <that-name> --yes

• Do not pass --run-as-user: with it, the parameters do not reach the script. Run as root, and inside the script write the files, then chown azureuser:azureuser and chmod 600 them.
• The parameters arrive as environment variables: printf '%s' "$PSENV" | base64 -d > .env.
• Start the run as the user: sudo -u azureuser -H setsid nohup <script> &.
• Delete the run command afterwards.

Gotchas

• pkill -f run-tests.sh over SSH kills your own SSH command, whose text contains the pattern. Use pkill -f "[r]un-tests.sh".
• zsh reads $name:a as a path modifier. "image-$svc:tag" becomes a file path. Write "image-${svc}:tag".
• Two HorizonDB tests sit close to their 60 s limit: the first test of delete-cascade and of pg-migrator sets up a fresh schema (about 55 s). Under the full parallel load they can time out. Re-run them alone with --with-horizondb --suite=delete-cascade --suite=pg-migrator before calling it a failure.
• Which keys the run uses:
  • Model calls go through the test catalog (packages/sdk/test/fixtures/model-providers.test.json), that is GitHub Copilot with GITHUB_TOKEN.
  • The HorizonDB phase also calls the embedding endpoint set by the HORIZON_EMBED_* values in .env.horizondb.
• A test that prints a database URL must hide the password. packages/sdk/test/helpers/local-env.js does; keep it that way, because the run log stays on the VM.

Tear down

az group delete --subscription "$SUB" -n "$RG" --yes --no-wait

This deletes the VM, its disk, IP and network, and any credentials still on it. Remove the SSH alias and key if you will not create the VM again.