Skip to content
ALTAIRA LABS
DocsBlog
OmniaOmnia

Run an Arena load test

Enterprise

Run an ArenaJob of type loadtest to replay a project’s scenarios at controlled concurrency against providers or agents, and fail the run when it breaks your SLO thresholds.

Prerequisites

Section titled “Prerequisites”
  • Arena is an Enterprise feature: the app chart has enterprise.enabled=true, and any license you install covers load testing. See Install with a License.
  • A Ready ArenaProject whose arena config holds the scenarios to replay.
  • A Provider, or a deployed AgentRuntime, in the same namespace to load.

Start a load test from the dashboard

Section titled “Start a load test from the dashboard”
  1. Open the project and select Run this project.
  2. In the dialog, select Load test. The dialog shows concurrency, virtual users, workers, duration, arrivals per second, ramp up and down, a budget and any number of thresholds.
  3. Tick the providers to load. To load a deployed agent, tick it under Deployed agents; agents are off by default.
  4. Select Run.

A field you leave blank is omitted from the job, so the defaults apply. Arrivals / s stays disabled until you enter a duration.

After a run finishes, select Rerun profile on its page to repeat the load profile, SLO gates, budget and session recording.

The remaining steps write the same job as YAML, which is the form CI uses.

Create the job

Section titled “Create the job”

  1. Write the job. The values here are examples; replace the names and numbers with your own.

    apiVersion: omnia.altairalabs.ai/v1alpha1
    kind: ArenaJob
    metadata:
      name: checkout-loadtest
      namespace: default
    spec:
      sourceRef:
        name: checkout-config
      type: loadtest
      trials: 500
      providers:
        default:
          - providerRef:
              name: gpt4-prod
      workers:
        replicas: 4
      loadTest:
        concurrency: 20
        vusPerWorker: 8
        thresholds:
          - metric: latency_p95
            operator: "<"
            value: "3s"
          - metric: error_rate
            operator: "<"
            value: "0.01"

    The job type defaults to evaluation, so set type: loadtest. The volume is scenarios × providers × trials, and spec.trials overrides any per-scenario trial count.

  2. Choose what to load. Each providers entry is a providerRef or an agentRef:

    providers:
      default:
        - agentRef:
            name: checkout-agent

    An agentRef entry drives the agent’s facade over WebSocket and is the only target that measures time-to-first-token. A providerRef entry measures total duration only.

  3. Size the load. concurrency caps the work items in flight across all workers, and the cap is divided among the worker pods. Set workers.replicas × loadTest.vusPerWorker to at least concurrency, or the virtual users become the limit. Both settings default to 1, which runs sequentially.

  4. To hold the load for a length of time instead of draining a fixed matrix, add a duration:

    loadTest:
      concurrency: 20
      duration: "10m"

    With a duration, the workers generate requests from the scenario × provider pool until the clock runs out, and trials is ignored.

  5. To offer a fixed arrival rate instead of holding a fixed concurrency, add rate to a job that has a duration:

    loadTest:
      concurrency: 50
      duration: "10m"
      rate: 20

    concurrency becomes a safety ceiling. Arrivals the workers cannot keep up with are dropped and counted, never queued; gate on them with an arrival_lag threshold.

  6. To ramp the load, add a ramp block with Go duration strings:

    loadTest:
      ramp:
        up: "2m"
        down: "30s"

  7. Gate on SLOs with loadTest.thresholds. Each rule has a metric, an operator (<, >, <=, >=) and a value. Latency and time-to-first-token values are durations ("3s", "500ms"), rate metrics are ratios from 0 to 1 ("0.01"), and total_cost is a number ("50.00"). See the ArenaJob CRD page for every metric and what it needs.

  8. Cap spend with budgetLimit at the top of spec, not under loadTest:

    spec:
      budgetLimit: "25.00"
      budgetCurrency: "USD"

    The value is a bare decimal with no currency symbol. When the run’s accumulated cost crosses the limit, the run stops, discards the queued work and fails. The cap is a ceiling, so requests already in flight finish.

    A budgetLimit written under spec.loadTest is pruned by the API server without an error, and the run has no cap.

  9. To store the conversations as sessions, set sessionRecording: true. It is false by default, which keeps session-api out of the path of a high-volume run. A datagen run records its conversations regardless, because its sessions are its corpus. In a workspace with several service groups, name the one to record into with spec.serviceGroup.

  10. Apply the job:

    Terminal window
    kubectl apply -f checkout-loadtest.yaml

Verify

Section titled “Verify”

The ArenaJob creates an ArenaRun for the execution. Its phase and results are on the run, not on the job.

Terminal window
kubectl get arenarun
kubectl get arenarun <run-name> -o yaml

Find <run-name> in the first command, or in the job’s status.lastRunRef. In status.result.summary, each threshold appears as a threshold:<metric> entry and thresholds_passed counts them. The run reaches Succeeded when every threshold passes. An enumerated run (no duration) also fails when any item fails.

See Monitor Arena jobs for progress and results.

Troubleshooting

Section titled “Troubleshooting”

A threshold reports unavailable and the run passes

Section titled “A threshold reports unavailable and the run passes”

A metric that could not be measured passes. A latency percentile needs a latency distribution, which a run keeps when it has a duration or when concurrency or vusPerWorker is above 1. A ttft_* metric also needs an agentRef entry, and arrival_lag needs loadTest.rate. Check that a rule you rely on in CI reports a number.

A threshold reports parse error and the run fails

Section titled “A threshold reports parse error and the run fails”

The value cannot be read for its metric, for example "3 seconds" or "$25". Use a duration, a ratio from 0 to 1, or a plain number as listed in step 7.

The run fails with BudgetExceeded

Section titled “The run fails with BudgetExceeded”

The run’s cost crossed budgetLimit. status.result.summary holds totalCost and drainedItems, the number of queued items discarded. Raise the limit or reduce the load.

The run fails with ServiceGroupAmbiguous

Section titled “The run fails with ServiceGroupAmbiguous”

The workspace has several service groups and the job names none. Set spec.serviceGroup.

The run stays Pending with JobCreationFailed

Section titled “The run stays Pending with JobCreationFailed”

With sessionRecording: true, the named service group does not exist or declares no session-api, so the controller cannot create the worker Job and retries with backoff. Fix spec.serviceGroup or the workspace’s service groups.

Section titled “Related pages”