Arena architecture

Arena is Omnia’s distributed testing framework for evaluating PromptKit bundles at scale. This page explains how a project, a job and a run fit together, why every result is tied to a git commit, and what the dev agent behind Try it does.

Overview

Arena provides:

• Distributed execution: workers share the scenario × provider matrix through a work queue.
• Git-based sources: a project is a hosted git repository, optionally seeded once from a ConfigMap, or an external https git remote.
• Revision tracking: every run records the commit it executed.
• Result aggregation: each run keeps its own phase, progress and result summary.
• A unified provider model: LLM providers and agents are interchangeable entries in the evaluation matrix.

Core concepts

PromptKit bundles

Arena tests PromptKit bundles - structured collections of prompts with versioning, templating, and parameter definitions. A bundle lives in the files of an ArenaProject.

Projects

An ArenaProject is a git repository of PromptKit content. The repository is either hosted in the cluster, on the workspace content volume, or an external https remote that the controller mirrors. The dashboard’s New project page offers a gallery of starting points for a new project.

The diagram shows where a project’s commits come from and how a run turns one of them into the files its workers read.

flowchart LR
    Ext["External remote<br/>https git host"] -->|mirrored| Repo
    Seed["ConfigMap seed<br/>first commit only"] -->|once| Repo
    Repo["Project repository<br/>arena/repos/PROJECT.git"] -->|run resolves a ref to a commit| Snap["Snapshot<br/>.arena/versions/SHA"]
    Snap --> Workers

The controller fetches an external remote on its poll interval (five minutes by default), and a hosted repository changes only through Omnia. After every seed, Sync or fetch the controller exports the default branch’s tip as the project’s head, which is what management-api reads for the dashboard to show what the project contains.

When a run starts, it resolves its job’s ref (the default branch when the job names none) to a commit SHA and exports that commit to a snapshot keyed by the SHA. It then records the SHA on the run and mounts the snapshot into its workers. Only commits in the repository can run: work nobody has committed and pushed is never executed.

Providers and agents

An ArenaJob references providers: the LLM backends or agents that execute prompts during a test run. Providers are organized into named groups (for example default and judge). Group names correspond to the arena config’s provider groups, the group: value on each providers: entry in the arena config file.

Each entry in a provider group is either:

• A providerRef pointing to a Provider CRD (direct LLM access).
• An agentRef pointing to an AgentRuntime CRD (the worker connects via WebSocket).

An agent takes the same place in the scenario × provider matrix as an LLM provider. Entries in an array-mode group form the matrix. Map-mode groups, which hold judges and self-play roles, do not add entries to it. There is no separate execution path for agents: an agent is another provider entry.

Jobs and runs

An ArenaJob is a template. It names the project through sourceRef, optionally a branch, tag or commit through ref, and maps provider groups to Provider or AgentRuntime CRDs. Each execution is an ArenaRun that the ArenaJob controller creates from a frozen copy of the job’s spec, either on the job’s schedule or by hand.

The split keeps results apart. A job’s status describes scheduling (active runs, the last run, the next scheduled time), and each run holds its own phase, progress and result. A scheduled job therefore never overwrites the previous run’s result, and editing a job never changes what an earlier run was told to do.

Architecture

The diagram shows which controller does what between a project’s repository and a run’s results.

flowchart TB
    SC["ArenaProject<br/>Controller"] -->|seed, fetch, export head| Repo["Project repository<br/>hosted or mirrored"]
    JC["ArenaJob<br/>Controller"] -->|creates| Run["ArenaRun"]
    RC["ArenaRun<br/>Controller"] -->|reconciles| Run
    RC -->|resolve ref, export commit| Repo
    RC -->|push work items| Q["Work Queue<br/>Redis"]
    W["Workers<br/>Pods"] -->|pull work items| Q
    W --> Res["Results"]

The ArenaRun controller partitions the scenario × provider matrix into work items and pushes them to the queue. Workers pull items from the queue, and the run aggregates what they report into its status.

See the ArenaProject and ArenaJob references for the YAML of each resource and every field. To follow a run, read the ArenaRun: kubectl get arenarun shows its phase, and the run’s status holds progress and the result summary.

Commit tracking

Every run records the commit it executed on status.source: the repository, the ref it asked for, and the SHA that ref resolved to. Snapshots are keyed by that SHA, so a run re-executed against the same SHA reads the same files, and a result can always be traced to the commit that produced it.

GitOps integration

Developers push prompt changes to the project’s repository. An ArenaJob, on a schedule or started by hand, runs against the branch or tag it names, and each run resolves that ref to the commit it finds. Results name that commit.

The diagram shows the loop from a push to a recorded result.

flowchart LR
    Dev[Developer] -->|git push| Repo[Project repository]
    Job[ArenaJob] -->|creates| Run[ArenaRun]
    Run -->|resolve ref| Repo
    Run --> Results

A deploy of an external project may only name refs matching spec.deploy.allowedRefs: the default branch and tags unless you widen it.

Trade-offs

Pinning every run to a commit makes results reproducible and traceable, and the price is that a prompt edit has to be committed and pushed before a run can test it. A job that names a branch tests whatever the branch held when each run started, and a job that names a SHA tests that commit every time.

A hosted repository suits a team with no git host. It has a single branch, main, so it needs no merging. The repository kind, hosted or external, is fixed once the project exists. An external remote gives you branches, tags and your own review process, at the cost of an https remote the controller must be able to fetch, plus a read Secret for a private repository.

Trying a project (the dev agent)

The project editor has three modes: Files, Builder and Try it. Try it chats with the project’s current source rather than with a commit, and it does not use the Arena workers. It deploys the source to one hidden dev agent, an ordinary AgentRuntime created through the deploy-intent API and reachable only on the management plane, and chats with it the way the agent console chats with any agent.

The runtime compiles the project when a session opens, so a prompt edit reaches the next session without a redeploy. The dev agent is deleted after 30 minutes unused, and the next visit to the project deploys it again.

Next steps

• ArenaProject CRD Reference: complete spec details
• ArenaJob CRD Reference: job and run details
• Try a project’s dev agent: where Try it runs