Workspace content layout

Omnia keeps per-workspace files on one shared volume: Arena project repositories, per-user working clones, exported snapshots, synced skills and the manifests that agents read. Most of it can be rebuilt. A small part cannot, and the directory names do not say which. This page explains the layout so you can tell authored work from derived content. The layout reference lists every path, writer and retention rule.

One volume, one writer per directory

Each workspace namespace has its own tree at /workspace-content/{workspace}/{namespace}/, and every directory in it has one component that writes it. The content falls into three groups:

• Authored: the bare git repository of each ArenaProject, and unpushed changes in a working clone.
• Derived: project snapshots, synced SkillSource and ArenaTemplateSource versions, template indexes and skill manifests. Each is produced from something else and is produced again when lost.
• Sandbox-owned: scratch and session directories that a sandbox pod writes.

The diagram shows which component acts on each shared directory.

flowchart LR
    Dashboard -->|content API| Operator
    Operator -->|create clone, edit, source control| WS["arena/workspaces/{project}/{user}"]
    Operator -->|push to a hosted project| Repos["arena/repos/{project}.git"]
    Operator -->|SkillSource sync, PromptPack manifests| Skills["skills/ and manifests/"]
    Arena["arena-controller"] -->|seed, fetch| Repos
    Arena -->|delete or flag idle clones| WS
    Operator -->|export a commit at run or deploy start| Sources
    Arena -->|export head and run commits| Sources["arena/sources/{project}/.arena/versions/{sha}"]
    Workers["Arena worker pods"] -->|read-only subPath| Sources
    Sandbox["Builder sandbox pod"] -->|read-write bind| WS
    Runtime["Agent runtime container"] -->|read-only| Skills

Neither the dashboard nor management-api mounts the volume. management-api reads and writes for the dashboard through the operator’s authenticated content API, which confines each request to the workspace’s namespace. Agent pods mount their own workspace’s PVC, and on a shared volume only their workspace’s subtree of it.

Two ways to address the same files

The operator mounts the whole share. A pod mounts a subPath of it, and the right subPath depends on how the storage is provisioned.

On a shared volume, every workspace PVC points at the same directory of the share, so a pod’s subPath carries the {workspace}/{namespace} prefix and isolation is a mount-time convention. On a scoped volume (workspaceContent.scoped), the storage layer gives each workspace namespace a volume rooted at its own subtree, and the subPath is relative to it. Adding the prefix there would point at a directory that does not exist. Scoped mode enforces isolation in the storage system.

Recorded content paths such as ArenaRun.status.source.contentPath are relative to the workspace subtree in both modes. Only the pod-side subPath differs. Isolate workspace content covers the admission-policy backstop.

How the content API confines a request

The operator is the one writer that can reach every workspace’s subtree, so each request is held by four safeguards before it touches the volume:

• Identity: for each request management-api mints a short-lived RS256 token with the audience omnia-operator. The operator verifies it against management-api’s JWKS endpoint. A token minted for another audience, such as the facade’s, fails.
• Role: the operator recomputes the caller’s role from the Workspace resource and never trusts a role claim in the token. The token supplies who the caller is and which workspace it is scoped to.
• Path: the operator rejects an absolute path and any .. that climbs out of the workspace subtree. It then resolves the path through a root opened on that subtree, which refuses a symlink that leads out of it, an absolute symlink included.
• Owner: the operator writes new files and directories as UID 65532, the same owner as the other server-side writers, so the layout stays uniform.

The operator and the arena controller run as non-root users: UID 65532 on the distroless images, with runAsNonRoot: true in the chart. The dashboard image is nginx running as UID 101, and management-api runs as UID 65532 on a distroless image; neither mounts the volume.

Every pod that writes the workspace content volume (the operator, the arena controller, arena workers, agents and sandboxes) runs as UID and GID 65532 with fsGroup: 65532 and fsGroupChangePolicy: OnRootMismatch. Kubelet then re-owns the volume only when its root does not already carry GID 65532, instead of walking every file on every pod start, which on a busy shared volume took tens of seconds. The policy is only safe while every writer uses that UID and GID: a writer running as another would leave files the others cannot change. Change the chart’s podSecurityContext for one of them only together with the rest, or set fsGroupChangePolicy: Always.

Why an Arena project is a git repository

A run needs to say exactly what it tested, so an ArenaProject is a git repository and a run resolves its ref to a commit SHA when it starts.

A hosted repository is a bare repository on the volume with one branch, main. Omnia never merges, so the hosted shape is kept to what needs no merge tool. An external project keeps its repository on a git host. The copy under arena/repos/ is a mirror the controller fetches every pollInterval, so runs of external and hosted projects read the same kind of local object store. The remote is the authoritative copy. Repository kind is immutable, because an external project force-fetches and prunes its mirror, which would destroy a hosted project’s history.

Why every user edits a private clone

Nobody edits a repository directly. Each user works in a clone at arena/workspaces/{project}/{user}, where {user} is the pseudonym from the caller’s verified token. A shared working copy would mix every user’s uncommitted edits, and it could not be cloned with one user’s git-host credential. Private clones trade disk for isolation. A hosted clone copies the repository’s object store, so it holds the whole history. An external clone comes from the remote with that user’s own git-host credential and never from the mirror, which was fetched with the project’s read credential and may hold what the user cannot see. External clones therefore need git-host Connections, an Enterprise feature.

The content API names a project’s files as arena/projects/{project}/…, an alias for the caller’s own clone. The real trees under arena/workspaces/ and arena/repos/ cannot be named in a request, nor can a clone’s .git directory. A sandbox bound to a clone can plant symlinks in it, so the API opens each request relative to the clone, and a planted link cannot lead to another user’s clone or to a repository.

Edits stay private until pushed. A run executes a commit of the repository, never a clone, so unpushed work exists in one place only.

Judging idle clones

A clone can hold the only copy of someone’s work, so age alone never expires it. The operator stamps status.workspaces[].lastActivity on the project when a user writes a file, runs a source-control verb that changes the clone, or starts or ends a builder sandbox on it. Reads do not count.

After the idle period (14 days by default), the arena controller asks git, under the clone’s index lock, what the clone holds that its repository lacks. A clone that holds nothing is deleted, and opening the project clones it afresh. A clone with uncommitted changes, untracked files that no .gitignore covers, a commit its upstream lacks or a branch with no upstream is flagged and kept. Using the clone clears the flag.

The judgement and the delete must not race a write, or the write is deleted with the clone. A file write through the content API takes the same index lock, and after waiting for it checks that the clone it opened is still the one at that path. A write that lost the race is refused with 409 rather than reported saved. A builder sandbox writes to the clone without the lock, so a clone that a live builder sandbox mounts is not judged at all, whatever its stamp says.

The controller requeues for the moment the oldest stamp runs out rather than scanning the volume on a timer.

Why runs read snapshots

A run reads one commit’s tree, exported to arena/sources/{project}/.arena/versions/{sha} and keyed by the full SHA. The export goes to a temporary sibling and is renamed into place, so a reader never sees half a snapshot, and two exporters racing for one commit end with the same directory.

The arena controller exports a run’s commit when it pins the run, and exports the default branch’s tip after every seed, sync or fetch, recording it as status.repository.head. Anything that shows a project’s contents reads that head. The operator also exports a commit when it starts a run or a deploy, so that bindings and the compiled pack come from what will execute. The SHA is the version.

A snapshot is a cache of a commit the repository still holds, so it can be removed. After each head export the arena controller keeps the head, every snapshot an unfinished run pins, the ten most recently used others and anything younger than an hour. A run records its snapshot only after exporting it, and the grace period covers that gap. A finished run’s contentPath may name a directory that has gone. The commit is the record, and a new run of it exports the snapshot again.

A deploy compiles its snapshot at once and never reads it again, so nothing deployed pins one. A hosted project is swept when pushed to, so runs of old commits can add snapshots until the next push or controller restart.

A project deleted and recreated under the same name starts with an empty repository and no snapshots. The repository and the snapshot tree each record the project’s UID, so a new project discards what an old one left. The layout reference also covers where a project’s display name comes from.

Why synced versions keep the version HEAD names

SkillSource and ArenaTemplateSource content goes through one syncer. It writes content-addressed versions under .arena/versions/ and a HEAD file naming the one being served. After each sync, versions beyond the ten most recent are removed.

HEAD is not always the newest. A sync whose content matches a version already on disk points HEAD back at it. A sweep by age alone would then delete the content the source is serving, and a later sync cannot restore it, because it writes a new version. So the version HEAD names is exempt, and a source whose HEAD is old holds one directory more than the cap. If HEAD cannot be read, nothing is removed. The cap is fixed in the operator and the arena controller, with no chart value.

Why sandboxes mount a clone

A codegen sandbox normally gets scratch space that nothing else reads. A sandbox claimed on a configured mount role gets a bind into a directory instead, read-write unless its SandboxClass says otherwise. The chart configures one role, project, which binds the declaring user’s own clone, arena/workspaces/{id}/{user}. The builder’s tools edit the files the project editor shows, so there is no second copy to reconcile and no promotion step. The sandbox runs real git in the clone, holds no credentials and never pushes. The user pushes from the dashboard through the operator, which refuses a hosted push that carries a commit authored by someone else.

Mounting a project is a privilege only the operator can grant. An agent’s ServiceAccount has no permissions on SandboxClaim. The chart installs a ValidatingAdmissionPolicy that refuses, for every identity but the operator’s ServiceAccount, to create a claim that mounts anything but scratch space (or binds a Connection, names a targetPath or is an enrolment) and to change any claim’s spec. The SandboxClaim controller checks a privileged claim against the policy once, before its first bind, and fails one the operator did not write with MountNotGranted; a claim that is Verified or already Bound is not judged again. Without the policy (Kubernetes older than 1.30), every sandbox gets scratch space.

The session declares the mount. The client’s first WebSocket message carries a declare naming the project, and the runtime passes it to the operator’s Sandbox API with a delegation token naming the person. The operator authorises it for that person, who must be verified and hold at least the role’s minRole (editor for project) in the workspace. A role’s path may not reach trees that other components own, such as arena/repos/ and skills/, because a sandbox can write a symlink into any directory it mounts. SandboxClaim lists the checks and refusal reasons.

Releasing or reaping a project claim ends the pod, not the project. The runtime remembers every (session, handle) it has seen bound to any mount other than scratch, a project mount included, with the declaration that asked for it, and acquires those handles with MustExist, so the operator refuses with SandboxEnded instead of creating a scratch sandbox that the agent would mistake for the project. The runtime then creates the claim again with the same mount: it declares the project again for the same user, with a fresh delegation token, or, for the session’s own workspace, asks for it plainly. The new sandbox mounts the same files; only the old pod’s own state is gone. A new claim that would mount anything else is refused.

Skills

Agent pods read skills in two steps. A SkillSource syncs content to skills/{source}/.arena/versions/{version}/. The PromptPack reconciler then resolves the pack’s spec.skills against what the sources have synced and writes manifests/{promptpack}.json, an index of each skill and where its files are. The write is atomic, so a reader never sees a half-written manifest. The manifest records locations relative to the workspace subtree, so one manifest works in either mount mode.

The runtime container mounts the workspace subtree read-only at /workspace-content, and the AgentRuntime reconciler sets OMNIA_PROMPTPACK_MANIFEST_PATH. At startup the runtime registers one PromptKit skill source per manifest entry. The mount exists only when the operator has a workspace content path and the pack declares a skill, so clusters without skills need no workspace content PVC.

Writing the manifest is best-effort: a failed write is logged. A pack can report SkillsResolved, SkillsValid and SkillToolsResolved healthy while its manifest is missing, and the agent then loads no skills until a reconcile succeeds. Check that the file exists, not only the conditions.

Concurrent edits

One user can collide with themselves: two tabs, or the editor and their builder sandbox, can have the same file open. The editor does not lock files. Each tab remembers the hash of the version it loaded and sends it as a precondition on save, and the content API answers 412 with the current hash on a mismatch. The check and the write happen under a per-path lock, so two saves cannot both pass the check. The content API runs on every operator replica, so the lock is a file on the shared volume rather than one held in memory: it is created with an exclusive create, which NFS honours, and a lock left by a replica that died is broken after 30 seconds. A delete or a move takes the same lock, so neither can slip between a save’s check and its write.

Trade-offs

• The volume needs storage that supports many writers (ReadWriteMany), because the operator, the arena controller and many pods mount it.
• Per-user clones cost disk, and flagged clones cost it until their users return. Unpushed work stays out of other users’ clones and out of runs.
• The operator and the arena controller must agree on the mount mode, or pods get a doubled path prefix.

Back up arena/repos/, and arena/workspaces/ if unpushed work matters.

Related

• Layout reference: every path, writer, reader and retention rule
• ArenaProject, SandboxClaim and SkillSource
• Isolate workspace content
• Troubleshoot workspace content
• Multi-tenancy architecture