Skip to content

Manage workspaces

Create a workspace for a team, grant people and pipelines access to it, and deploy agents into its namespace.

  • The Omnia operator is installed.
  • kubectl access to the cluster.
  • For production, an identity provider that your dashboard authentication is set up against.
  1. Write a minimal Workspace. The controller creates the namespace because create is true.

    apiVersion: omnia.altairalabs.ai/v1alpha1
    kind: Workspace
    metadata:
    name: my-team
    spec:
    displayName: "My Team"
    namespace:
    name: omnia-my-team
    create: true
  2. Apply it.

    Terminal window
    kubectl apply -f workspace.yaml

    The controller creates the namespace, one ServiceAccount per role (owner, editor, viewer) and the RBAC bindings for them.

  3. Add a service group before you run real agents. Without one, agents in the workspace record no sessions, so token usage, cost and memory are unavailable. See Configure workspace service groups.

The dashboard shows a user only the workspaces on which they resolve to a role. A workspace with no roleBindings, directGrants or anonymousAccess reaches Ready but is visible to nobody. A dashboard user’s role is the highest of the roleBindings[].groups entries that match their groups, an unexpired directGrants entry and anonymousAccess. anonymousAccess is a floor for every signed-in user as well as the role of unauthenticated visitors. ServiceAccount entries in roleBindings give a pipeline Kubernetes access and no dashboard role. See the Workspace reference for how roles resolve.

  1. Map IdP groups to roles. A group name must equal the name in the user’s group claim exactly, including case.

    spec:
    roleBindings:
    - groups:
    - "cs-admins@acme.com"
    role: owner
    - groups:
    - "cs-engineers@acme.com"
    - "support-team"
    role: editor
    - groups:
    - "cs-contractors@acme.com"
    role: viewer
  2. Optional: give a CI/CD ServiceAccount a role. The controller creates a Kubernetes RoleBinding in the workspace namespace for each ServiceAccount, so the pipeline can call the Kubernetes API directly.

    spec:
    roleBindings:
    - serviceAccounts:
    - name: argocd-application-controller
    namespace: argocd
    role: editor
  3. Optional: grant one user a role for a limited time. Use groups for lasting access. The expiry below is illustrative.

    spec:
    directGrants:
    - user: oncall@acme.com
    role: owner
    expires: "2027-01-01T00:00:00Z"
  4. Optional, development only: give unauthenticated visitors a role, and every signed-in user at least that role. Use this when the dashboard runs in anonymous mode.

    spec:
    anonymousAccess:
    enabled: true
    role: viewer

    Never set editor or owner in production: anyone who reaches the dashboard, signed in or not, gets that role.

  1. Set spec.quotas on the Workspace. The controller writes it into a ResourceQuota named workspace-<workspace-name>-quota in the workspace namespace.

    spec:
    quotas:
    compute:
    requests:
    cpu: "50"
    memory: 100Gi
    objects:
    pods: 200
    persistentVolumeClaims: 20
  2. Check that the quota is in force.

    Terminal window
    kubectl get workspace my-team -o jsonpath='{.status.conditions[?(@.type=="ResourceQuotaReady")]}'
    kubectl describe resourcequota workspace-my-team-quota -n omnia-my-team

    ResourceQuotaReady=False with reason ResourceQuotaFailed carries the API server’s error in its message, and the workspace phase is Degraded until it clears. Reason ResourceQuotaConflict means a ResourceQuota of that name, not made by the operator, is already in the namespace. The operator leaves it alone, so delete or rename it.

  3. Give every SandboxClass used in the workspace resources.requests (or limits, which Kubernetes copies to requests). Under a compute quota the API server rejects a pod whose containers have no request for a capped resource. Omnia’s agent and service containers carry default requests.

To remove the quota, delete spec.quotas. The controller deletes the ResourceQuota.

status.quotas.resourceQuota on the Workspace shows the limits and what the namespace uses against them:

Terminal window
kubectl get workspace my-team -o jsonpath='{.status.quotas.resourceQuota}'
  1. Set spec.quotas.agents on the Workspace.

    spec:
    quotas:
    agents:
    maxAgentRuntimes: 20
    maxReplicasPerAgent: 5

    maxAgentRuntimes caps how many AgentRuntimes are admitted, oldest first. maxReplicasPerAgent caps how many replicas each one runs, autoscaled or not.

  2. Check the usage.

    Terminal window
    kubectl get workspace my-team -o jsonpath='{.status.quotas.agents}'
    kubectl get workspace my-team -o jsonpath='{.status.conditions[?(@.type=="AgentQuotaWithinLimits")]}'

    admitted lists the AgentRuntimes that may run, waiting those still waiting for admission, and clamped those running at the replica cap.

An AgentRuntime the count limit has not admitted is not started: it stays Pending with WithinWorkspaceQuota=False, and starts once an older one is deleted or the limit is raised. Lowering maxAgentRuntimes stops nothing that is already admitted.

An AgentRuntime that asks for more replicas than maxReplicasPerAgent, through spec.runtime.replicas or an autoscaler’s maxReplicas, runs at the cap. Lowering the cap scales running agents down to it.

With the admission webhook enabled (webhook.enabled), the API server also refuses a create beyond maxAgentRuntimes, and an explicit spec.runtime.replicas above maxReplicasPerAgent.

  1. Set spec.costControls on the Workspace. The workspace needs a service group with a session-api, because spend is read from it.

    spec:
    costControls:
    dailyBudget: "100.00" # USD, per UTC day
    monthlyBudget: "2000.00" # USD, per UTC calendar month
    budgetExceededAction: pauseJobs # warn | pauseJobs | block
    alertThresholds:
    - percent: 80
    - percent: 95

    Spend counts every provider call the workspace’s agents, try-it sessions and Arena runs made, plus session-less usage such as embeddings, across every service group. Service groups that share a database are counted once only when they name the same database Secret or session-api URL. Two Secrets pointing at one database double the spend, so give such groups one Secret.

  2. Check the spend. The operator reads it about every five minutes.

    Terminal window
    kubectl get workspace my-team -o jsonpath='{.status.costUsage}'
    kubectl get workspace my-team -o jsonpath='{.status.conditions[?(@.type=="WithinBudget")]}'

    dailyPercent and monthlyPercent are the spend as a percent of each budget.

When the day’s or the month’s spend reaches its budget, WithinBudget turns False and the operator emits a Warning event. What else happens depends on budgetExceededAction:

  • warn does nothing more.
  • pauseJobs holds every ArenaRun that has not started. It stays Pending with reason WorkspaceBudgetExceeded. Runs already running finish. Scheduled ArenaJobs skip their ticks. A held run takes no arena quota slot and is not counted against maxJobsPerDay; when the hold lifts, waiting runs are admitted oldest first.
  • block holds ArenaRuns too, and the agents refuse new work. A WebSocket client gets the error code BUDGET_EXCEEDED, an A2A message fails, a function call gets 429 budget_exceeded, and an MCP tool call gets a budget_exceeded tool error. Open conversations carry on. Each agent pod reads its Workspace at most every 30 seconds, and admits all new work when it cannot read it, for example when it has no read permission on the Workspace or the API server is down.

The hold lifts at the next UTC midnight for a daily budget, or the first of the next month for a monthly one, or as soon as you raise the budget.

Each alert threshold emits one CostAlertThreshold Warning event per period, the first time spend reaches it. Alerts are at most once: an event lost to a failed status write is not repeated. The notify addresses are not emailed.

Terminal window
kubectl get events --field-selector involvedObject.name=my-team,reason=CostAlertThreshold

Arena is Enterprise: the arena controller enforces spec.quotas.arena, and a Core install ignores it.

  1. Set spec.quotas.arena on the Workspace.

    spec:
    quotas:
    arena:
    maxConcurrentJobs: 3
    maxJobsPerDay: 50
    maxWorkersPerJob: 4

    Every limit counts ArenaRuns, the executions of an ArenaJob, not ArenaJobs. A manual ArenaJob creates one run; a scheduled one creates a run at every firing.

    • maxConcurrentJobs caps how many runs are admitted and unfinished at once. A run holds its slot from admission until it is Succeeded, Failed or Cancelled. A run that has not started its workers 15 minutes after admission gives its slot back and waits again, behind the other waiting runs.
    • maxJobsPerDay caps how many runs are admitted in any rolling 24 hours, not per calendar day, up to 1000. A run is charged once, when it is admitted, however long it runs.
    • maxWorkersPerJob caps the worker replicas of one run.
  2. Check the usage.

    Terminal window
    kubectl get workspace my-team -o jsonpath='{.status.quotas.arena}'
    kubectl get workspace my-team -o jsonpath='{.status.conditions[?(@.type=="ArenaQuotaWithinLimits")]}'

    running and startedLast24h are the counts the limits apply to. admitted lists the unfinished runs the quota has admitted, waiting those still waiting, in the order they will be admitted, stalled those whose admission was released because they did not start, and clamped those running with fewer workers than they ask for. starts is the record of the last 24 hours of admissions that maxJobsPerDay is counted from. It is kept on the Workspace, so a controller restart, or an ArenaJob pruning its finished runs, does not reset the count. Removing maxJobsPerDay discards it.

No limit fails or stops a run. A run beyond maxConcurrentJobs or maxJobsPerDay waits, Pending with WithinWorkspaceQuota=False. Waiting runs are admitted oldest first, when an earlier run finishes or when the oldest admission is 24 hours old. A scheduled ArenaJob is the exception: when the runs already started or waiting use up the day’s allowance, it skips that firing with a RunSkipped event rather than adding a run that could not start until the window moves on.

A run asking for more workers than maxWorkersPerJob runs at the cap, and its status.workers shows the count it ran with. The cap applies when the run starts, so lowering it never shrinks a running run. Lowering maxConcurrentJobs or maxJobsPerDay stops nothing that is already admitted.

With the arena controller’s licence webhooks enabled, the API server also refuses an ArenaJob whose explicit spec.workers.replicas is above maxWorkersPerJob.

Set the environment tier and the tags in the Workspace spec. The controller writes both onto the workspace namespace as labels.

spec:
environment: production # development | staging | production
defaultTags:
team: "customer-support"
cost-center: "CC-1234"

To restrict traffic to and from the namespace, set spec.networkPolicy.isolate: true. See Configure workspace network isolation.

Create the AgentRuntime in the workspace namespace and select a service group.

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: AgentRuntime
metadata:
name: support-bot
namespace: omnia-my-team
spec:
serviceGroup: default
promptPackRef:
name: support-prompts
providers:
- name: default
providerRef:
name: claude-provider
  1. Read the workspace status. Workspace is cluster-scoped, so there is no -n.

    Terminal window
    kubectl get workspace my-team -o yaml

    status.phase is Ready, status.namespace.name is the namespace, and status.serviceAccounts names workspace-my-team-owner-sa, workspace-my-team-editor-sa and workspace-my-team-viewer-sa.

  2. Confirm management-api serves the workspace to a signed-in user. Open /api/v1/workspaces on the dashboard URL. An empty {"workspaces":[],"count":0} means the user resolves to no role, so check the bindings above.

  3. Switch workspaces in the dashboard with the selector in the header, which reads “Select workspace” until you pick one. It lists only the workspaces you have a role on.

The workspace storage PVC is not bound yet. Read the StorageReady and Ready conditions.

Terminal window
kubectl describe workspace my-team

Read the condition whose status is False. NamespaceReady with reason NamespaceConflict means another Workspace owns the namespace, and a message that the namespace does not exist means spec.namespace.create is false. The operator logs name the failing step.

Terminal window
kubectl logs -n omnia-system deploy/omnia-controller-manager
  1. Read the groups in the user’s token and compare them with roleBindings[].groups. They must match exactly.
  2. Check whether directGrants has an unexpired entry for the user’s email or username.
  3. Check whether anonymousAccess is enabled, if you are on a development install.

A pipeline ServiceAccount is refused by the Kubernetes API

Section titled “A pipeline ServiceAccount is refused by the Kubernetes API”
  1. Confirm the RoleBinding exists in the workspace namespace.

    Terminal window
    kubectl get rolebindings -n omnia-my-team
  2. Confirm the ServiceAccount name and namespace in roleBindings match the pipeline’s.

Terminal window
kubectl describe resourcequota -n omnia-my-team

Raise the limit in spec.quotas (or spec.hard on a ResourceQuota of your own), or remove unused resources.

An agent stays Pending with WithinWorkspaceQuota=False

Section titled “An agent stays Pending with WithinWorkspaceQuota=False”
Terminal window
kubectl get agentruntime support-bot -n omnia-my-team \
-o jsonpath='{.status.conditions[?(@.type=="WithinWorkspaceQuota")].message}'

AgentRuntimeQuotaExceeded means the Workspace has not admitted it: maxAgentRuntimes older AgentRuntimes are admitted. Delete one or raise the limit, and the agent starts. ReplicasClamped does not hold an agent; it only says it runs at the cap.

An ArenaRun stays Pending with WorkspaceBudgetExceeded

Section titled “An ArenaRun stays Pending with WorkspaceBudgetExceeded”

The workspace is over a budget and its budgetExceededAction is pauseJobs or block. Read the spend and the condition.

Terminal window
kubectl get workspace my-team -o jsonpath='{.status.costUsage}'
kubectl get workspace my-team -o jsonpath='{.status.conditions[?(@.type=="WithinBudget")].message}'

Raise the budget or wait for the period to roll over, and the run starts within a minute.

Read the CostUsageAvailable condition. NoSessionAPI means no service group has a session-api, so no spend is recorded. SessionAPIUnavailable means the last read failed; the operator log line workspace spend read failed carries the error. A WithinBudget reason of InvalidBudget means a budget is not a positive number.

An Arena run stays Pending with WithinWorkspaceQuota=False

Section titled “An Arena run stays Pending with WithinWorkspaceQuota=False”
Terminal window
kubectl get arenarun nightly-eval-x7k2p -n omnia-my-team \
-o jsonpath='{.status.conditions[?(@.type=="WithinWorkspaceQuota")].message}'

WaitingForConcurrentRunSlot means maxConcurrentJobs runs are running: the run starts when one of them finishes. WaitingForDailyRunQuota means maxJobsPerDay runs were admitted in the last 24 hours: the message gives the time the next slot opens. WaitingForQuotaAdmission means the quota has not counted the run yet, which takes a moment after it is created. AdmissionReleased means the run was admitted but did not start its workers within 15 minutes, so it gave its slot back: read its other conditions (SourceValid, JobCreated) for why, and fix that. Cancel the run (spec.cancelled: true) to take it out of line. WorkersClamped does not hold a run; it only says the run executes at the worker cap.

For network, content, project and skill problems, see Configure workspace network isolation and Troubleshoot workspace content.