Skip to content

Configure pod overrides

Set podOverrides on an AgentRuntime, a Workspace service group or an ArenaJob to change the pods the operator generates for it.

  • The CRD that generates the pod, and permission to edit it.
  • For a CSI secret store: the CSI driver and a SecretProviderClass in the pod’s namespace.
  • For workload identity: a ServiceAccount that is federated with your cloud.

podOverrides is accepted in these fields. podOverrides lists the pod each one affects and how every field merges.

GoalField
Agent pod (facade and runtime)AgentRuntime spec.podOverrides
session-apiWorkspace spec.services[].session.podOverrides
memory-apiWorkspace spec.services[].memory.podOverrides
Eval-workerWorkspace spec.services[].evalWorker.podOverrides
Eval-worker for the agent’s service groupAgentRuntime spec.evals.podOverrides (the Workspace field takes precedence)
Arena workersArenaJob spec.workers.podOverrides
  1. Add podOverrides to the service group’s session block. This example uses Azure workload identity and the Secrets Store CSI driver. The values are illustrative.

    spec:
    services:
    - name: default
    mode: managed
    session:
    database:
    secretRef:
    name: session-db
    podOverrides:
    serviceAccountName: workload-identity-sa
    annotations:
    azure.workload.identity/use: "true"
    extraVolumes:
    - name: kv-secrets
    csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
    secretProviderClass: session-db-spc
    extraVolumeMounts:
    - name: kv-secrets
    mountPath: /mnt/secrets-store
    readOnly: true
    extraEnvFrom:
    - secretRef:
    name: session-db # created by the SecretProviderClass secretObjects
  2. Apply the Workspace.

    Terminal window
    kubectl apply -f workspace.yaml

Run an agent on GPU nodes with its own identity

Section titled “Run an agent on GPU nodes with its own identity”
  1. Add podOverrides to the AgentRuntime. This example uses AWS IRSA, an NVIDIA node label and a private registry. The values are illustrative.

    apiVersion: omnia.altairalabs.ai/v1alpha1
    kind: AgentRuntime
    metadata:
    name: vision-agent
    spec:
    promptPackRef:
    name: vision
    facades:
    - type: websocket
    podOverrides:
    serviceAccountName: vision-irsa
    nodeSelector:
    nvidia.com/gpu.product: A100
    tolerations:
    - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
    imagePullSecrets:
    - name: ecr-pull
  2. Apply it.

    Terminal window
    kubectl apply -f vision-agent.yaml

To set affinity, use spec.runtime.affinity. podOverrides has no affinity field.

  1. Add podOverrides under spec.workers:

    apiVersion: omnia.altairalabs.ai/v1alpha1
    kind: ArenaJob
    metadata:
    name: eval-run-1
    spec:
    sourceRef:
    name: my-arena-source
    workers:
    replicas: 5
    podOverrides:
    extraVolumes:
    - name: provider-keys
    csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
    secretProviderClass: arena-providers
    extraEnvFrom:
    - secretRef:
    name: arena-provider-keys
  2. Apply the ArenaJob.

Find the Deployment for the workload, then read the ServiceAccount and volumes the operator rendered.

Terminal window
kubectl get deployments -n <namespace>
kubectl get deployment <name> -n <namespace> \
-o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}{.spec.template.spec.volumes[*].name}{"\n"}'

The output lists your serviceAccountName and the volume names you added. For an ArenaJob, inspect a worker pod with kubectl get pod <pod> -o yaml.

Operator-set labels win on key collision because Service selectors depend on them. Use a different label key.

An environment variable is missing from the agent pod

Section titled “An environment variable is missing from the agent pod”

The operator drops reserved keys such as OMNIA_PACK_MODE from extraEnv and sets the ReservedEnvClean condition to False.

Terminal window
kubectl describe agentruntime <name>

The agent runs as the wrong ServiceAccount

Section titled “The agent runs as the wrong ServiceAccount”

An AgentRuntime’s ServiceAccount is podOverrides.serviceAccountName, else the Workspace’s spec.runtime.serviceAccountName, else <agentruntime>-facade.