Controller reconciliation

Omnia’s controllers decide what state each resource is in and write that state to status. This page explains how the AgentRuntime, PromptPack and ToolRegistry controllers get there, and why they are built around watches and not polling.

Reconciliation pattern

Omnia follows the Kubernetes controller reconciliation pattern. A controller watches resources for changes and adds each changed resource to a work queue. It then reconciles each queued resource toward its desired state and requeues the resource when a step fails transiently.

The design leans on the watch. A controller that watches every object its decision depends on never needs a timer to notice that something changed, so most Omnia reconcilers return without a requeue interval. When a reconcile is blocked on a Provider that is not Ready, a Provider that lacks a required capability, or a PromptPack that failed validation, it records the reason in a condition and returns without an error. The watch on that object brings the reconcile back when the object changes. A reference to an object that does not exist is different: the reconciler returns an error, and controller-runtime retries it with backoff (see Error handling).

AgentRuntime controller

An AgentRuntime depends on more objects than any other kind: a PromptPack, optionally a ToolRegistry, one or more Providers, plus Secrets, the Workspace and several others. The controller watches all of them and maps each change back to the AgentRuntimes that reference it. It also owns the Deployment, Service and HorizontalPodAutoscaler it creates, so a change to those re-triggers it too.

Reconciliation flow

The controller resolves references first and builds nothing until every input is usable.

flowchart TD
    A[AgentRuntime or a watched dependency changes] --> B[Resolve PromptPack]
    B -->|no match, invalid ref or invalid pack| F[Phase Failed]
    B -->|valid| C{toolRegistryRef set?}
    C -->|Yes| D[Fetch ToolRegistry]
    C -->|No| E[Resolve Providers]
    D -->|Found| E
    D -->|Not found| N[ToolRegistryReady false, continue without tools]
    N --> E
    E -->|not Ready or missing capability| P[Phase Pending]
    E -->|not found or role mismatch| F
    E -->|usable| G[Create or update Deployment and Service]
    G --> H{Ready replicas > 0?}
    H -->|Yes| R[Phase Running]
    H -->|No| P

A missing ToolRegistry does not fail the agent. The reference is optional, so the controller records ToolRegistryReady=False and builds the Deployment without tools. A missing PromptPack or Provider is different: an agent without them cannot serve a conversation, so the phase is Failed or Pending and no Deployment is built from a half-resolved spec.

The phase says what the controller expects next:

• Failed is set for a promptPackRef that matches no pack, an invalid promptPackRef, a pack that failed schema validation, a Provider that does not exist, and a Provider whose role does not match. It is also set for a providerRef naming another namespace.
• Pending is set for a Provider that is not Ready and for a Provider that lacks a required capability.

Failed does not mean permanent. A promptPackRef that matches no pack or a providerRef to a missing Provider returns an error and is retried with backoff, so the agent leaves Failed once the object is created. An invalid promptPackRef, a failed pack and a role mismatch return no error and wait for an edit to the spec or the referenced object. See the AgentRuntime reference for the phase and condition values.

The controller gathers every input to the pod template before writing the Deployment. Kubernetes hashes the pod template, so an input that arrives one reconcile late would replace a pod that had only started. Waiting costs less than a second pod lifecycle.

What the controller builds

The Deployment and Service, their environment variables, volumes and status fields are listed in the AgentRuntime reference.

PromptPack controller

A PromptPack is an immutable version of a pack’s content. The controller’s job is to decide whether that version is usable and whether it is the version its pack serves on its channel. It does not roll anything out to agents, and it holds no traffic weights; rollout lives on the AgentRuntime, under spec.rollout, and is driven by the AgentRuntime controller.

Reconciliation flow

The controller watches PromptPacks and the ConfigMaps they point at, and re-reconciles older sibling versions when a newer one changes phase.

flowchart TD
    A[PromptPack or its ConfigMap changes] --> B{Live source?}
    B -->|Yes| L[SourceValid from spec alone, Phase Active]
    B -->|No| C[Validate source: ConfigMap with pack.json]
    C -->|Invalid| F[Phase Failed, SourceValidationFailed event]
    C -->|Valid| D[Validate pack.json against schema]
    D -->|Invalid| F2[Phase Failed, SchemaValidationFailed event]
    D -->|Valid| E[Record content digest, resolve skills]
    E --> G[List sibling versions of the packName]
    G -->|Highest on its channel| H[Phase Active, activeVersion set]
    G -->|Otherwise| I[Phase Superseded]
    H --> J[Set AgentsNotified condition]
    I --> J

The AgentsNotified condition reports how many AgentRuntimes reference the pack. The controller does not push anything to them. Each AgentRuntime has its own watch on PromptPacks and picks up a new version by itself.

Validation

Validate source and Validate pack.json against schema in the diagram are two phases, and each has its own condition. Source validation checks that the referenced ConfigMap exists and contains a pack.json key. Schema validation checks that pack.json conforms to the published PromptPack JSON Schema.

Keeping them apart means a missing ConfigMap and a malformed pack produce different conditions and different events, so the failure points at the thing to fix. A failure in either phase emits a Kubernetes event carrying the validator’s own error message, not a generic one:

kubectl describe promptpack my-prompts

# Events:
#   Warning  SchemaValidationFailed  invalid pack.json: <validator errors, joined with "; ">

A source failure clears the schema condition to Unknown, since there is nothing to validate.

Versions and channels

Several PromptPack objects can share one packName, one per published version. The controller marks a version Active when it is the highest on the stable or the prerelease channel, and Superseded otherwise. A newer version that has not been validated yet cannot supersede a working older one, so publishing a broken version leaves the running version in place. Phase transitions on a newer sibling re-enqueue the older ones, which is how they converge to Superseded once the newer version validates.

Two versions that differ only in build metadata have equal semver precedence, so the one created later serves the channel and the other reports DuplicateVersion. The PromptPack reference lists the phases and conditions.

Rollout of a new pack version

Moving an agent from one pack version to another is the AgentRuntime controller’s work: it resolves the promptPackRef, runs the rollout engine for spec.rollout, and records progress in status.rollout. Putting rollout on the AgentRuntime rather than the pack lets two agents pin or track the same pack differently. The AgentRuntime reference describes the strategies.

ToolRegistry controller

Reconciliation flow

The controller reads handler definitions; it does not look up Services or read annotations. Handlers that carry an explicit tool definition contribute that tool. Self-describing handlers contribute a placeholder, because their real tools are only known to the runtime when it connects.

flowchart TD
    A[ToolRegistry spec changes] --> B[Validate each handler]
    B -->|Errors| X[Phase Failed, HandlersValid false]
    B -->|Valid| C{Handler type}
    C -->|http, grpc, client| D[Use the explicit tool definition]
    C -->|mcp, openapi| E[Add a placeholder entry]
    C -->|platform| P[Add the platform service's tool list]
    D --> F[Optional endpoint probe]
    E --> F
    P --> F
    F --> G{Available tools}
    G -->|All| H[Phase Ready]
    G -->|Some| I[Phase Degraded]
    G -->|None| J[Phase Failed]
    H --> K[Write discoveredTools]
    I --> K
    J --> K

Because discovery reads the spec and not the cluster, the controller reconciles on spec changes only. It writes its own status on every pass, and without that filter its own writes would re-enqueue it in a loop. It additionally watches ToolPolicies, SandboxClasses and Connections, because the PolicyBound, SandboxesValid and SandboxConnectionsValid conditions are judged against them.

What “Ready” means

A Ready phase means the configuration is valid and the handlers produced tools, not that the backends are up. Degraded appears only when probing is enabled and finds some endpoints unreachable. This is a trade-off: probing every endpoint on a timer makes the status truthful but adds load and a requeue loop, so it is off by default. The ToolRegistry reference lists the phases.

Error handling

The controllers split failures into two kinds, and the split decides whether a requeue helps.

When a call to the API server or another dependency fails, the reconciler returns the error. controller-runtime then requeues the object with its default per-item exponential backoff, so a flapping dependency is retried with growing gaps and not hammered.

When the failure is in the spec, a retry reads the same spec and fails the same way. The AgentRuntime controller records the condition and returns without an error for a promptPackRef that names a live pack in a rollout or a pack name with both live and published versions, a role-mismatched Provider, and a PromptPack that failed schema validation. All three set the phase to Failed. A Provider that is not Ready or lacks a capability returns without an error and sets Pending. A spec edit is itself a watch event, so waiting for one costs nothing, while a timer would only re-read what the watch already delivers.

A reference to an object that does not exist is retried. A promptPackRef that matches no PromptPack, a providerRef to a missing Provider, and a providerRef naming another namespace set Failed and return the error, so controller-runtime retries them with backoff.

Concurrency

The AgentRuntime, ToolRegistry, Workspace, MemoryPolicy, SessionRetentionPolicy and ScheduledFunction controllers each run up to three reconciles at once; the PromptPack controller uses controller-runtime’s default of one. No operator flag changes these values. Each object is reconciled by at most one worker at a time, which is why the code can read an object, change it and write it back without locking.

Requeuing

Success is not requeued by default. Timers appear only where the answer can change without any Kubernetes event:

• ToolRegistry endpoint probing, at the configured probe interval.
• OIDC JWKS refresh for an AgentRuntime with spec.externalAuth.oidc.
• Rollout steps and rollout analysis, which wait on time or on metrics.
• The idle deadline of an AgentRuntime that can be reaped.
• Provider endpoint health checks, every 30 seconds while an endpoint is unreachable.

Everything else waits for a watch event. A reference to an object that does not exist is retried with controller-runtime’s backoff.

Related

• AgentRuntime reference
• PromptPack reference
• ToolRegistry reference