# Controller reconciliation Omnia's controllers decide what state each resource is in and write that state to `status`. This page explains how the AgentRuntime, PromptPack and ToolRegistry controllers get there, and why they are built around watches and not polling. ## Reconciliation pattern Omnia follows the Kubernetes controller reconciliation pattern. A controller watches resources for changes and adds each changed resource to a work queue. It then reconciles each queued resource toward its desired state and requeues the resource when a step fails transiently. The design leans on the watch. A controller that watches every object its decision depends on never needs a timer to notice that something changed, so most Omnia reconcilers return without a requeue interval. When a reconcile is blocked on a Provider that is not `Ready`, a Provider that lacks a required capability, or a PromptPack that failed validation, it records the reason in a condition and returns without an error. The watch on that object brings the reconcile back when the object changes. A reference to an object that does not exist is different: the reconciler returns an error, and controller-runtime retries it with backoff (see [Error handling](#error-handling)). ## AgentRuntime controller An AgentRuntime depends on more objects than any other kind: a PromptPack, optionally a ToolRegistry, one or more Providers, plus Secrets, the Workspace and several others. The controller watches all of them and maps each change back to the AgentRuntimes that reference it. It also owns the Deployment, Service and HorizontalPodAutoscaler it creates, so a change to those re-triggers it too. ### Reconciliation flow The controller resolves references first and builds nothing until every input is usable. ```mermaid flowchart TD A[AgentRuntime or a watched dependency changes] --> B[Resolve PromptPack] B -->|no match, invalid ref or invalid pack| F[Phase Failed] B -->|valid| C{toolRegistryRef set?} C -->|Yes| D[Fetch ToolRegistry] C -->|No| E[Resolve Providers] D -->|Found| E D -->|Not found| N[ToolRegistryReady false, continue without tools] N --> E E -->|not Ready or missing capability| P[Phase Pending] E -->|not found or role mismatch| F E -->|usable| G[Create or update Deployment and Service] G --> H{Ready replicas > 0?} H -->|Yes| R[Phase Running] H -->|No| P ``` A missing ToolRegistry does not fail the agent. The reference is optional, so the controller records `ToolRegistryReady=False` and builds the Deployment without tools. A missing PromptPack or Provider is different: an agent without them cannot serve a conversation, so the phase is `Failed` or `Pending` and no Deployment is built from a half-resolved spec. The phase says what the controller expects next: - `Failed` is set for a `promptPackRef` that matches no pack, an invalid `promptPackRef`, a pack that failed schema validation, a Provider that does not exist, and a Provider whose role does not match. It is also set for a `providerRef` naming another namespace. - `Pending` is set for a Provider that is not `Ready` and for a Provider that lacks a required capability. `Failed` does not mean permanent. A `promptPackRef` that matches no pack or a `providerRef` to a missing Provider returns an error and is retried with backoff, so the agent leaves `Failed` once the object is created. An invalid `promptPackRef`, a failed pack and a role mismatch return no error and wait for an edit to the spec or the referenced object. See the [AgentRuntime reference](/reference/core/agentruntime/) for the phase and condition values. The controller gathers every input to the pod template before writing the Deployment. Kubernetes hashes the pod template, so an input that arrives one reconcile late would replace a pod that had only started. Waiting costs less than a second pod lifecycle. ### What the controller builds The Deployment and Service, their environment variables, volumes and status fields are listed in the [AgentRuntime reference](/reference/core/agentruntime/). ## PromptPack controller A PromptPack is an immutable version of a pack's content. The controller's job is to decide whether that version is usable and whether it is the version its pack serves on its channel. It does not roll anything out to agents, and it holds no traffic weights; rollout lives on the AgentRuntime, under `spec.rollout`, and is driven by the AgentRuntime controller. ### Reconciliation flow The controller watches PromptPacks and the ConfigMaps they point at, and re-reconciles older sibling versions when a newer one changes phase. ```mermaid flowchart TD A[PromptPack or its ConfigMap changes] --> B{Live source?} B -->|Yes| L[SourceValid from spec alone, Phase Active] B -->|No| C[Validate source: ConfigMap with pack.json] C -->|Invalid| F[Phase Failed, SourceValidationFailed event] C -->|Valid| D[Validate pack.json against schema] D -->|Invalid| F2[Phase Failed, SchemaValidationFailed event] D -->|Valid| E[Record content digest, resolve skills] E --> G[List sibling versions of the packName] G -->|Highest on its channel| H[Phase Active, activeVersion set] G -->|Otherwise| I[Phase Superseded] H --> J[Set AgentsNotified condition] I --> J ``` The `AgentsNotified` condition reports how many AgentRuntimes reference the pack. The controller does not push anything to them. Each AgentRuntime has its own watch on PromptPacks and picks up a new version by itself. ### Validation `Validate source` and `Validate pack.json against schema` in the diagram are two phases, and each has its own condition. Source validation checks that the referenced ConfigMap exists and contains a `pack.json` key. Schema validation checks that `pack.json` conforms to the [published PromptPack JSON Schema](https://promptpack.org/schema/latest/promptpack.schema.json). Keeping them apart means a missing ConfigMap and a malformed pack produce different conditions and different events, so the failure points at the thing to fix. A failure in either phase emits a Kubernetes event carrying the validator's own error message, not a generic one: ```bash kubectl describe promptpack my-prompts # Events: # Warning SchemaValidationFailed invalid pack.json: ``` A source failure clears the schema condition to `Unknown`, since there is nothing to validate. ### Versions and channels Several PromptPack objects can share one `packName`, one per published version. The controller marks a version `Active` when it is the highest on the stable or the prerelease channel, and `Superseded` otherwise. A newer version that has not been validated yet cannot supersede a working older one, so publishing a broken version leaves the running version in place. Phase transitions on a newer sibling re-enqueue the older ones, which is how they converge to `Superseded` once the newer version validates. Two versions that differ only in build metadata have equal semver precedence, so the one created later serves the channel and the other reports `DuplicateVersion`. The [PromptPack reference](/reference/core/promptpack/) lists the phases and conditions. ### Rollout of a new pack version Moving an agent from one pack version to another is the AgentRuntime controller's work: it resolves the `promptPackRef`, runs the rollout engine for `spec.rollout`, and records progress in `status.rollout`. Putting rollout on the AgentRuntime rather than the pack lets two agents pin or track the same pack differently. The [AgentRuntime reference](/reference/core/agentruntime/#rollout) describes the strategies. ## ToolRegistry controller ### Reconciliation flow The controller reads handler definitions; it does not look up Services or read annotations. Handlers that carry an explicit tool definition contribute that tool. Self-describing handlers contribute a placeholder, because their real tools are only known to the runtime when it connects. ```mermaid flowchart TD A[ToolRegistry spec changes] --> B[Validate each handler] B -->|Errors| X[Phase Failed, HandlersValid false] B -->|Valid| C{Handler type} C -->|http, grpc, client| D[Use the explicit tool definition] C -->|mcp, openapi| E[Add a placeholder entry] C -->|platform| P[Add the platform service's tool list] D --> F[Optional endpoint probe] E --> F P --> F F --> G{Available tools} G -->|All| H[Phase Ready] G -->|Some| I[Phase Degraded] G -->|None| J[Phase Failed] H --> K[Write discoveredTools] I --> K J --> K ``` Because discovery reads the spec and not the cluster, the controller reconciles on spec changes only. It writes its own status on every pass, and without that filter its own writes would re-enqueue it in a loop. It additionally watches ToolPolicies, SandboxClasses and Connections, because the `PolicyBound`, `SandboxesValid` and `SandboxConnectionsValid` conditions are judged against them. ### What "Ready" means A `Ready` phase means the configuration is valid and the handlers produced tools, not that the backends are up. `Degraded` appears only when [probing](/reference/core/toolregistry/#probing) is enabled and finds some endpoints unreachable. This is a trade-off: probing every endpoint on a timer makes the status truthful but adds load and a requeue loop, so it is off by default. The [ToolRegistry reference](/reference/core/toolregistry/) lists the phases. ## Error handling The controllers split failures into two kinds, and the split decides whether a requeue helps. When a call to the API server or another dependency fails, the reconciler returns the error. controller-runtime then requeues the object with its default per-item exponential backoff, so a flapping dependency is retried with growing gaps and not hammered. When the failure is in the spec, a retry reads the same spec and fails the same way. The AgentRuntime controller records the condition and returns without an error for a `promptPackRef` that names a live pack in a rollout or a pack name with both live and published versions, a role-mismatched Provider, and a PromptPack that failed schema validation. All three set the phase to `Failed`. A Provider that is not `Ready` or lacks a capability returns without an error and sets `Pending`. A spec edit is itself a watch event, so waiting for one costs nothing, while a timer would only re-read what the watch already delivers. A reference to an object that does not exist is retried. A `promptPackRef` that matches no PromptPack, a `providerRef` to a missing Provider, and a `providerRef` naming another namespace set `Failed` and return the error, so controller-runtime retries them with backoff. ## Concurrency The AgentRuntime, ToolRegistry, Workspace, MemoryPolicy, SessionRetentionPolicy and ScheduledFunction controllers each run up to three reconciles at once; the PromptPack controller uses controller-runtime's default of one. No operator flag changes these values. Each object is reconciled by at most one worker at a time, which is why the code can read an object, change it and write it back without locking. ## Requeuing Success is not requeued by default. Timers appear only where the answer can change without any Kubernetes event: - ToolRegistry endpoint probing, at the configured probe interval. - OIDC JWKS refresh for an AgentRuntime with `spec.externalAuth.oidc`. - Rollout steps and rollout analysis, which wait on time or on metrics. - The idle deadline of an AgentRuntime that can be reaped. - Provider endpoint health checks, every 30 seconds while an endpoint is unreachable. Everything else waits for a watch event. A reference to an object that does not exist is retried with controller-runtime's backoff. ## Related - [AgentRuntime reference](/reference/core/agentruntime/) - [PromptPack reference](/reference/core/promptpack/) - [ToolRegistry reference](/reference/core/toolregistry/)