Schedule functions

A ScheduledFunction fires a function-mode AgentRuntime on a cron schedule with pre-defined parameters. It is the right shape for recurring work that has no caller: a nightly digest, an hourly sync, a periodic sweep of a queue.

apiVersion: omnia.altairalabs.ai/v1alpha1
kind: ScheduledFunction
metadata:
  name: daily-digest
  namespace: ws-acme
spec:
  functionRef:
    name: daily-digest          # a mode: function AgentRuntime in this namespace
  schedule:
    cron: "0 6 * * *"
    timezone: "Europe/London"
    concurrencyPolicy: Forbid   # Allow | Forbid
  input:
    region: emea                # frozen parameters, validated against the function's inputSchema
  runAs:
    systemUser: {}

Every fire is recorded like any other function invocation: one sessions row, tagged function, with its messages, tool calls and token usage.

Carrying state between runs

Most recurring work needs to know where it got to last time. Declare a cursor: a slice of the function’s output that is carried into the next run’s input.

spec:
  input:
    region: emea
  state:
    cursorFrom: "/cursor"       # JSON Pointer into the fire's output
    cursorInto: "/cursor"       # JSON Pointer into the next fire's input
    initial:                    # used before the first successful fire
      since: "2026-01-01T00:00:00Z"
    store:
      configMap: {}             # defaults to a ConfigMap named <schedule>-state

Your function receives {"region":"emea","cursor":{"since":"..."}} and returns a new cursor in its output. Omnia stores it and splices it into the next run.

The cursor is a resumption pointer, never a system of record

This is the one rule to internalise. Legitimate cursor contents are a watermark timestamp, a last-processed id, or a page token — enough to know where to resume. Never put the only copy of a business fact in it.

If the cursor store is lost, the correct outcome is that the next fire reprocesses a window. It must never be that data is gone. Cursors are capped at 64 KiB and a larger one fails loudly, because a cursor approaching that size is almost always someone using it as a database.

The cursor only advances on a clean success

A fire advances the cursor only on HTTP 200 with output that validates against the function’s outputSchema. Every other outcome — a runtime error, schema-invalid output, a timeout, or an operator restart that lost track of the fire — leaves the stored cursor exactly as it was, so the next fire retries the same window.

That is deliberate. Reprocessing a window is recoverable; silently skipping one is not. It also means a fire whose cursor could not be saved is reported as Failed, not Succeeded, even if the function itself returned 200.

Cross-run memory

The cursor carries exact state. If the function also has memory enabled, it gets accumulated context for free: each schedule runs as a stable principal (system:schedule:<workspace>/<name>), so what one fire writes, the next can recall. Memory is an enhancement — the cursor works with memory switched off.

Your function must be idempotent within a fire slot

Omnia matches Kubernetes CronJob’s DST semantics exactly. In a DST-observing timezone:

• on the fall-back day a daily schedule fires twice;
• on the spring-forward day it does not fire at all.

There is no guard against this, on purpose. Suppressing the repeated wall-clock hour would be wrong for hourly-or-finer schedules, where that hour is a genuine extra hour of data that must be processed — so the rule cannot be applied uniformly without a special case harder to reason about than the behaviour it hides.

A watermark-driven function absorbs this by itself: the second fire processes an empty window. The exposure is functions with non-idempotent side effects — one that emails, or writes to a CRM. Make those idempotent within a fire slot, or use UTC.

When a run does not happen

A tick that does not fire records status.lastSkipReason and emits an event. A skip is not a fire: it never touches lastFireTime, lastOutcome or consecutiveFailures, so a suspended schedule never looks like a failing one.

It does advance status.lastScheduleTime — see the three timestamps. A declined tick has still been answered, and a tick that stays unanswered is one the controller re-decides on every reconcile.

Reason	Meaning
Suspended	spec.suspend: true.
MissedDeadline	The tick was older than startingDeadlineSeconds — typically the operator was down. Schedules resume rather than storming through every tick they slept past.
ConcurrencyForbidden	A fire is still in flight and concurrencyPolicy: Forbid.
DispatchCapReached	The operator’s global in-flight dispatch cap was full. The fire is skipped, never queued: a backlog of stale fires is worse than a visibly missed one when the next tick is minutes away.
DispatchStartFailed	The tick was due and allowed, but the dispatch could not be started at all — typically the cursor store could not be read. The cause is in the skip event and on the Ready condition.

A DispatchCapReached skip is the one case that does not consume a trigger-now request: nothing about your schedule caused it, so the “Run now” stays pending and is retried on the next reconcile.

The dispatch cap is global, not per-workspace

One number covers every schedule in the cluster (--scheduled-function-max-concurrent-dispatches, default 10). What it protects is the operator: one workspace’s fan-out must not saturate dispatch goroutines and slow every other reconciler down.

There is deliberately no per-workspace quota. Omnia has no workspace quota surface to hang one from — spec.quotas and spec.costControls are declared but unenforced — so a schedules-only quota would be a one-off in the wrong place. If workspaces ever stop being trusted internal teams, that quota surface is the prerequisite, not something to smuggle in here (#1781).

Allow tracks only one fire

concurrencyPolicy: Allow lets a due tick start while an earlier fire is still running, but status.inFlight holds a single fire. With overlapping fires:

• inFlight and lastInvocationID describe whichever fire touched them last;
• an operator restart can only resolve one of them (the rest settle as whatever their own goroutines managed before the restart);
• two concurrent cursor advances are last-write-wins, so the cursor can move backwards — safe, because a re-processed window is recoverable, but not free.

Prefer Forbid for any schedule that carries a cursor.

Why there is no Replace policy

concurrencyPolicy accepts Allow and Forbid only. Replace would be a promise Omnia cannot keep: a function invocation is an HTTP call, and abandoning the response does not stop the runtime spending provider tokens to completion. There is nothing to cancel.

Running one now

Set the trigger annotation to any new value — the dashboard’s Run now does this for you:

kubectl annotate scheduledfunction daily-digest -n ws-acme \
  omnia.altairalabs.ai/trigger-now="$(date -u +%FT%TZ)" --overwrite

A manual fire is a real fire: same cursor semantics, same audit row, and it obeys concurrencyPolicy. It runs once per annotation value, however many times the controller reconciles. A suspended schedule ignores it.

For an ad-hoc invocation that does not touch stored state, use the function’s test panel in the dashboard instead — that is the dry run.

Checking on a schedule

kubectl get scheduledfunction -n ws-acme

NAME           FUNCTION       SCHEDULE    SUSPENDED   LAST        NEXT   AGE
daily-digest   daily-digest   0 6 * * *   false       Succeeded   6h     9d

status.state points at where the cursor lives — the provider, the reference and when it was last written. It never holds the cursor value itself; read the ConfigMap for that.

The three timestamps

Field	Meaning
lastScheduleTime	The last tick the schedule handled — whether it fired that tick or declined it. This is the scheduling anchor: nextFireTime is computed forward from here.
lastFireTime	The last tick that actually fired. A skip never touches it.
lastSkipTime	The last tick that was declined, with lastSkipReason saying why.

lastScheduleTime is the one to read when you want to know what the schedule believes it has already dealt with; the other two say which answer it gave. They part company whenever a tick is declined: a suspended schedule advances lastScheduleTime on every tick it sleeps through while lastFireTime stays where the last real fire left it.

It is also why nextFireTime holds still between reconciles. Computing it from the current time instead would make a relative schedule (@every 5m) answer a new instant every time the controller looked at it, and — since a status write is itself a watch event — each answer would provoke the next one.

Metrics for dashboards and alerting:

• omnia_scheduled_function_fires_total{schedule,namespace,outcome}
• omnia_scheduled_function_fire_duration_seconds{schedule,namespace}
• omnia_scheduled_function_seconds_since_last_success{schedule,namespace}
• omnia_scheduled_function_cursor_write_failures_total{schedule,namespace}

The two conditions

A schedule reports two conditions, because there are two different questions:

Condition	Question it answers
SpecValid	Can this schedule fire at all? False means the spec, or something it references, is broken: a missing functionRef, an input that fails the function’s inputSchema, a cursor pointer that is not a JSON Pointer, an outputSchema that will not compile, or a function that publishes no management-plane endpoint (spec.facades[].managementPlane).
Ready	Did the last fire work? False means the last fire failed, timed out, was lost to an operator restart, or could not be started. It also mirrors a SpecValid failure, so a reader who checks only Ready never misses one.

Ready is derived from recorded fire history on every reconcile, so a schedule that fails every fire stays Ready=False — it does not flicker back to True between ticks.

Troubleshooting

Ready=False with CronInvalid. The controller parses spec.schedule.cron, and its verdict is the one that counts — the CRD’s own length check screens almost nothing. Descriptor forms are accepted (@daily, @hourly, @every 5m) alongside standard five-field expressions.

lastOutcome: Unknown. The operator restarted while this fire was in flight, so its result was never observed. The cursor was deliberately not advanced, so the next fire retries the same window. Unknown does not increment consecutiveFailures — the fire may well have succeeded, and inflating a health counter on an ambiguity would trip alerting wrongly.

The schedule never fires. Check status.conditions first — every reason a schedule cannot fire is reported there:

• SpecValid=False names the spec problem: an invalid functionRef, input that fails the function’s inputSchema, a malformed cursor pointer, an outputSchema that will not compile, an unimplemented state-store provider, or ManagementEndpointMissing — the function publishes no management-plane twin, so there is nowhere to dispatch to. That last one is what you get if the function’s facade has managementPlane disabled.
• Ready=False with DispatchStartFailed means the tick was due but the dispatch could not be started (an unreadable cursor ConfigMap, say); the cause is in the condition message and in the skip event.

If both conditions are True and nothing fires, check status.nextFireTime and status.lastSkipReason.