Operate memory embeddings

Check that memory vectors are caught up, and drain the backlog or tune the drain rate when they are not.

Embedding is asynchronous. A memory write succeeds before its vector exists, and institutional batch ingest returns 202. Until the backlog is drained, lexical retrieval works but semantic retrieval is incomplete.

Prerequisites

A workspace memory service with an embedding Provider. Without one, POST /admin/reembed and POST /admin/reembed/calibrate return 503.
A reachable memory-api with MEMORY_API set. The memory tutorial shows how to port-forward it.
internalServiceAuth.enabled is true by default. Add -H "Authorization: Bearer $TOKEN" to each curl below, with a ServiceAccount token as described in Authentication and scope.
Prometheus scraping the memory service on port 9090, if you want the metrics below.

Check embedding health

Watch these gauges and counters. Memory metrics lists every memory metric and its labels.

omnia_memory_embedding_coverage{workspace}: fraction of live entities whose latest active observation has an embedding.
omnia_memory_reembed_backlog{workspace}: active observations awaiting an embedding for the current model.
omnia_memory_reembed_rows_total{model}: rows the worker has embedded. Rate this for throughput.
omnia_memory_embed_requests_total{outcome} and omnia_memory_embed_duration_seconds: provider call outcomes and latency.

Rate omnia_memory_reembed_rows_total instead of reading the slope of the backlog. An ingest in progress holds the backlog flat while the worker runs at full speed.

The chart ships a memory pipeline Grafana dashboard at charts/omnia/dashboards/omnia-memory-pipeline.json. When coverage is below 0.7, the Memory Galaxy uses the lexical basis.

For the state of the schema and backfill in one response, call GET /admin/embedding-status. It returns state, current_dimension, current_type, configured_dimension, pending_target_dimension and reembed_backlog.

curl -fsS "$MEMORY_API/admin/embedding-status"

Where embedding spend is recorded

Embedding usage is recorded in provider_usage with source: embedding and in omnia_embedding_tokens_total.

Drain the backlog

The periodic worker drains at startup and then every 60 minutes. To drain now, repeat the bounded admin request until embedded is 0:

while :; do
  response=$(curl -fsS -X POST "$MEMORY_API/admin/reembed" \
    -H 'Content-Type: application/json' \
    -d '{"max_batches":20}') || break
  echo "$response"
  embedded=$(printf '%s' "$response" | jq -r '.embedded // 0')
  [ "$embedded" -eq 0 ] && break
done

A batch holds up to 500 rows by default (batchSize), and one request accepts at most 200 batches. The response reports embedded, batches and, when the memory-api knows its workspace, remaining.

503: the memory-api has no embedding provider.
502: the provider failed during a batch. The body still reports what was embedded before the failure, with the failure in error.

Tune the drain rate

The worker embeds several batches at once. The best concurrency depends on the database’s CPU allocation, the provider’s latency and the vector width, so measure it.

Run the calibration sweep:
Terminal window
```
curl -fsS -X POST "$MEMORY_API/admin/reembed/calibrate" \
  -H 'Content-Type: application/json' \
  -d '{"sample_size":200}'
```
The sweep embeds a sample of the real backlog at concurrency 1, 2, 4 and 8, and returns each rate plus a suggested value. It returns 503 without an embedding provider. The values below are illustrative.
```
{
  "points": [
    {"concurrency": 1, "rows": 200, "seconds": 2.2, "rows_per_sec": 90.9},
    {"concurrency": 4, "rows": 200, "seconds": 0.9, "rows_per_sec": 222.2}
  ],
  "suggested": 4
}
```
The sweep consumes backlog rather than synthetic work. A tie resolves to the lower concurrency. If nothing needed embedding, the response carries a note and suggested stays at the default.
Set the result on the memory service in the Workspace:
```
services:
  - name: default
    memory:
      database:
        secretRef:
          name: omnia-postgres-memory
      reembed:
        concurrency: 4
```
concurrency defaults to 2 and accepts 1 to 64. batchSize defaults to 500 and accepts 1 to 2048. It bounds how many rows one provider request carries.

Verify

Check that the backlog is falling and coverage is rising.

Query GET /admin/embedding-status and confirm reembed_backlog is 0 and state is ready. While rows remain, state is reembedding.
Confirm omnia_memory_embedding_coverage for the workspace approaches 1.
After changing concurrency, confirm omnia_memory_reembed_concurrency{model} shows the new value and the rate of omnia_memory_reembed_rows_total has risen.

Troubleshooting

Embedding request errors are rising

omnia_memory_embed_requests_total{outcome="error"} counts failed provider calls. The provider is reachable but failing, which stalls the drain. Check the embedding Provider’s credentials and rate limits.

The backlog jumped after a model change

A model change at the same dimension starts a background re-embed. memory-api reads the Provider only at startup, and the operator rolls it when the Provider’s spec changes. A change to a different dimension also requires one-shot consent. See Change the memory embedding model.