Migrate memory to the brain store

Move one Workspace service group’s memory from the legacy store to the brain store. The migration runs offline, as a Kubernetes Job, while memory-api is scaled to zero. Agents cannot read or write memory until it finishes.

Prerequisites

• A Workspace service group with memory configured and memory-api running on the legacy store.
• pgvector 0.8.0 or later on the memory database. The migrator refuses an older one.
• The same OMNIA_PSEUDONYM_HMAC_KEY that memory-api runs with. The Job copies memory-api’s environment, so a key set through podOverrides.extraEnv is carried over. The migrator records the key’s fingerprint, and memory-api refuses to start on the brain store under a different key.
• A physical backup or snapshot of the memory database. Rollback restores the legacy tables, but a backup is the only copy that survives a mistake.

Check the plan first

Run the migrator in dry-run mode against the database. It reads, writes nothing, and does not need memory-api to be down:

kubectl run memory-migrate-plan -n <namespace> --rm -i --restart=Never \
  --image=<memory-api image> --overrides='{"spec":{"serviceAccountName":"memory-<workspace>-<group>"}}' \
  --command -- /memory-migrate plan --workspace=<workspace> --service-group=<group>

The JSON result reports:

• counts.copied: live user-tier, ingested and operator-written institutional items that are copied.
• counts.queued: legacy agent-tier items, and institutional items that neither ingest nor an operator wrote. They go to the shared-write gate’s queue, not into the store.
• counts.historyObservations and counts.relations: the history and edges that are converted.
• estimate: the storage the brain store adds, roughly (±50%).
• problems: anything that would stop the migration, such as an old pgvector, a pseudonym key mismatch or memory-api still running.

Migrate

Set the service group’s store to brain:

spec:
  services:
    - name: default
      memory:
        database:
          secretRef:
            name: memory-postgres
        providerRef:
          name: memory-embedding
        store: brain

The operator then:

1. Scales memory-api to zero replicas.
2. Creates the Job memory-migrate-<workspace>-<group> from memory-api’s own pod template: the same image, ServiceAccount, environment and podOverrides. Only the first container runs. A database proxy must be a native sidecar (an init container with restartPolicy: Always), or the Job cannot complete. The same holds for an injected sidecar, such as a service mesh proxy or a secrets agent: run it in native-sidecar mode, or the Job pod never finishes.
3. Starts memory-api with MEMORY_STORE=brain when the Job succeeds.

Follow it on the service group’s MemoryMigration condition, and in the Job’s log:

kubectl get workspace <workspace> -o jsonpath='{.status.services[?(@.name=="<group>")].conditions}'
kubectl logs -n <namespace> job/memory-migrate-<workspace>-<group> -f

The operator trusts a succeeded Job only when the migrator’s marker says the database’s migration finished: the Job’s pod writes the database’s migration state, read back from memory_migrate_state, as its termination message. When that marker cannot be read (the pod is gone) or the Job failed, the operator asks the database with a short Job, memory-migrate-status-<workspace>-<group> (memory-migrate status), before deciding anything. A status Job that cannot answer (an unreachable database) is retried with back-off, from 10 seconds doubling to at most 10 minutes; meanwhile the condition reads StatusUnavailable and memory-api stays at zero.

memory-api in brain mode refuses to start while a migration has started and not finished. Finish it, or undo it with memory-migrate abort --confirm.

The migration Job is never interrupted. If store changes while it runs, the operator lets it finish, then acts on what the database says.

A database already on the brain store is refused. If any brain table holds rows the migration did not write (memory-api has run with MEMORY_STORE=brain), the migrator refuses and drops nothing, the cleanup is never run, and the condition reads BrainDataPresent. There is nothing to migrate; to move the legacy store anyway, clear the brain tables by hand first. The Job is kept while store is brain; deleting it makes the operator run it again, which finds the database migrated and succeeds at once. memory-api is scaled to zero for those seconds.

The Job and its cleanup Job are bounded by activeDeadlineSeconds, 24 hours unless the group sets memory.migrationActiveDeadlineSeconds.

What the migration does

1. Checks pgvector, and that any existing brain memory_meta agrees with the pseudonym key, embedding model and dimension. The model is the service group’s embedding Provider, named as memory-api names it; the dimension is the legacy vectors’. --embedding-model and --embedding-dim override them.
2. Creates the brain tables beside the legacy ones, with no indexes yet.
3. Creates a subject per pseudonymous user and a collection per agent.
4. Fits the cluster hierarchy’s leaves on a sample of the legacy vectors, then copies every live item with its vector and its leaf, one partition at a time. Live means not forgotten, not expired, not superseded. A vector made by a different embedding model is left out, and the embedding worker re-embeds that item.
5. Checks that every live item arrived. Two legacy items whose keys differ only in case, such as two URLs, collide in the brain store, which compares keys in lower case. The migration then stops before the last step and names the items; fix them in the legacy store, or run the migrator with --accept-conflicts to migrate without the ones that lost. The operator’s Job never passes it.
6. Converts superseded observations to revision history, part_of relations to parent documents, other relations to edges, and promotion sources to lineage. An edge between two different users is dropped: it is itself personal data.
7. Queues the legacy agent-tier items for the gate. They were written before de-identification existed, so they are not copied. They were already shared, so the gate de-identifies them without asking for corroboration again. Each names the users its promotion sources belonged to, so erasing one of them deletes it.
8. Settles the cluster hierarchy (counts, the levels above the leaves, the few leaves that came out too large), marks outliers, then builds the indexes.
9. Renames the legacy tables to memory_legacy_* and drops the 2D Galaxy projections, in one transaction. Until this step the legacy store is exactly as it was.

Every step commits with a record in memory_migrate_state. A Job pod that crashes is retried, and the retry resumes at the first unfinished step without duplicating a row.

Downtime

Downtime runs from memory-api scaling to zero until the Job succeeds. Measured on a laptop at 50,000 items of 1,536 dimensions:

Part	5×10⁴ items	Grows with
Fitting the leaves	~3 s	the sample (at most 32,768 vectors) × leaves
Copy, with each item’s leaf	~5 s	items × √leaves
Settling the hierarchy, outliers	~3 s	items
Indexes (HNSW, full text, scope)	~5 s	items, more than linearly

At 2×10⁷ items expect hours, not minutes: plan a maintenance window, and measure a copy of the real database first with BRAINMIGRATE_MEASURE=1 (internal/memory/brainmigrate). The index build uses the database’s memory, set with --maintenance-work-mem (default 512MB); raise it on a large database so the HNSW graph fits.

Shared knowledge that came from the legacy agent tier is missing after the migration. It returns as the gate publishes the queued items. The gate calls a model per item, so a large agent tier can take hours.

If the migration fails

The Job retries three times. When it fails for good (or succeeds without the marker), the operator keeps memory-api at zero and runs the cleanup Job memory-migrate-abort-<workspace>-<group> (memory-migrate abort --confirm). It drops the unfinished migration’s brain tables, which hold a copy of every migrated user that an erasure on the legacy store would otherwise miss. Then memory-api comes back on the legacy store, which the migrator has not changed, and the MemoryMigration condition reads Failed. If the cleanup fails too (CleanupFailed), memory-api still comes back: legacy erasures also reach the brain tables. Read the Job’s log, fix the cause, then delete the Job to try again:

kubectl delete job memory-migrate-<workspace>-<group> -n <namespace>

If legacy memory-api wrote anything in between, the next run notices and starts the migration over.

Roll back

Rollback restores the legacy store and drops the brain store, with everything memory-api wrote to it since the migration. It is never automatic: setting store back to legacy alone leaves memory-api refusing to start, because the legacy tables are renamed.

1. Set store: legacy on the service group. The operator reads the database’s state; on a migrated database it deletes the migration Jobs (so a later brain migrates afresh), holds memory-api at zero rather than letting it crash-loop, creates the memory-migrate-rollback-<workspace>-<group> Job suspended, and sets the MemoryMigration condition to RollbackRequired with the command that starts it. The Job is memory-api’s own pod (image, pull policy and secrets, environment, sidecars) running /memory-migrate rollback --confirm.

2. Start it by unsuspending it:

   kubectl --context <context> patch job memory-migrate-rollback-<workspace>-<group> \
     -n <namespace> --type merge -p '{"spec":{"suspend":false}}'

3. Nothing else. When the Job succeeds the operator deletes it and the status Jobs and starts memory-api on the legacy store. A Job that fails is kept, and the condition names it so its logs can be read; delete it to get a fresh suspended one.

Rollback drops only what a migration created. It refuses, dropping nothing, when no migration is recorded, when another service group’s memory-api uses the same database, or when the brain tables hold another workspace’s rows.

Drop the legacy tables

Once the brain store has held, drop the renamed legacy tables. There is no rollback after this:

kubectl run memory-migrate-drop -n <namespace> --rm -i --restart=Never \
  --image=<memory-api image> --overrides='{"spec":{"serviceAccountName":"memory-<workspace>-<group>"}}' \
  --command -- /memory-migrate drop-legacy --confirm --workspace=<workspace> --service-group=<group>

Until they are dropped, the legacy tables hold every user’s legacy rows, forgotten ones included. Erasure (DSAR) deletes a user’s rows from them too, so a rollback never brings an erased user back.

Troubleshooting

The Job waits, then fails with “still holds the memory service lock”

Every memory-api holds the memory service lock while it runs, and the migrator needs it alone. A memory-api from before this release does not take the lock, so the operator scales memory-api to zero before the Job starts. Check that no other process runs memory-api against the same database.

memory-api refuses to start with “this database was migrated to the brain store”

memory-api runs on the legacy store against a migrated database. Set store: brain, or roll back.

The migration fails with “legacy vectors are N-dimensional, the target is M”

The embedding Provider’s dimension differs from the legacy vectors’. Run the migrator with --allow-reembed to copy without vectors, or point the Provider back at the legacy model and change model after the migration.

Related pages

• Configure workspace memory
• Operate memory embeddings
• Workspace reference
• Memory API reference