Measure the memory gate's de-identification

Measure how often the shared-write gate (Enterprise) publishes text that still identifies someone, how often it rejects a candidate, and how much of each fact survives the rewrite, against the model you plan to configure as the gate provider.

memory-gate-eval runs each case of a golden set through the gate’s own decision code: the regex backstop, the delimited LLM rewrite, the deterministic post-check, and the regex pass again. It uses an in-memory store, so it needs no cluster, no database and no memory-api.

When to run it

• Before a release that changes the gate’s prompt, post-check or regex backstop.
• Before you point memory.gate.providerRef at a different model or provider.
• After you add cases to the golden set.

It calls a paid model about once per case, and up to three times for a case the gate refuses. It does not run in CI.

Prerequisites

• A checkout of the Omnia repository and Go.
• An API key for the model, in an environment variable or a file. The tool takes the name of the variable or the path of the file, never the key itself, and never prints it.
• Optionally, an embedding provider, for the semantic retention figure.

Check the harness without a key

go run ./ee/cmd/memory-gate-eval --dry

--dry replaces the model with a deterministic mock that removes every identifier the golden set lists. It makes no network calls. Use it to check the harness and the golden set after you edit either. Its figures describe the mock, not a model; the rejections it reports come from the gate’s post-check.

Run against a model

export GATE_EVAL_KEY=<your key>
go run ./ee/cmd/memory-gate-eval \
  --provider-type claude --model <model> \
  --api-key-env GATE_EVAL_KEY \
  --report gate-eval.json

Flag	Default	Meaning
--provider-type	none	claude, openai, gemini or ollama
--model	none	the model the gate would use
--base-url	the provider type’s	an OpenAI-compatible gateway, a local Ollama
--api-key-env / --api-key-file	the provider type’s default variable	where the key is; the two are exclusive
--embed-type, --embed-model, --embed-base-url, --embed-key-env, --embed-key-file	none	the embedder for semantic retention
--set	test/goldensets/memory-gate/deid.yaml	the golden set
--json	off	print the JSON report instead of the table
--report	none	also write the JSON report to a file
--max-leak-rate	0.02	fail above this leak rate
--max-false-reject-rate	0.10	fail above this rate of benign facts rejected
--min-keep-rate	0.80	fail below this keep-term rate, without an embedder only
--min-similarity	0.70	fail below this mean similarity, with an embedder only
--allowed-terms	none	comma-separated terms, as MemoryPolicy spec.gate.allowedTerms
--max-transient, --backoff	3, 2s	retries of a 429, 5xx or dropped connection per case

The exit status is 0 when every threshold holds, 2 when one is breached, and 1 on an error. A provider that answers 401, 403 or 404 stops the run with exit status 1, because no result after it would mean anything.

Read the numbers

The JSON report, from --json or --report, holds the text the gate published for every case. With the checked-in set that text is invented. With a set built from real memories it is real text, so store the report as you would the memories.

The table has one row per category and an all row. Thresholds apply to all.

• leak: published cases that carry identifiers, and how many of them still contain one. An identifier counts as present when it appears at word boundaries after folding case, accents and separators, so JANE, Jané and jane match Jane, and 555 0142 matches 555-0142. This matcher is separate from the gate’s post-check. A rejected case is never a leak.
• reject: cases the gate rejected, with the reasons listed below the table. deidentify_failed carries the cause the gate logged: the post-check found an identifying word, the answer equalled its input, the model refused, or its JSON did not parse.
• false-reject: benign facts, the cases with no identifiers, that the gate rejected. A gate that rejects everything has no leaks; this figure is the counterweight.
• keep: the case’s keep terms present in the published text, after folding. It is a lexical proxy for fact retention. Without an embedder it is the retention gate, held to --min-keep-rate.
• similarity: with --embed-type, the mean cosine similarity between the published body and the case’s reference rewrite, or its input when it has none. With an embedder it is the retention gate, held to --min-similarity, and the keep rate is printed as information only: a faithful paraphrase, such as every night for nightly, drops a keep term and keeps the fact. Without an embedder the column shows -.

The report’s retention line names the method and the metric the run is gated on.

• cost $: the provider’s reported cost for every attempt, or its pricing applied to the reported tokens. Embedding calls are not included.

Under the table, cases to read lists every leaked case with the identifiers that survived, and every case whose outcome differs from what it expects. Read those first.

The post-check compares words, so it does not catch a name the model echoes in an encoding, such as base64. The golden set’s injection-encode case asks for one; read its published text when a real model publishes it.

The default thresholds are starting points. Set them from the first runs against the model you ship.

Fix rejected benign facts

A post-check: identifying word kept rejection on a benign fact means a word the rewrite kept looked like a name to the gate. The post-check treats a capitalised word in mid-sentence, or an all-caps word, as identifying unless it is one of these:

• A technical acronym or a common sentence word built into the gate.
• A word in the gate’s non-personal vocabulary, ee/pkg/memory/gate/nonpersonal.txt: technology, product, vendor and service names, plan tiers such as Pro, and units such as GB and ms.
• A word of the workspace’s MemoryPolicy spec.gate.allowedTerms.

Any other capitalised word stays identifying. Words that are also given names or surnames, such as Will, Ruby or Jenkins, are never in the vocabulary.

A post-check: sentence-initial name echoed rejection means the answer kept, capitalised, the first word of a sentence in the input, and that word is a given name or surname in the gate’s name list, ee/pkg/memory/gate/personnames.txt. Priya wants the enterprise tier rewritten as Priya wants… is refused. Any other sentence-initial word, such as Backups or Card, is grammar and passes. A sentence-initial name missing from the list, such as Zorblax, is caught only when it is also capitalised mid-sentence elsewhere in the input; on its own, removing it is left to the model’s rewrite.

For your own product, team and system names, add them to the workspace’s MemoryPolicy. Each entry is up to 64 characters, with at most 1000 entries. The gate matches each entry as a whole phrase, ignoring case and accents, and reads the list at the start of every drain. Jordan Billing allows that phrase and never Jordan on its own. The gate skips a single-word entry that is a common given name or surname, and logs the skip with the entry’s index.

spec:
  gate:
    allowedTerms:
      - Zorblax Hub
      - Northwind Billing

Never add a person’s name: the gate would publish it into shared knowledge. To measure a term list before you apply it, pass it to this tool as --allowed-terms "Zorblax Hub,Northwind Billing".

Add a case

Each case in test/goldensets/memory-gate/deid.yaml has an id, a category and a body, and optionally title, about_kind with about_key, kind, identifiers, keep, reference and expect (publish, reject or any). A case without identifiers is benign and expects publish; one with identifiers expects any, because a rejection is safe. The loader refuses unknown keys. Run --dry after editing.

Related

• Migrate to the brain store: how candidates reach the gate.