Write an S3 memory populator

Sync files from an S3 prefix into institutional memory by running the example populator and adapting its sync loop.

The populator owns the S3 credentials and the extraction policy. memory-api owns the document hierarchy, chunking, embeddings and retrieval.

The diagram shows what the populator does and what memory-api does with the result.

flowchart LR
    S3[S3 bucket and prefix] --> Pop[s3-memory-populator]
    Pop -->|"POST /api/v1/institutional/ingest/batch"| API[memory-api]
    API --> Store[Document, section, chunk]
    Store --> Embed[Async embedding worker]
    Embed --> Recall[Memory recall]

Prerequisites

• An Enterprise install. On Core the batch endpoint returns 403 enterprise_required.
• The URL of the workspace’s memory-api. See Ingest documents at scale for the authentication and workspace_id rules.
• AWS credentials that can list and read the prefix. The example uses the standard AWS SDK credential chain.
• A Go toolchain and a checkout of the repository. The example lives in demos/s3-memory-populator.

Run the example

The example reads .txt, .md, .markdown, .csv, .json, .yaml, .yml and .log objects and skips every other extension.

1. Export the configuration. MEMORY_API_URL, WORKSPACE_ID and S3_BUCKET are required.

   export MEMORY_API_URL=http://memory-api.example:8080
   export WORKSPACE_ID=<workspace-name-or-uid>
   export S3_BUCKET=company-handbook
   export S3_PREFIX=published/
   export MEMORY_SOURCE=s3-handbook

   S3_REGION defaults to us-east-1, and MEMORY_SOURCE defaults to s3.

2. If you use MinIO or LocalStack, point the example at it:

   export S3_ENDPOINT=http://localhost:9000
   export S3_FORCE_PATH_STYLE=true

3. If memory-api requires a ServiceAccount token, set MEMORY_TOKEN_PATH to the projected token file. The example sends its contents as a bearer token.

4. Run the populator:

   go run ./demos/s3-memory-populator

Adapt the sync loop

The loop in demos/s3-memory-populator/main.go follows this contract. Keep each step when you replace S3 with another source.

1. List the source and collect a stable URL for every document. The example uses s3://<bucket>/<key>.
2. Fetch and extract the content. Send sections[] when the source exposes headings. Otherwise send text, and memory-api splits it on Markdown headings. The example sends text.
3. Compute content_hash over the extracted content. memory-api skips a document whose hash, source and chunking settings match the stored ones.
4. Send documents in batches of at most 100 to POST /api/v1/institutional/ingest/batch. The example sends 20 at a time.
5. Put every document URL the source still has in present on the final request. memory-api forgets the documents of that kind and source that are not listed, along with their chunks and sections. Edges stay until the tombstone worker deletes the forgotten rows.

{
  "source": "s3-handbook",
  "documents": [{
    "title": "returns.md",
    "url": "s3://company-handbook/published/returns.md",
    "site": "s3://company-handbook",
    "text": "# Returns\nUnused items may be returned within 30 days.",
    "modified_at": "2026-09-13T00:00:00Z",
    "content_hash": "sha256:<hash-of-text>"
  }],
  "present": {
    "kind": "document",
    "keys": ["s3://company-handbook/published/returns.md"]
  }
}

The values are illustrative. See Memory API for every field.

If listing the source fails part way, fail the run and do not send present. A manifest built from a partial listing makes memory-api forget valid documents. The example returns an error from the listing call before it posts the manifest.

Prepare for production

The example already pages through ListObjectsV2 results, rejects objects over 16 MiB, and logs JSON with log/slog. Add the rest before you rely on it:

• Retry with backoff on S3 and memory-api errors.
• A concurrency limit on object fetches.
• Extraction chosen by content type. The example chooses by file extension.
• Source metrics.

Give the populator S3 credentials only. It needs no database credentials, because memory-api is the only writer.

Verify

1. Read the sync complete log line. It carries the bucket, prefix, documents, accepted and skipped keys.

2. Run the populator again with no source changes. Every document reports under skipped.

3. List the documents the source wrote:

   curl -fsS "$MEMORY_API_URL/api/v1/institutional/documents?source=s3-handbook"

Ingest does not wait for vectors. See Operate memory embeddings to monitor coverage and drain the re-embed backlog.

Troubleshooting

The run exits with “memory-api rejected N documents”

The error text carries the first rejection reason. Fix the cause and run again; unchanged documents are skipped.

The run logs “skip object”

The object has an extension outside the supported list, exceeds 16 MiB, or failed to download. The skipped object stays in the present manifest, so its earlier version is not forgotten.

The run exits with “memory-api returned 403 Forbidden”

An Enterprise install is required, or memory-api needs a valid ServiceAccount token in MEMORY_TOKEN_PATH.

Related pages

• Ingest documents at scale documents the batch contract.
• Ingest an S3 document corpus into memory walks through the example end to end.
• Memory API lists every endpoint.