Skip to content
ALTAIRA LABS
DocsBlog
OmniaOmnia

Ingest an S3 document corpus into memory

In this tutorial we sync two Markdown files from an S3 prefix into institutional memory with demos/s3-memory-populator, then recall their content through memory-api. The populator fetches and hashes the files. memory-api chunks, stores and indexes them.

The sequence diagram shows the path we build.

sequenceDiagram
    participant P as s3-memory-populator
    participant S as S3
    participant M as memory-api
    participant Y as You (curl)
    P->>S: List and fetch files under the prefix
    P->>M: POST /api/v1/institutional/ingest/batch
    M-->>P: 202 with accepted and skipped counts
    Y->>M: POST /admin/reembed
    Y->>M: POST /api/v1/memories/retrieve
    M-->>Y: Institutional-tier memories

Prerequisites

Section titled “Prerequisites”
  • An Enterprise install of Omnia with an embedding provider configured for the workspace.
  • The workspace’s memory-api, reachable from your machine, with internalServiceAuth.enabled: false on the chart. The curl commands below send no token. See Memory API.
  • An S3 bucket, the aws CLI, and AWS credentials that can list and read one prefix.
  • Go 1.27.1 or newer, curl and jq.
  • A clone of this repository.

Set the connection variables

Section titled “Set the connection variables”

Export the variables. WORKSPACE_ID is the Workspace UID, which every memory-api route accepts.

Terminal window
export MEMORY_API_URL=http://memory-api.example:8080
export WORKSPACE_ID=<workspace-uid>
export S3_BUCKET=company-handbook
export S3_REGION=<bucket-region>
export S3_PREFIX=tutorial/
export MEMORY_SOURCE=tutorial-s3

Upload two files

Section titled “Upload two files”

Create the files and upload them under the prefix.

Terminal window
printf '# Returns\n\nItems can be returned within a 30-day return window.\n' > returns.md
printf '# Shipping\n\nStandard shipping takes five working days.\n' > shipping.md
aws s3 cp returns.md "s3://$S3_BUCKET/${S3_PREFIX}returns.md"
aws s3 cp shipping.md "s3://$S3_BUCKET/${S3_PREFIX}shipping.md"

Each aws s3 cp prints an upload: line.

Run the populator

Section titled “Run the populator”

Run the populator from the repository root.

Terminal window
go run ./demos/s3-memory-populator

The populator sends a content_hash for every file, then a final present manifest listing every key it found. It ends with one log line:

{"time":"2026-10-01T10:00:00.000000000Z","level":"INFO","msg":"sync complete","bucket":"company-handbook","prefix":"tutorial/","documents":2}

A second run is safe: memory-api skips documents whose hash, source and chunking are unchanged. If you delete a file from S3 and run the populator, memory-api forgets that document and its chunks, because the manifest omits its key.

List the documents

Section titled “List the documents”

List the documents this source wrote.

Terminal window
curl -fsS "$MEMORY_API_URL/api/v1/institutional/documents?workspace=$WORKSPACE_ID&source=$MEMORY_SOURCE" \
  | jq '{total, documents: [.documents[] | {title, url, chunk_count, embedded_count}]}'

The output has one entry per file. The chunks have no vectors yet, so embedded_count is 0:

{
  "total": 2,
  "documents": [
    {
      "title": "returns.md",
      "url": "s3://company-handbook/tutorial/returns.md",
      "chunk_count": 1,
      "embedded_count": 0
    },
    {
      "title": "shipping.md",
      "url": "s3://company-handbook/tutorial/shipping.md",
      "chunk_count": 1,
      "embedded_count": 0
    }
  ]
}

Embed the chunks

Section titled “Embed the chunks”

Ingest returns before embedding happens. Call POST /admin/reembed until a call embeds nothing.

Terminal window
while :; do
  response=$(curl -fsS -X POST "$MEMORY_API_URL/admin/reembed" \
    -H 'Content-Type: application/json' \
    -d '{"max_batches":20}')
  echo "$response"
  [ "$(printf '%s' "$response" | jq -r '.embedded')" -eq 0 ] && break
done

The loop prints one response per call. The last one reports embedded as 0:

{"embedded":6,"batches":2,"remaining":0}
{"embedded":0,"batches":1,"remaining":0}

The first embedded count depends on how many rows your two files produce.

Recall a phrase

Section titled “Recall a phrase”

Retrieve memories for the phrase “return window”.

Terminal window
curl -fsS -X POST "$MEMORY_API_URL/api/v1/memories/retrieve" \
  -H 'Content-Type: application/json' \
  -d "{\"workspace_id\":\"$WORKSPACE_ID\",\"query\":\"return window\",\"limit\":5}" \
  | jq '.memories[0].tier'

The top result comes from the institutional tier:

"institutional"

The text in these rows is authoritative workspace content: the populator sends the extracted text itself, and memory-api stores it. See Memory for how institutional memory fits with the other tiers.

Next steps

Section titled “Next steps”