Ingest OpenTelemetry traces into sessions

Enable session-api’s OTLP listeners so agents in a workspace namespace that are instrumented with OpenTelemetry GenAI conventions record their conversations as Omnia sessions.

flowchart LR
  A["Agent in the workspace namespace (OTLP exporter)"] -->|"HTTP :4318 /v1/traces or gRPC :4317"| S["session-api"]
  S --> T["OTLP transformer"]
  T --> P[("Postgres")]

This is for agents that run in the workspace namespace without an Omnia runtime. Omnia’s own agents record their sessions directly, and their traces go to Tempo. The listeners admit only the workspace namespace, so agents outside the cluster do not send traces this way.

Prerequisites

A Workspace with a managed spec.services[] group that configures session. See Workspace.
kubectl access to the workspace namespace.
A sender that can attach an Authorization header. The listeners require a ServiceAccount token by default; see Authentication.

Enable the listeners

Set otlp.enabled on the group’s session.
```
spec:
  services:
    - name: default
      session:
        database:
          secretRef:
            name: session-postgres
        otlp:
          enabled: true
```
The operator restarts session-api with its OTLP listeners on, gRPC on 4317 and HTTP on 4318. It adds both ports to the group’s session-<workspace>-<group> Service as otlp-grpc and otlp-http. With internal service authentication on, it also opens both ports in the workspace’s service-auth NetworkPolicy, to the workspace namespace only.
Mint a token for a ServiceAccount in the workspace namespace, with the audience omnia-session-api (the chart’s internalServiceAuth.audience).
Terminal window
```
kubectl --context <context> -n <workspace-namespace> create token <service-account> --audience omnia-session-api
```
Pass it to the sender as Authorization: Bearer <token>. With internalServiceAuth.enabled: false the listeners accept unauthenticated requests and the operator creates no NetworkPolicy, so restrict the two ports to the workspace namespace with a network policy of your own.

Send traces

Point the exporter at http://session-<workspace>-<group>.<workspace-namespace>:4318/v1/traces. The HTTP listener accepts application/json and application/x-protobuf (exact Content-Type, optionally gzip-encoded, up to 4 MB).

Set gen_ai.conversation.id on every span to an ID that identifies the conversation. Spans with the same ID land in one session. A UUID becomes the session ID. Any other value, such as thread-42, maps to a fixed UUID derived from it, and the session keeps the original ID as the tag otlp-session-id:thread-42.

From the OpenTelemetry Python SDK

pip install opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http

The Python SDK cannot set the structured gen_ai.input.messages attribute, so this example uses the indexed attributes, which Omnia also reads.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "my-agent",
    "service.namespace": "<workspace-namespace>",
})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(
    endpoint="http://session-<workspace>-<group>.<workspace-namespace>:4318/v1/traces",
    headers={"Authorization": "Bearer <token>"},
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("my-agent")
with tracer.start_as_current_span("chat gpt-4") as span:
    span.set_attribute("gen_ai.conversation.id", "thread-42")
    span.set_attribute("gen_ai.request.model", "gpt-4")
    span.set_attribute("gen_ai.provider.name", "openai")
    span.set_attribute("gen_ai.prompt.0.role", "user")
    span.set_attribute("gen_ai.prompt.0.content", "Hello")
    span.set_attribute("gen_ai.completion.0.role", "assistant")
    span.set_attribute("gen_ai.completion.0.content", "Hi there")
    span.set_attribute("gen_ai.usage.input_tokens", 9)
    span.set_attribute("gen_ai.usage.output_tokens", 3)

provider.shutdown()

From LangChain through OpenLLMetry

OpenLLMetry auto-instruments LangChain, OpenAI, Anthropic and more, and emits the indexed attribute format.

pip install traceloop-sdk

from traceloop.sdk import Traceloop

Traceloop.init(
    app_name="my-langchain-agent",
    api_endpoint="http://session-<workspace>-<group>.<workspace-namespace>:4318",
    headers={"Authorization": "Bearer <token>"},
    disable_batch=False,
)

Set TRACELOOP_TRACE_CONTENT=true to include message content. Without it only metadata and token counts are captured.

From AWS Bedrock through ADOT

Run the ADOT collector in the workspace namespace and configure its OTLP HTTP exporter:

exporters:
  otlphttp:
    endpoint: "http://session-<workspace>-<group>.<workspace-namespace>:4318"
    headers:
      Authorization: "Bearer <token>"

service:
  pipelines:
    traces:
      exporters: [otlphttp]

For Python agents using the OpenTelemetry SDK directly, set OTEL_EXPORTER_OTLP_ENDPOINT to the same URL and OTEL_SERVICE_NAME to the agent name.

The session ID sources and the attributes Omnia reads are in OTLP ingestion.

Token counts

A span that carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, or the older gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens, records one provider call on the session. The session’s totalInputTokens and totalOutputTokens add those calls up, and the span’s assistant message carries the same counts. This works without message content. Omnia does not price these calls: each records a cost of 0, so the session’s estimatedCostUSD does not change.

Verify

Call session-api’s JSON API with the same token. The list route requires namespace.

curl -s -H "Authorization: Bearer <token>" \
  "http://session-<workspace>-<group>.<workspace-namespace>:8080/api/v1/sessions?namespace=<workspace-namespace>&tag=otlp-session-id:thread-42"
curl -s -H "Authorization: Bearer <token>" \
  "http://session-<workspace>-<group>.<workspace-namespace>:8080/api/v1/sessions/<session-id>/messages"

The first call lists the session with the agentName you set in service.name, totalInputTokens: 9 and totalOutputTokens: 3. Use its id in the second call to read the messages. A conversation ID that is a UUID is the session ID, so for those the tag filter is not needed. The same sessions appear in the dashboard beside sessions from Omnia agents.

Troubleshooting

The sender gets 401 or 403

The token is missing, has the wrong audience, or belongs to a ServiceAccount outside the allowed namespaces. Mint it again as in step 2.

The sender gets 415

The HTTP listener matches Content-Type exactly. application/json; charset=utf-8 is rejected.

Connection refused or timeout on 4317 or 4318

Check that the Service lists the otlp-grpc and otlp-http ports and that the session-api container has --otlp-enabled=true in its arguments.

kubectl --context <context> -n <workspace-namespace> get service session-<workspace>-<group> -o wide
kubectl --context <context> -n <workspace-namespace> describe deploy session-<workspace>-<group>

With internal service authentication on, the service-auth NetworkPolicy admits the OTLP ports from the workspace namespace only. Run the sender, or a collector that forwards for it, in the workspace namespace.

Sessions exist but have no messages

Most frameworks leave out message content by default. Set TRACELOOP_TRACE_CONTENT=true for OpenLLMetry, or OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true for the OpenTelemetry Python OpenAI instrumentation.

Each trace creates a separate session

Set gen_ai.conversation.id to the same value on every span of a conversation. Spans without one of the session ID attributes are skipped.