Trazio by CerebrixOS

← All guides

Application observability

The situation. After a release, one service starts erroring and slowing down. You want to know which service, how bad, and which trace shows why. For AI agents and workflows you also want each run's steps, tool calls and quality scores.

Set the five values first. Most applications need no SDK: keep your OpenTelemetry instrumentation and change where it exports.

1. What the pipeline needs

Your administrator sets up accepted sources otel.* (plus any application families you use), attaches the observability Rule Pack, and enables correlation if you want traces and runs linked.

2. OpenTelemetry Collector or Grafana Alloy

Collectors that can't mint a JWT use the write-only protocol token.

exporters:
  otlphttp/platform:
    endpoint: https://us.trazio.cerebrixos.com
    headers:
      Authorization: "Bearer ${env:PLATFORM_PROTOCOL_TOKEN}"
processors:
  batch: {}
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/platform]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/platform]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/platform]

The exporter adds /v1/logs, /v1/metrics or /v1/traces for you. JSON and protobuf both work. If a source emits OTLP over gRPC, put a Collector or Alloy gateway in front.

Other shippers. Working configs for Fluent Bit, Vector and Alloy are in examples/ (fluent-bit.conf, vector.yaml, alloy/, otel-collector-config.yaml). For a tidy end-to-end check, run the compatibility kit. It must pass auth rejection, NDJSON, OTLP logs, metrics and traces, HEC, Loki and Elastic:

node scripts/compatibility-test.mjs \
  --ingest-url "$PLATFORM_BASE" \
  --tenant-id "$TENANT_ID" \
  --pipeline-id "$PIPELINE_ID" \
  --protocol-token "$PLATFORM_PROTOCOL_TOKEN"

3. Python, Node.js and AI agents

If your app or agent framework already exports OTLP or OpenInference, change only the exporter endpoint and headers. That avoids duplicate spans and keeps the framework's own trace tree.

Add the SDK only when code must set an explicit Run and goal_id, give an event a deterministic identity, add an evaluation, or attach an artifact.

pip install trazio-telemetry
npm install trazio-telemetry

Python

import os
from trazio_telemetry import Client, Privacy

client = Client(
    base_url=os.environ["PLATFORM_BASE"],
    tenant_id=os.environ["TENANT_ID"],
    pipeline_id=os.environ["PIPELINE_ID"],
    protocol_token=os.environ["PLATFORM_PROTOCOL_TOKEN"],
    tenant_jwt=os.environ["TENANT_JWT"],
    privacy=Privacy(capture_content=False),
)

with client.run("support-agent", goal_id="refund-request-42") as run:
    with run.span("lookup-order", kind="RETRIEVER"):
        pass
    run.evaluate("groundedness", score=0.97, label="pass")

client.flush()

Node.js

import { Client, Privacy } from "trazio-telemetry";

const client = new Client({
  baseUrl: process.env.PLATFORM_BASE,
  tenantId: process.env.TENANT_ID,
  pipelineId: process.env.PIPELINE_ID,
  protocolToken: process.env.PLATFORM_PROTOCOL_TOKEN,
  tenantJwt: process.env.TENANT_JWT,
  privacy: new Privacy({ captureContent: false }),
});

const run = client.startRun("support-agent", { goalId: "refund-request-42" });
await run.withSpan("lookup-order", { kind: "RETRIEVER" }, async () => {});
run.evaluate("groundedness", { score: 0.97, label: "pass" });
run.end();
await client.flush();

The protocol token is write-only and carries OTLP. Events, queries, evaluations and artifacts need a short-lived JWT. Tenant or pipeline names inside your telemetry are routing hints only: the credential decides.

Model each agent run as one trace. A root AGENT or WORKFLOW span, with child CHAIN, LLM, RETRIEVER, TOOL, GUARDRAIL and EVALUATOR spans. Keep the framework's own trace, span, workflow-run, tool-call and conversation IDs, because those make exact links in the evidence graph.

Privacy. Prompts, completions, tool arguments and tool results are not captured by default. Authorization, cookie, password, secret, token, API-key and private-key fields are always redacted. Turn content capture on only after your administrator approves masking, residency, retention and access.

Frameworks. For Temporal, use its replay-safe OpenTelemetry interceptor and export through the Collector above; don't call the network from workflow code. For Langflow, keep its native tracing and use the SDK only for explicit evaluations or artifact links.

4. Check it landed

sleep 5
curl --fail-with-body -sS \
  "$PLATFORM_BASE/v1/query/$TENANT_ID/$PIPELINE_ID?limit=100&include_ai=true" \
  -H "Authorization: Bearer $TENANT_JWT" \
  | jq '.results[]? | select(.metadata.source_type | startswith("otel."))'

Confirm the tenant and pipeline in the result, the source family and the rule result.

5. Rules that fire

The standard observability pack reads normalized OpenTelemetry signals:

The pack accepts only severity, message, error_rate, latency_ms and status_code on otel.logs, otel.metrics and otel.traces. OTLP records from a collector are mapped onto those names, so normalize your own field names to match. Send a test event for the second rule:

curl --fail-with-body -sS -X POST \
  "$PLATFORM_BASE/v1/ingest/$TENANT_ID/$PIPELINE_ID" \
  -H "Authorization: Bearer $TENANT_JWT" \
  -H 'Idempotency-Key: checkout-latency-001' \
  -H 'Content-Type: application/json' \
  -d '{"source_type":"otel.traces","latency_ms":6200,"message":"checkout p99 latency 6.2s"}' | jq

6. Ask about it

Assistant.

• "Which service started erroring after the last deploy?"
• "Show the slowest checkout traces from the last hour."
• "Which agent runs scored below 0.8 on groundedness?"

Python.

client.semantic_query("Show high error rates and slow requests in the last 2 hours")
client.list_runs()
client.get_run("<run-id>")

Reliability

The SDK keeps a small local retry buffer for failed OTLP batches. For durable, high-volume delivery use the Collector's file-backed queue or Alloy. Don't encode large files into trace attributes: use the artifact API or the edge uploader instead.

Troubleshooting