Trazio by CerebrixOS

← All guides

Cloud and Kubernetes

The situation. A deployment fails to sync and health goes degraded, and a privileged workload appears in a cluster nobody expected it in. You want both caught, and a way to ask "what changed before the outage?"

Set the five values first. This guide covers Kubernetes audit and workload events and Argo CD deployment events. For services' own logs, metrics and traces, see Application observability.

1. What the pipeline needs

Your administrator sets up accepted sources kubernetes.* and argocd (plus otel.* if you send traces too), attaches the cloud-kubernetes Rule Pack, and enables correlation if you want related events linked.

2. Send events

Kubernetes audit logs and Argo CD notifications usually already flow through a log shipper or webhook. Point that shipper at the platform with the write-only protocol token, using the protocol it supports (formats). To send a single event directly:

curl

curl --fail-with-body -sS -X POST \
  "$PLATFORM_BASE/v1/ingest/$TENANT_ID/$PIPELINE_ID" \
  -H "Authorization: Bearer $TENANT_JWT" \
  -H "Idempotency-Key: k8s-audit-8841" \
  -H 'Content-Type: application/json' \
  -d '{"source_type":"kubernetes.audit","event_type":"workload_created","verb":"create","resource":"pods","privileged":true,"namespace":"payments","message":"Privileged pod created in payments"}' | jq

Python

client.record_event({
    "source_type": "argocd",
    "event_type": "sync_failed",
    "sync_status": "Failed",
    "health": "Degraded",
    "namespace": "checkout",
    "message": "checkout failed to sync and is degraded",
}, idempotency_key="argocd-checkout-sync-210")

Node.js

await client.recordEvent({
  source_type: "argocd",
  event_type: "sync_failed",
  sync_status: "Failed",
  health: "Degraded",
  namespace: "checkout",
  message: "checkout failed to sync and is degraded",
}, { idempotencyKey: "argocd-checkout-sync-210" });

Cluster metrics and traces. Run the OpenTelemetry Collector in the cluster and export to the platform. The config is in Application observability.

3. Check it landed

sleep 5
curl --fail-with-body -sS \
  "$PLATFORM_BASE/v1/query/$TENANT_ID/$PIPELINE_ID?limit=100&include_ai=true" \
  -H "Authorization: Bearer $TENANT_JWT" \
  | jq '.results[]? | select(.metadata.source_type | test("kubernetes|argocd"))'

Check rule_evaluation on each result.

4. Rules that fire

The standard cloud-kubernetes pack:

The pack accepts only these fields, so send nothing else: for kubernetes.audit, event_type, verb, resource, namespace, privileged and message; for argocd, event_type, sync_status, health, namespace and message. (Envelope fields such as severity, device_id and run_id are separate.) The two example events in step 2 trigger one rule each. Add your own, for example alerting on a new service account with cluster-admin, and dry-run it first:

client.dry_run_rule_pack(pack, [{"source_type": "kubernetes.audit", "verb": "create", "privileged": True}])

5. Ask about it

Assistant.

• "What changed in the payments namespace before the outage?"
• "Which deployments failed to sync today?"
• "Were any privileged workloads created this week?"

Python.

client.semantic_query("Show critical Kubernetes events from the last 24 hours")

To see what happened around a failed deployment, use the evidence graph. Events that share a run or trace ID link as exact edges:

curl --fail-with-body -sS \
  "$PLATFORM_BASE/v1/query/$TENANT_ID/$PIPELINE_ID/graph?event_id=$EVENT_ID&hops=2&limit=100" \
  -H "Authorization: Bearer $TENANT_JWT" | jq

Troubleshooting