Data Sources
The AI Observability module collects AI environment telemetry from multiple sources and normalizes it into a unified set of gen_ai_* runtime indexes based on the OpenTelemetry GenAI data model.
Normalized data is used by dashboards, inventory, and global Service Monitor Toolkit metrics.
Connected Sources
| Source | Type | Collection Route | What is collected | Where it goes |
|---|---|---|---|---|
| DCGM Exporter /metrics | GPU | SAF Beat Scriptbeat - Logstash | GPU util, memory, temperature, ECC, XID errors, energy consumption | gen_ai_gpu_metrics |
| vLLM/metrics | Inference runtime | SAF Beat Scriptbeat - Logstash | Queue depth, latency, KV cache, prefix cache | gen_ai_metrics |
Claude Code (.claude/projects) | Local client | SAF Beat Filebeat - Logstash | Requests, responses, tokens, session events | gen_ai_events |
Claude Desktop (Library/Logs/Claude) | Local client | SAF Beat Filebeat - Logstash | Claude Desktop application events and errors (macOS) | gen_ai_events |
Codex (.codex/sessions) | Local client | SAF Beat Filebeat - Logstash | Requests, responses, tokens, TTFT, reasoning, cache | gen_ai_events, gen_ai_cost |
| LiteLLM_SpendLogs PostgreSQL | LLM gateway | Logstash JDBC | Tokens, cost, provider, model, duration | gen_ai_cost, gen_ai_events |
| AI agents OTLP traces | AI agent | OTLP-domain | Agent steps, tool calls, spans, errors | gen_ai_traces |
| AI agents OTLP logs | AI agent | OTLP-domain | Questions, answers, diagnostics | gen_ai_events |
| AI agents OTLP metrics | AI agent | OTLP-domain | Request rate, error rate, custom latency | gen_ai_metrics |
| LiteLLM OTLP callback | LLM gateway | OTLP-domain | Request traces, latency, routing | gen_ai_traces |
Index Model
The module uses a unified runtime index set with wildcard patterns:
| Index | Time field | Sources | Purpose |
|---|---|---|---|
gen_ai_events* | @timestamp | LiteLLM, AI agents, Claude Code, Codex | LLM requests, agent events and local client events, responses, errors |
gen_ai_traces* | startTime | LiteLLM OTel, AI agents | Traces and spans of AI requests and agent steps |
gen_ai_metrics* | @timestamp | vLLM, AI agents | Performance metrics of inference services and agents |
gen_ai_cost* | @timestamp | LiteLLM SpendLogs, Codex | Costs and usage by models, providers, commands |
gen_ai_gpu_metrics* | @timestamp | DCGM Exporter | Hardware GPU metrics: load, memory, temperature, errors |
Notes
- ISM policies manage index lifecycle (rollover, retention) and are included in the content module
- SA Data Collector, SAF Beat, and otlp_collector configurations are included in the data collection configurations