Data Sources
The AI Observability module collects AI infrastructure telemetry from multiple sources and normalizes it into a unified set of gen_ai_* runtime indexes based on the OpenTelemetry GenAI data model.
Normalized data is used by dashboards, inventory, and global Service Monitor Toolkit metrics.
Connected Sources
| Source | Type | Collection Route | What is collected | Where it goes |
|---|---|---|---|---|
| DCGM Exporter /metrics | GPU | SAF Beat Scriptbeat - Logstash | GPU util, memory, temperature, ECC, XID errors, energy consumption | gen_ai_gpu_metrics |
| vLLM/metrics | Inference runtime | SAF Beat Scriptbeat - Logstash | Queue depth, latency, KV cache, prefix cache | gen_ai_metrics |
Claude Code (.claude/projects) | Local client | SAF Beat Filebeat - Logstash | Requests, responses, tokens, session events | gen_ai_events |
| Claude Code OTLP telemetry | Local client | OTLP domain | claude_code.* metrics: sessions, tokens, costs, code lines, commits, pull requests, edit decisions, and active time. Events: user_prompt, api_request, api_error, tool_decision, tool_result | gen_ai_metrics, gen_ai_events |
| Claude Code subscription limits | Local client | Limit collector - OTLP domain | claude_code.rate_limits event: five-hour and weekly window utilization, model-specific limit, and window reset time | gen_ai_events |
Claude Desktop (Library/Logs/Claude) | Local client | SAF Beat Filebeat - Logstash | Claude Desktop application events and errors (macOS) | gen_ai_events |
Codex (.codex/sessions) | Local client | SAF Beat Filebeat - Logstash | Requests, responses, tokens, TTFT, reasoning, cache | gen_ai_events, gen_ai_cost |
| LiteLLM_SpendLogs PostgreSQL | LLM gateway | Logstash JDBC | Tokens, cost, provider, model, duration | gen_ai_cost, gen_ai_events |
| AI agents OTLP traces | AI agent | OTLP-domain | Agent steps, tool calls, spans, errors | gen_ai_traces |
| AI agents OTLP logs | AI agent | OTLP-domain | Questions, answers, diagnostics | gen_ai_events |
| AI agents OTLP metrics | AI agent | OTLP-domain | Request rate, error rate, custom latency | gen_ai_metrics |
| LiteLLM OTLP callback | LLM gateway | OTLP-domain | Request traces, latency, routing | gen_ai_traces |
The Claude Code client sends metrics and events over OTLP regardless of the subscription plan. Prompt text in OTel events is hidden by default (<REDACTED>); only the .claude/projects file-based channel provides full conversation text. The channels are linked by session_id at the session level and request_id at the request level.
Subscription-limit utilization is not part of the client telemetry. A limit collector on the member's machine collects it: a launchd agent queries the subscription reporting endpoint every 15 minutes and sends an event to the OTLP domain. The terminal client's status-line command provides a second channel. The snapshot source is recorded in limit_source as usage_api or statusline.
Index Model
The module uses a unified runtime index set with wildcard patterns:
| Index | Time field | Sources | Purpose |
|---|---|---|---|
gen_ai_events* | @timestamp | LiteLLM, AI agents, Claude Code, Codex | LLM requests, agent and local client events, responses, errors, and Claude subscription-limit snapshots |
gen_ai_traces* | startTime | LiteLLM OTel, AI agents | Traces and spans of AI requests and agent steps |
gen_ai_metrics* | @timestamp | vLLM, AI agents, Claude Code | Performance metrics of inference services and agents, and Claude Code usage metrics |
gen_ai_cost* | @timestamp | LiteLLM SpendLogs, Codex | Costs and usage by models, providers, commands |
gen_ai_gpu_metrics* | @timestamp | DCGM Exporter | Hardware GPU metrics: load, memory, temperature, errors |
The gen_ai_events template explicitly defines numeric OTel event attributes and limit-snapshot fields so they can be aggregated:
log.attributes.input_tokens,output_tokens,cache_read_tokens,cache_creation_tokens- request tokenslog.attributes.cost_usd,duration_ms,ttft_ms- cost, duration, and time to first tokenlog.attributes.prompt_length,tool_input_size,tool_output_size- prompt and tool data sizeslog.attributes.limit_*- limit-window utilization percentage, reset time, and time until reset
Module Lookups
Nine claude_code_* lookups supplement Claude Code telemetry. Module jobs build seven of them from gen_ai_* indexes. Six are equivalents of Analytics API reports, Compliance API reports, and the subscription spend report: their structure follows Anthropic reports, but their data is calculated from module telemetry.
| Lookup | Contents | Populated by |
|---|---|---|
claude_code_analytics_daily | an equivalent of the Analytics API daily summary: sessions, code lines, commits, pull requests, edit decisions, cost, and active time per member | job using gen_ai_metrics |
claude_code_analytics_models_daily | an equivalent of the Analytics API model breakdown: four token types and cost | job using gen_ai_metrics |
claude_code_analytics_tools_daily | an equivalent of the Analytics API tool breakdown: accepted and rejected edits | job using gen_ai_metrics |
claude_code_activity_daily | an equivalent of the Compliance API audit feed: daily event counts by member, event type, and channel | job using gen_ai_events |
claude_code_sessions | an equivalent of the Compliance API session list: session metadata and member identity | two jobs using gen_ai_events and gen_ai_metrics |
claude_code_spend_report | an equivalent of the subscription spend report: daily requests, tokens, and costs by member and model | job using gen_ai_metrics |
claude_code_identity | mapping of an OS account and host to a member address | job using gen_ai_events and gen_ai_metrics |
claude_code_members | organization members: address, name, role, seat type, status, and join date | administrator |
claude_code_seat_pricing | monthly seat price, included tokens, and USD equivalent of limits by seat type | installation utility adds Team plan rows; an administrator adjusts them to the contract |
Costs in these equivalent-report lookups are estimates. In claude_code_spend_report, gross_spend contains the list-price cost from client telemetry, while net_spend contains that cost with the contract discount specified in the job.
Notes
- ISM policies manage index lifecycle (rollover, retention) and are included in the content module
SA Data Collector,SAF Beat, andotlp_collectorconfigurations and the subscription-limit collector are included in the data collection configurations