Skip to main content
Version: 6.2

AI Observability

Purpose​

The AI Observability module is designed for observability of AI infrastructure in Search Anywhere Framework.
It collects AI environment telemetry, normalizes it into a unified set of gen_ai_* indexes, and provides ready boxed content for operations.

The module covers the following observability directions:

  • LLM gateway — requests, model routing, errors, latency
  • Inference runtime — performance and load of inference service
  • GPU environment — graphics card status: temperature, memory, load, errors
  • AI agents — questions, answers, agent steps, tool calls, traces
  • local AI clients — monitoring Claude Code and Codex as telemetry sources
  • LLM costs — tokens and spending by model and provider
  • service availability — an overall operational view of AI components

For Claude Code, the module provides organization-level usage analytics: members and subscription seats, work outcomes, spending, and subscription limits. Module jobs calculate equivalents of Analytics API reports, Compliance API reports, and subscription spend reports from collected telemetry and store the results in claude_code_* lookups. Subscription-limit monitoring jobs register incidents in Incident Manager.

AI Monitoring Architecture

Module Components​

  1. Dashboards — 8 dashboards for all AI observability directions
  2. Inventory — automatically maintained GPU asset
  3. Service Monitor Toolkit global metrics — 6 key operational metrics for end-to-end monitoring
  4. Data sources — unified set of gen_ai_* indexes with normalized fields

Documentation Sections​