Version: 6.2

Data Collection Setup

General Description
- Data Collection Configuration Structure
Setup Sequence
Config and Index Correspondence
End-to-end Flow Verification

General Description

The AI Observability module data collection configurations are designed to deploy and configure the AI telemetry collection infrastructure.

The installation utility installs the gen_ai_* index templates and ingest pipelines before data collection starts. Without the pipeline, writes to a gen_ai_* index are rejected.

Data Collection Configuration Structure

The data collection configurations are supplied in source-specific packages:

Package	Data collected	Contents
`nvidia_dcgm_metrics`	DCGM Exporter GPU metrics	`nvidia_dcgm_metrics.conf`, `scriptbeat_nvidia_dcgm_metrics` application
`vllm_metrics`	vLLM metrics	`vllm_metrics.conf`, `scriptbeat_vllm_metrics` application
`claude_code`	Claude Code sessions, Claude Desktop logs, subscription limits	`claude_code.conf`, `filebeat_claude_code` application, `ratelimit_collector/` limit collector
`codex`	Codex sessions	`codex.conf`, `filebeat_codex` application
`litellm`	LiteLLM spend logs and gateway OTLP export	`litellm_spendlogs.conf`, `llm_gateway/litellm/` overlay configurations
`opentelemetry_gen_ai`	OTLP telemetry from AI agents, the LLM gateway, and Claude Code	`otlp_collector/`: OpenTelemetry Collector and Data Prepper

The `SA Data Collector` configurations are in each package's `sm_data_collector_pipelines/` directory, `SAF Beat` applications in `SafBeatManager/apps/`, and sample input and normalized data in `sample/`.

Setup Sequence

Step 1. Setting up SA Data Collector

note

SA Data Collector is based on Logstash, hereinafter the alternative name may be used in the text.

SA Data Collector receives data from SAF Beat and from PostgreSQL LiteLLM.

1.1. Placing pipeline configs

Place the contents of sm_data_collector_pipelines/ in /app/logstash/config/conf.d.

1.2. Keystore keys

The following keys must be present in logstash.keystore:

Key	Purpose
`ES_PWD`	Password of the `logstash` user for Search Anywhere Framework
`PG_PWD`	Password of the PostgreSQL LiteLLM user (`litellm_spendlogs.conf`)
`JDBC_DRIVER_LIBRARY`	Path to PostgreSQL JDBC driver (`litellm_spendlogs.conf`)

sudo -u logstash /app/logstash/bin/logstash-keystore add ES_PWD
sudo -u logstash /app/logstash/bin/logstash-keystore add PG_PWD
sudo -u logstash /app/logstash/bin/logstash-keystore add JDBC_DRIVER_LIBRARY

1.3. Connecting pipelines in `pipelines.yml`

Add to /app/logstash/config/pipelines.yml:

### AI Observability

- pipeline.id: nvidia_dcgm_metrics
  path.config: "/app/logstash/config/conf.d/nvidia_dcgm_metrics.conf"

- pipeline.id: vllm_metrics
  path.config: "/app/logstash/config/conf.d/vllm_metrics.conf"

- pipeline.id: claude_code
  path.config: "/app/logstash/config/conf.d/claude_code.conf"
  pipeline.workers: 1

- pipeline.id: codex
  path.config: "/app/logstash/config/conf.d/codex.conf"
  pipeline.workers: 1

- pipeline.id: litellm_spendlogs
  path.config: "/app/logstash/config/conf.d/litellm_spendlogs.conf"

1.4. Replacing placeholders in configs

In the placed *.conf files, replace:

HOST:PORT in the output.opensearch block — Search Anywhere Framework address and port
PORT in the input.beats block — port for receiving data from SAF Beat
PG_HOST, PG_PORT, PG_DB in litellm_spendlogs.conf — connection parameters to PostgreSQL LiteLLM

The cost of Codex calls is calculated in codex.conf according to the price table (USD per 1M tokens) — check and correct if necessary for the used models.

1.5. Restarting Logstash

systemctl restart logstash

Step 2. Setting up SAF Beat Manager

SAF Beat Manager manages collection agents on GPU hosts and other workstations.

2.1. Placing applications

Place directories from SAFBeatManager/apps/ in /app/SAFBeatManager/apps, and Beat binary distributions — in /app/SAFBeatManager/binaries:

/app/SAFBeatManager/binaries/scriptbeat-oss-<version>-linux-x86_64.tar.gz
/app/SAFBeatManager/binaries/filebeat-oss-<version>-linux-x86_64.tar.gz

2.2. Application parameters

In all Beat configurations, replace HOST:PORT with the Logstash address and port.

Local client files (filebeat_claude_code, filebeat_codex):
Replace <HOME> with the user's home directory depending on the OS:

OS	Path
macOS	`/Users/<user>`
Linux	`/home/<user>`
Windows	`C:\Users\<user>`

Application	Data source	What is collected
`filebeat_claude_code`	`<HOME>/.claude/projects/*/.jsonl`	Claude Code sessions: requests, responses, tokens
`filebeat_claude_code`	`<HOME>/Library/Logs/Claude/main.log` and `<HOME>/Library/Logs/Claude/claude.ai-web.log`	Claude Desktop events (macOS)
`filebeat_codex`	`<HOME>/.codex/sessions/*/.jsonl`	Codex sessions (layout `YYYY/MM/DD/rollout-*.jsonl`)

note

Claude Desktop logs (Library/Logs/Claude/*.log) are relevant for macOS. On other OS, if Claude Desktop is not used, this input can be disabled — collecting Claude Code sessions (.claude/projects) is not affected by this.

GPU hosts (scriptbeat_gpu_metrics):
GPU metrics are collected from DCGM Exporter. The collect_gpu_metrics.py script scrapes its HTTP endpoint /metrics (Prometheus). DCGM Exporter is installed on the GPU host as a native service via systemd.

Configure through script environment variables:

Variable	Purpose	Default
`DCGM_EXPORTER_URL`	Primary `/metrics` address	`http://127.0.0.1:9401/metrics`
`DCGM_EXPORTER_FALLBACK_URL`	Backup address	`http://127.0.0.1:9400/metrics`
`DCGM_KEEP_METRICS`	List of `DCGM_FI_DEV_*` metrics (empty value means all metrics)	Reference set including `XID_ERRORS`, `TOTAL_ENERGY_CONSUMPTION`

2.3. Connecting groups in `serverclasses.yml`

Add to /app/smartBeatManager/etc/serverclasses.yml (replace GPU_HOSTS and WORKSTATION_HOSTS with real host addresses or masks):

- name: ai_gpu_metrics
  apps:
    - scriptbeat_nvidia_dcgm_metrics
    - scriptbeat_vllm_metrics
  binaries:
    - scriptbeat-oss-<version>-linux-x86_64.tar.gz
  filters:
    - GPU_HOSTS

- name: ai_local_clients
  apps:
    - filebeat_claude_code
    - filebeat_codex
  binaries:
    - filebeat-oss-<version>-linux-x86_64.tar.gz
  filters:
    - WORKSTATION_HOSTS

2.4. Restarting SAF Beat Manager

systemctl restart SAFBeatManager

Step 3. Setting up OTLP domain

The OTLP domain receives telemetry from AI agents, LLM gateways, and Claude Code clients that can export OpenTelemetry, as well as subscription-limit snapshots from the limit collector.

Data flow:

AI agents / LLM gateway / Claude Code
        │  OTLP (gRPC/HTTP)
        ▼
OpenTelemetry Collector
        │  OTLP gRPC
        ▼
Data Prepper
        │  OpenSearch bulk ingest
        ▼
gen_ai_traces / gen_ai_events / gen_ai_metrics

3.1. Starting the receiver

Fill otlp_collector/.env according to the .env.example sample (ports, Search Anywhere Framework address and password) and start:

docker compose -f otlp_collector/docker-compose.yml up -d

Variables in .env:

Variable	Value
`SMART_MONITOR_HOST`	`SA Data Storage` address
`SMART_MONITOR_PORT`	`SA Data Storage` port, usually `9200`
`SMART_MONITOR_USERNAME`	user allowed to write to `gen_ai_*`
`SMART_MONITOR_PASSWORD`	that user's password
`SMART_MONITOR_SKIP_SSL_VERIFY`	`true` to skip `SA Data Storage` certificate verification, `false` to verify it
`OTEL_COLLECTOR_HOST`	address on which the collector listens for OTLP
`OTEL_COLLECTOR_GRPC_PORT`	OTLP/gRPC port, usually `4317`
`OTEL_COLLECTOR_HTTP_PORT`	OTLP/HTTP port, usually `4318`
`DATA_PREPPER_TRACES_PORT`	traces port, `22890`
`DATA_PREPPER_METRICS_PORT`	metrics port, `22891`
`DATA_PREPPER_LOGS_PORT`	events port, `22892`
`DATA_PREPPER_GPU_METRICS_PORT`	GPU metrics port, `22893`
`DEPLOYMENT_ENVIRONMENT`	environment name in `deployment_environment`

Each DATA_PREPPER_*_PORT variable sets the port across the collector exporter, the port published by docker-compose.yml, and the Data Prepper source. The first data may arrive up to a minute later: the collector starts before the Data Prepper sources and reconnects after a delay.

Updating from 6.1

In 6.2, all files in otlp_collector changed and data-prepper/render_pipelines.sh was added. Replace the whole directory and transfer values into a new .env based on .env.example. SMART_MONITOR_VERIFY_SSL was renamed to SMART_MONITOR_SKIP_SSL_VERIFY; the old name is not supported. With a version 6.1 .env, the Data Prepper container stops and reports the missing variable. OTEL_TRACES_PORT, OTEL_METRICS_PORT, and OTEL_LOGS_PORT were removed; DATA_PREPPER_*_PORT variables take their place. After replacing the directory, recreate the containers:

docker compose -f otlp_collector/docker-compose.yml up -d --force-recreate

3.2. Verification

docker compose -f otlp_collector/docker-compose.yml ps

Both containers (otel-collector, data-prepper) should be in running status.

Step 4. Connecting LLM gateway

llm_gateway/ — these are overlay configs that enable telemetry export to the OTLP domain on your already running gateway. The package does not install the gateway itself.

On the example of LiteLLM (llm_gateway/litellm/):

Transfer the contents of host-config.yaml to the LiteLLM config (key fields — success_callback/failure_callback: ["otel"] and prices *_cost_per_token)
Set environment variables according to the .env.example sample — main: OTEL_EXPORTER_OTLP_ENDPOINT points to the HTTP receiver otlp_collector (default port :4318)
The method of delivering variables depends on the LiteLLM startup method:
- docker-compose — connect otel.override.yml as an override file:
```
docker compose -f docker-compose.yml -f llm_gateway/litellm/otel.override.yml up -d
```
- systemd / other startup — specify variables in the service environment; otel.override.yml is not needed in this case

Please note!

Gateway and inference are shown on the example of LiteLLM and vLLM. If you have a different gateway or inference, llm_gateway/, vllm_metrics and litellm_spendlogs serve as examples: the gen_ai.* data model remains unchanged, only the method of enabling export and extracting metrics/cost is adapted to the specific tool.

Step 5. Connecting Claude Code

Claude Code exports metrics and events over OTLP regardless of the subscription plan. The receiver is the OTLP domain from step 3: claude_code.* metrics are written to gen_ai_metrics, while user_prompt, api_request, api_error, tool_decision, and tool_result events are written to gen_ai_events.

5.1. Client variables

Set these environment variables on the member's machine:

CLAUDE_CODE_ENABLE_TELEMETRY=1
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/json
OTEL_EXPORTER_OTLP_ENDPOINT=http://<collector_address>:4318

Members do not set the deployment_environment attribute: the collector sets it from DEPLOYMENT_ENVIRONMENT.

5.2. Rolling out to members

Set the variables in the env block of Claude Code settings using one of three methods, in order of increasing scope:

In the member's ~/.claude/settings.json
In server-managed settings in the claude.ai admin console for Team and Enterprise plans, together with forceLoginOrgUUID so sessions under personal accounts are covered by the policy
In managed-settings.json deployed through MDM or a script for clients that do not sign in through claude.ai

Run /status in Claude Code and check Setting sources to see which settings apply on the machine.

5.3. Verification

The client sends events every 5 seconds and metrics every 60 seconds, so a short claude -p run ends before the first metric export. Shorten the interval for one test run:

OTEL_METRIC_EXPORT_INTERVAL=3000 claude -p "reply with one word: ready"

Within a minute, claude_code.* metrics should appear in gen_ai_metrics with the member's address in metric.attributes.user_email.

Step 6. Collecting Subscription Limit Utilization

Claude Code telemetry shows spending, but not subscription-limit utilization or the window reset time. The ratelimit_collector/ component in the claude_code package collects these values: it sends a claude_code.rate_limits event as an OTLP log to the collector from step 3, which writes it to gen_ai_events. The collector runs only on macOS: it reads the subscription token from Keychain and is scheduled by launchd. Claude Code version 2.1.251 or later is required.

6.1. Placing scripts

Keep the scripts together because the polling script imports functions from the status-line script:

mkdir -p ~/.claude/scripts
cp ratelimit_collector/rate_limit_poller.py ratelimit_collector/statusline_rate_limits.py ~/.claude/scripts/

Run a one-time check without sending data:

python3 ~/.claude/scripts/rate_limit_poller.py --dry-run

The check prints a snapshot with utilization percentages for both windows, reset times, and the model-specific limit.

6.2. Scheduled polling agent

The com.volgablob.claude.ratelimit-poller.plist agent polls every 15 minutes. Replace the username in it, place it in ~/Library/LaunchAgents/, and load it:

P=~/Library/LaunchAgents/com.volgablob.claude.ratelimit-poller.plist
sed "s/&lt;user&gt;/$USER/g" ratelimit_collector/com.volgablob.claude.ratelimit-poller.plist > $P
plutil -lint $P && launchctl load $P

The first run requests Keychain access. Select Always Allow; otherwise, the agent cannot retrieve the token in the background. An agent in ~/Library/LaunchAgents/ loads automatically when the user signs in.

The collector obtains its endpoint from CLAUDE_RL_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT, then from the env block in Claude Code settings.

6.3. Status line and probe

The status line is a second source of snapshots. Configure it with the statusLine command in ~/.claude/settings.json or in the organization's managed settings. Only the Claude Code command-line client runs it; the desktop application does not. Install rate_limit_probe.py only where polling is unavailable: each probe session sends a real model request and consumes the limit. Installation instructions for both channels are in ratelimit_collector/README.md.

6.4. Verification

launchctl list | grep ratelimit

The command shows the agent. A line reading отправлено appears in ~/.claude/.rate_limits.poller.log every 15 minutes.

Please note!

The collector calls https://api.anthropic.com/api/oauth/usage, an endpoint not covered by Anthropic's public documentation. Its response format may change when the client is updated.

Config and Index Correspondence

Configuration file / component	Target index
`nvidia_dcgm_metrics.conf`	`gen_ai_gpu_metrics`
`vllm_metrics.conf`	`gen_ai_metrics`
`claude_code.conf`	`gen_ai_events`
`codex.conf`	`gen_ai_events`, `gen_ai_cost`
`litellm_spendlogs.conf`	`gen_ai_cost`, `gen_ai_events`
`otlp_collector` (OTel Collector + Data Prepper)	`gen_ai_traces`, `gen_ai_events`, `gen_ai_metrics`
`ratelimit_collector` through `otlp_collector`	`gen_ai_events`

End-to-end Flow Verification

After deploying all components:

Make sure that new documents appear in gen_ai_gpu_metrics* and gen_ai_metrics* in Smart Monitor
Check for events in gen_ai_events* from local AI clients (Claude Code, Codex) and AI agents
Send a request through the LiteLLM API — check for traces in gen_ai_traces*
Check for cost data in gen_ai_cost*
Check for claude_code.* metrics in gen_ai_metrics* and subscription-limit snapshots in gen_ai_events* (log.attributes.event_name = rate_limits)
Open the module dashboards and make sure the data is displayed correctly

Table of Contents​

General Description​

Data Collection Configuration Structure​

The SA Data Collector configurations are in each package's sm_data_collector_pipelines/ directory, SAF Beat applications in SafBeatManager/apps/, and sample input and normalized data in sample/.​

Setup Sequence​

Step 1. Setting up SA Data Collector​

1.1. Placing pipeline configs​

1.2. Keystore keys​

1.3. Connecting pipelines in pipelines.yml​

1.4. Replacing placeholders in configs​

1.5. Restarting Logstash​

Step 2. Setting up SAF Beat Manager​

2.1. Placing applications​

2.2. Application parameters​

2.3. Connecting groups in serverclasses.yml​

2.4. Restarting SAF Beat Manager​

Step 3. Setting up OTLP domain​

Step 4. Connecting LLM gateway​

Step 5. Connecting Claude Code​

Step 6. Collecting Subscription Limit Utilization​

Config and Index Correspondence​

End-to-end Flow Verification​

Table of Contents