Version: 6.1

Data Collection Setup

General Description
- Data Collection Configuration Structure
Setup Sequence
Config and Index Correspondence
End-to-end Flow Verification

General Description

The AI Observability module data collection configurations are designed to deploy and configure the AI telemetry collection infrastructure.

Data Collection Configuration Structure

ai_observability/
├── sm_data_collector_pipelines/      # (input + filter + output)
│   ├── gpu_metrics.conf              # GPU metrics → gen_ai_gpu_metrics
│   ├── vllm_metrics.conf             # vLLM metrics → gen_ai_metrics
│   ├── claude_code.conf              # Local Claude Code events → gen_ai_events
│   ├── codex.conf                    # Local Codex events → gen_ai_events, gen_ai_cost
│   └── litellm_spendlogs.conf        # LiteLLM cost/usage → gen_ai_cost, gen_ai_events
├── sample/                           # Examples of input and normalized data
├── SAFBeatManager/
│   └── apps/
│       ├── scriptbeat_gpu_metrics    # Scriptbeat: scrape /metrics DCGM Exporter
│       ├── scriptbeat_vllm_metrics   # Scriptbeat: scrape /metrics vLLM
│       ├── filebeat_claude_code      # Filebeat: reading Claude Code / Claude Desktop logs
│       └── filebeat_codex            # Filebeat: reading Codex sessions
├── otlp_collector/                   # OpenTelemetry Collector + Data Prepper (docker-compose)
│   ├── docker-compose.yml
│   ├── otel-collector/config.yaml
│   ├── data-prepper/pipelines.yaml
│   └── .env.example
└── llm_gateway/
    └── litellm/                      # Overlay configs for enabling OTLP export in LiteLLM

Setup Sequence

Step 1. Setting up SA Data Collector

note

SA Data Collector is based on Logstash, hereinafter the alternative name may be used in the text.

SA Data Collector receives data from SAF Beat and from PostgreSQL LiteLLM.

1.1. Placing pipeline configs

Place the contents of sm_data_collector_pipelines/ in /app/logstash/config/conf.d.

1.2. Keystore keys

The following keys must be present in logstash.keystore:

Key	Purpose
`ES_PWD`	Password of the `logstash` user for Search Anywhere Framework
`PG_PWD`	Password of the PostgreSQL LiteLLM user (`litellm_spendlogs.conf`)
`JDBC_DRIVER_LIBRARY`	Path to PostgreSQL JDBC driver (`litellm_spendlogs.conf`)

sudo -u logstash /app/logstash/bin/logstash-keystore add ES_PWD
sudo -u logstash /app/logstash/bin/logstash-keystore add PG_PWD
sudo -u logstash /app/logstash/bin/logstash-keystore add JDBC_DRIVER_LIBRARY

1.3. Connecting pipelines in `pipelines.yml`

Add to /app/logstash/config/pipelines.yml:

### AI Observability

- pipeline.id: gpu_metrics
  path.config: "/app/logstash/config/conf.d/gpu_metrics.conf"

- pipeline.id: vllm_metrics
  path.config: "/app/logstash/config/conf.d/vllm_metrics.conf"

- pipeline.id: claude_code
  path.config: "/app/logstash/config/conf.d/claude_code.conf"
  pipeline.workers: 1

- pipeline.id: codex
  path.config: "/app/logstash/config/conf.d/codex.conf"
  pipeline.workers: 1

- pipeline.id: litellm_spendlogs
  path.config: "/app/logstash/config/conf.d/litellm_spendlogs.conf"

1.4. Replacing placeholders in configs

In the placed *.conf files, replace:

HOST:PORT in the output.opensearch block — Search Anywhere Framework address and port
PORT in the input.beats block — port for receiving data from SAF Beat
PG_HOST, PG_PORT, PG_DB in litellm_spendlogs.conf — connection parameters to PostgreSQL LiteLLM

The cost of Codex calls is calculated in codex.conf according to the price table (USD per 1M tokens) — check and correct if necessary for the used models.

1.5. Restarting Logstash

systemctl restart logstash

Step 2. Setting up SAF Beat Manager

SAF Beat Manager manages collection agents on GPU hosts and other workstations.

2.1. Placing applications

Place directories from SAFBeatManager/apps/ in /app/SAFBeatManager/apps, and Beat binary distributions — in /app/SAFBeatManager/binaries:

/app/SAFBeatManager/binaries/scriptbeat-oss-<version>-linux-x86_64.tar.gz
/app/SAFBeatManager/binaries/filebeat-oss-<version>-linux-x86_64.tar.gz

2.2. Application parameters

In all Beat configurations, replace HOST:PORT with the Logstash address and port.

Local client files (filebeat_claude_code, filebeat_codex):
Replace <HOME> with the user's home directory depending on the OS:

OS	Path
macOS	`/Users/<user>`
Linux	`/home/<user>`
Windows	`C:\Users\<user>`

Application	Data source	What is collected
`filebeat_claude_code`	`<HOME>/.claude/projects/*/.jsonl`	Claude Code sessions: requests, responses, tokens
`filebeat_claude_code`	`<HOME>/Library/Logs/Claude/main.log` and `*.log`	Claude Desktop events (macOS)
`filebeat_codex`	`<HOME>/.codex/sessions/*/.jsonl`	Codex sessions (layout `YYYY/MM/DD/rollout-*.jsonl`)

note

Claude Desktop logs (Library/Logs/Claude/*.log) are relevant for macOS. On other OS, if Claude Desktop is not used, this input can be disabled — collecting Claude Code sessions (.claude/projects) is not affected by this.

GPU hosts (scriptbeat_gpu_metrics):
GPU metrics are collected from DCGM Exporter. The collect_gpu_metrics.py script scrapes its HTTP endpoint /metrics (Prometheus). DCGM Exporter is installed on the GPU host as a native service via systemd.

Configure through script environment variables:

Variable	Purpose	Default
`DCGM_EXPORTER_URL`	Main `/metrics` address	`http://127.0.0.1:9400/metrics`
`DCGM_EXPORTER_FALLBACK_URL`	Backup address	`http://127.0.0.1:9401/metrics`
`DCGM_KEEP_METRICS`	List of `DCGM_FI_DEV_*` metrics (empty = all)	Reference set including `XID_ERRORS`, `TOTAL_ENERGY_CONSUMPTION`

2.3. Connecting groups in `serverclasses.yml`

Add to /app/SAFBeatManager/etc/serverclasses.yml (replace GPU_HOSTS and WORKSTATION_HOSTS with real host addresses or masks):

- name: ai_gpu_metrics
  apps:
    - scriptbeat_gpu_metrics
    - scriptbeat_vllm_metrics
  binaries:
    - scriptbeat-oss-<version>-linux-x86_64.tar.gz
  filters:
    - GPU_HOSTS

- name: ai_local_clients
  apps:
    - filebeat_claude_code
    - filebeat_codex
  binaries:
    - filebeat-oss-<version>-linux-x86_64.tar.gz
  filters:
    - WORKSTATION_HOSTS

2.4. Restarting SAF Beat Manager

systemctl restart SAFBeatManager

Step 3. Setting up OTLP domain

The OTLP domain receives telemetry from AI agents and LLM gateways that can export OpenTelemetry.

Data flow:

AI agents / LLM gateway
        │  OTLP (gRPC/HTTP)
        ▼
OpenTelemetry Collector
        │  OTLP gRPC
        ▼
Data Prepper
        │  OpenSearch bulk ingest
        ▼
gen_ai_traces / gen_ai_events / gen_ai_metrics

3.1. Starting the receiver

Fill otlp_collector/.env according to the .env.example sample (ports, Search Anywhere Framework address and password) and start:

docker compose -f otlp_collector/docker-compose.yml up -d

3.2. Verification

docker compose -f otlp_collector/docker-compose.yml ps

Both containers (otel-collector, data-prepper) should be in running status.

Step 4. Connecting LLM gateway

llm_gateway/ — these are overlay configs that enable telemetry export to the OTLP domain on your already running gateway. The package does not install the gateway itself.

On the example of LiteLLM (llm_gateway/litellm/):

Transfer the contents of host-config.yaml to the LiteLLM config (key fields — success_callback/failure_callback: ["otel"] and prices *_cost_per_token)
Set environment variables according to the .env.example sample — main: OTEL_EXPORTER_OTLP_ENDPOINT points to the HTTP receiver otlp_collector (default port :4318)
The method of delivering variables depends on the LiteLLM startup method:
- docker-compose — connect otel.override.yml as an override file:
```
docker compose -f docker-compose.yml -f llm_gateway/litellm/otel.override.yml up -d
```
- systemd / other startup — specify variables in the service environment; otel.override.yml is not needed in this case

Please note!

Gateway and inference are shown on the example of LiteLLM and vLLM. If you have a different gateway or inference, llm_gateway/, vllm_metrics and litellm_spendlogs serve as examples: the gen_ai.* data model remains unchanged, only the method of enabling export and extracting metrics/cost is adapted to the specific tool.

Config and Index Correspondence

Configuration file / component	Target index
`gpu_metrics.conf`	`gen_ai_gpu_metrics`
`vllm_metrics.conf`	`gen_ai_metrics`
`claude_code.conf`	`gen_ai_events`
`codex.conf`	`gen_ai_events`, `gen_ai_cost`
`litellm_spendlogs.conf`	`gen_ai_cost`, `gen_ai_events`
`otlp_collector` (OTel Collector + Data Prepper)	`gen_ai_traces`, `gen_ai_events`, `gen_ai_metrics`

End-to-end Flow Verification

After deploying all components:

Make sure that new documents appear in gen_ai_gpu_metrics* and gen_ai_metrics* in Search Anywhere Framework
Check for events in gen_ai_events* from local AI clients (Claude Code, Codex) and AI agents
Send a request through the LiteLLM API — check for traces in gen_ai_traces*
Check for cost data in gen_ai_cost*
Open the module dashboards and make sure the data is displayed correctly

Table of Contents​

General Description​

Data Collection Configuration Structure​

Setup Sequence​

Step 1. Setting up SA Data Collector​

1.1. Placing pipeline configs​

1.2. Keystore keys​

1.3. Connecting pipelines in pipelines.yml​

1.4. Replacing placeholders in configs​

1.5. Restarting Logstash​

Step 2. Setting up SAF Beat Manager​

2.1. Placing applications​

2.2. Application parameters​

2.3. Connecting groups in serverclasses.yml​

2.4. Restarting SAF Beat Manager​

Step 3. Setting up OTLP domain​

Step 4. Connecting LLM gateway​

Config and Index Correspondence​

End-to-end Flow Verification​

Table of Contents