Skip to main content
Version: 6.2

Diagnosing SA Data Collector Issues

This article describes how to diagnose issues with SA Data Collector (hereafter Data Collector). Examples were verified on version 8.9.0. API responses shown as examples of normal operation were obtained from a test environment; responses and log examples under "problem" are modeled for illustration and are not an exact copy of a specific version's output.

Conventions​

  • DATA_COLLECTOR_HOME - Data Collector home directory (LS_PATH_HOME), usually /app/logstash/
  • DATA_COLLECTOR_DATA - Data Collector data directory (LS_PATH_DATA, the path.data setting), including the queue and Dead Letter Queue; usually a separate directory /app/data/logstash/, not a subdirectory of ${DATA_COLLECTOR_HOME}
  • DATA_COLLECTOR_LOGS - Data Collector log directory (LS_PATH_LOGS, the path.logs setting); usually a separate directory /app/logs/logstash/
  • PIPELINE_ID - the pipeline ID; set by pipeline.id in ${DATA_COLLECTOR_HOME}/config/pipelines.yml, not in the pipeline's .conf configuration file
info

In a standard Search Anywhere Framework installation, DATA_COLLECTOR_DATA and DATA_COLLECTOR_LOGS are separate directories, not nested inside DATA_COLLECTOR_HOME. The actual paths are specified in ${DATA_COLLECTOR_HOME}/config/logstash.yml (path.data and path.logs) or in the ExecStart and Environment settings of the logstash.service unit. Default values are set during installation (see Installing SA Data Collector).

The commands in this article use the variables listed above. They are not set in the session by default: before running the commands, replace them with the actual paths or set the variables in your session:

export DATA_COLLECTOR_HOME=/app/logstash
export DATA_COLLECTOR_DATA=/app/data/logstash
export DATA_COLLECTOR_LOGS=/app/logs/logstash

Data Collector Diagnostic API​

Data Collector provides a separate HTTP API for diagnostics. By default, it is available locally on the Data Collector host on port 9600.

Note!

The Data Collector diagnostic API (path _node/stats/..., port 9600) is unrelated to the Data Storage API (path _nodes/stats/..., port 9200, available through the Dev Tools console). Requesting the flow metric through the Data Storage API returns an error:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/_nodes/stats/flow] contains unrecognized metric: [flow]"}],"status":400}}

Run diagnostic requests directly on the Data Collector host, not through the Dev Tools console.

Check API availability:

curl -s "http://localhost:9600?pretty"

Example response when the service is healthy:

{
  "host": "sm-logstash",
  "version": "8.9.0",
  "http_address": "127.0.0.1:9600",
  "status": "green",
  "pipeline": {
    "workers": 4,
    "batch_size": 125,
    "batch_delay": 50
  }
}

Delay or Loss of Events​

If the data collection and parsing recommendations are followed but Data Collector is overloaded and events are delayed or lost, check the metrics below.

Backpressure​

The queue_backpressure metric should be close to zero. It increases when Data Collector cannot keep up with incoming data and slows or pauses processing. Possible causes include limited throughput, high CPU load, or insufficient memory.

curl -s "http://localhost:9600/_node/stats/flow?pretty"

Example response during normal operation: queue_backpressure.current is close to zero:

{
  "flow": {
    "input_throughput": { "current": 1842.3, "lifetime": 1790.5 },
    "filter_throughput": { "current": 1840.1, "lifetime": 1788.9 },
    "output_throughput": { "current": 1839.6, "lifetime": 1788.2 },
    "queue_backpressure": { "current": 0.02, "lifetime": 0.03 },
    "worker_concurrency": { "current": 1.1, "lifetime": 1.0 }
  }
}

Example under overload: input_throughput remains high, output_throughput drops sharply, queue_backpressure approaches 1, and worker_concurrency approaches pipeline.workers (all workers are waiting to write to output):

{
  "flow": {
    "input_throughput": { "current": 1850.7, "lifetime": 1790.5 },
    "filter_throughput": { "current": 420.3, "lifetime": 1620.1 },
    "output_throughput": { "current": 180.9, "lifetime": 1600.4 },
    "queue_backpressure": { "current": 0.94, "lifetime": 0.41 },
    "worker_concurrency": { "current": 3.98, "lifetime": 2.1 }
  }
}

What to Do​

  1. Check whether the problem is on the Data Storage side. High queue_backpressure with falling output_throughput means Data Collector is waiting for writes to complete. Check the load on Data Storage nodes. If Data Storage is the cause, fix it there: adding resources to Data Collector will not help

  2. If Data Storage responds normally and worker_concurrency reaches pipeline.workers, increase pipeline.workers or pipeline.batch.size in ${DATA_COLLECTOR_HOME}/config/pipelines.yml (or in ${DATA_COLLECTOR_HOME}/config/logstash.yml if the setting is shared across pipelines). Use the number of host CPU cores as a guide and do not exceed it. Restart the service

    systemctl restart logstash.service

    Change one setting at a time. Compare the flow metrics before and after each change to determine its effect

  3. If there is no configuration bottleneck, check resource usage on the Data Collector host. To inspect per-thread load and identify the thread using the CPU, run

    top -H

    To inspect disk utilization and latency for 5 seconds, run the following command (relevant for a persisted queue; iostat is part of the sysstat package)

    iostat -x 1 5

    If CPU or memory is insufficient, check the JVM heap settings: -Xms and -Xmx in ${DATA_COLLECTOR_HOME}/config/jvm.options must match. The default heap size in a Search Anywhere Framework installation is half of the host's available RAM (see Setting Logstash Parameters). If resources remain insufficient, increase server resources or add a Data Collector node

Throughput​

Throughput shows how much data Data Collector processes per unit of time. A drop below the target may indicate unsuitable configuration or insufficient resources. The metric also helps assess the load on Data Storage, which receives data from Data Collector. If available resources are underused, increase pipeline.workers or pipeline.batch.size in the pipeline configuration.

curl -s "http://localhost:9600/_node/stats/flow?pretty"
curl -s "http://localhost:9600/_node/stats/pipelines?pretty"

A sign of a problem in the _node/stats/pipelines response is a high queue_push_duration_in_millis relative to duration_in_millis: events wait a long time in the queue before filters process them and send them to output:

{
  "pipelines": {
    "main": {
      "events": {
        "in": 216610,
        "filtered": 148200,
        "out": 148200,
        "duration_in_millis": 365495,
        "queue_push_duration_in_millis": 312466
      }
    }
  }
}

What to Do​

See Backpressure. High queue_push_duration_in_millis and low throughput usually indicate the same cause: the output plugin cannot keep up with incoming events.

Queue​

The Data Collector queue stores events awaiting processing. Two queue types are available: memory and persisted. The in-memory queue (memory) processes events faster, but events may be lost if the service fails or restarts. The disk-backed queue (persisted) saves events to disk but may reduce processing speed. Choose the queue type based on the data stream and durability requirements. Ensure that sufficient memory or disk space is available for the selected queue type.

curl -s "http://localhost:9600/_node/stats/pipelines?pretty"

Example response during normal operation of a persisted queue: queue_size_in_bytes is much lower than max_queue_size_in_bytes, and sufficient disk space is available:

{
  "queue": {
    "type": "persisted",
    "capacity": {
      "max_unread_events": 0,
      "page_capacity_in_bytes": 67108864,
      "max_queue_size_in_bytes": 1073741824,
      "queue_size_in_bytes": 3885
    },
    "data": {
      "path": "/app/data/logstash/queue/main",
      "free_space_in_bytes": 936886480896
    },
    "events_count": 12
  }
}

Example of an almost full queue: queue_size_in_bytes is close to max_queue_size_in_bytes, and little free disk space remains:

{
  "queue": {
    "type": "persisted",
    "capacity": {
      "max_unread_events": 0,
      "page_capacity_in_bytes": 67108864,
      "max_queue_size_in_bytes": 1073741824,
      "queue_size_in_bytes": 1070572288
    },
    "data": {
      "path": "/app/data/logstash/queue/main",
      "free_space_in_bytes": 52428800
    },
    "events_count": 184302
  }
}

In this state, Data Collector stops accepting new events, and messages about the full queue appear in the log.

info

For a persisted queue, also check the free space available for ${DATA_COLLECTOR_DATA}/queue/<PIPELINE_ID>. When space runs out, the pipeline stops accepting new events.

What to Do​

  1. Queue growth is usually a consequence, not the cause. First fix the issue on the output side (see Backpressure and Errors Sending Data to Data Storage)
  2. To temporarily increase queue capacity while resolving the cause, raise the queue.max_bytes limit in ${DATA_COLLECTOR_HOME}/config/pipelines.yml or ${DATA_COLLECTOR_HOME}/config/logstash.yml. Ensure sufficient disk space for the new limit, then restart the service
  3. Choose the queue type according to your requirements:
    • if event loss is unacceptable, use queue.type: persisted
    • if processing speed matters more and occasional event loss is acceptable, use queue.type: memory

Pipeline Configuration and Startup Errors​

Before starting a pipeline or after changing its configuration, check the syntax without starting it. Run the check as the service user (logstash), not as root. Otherwise, bin/logstash will create files in path.data and path.logs owned by root, which may cause a permissions error the next time the service starts:

sudo -u logstash ${DATA_COLLECTOR_HOME}/bin/logstash --config.test_and_exit -f <PIPELINE_FILE> --path.settings ${DATA_COLLECTOR_HOME}/config

If the Data Collector service fails to start or keeps restarting, check:

  • the service status

    systemctl status logstash.service

    Example of a failure: the service keeps restarting

    ● logstash.service - logstash
       Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
       Active: activating (auto-restart) (Result: exit-code) since Tue 2026-10-07 11:05:02 UTC; 3s ago
      Process: 20911 ExecStart=/app/logstash/bin/logstash (code=exited, status=1/FAILURE)

  • the main service log

    tail -n 100 ${DATA_COLLECTOR_LOGS}/logstash-plain.log

    Example of a configuration error in the log

    [2026-10-26T12:07:20,575][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:test_config_error, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of '\\\"', any character, '\"' at line 17, column 1 (byte 169) after filter {\nmutate {\n  add_field => { \"test\" => \"test }\n}\n}\n\noutput {\n    stdout {}\n}\n", :backtrace=>["/app/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "..."]}

One common cause of service failure is the absence of configured pipelines in ${DATA_COLLECTOR_HOME}/config/conf.d (see Creating a Standard Logstash Pipeline).

Another common cause is incorrect permissions on service files and directories, for example after a configuration change by another user or a restore from backup. The Data Collector service runs as logstash and needs write access to the following directories:

  • ${DATA_COLLECTOR_HOME}/config/conf.d - pipeline files
  • ${DATA_COLLECTOR_LOGS} - log files
  • ${DATA_COLLECTOR_DATA} - queue, Dead Letter Queue, sincedb

Example of a permissions error in the log (only the path in rb_sysopen differs for configuration, logs, and path.data):

[FATAL][logstash.runner] An unexpected error occurred! {:error=>#<Errno::EACCES: Permission denied @ rb_sysopen - /app/data/logstash/queue/main/checkpoint.head>, ...}

Check permissions with:

ls -la ${DATA_COLLECTOR_HOME}/config/conf.d ${DATA_COLLECTOR_LOGS} ${DATA_COLLECTOR_DATA}

What to Do​

  1. Fix the configuration based on the error from --config.test_and_exit or the log (the example shows a syntax error on line 17)

  2. If there are no pipelines, create them by following Creating a Standard Logstash Pipeline

  3. If permissions are the cause, recursively set the owner of all three directories to logstash:logstash or to the user and group that run the service

    chown -R logstash:logstash ${DATA_COLLECTOR_HOME}/config/conf.d ${DATA_COLLECTOR_LOGS} ${DATA_COLLECTOR_DATA}

  4. Restart the service and confirm that it remains in the active (running) state

    systemctl restart logstash.service
    systemctl status logstash.service

Dead Letter Queue​

The Dead Letter Queue (DLQ) stores events that could not be written through output, for example because of a mapping conflict in Data Storage. Enable it with dead_letter_queue.enable: true in ${DATA_COLLECTOR_HOME}/config/pipelines.yml. Events are stored in:

${DATA_COLLECTOR_DATA}/dead_letter_queue/<PIPELINE_ID>/

Growing data volume in this directory indicates recurring errors writing to Data Storage: a mapping mismatch, an exceeded index limit, or Data Storage being unavailable. The dead_letter_queue.max_bytes setting limits the DLQ size. After the limit is reached, new events are dropped without being written.

What to Do​

  1. Inspect the DLQ contents with an auxiliary pipeline to identify the original events and the failure reason. The rubydebug codec requires metadata => true: without it, the @metadata field containing the reason will not appear in the output. The reason is stored at [@metadata][dead_letter_queue][reason]

    input {
      dead_letter_queue {
        path => "/app/data/logstash/dead_letter_queue/<PIPELINE_ID>"
      }
    }
    output {
      stdout { codec => rubydebug { metadata => true } }
    }

  2. Fix the cause of the failure

  3. After fixing the cause, resend events from the DLQ if needed by setting the auxiliary pipeline's output to the target Data Storage index instead of stdout

  4. If the accumulated events are no longer needed, clear the DLQ in one of two ways

    • stop the pipeline and manually delete the contents of its DLQ directory; they are not removed automatically
    • without stopping the pipeline, enable clean_consumed => true for the dead_letter_queue plugin in the auxiliary pipeline's input block; it removes consumed segment files without touching those the pipeline may currently be reading

Errors Sending Data to Data Storage​

If data does not reach Data Storage, check ${DATA_COLLECTOR_LOGS}/logstash-plain.log. All pipelines write to this file, and each entry contains a pipeline_id. With multiple pipelines, filter entries by the relevant pipeline_id to identify the source of the error:

grep 'pipeline_id:main' ${DATA_COLLECTOR_LOGS}/logstash-plain.log

Check both connection retry warnings (retrying) and connection errors. The error text depends on the cause:

[WARN][logstash.outputs.opensearch] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://192.0.2.10:9200/", :exception=>Manticore::ConnectTimeout, :message=>"connect timed out after 5000 milliseconds"}
[ERROR][logstash.outputs.opensearch] Failed to install template {:message=>"PUT https://192.0.2.10:9200/_template/... failed: [401] {\"error\":\"unauthorized\"}"}
[ERROR][logstash.outputs.opensearch] SSL Exception {:message=>"javax.net.ssl.SSLHandshakeException: PKIX path building failed: unable to find valid certification path to requested target"}

Typical causes and actions:

  • the Data Storage host or port in the pipeline's output block is unreachable (indicated by ConnectTimeout or Connection refused) - check connectivity with curl -v https://<data_storage_host>:9200 and verify that the address in output matches the actual Data Storage address
  • invalid Data Storage credentials (indicated by a 401 or 403 error in the log) - check the username, password, or API key configured in the pipeline's output block
  • a TLS certificate validation error (indicated by SSLHandshakeException and PKIX path building failed) - check the certificate chain with curl -v --cacert <path to CA certificate> https://<data_storage_host>:9200 without disabling validation, replace the certificate following Replacing Old Certificates, and restart the service
  • Data Storage cannot keep up with incoming data (rejected increases in the thread pool or the circuit breaker trips) - resolve the cause on the Data Storage side (see Diagnosing Data Storage Issues). Increasing Data Collector timeouts or retry counts only masks the issue

After fixing the cause, confirm that retrying warnings no longer appear in the log and that the data volume in the queue and DLQ is decreasing.