Skip to main content
Version: 6.2

Diagnosing SA Data Storage Issues

This article describes how to diagnose SA Data Storage (hereafter Data Storage). Default thresholds and limits are based on the dedicated articles linked in the text and under "Useful Links". Example responses for normal operation were obtained from a test environment; responses and examples under "problem" are modeled for illustration.

Conventions​

  • OS_LOGS - Data Storage log directory; separate from the Data Storage home directory, usually /app/logs/opensearch/

Before running the commands, replace the variable with the actual path for your installation. Its default value is set during installation (see Installing Search Anywhere Framework Master Node and Search Anywhere Framework Data Storage).

info

Run all requests below in the Dev Tools console (Main Menu - Settings - Dev Tools) or send equivalent requests to Data Storage with curl.

Cluster State​

Check the overall cluster state with:

GET _cluster/health

Key response fields:

  • status - green (all shards allocated), yellow (replicas not allocated), or red (primary shards not allocated, some data unavailable)
  • unassigned_shards - the number of unassigned shards
  • number_of_nodes and number_of_data_nodes - the actual number of cluster nodes; compare with the expected count

Example response during normal operation:

{
  "cluster_name": "smartmonitor",
  "status": "green",
  "number_of_nodes": 5,
  "number_of_data_nodes": 2,
  "active_primary_shards": 182,
  "active_shards": 364,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "active_shards_percent_as_number": 100.0
}

Example of a failure: some primary shards are unallocated, and some data is unavailable:

{
  "cluster_name": "smartmonitor",
  "status": "red",
  "number_of_nodes": 4,
  "number_of_data_nodes": 2,
  "active_primary_shards": 178,
  "active_shards": 350,
  "relocating_shards": 0,
  "initializing_shards": 2,
  "unassigned_shards": 6,
  "delayed_unassigned_shards": 6,
  "number_of_pending_tasks": 0,
  "active_shards_percent_as_number": 97.2
}

If status is not green, see Shard Allocation and Disk Space and Problem Indices.

Shard Allocation and Disk Space​

Check shard allocation across nodes and disk usage:

GET _cat/allocation?v

Example during normal operation: disk usage is approximately even across nodes:

shards disk.indices disk.used disk.avail disk.total disk.percent host         node
   182       45.2gb    52.1gb    147.9gb     200gb           26 192.0.2.10 smos-datawarm1
   180       44.8gb    51.6gb    148.4gb     200gb           25 192.0.2.11 smos-datawarm2

Example of a failure: one node has exceeded the high Disk Watermark threshold and stopped accepting new shards (shards = 0, disk.percent close to 100):

shards disk.indices disk.used disk.avail disk.total disk.percent host         node
   182       45.2gb    52.1gb    147.9gb     200gb           26 192.0.2.10 smos-datawarm1
     0      178.9gb   189.3gb      9.1gb     200gb           95 192.0.2.11 smos-datawarm2

Get a list of unassigned shards with brief reasons without querying each shard separately:

GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason

To get a detailed explanation for the first unassigned shard found, run the request without specifying a shard:

GET _cluster/allocation/explain

If you need an explanation for a specific shard found with _cat/shards, specify its parameters:

GET _cluster/allocation/explain
{
  "index": "<INDEX_NAME>",
  "shard": 0,
  "primary": true
}

Example response when the disk threshold is reached: the deciders field contains disk_threshold:

{
  "index": "winlog-000123",
  "shard": 0,
  "primary": true,
  "current_state": "unassigned",
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_name": "smos-datawarm2",
      "node_decision": "no",
      "deciders": [
        {
          "decider": "disk_threshold",
          "decision": "NO",
          "explanation": "the node is above the high watermark cluster setting [cluster.routing.allocation.disk.watermark.high=90%], having less than the minimum required [10%] free disk on node, actual free: [4.5%]"
        }
      ]
    }
  ]
}

One common reason for unassigned shards is the disk threshold being reached on data nodes. For details about the mechanism and protection levels, see Disk Watermark Configuration.

What to Do​

  1. Free up disk space on the affected node: delete old indices manually or speed up the ISM policy. This addresses the cause, not just the symptom

  2. If space has been freed but shards are not reallocated automatically, retry allocation

    POST _cluster/reroute?retry_failed=true

  3. Raise the cluster.routing.allocation.disk.watermark.* thresholds only temporarily if no free space remains and you need time to expand the disk or add a node (see Disk Watermark Configuration). Restore the previous values after resolving the cause

info

If _cluster/allocation/explain points to cluster.routing.allocation.enable rather than insufficient disk space, shard allocation was explicitly disabled earlier, for example during cluster maintenance. Check which block contains the setting: transient or persistent:

GET _cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation.enable

Change the setting in the same block where it was set. The transient block takes priority over persistent, so a PUT to persistent will not override a value from transient. If the setting is in persistent:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}

If the setting is in transient, change it in that block. The null value removes the transient override, after which the value from persistent or the default applies:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": null
  }
}

If a primary shard is lost permanently, for example with the node holding its only copy, the standard steps above will not help. Data can only be recovered from a snapshot (see Configuring Search Anywhere Framework Cluster Snapshots).

Last resort - data loss

If there is no snapshot, and keeping the index working matters more than the recent data in the lost shard, you can force-allocate an empty primary:

POST _cluster/reroute
{
  "commands": [
    {
      "allocate_empty_primary": {
        "index": "<INDEX_NAME>",
        "shard": 0,
        "node": "<NODE_NAME>",
        "accept_data_loss": true
      }
    }
  ]
}

An index can have several primary shards (0, 1, 2, and so on). The command requires a specific shard (shard) and node (node), not just an index.

If a node retains a stale copy of the shard, for example after temporary unavailability, use allocate_stale_primary instead of allocate_empty_primary with the same parameters. It promotes the retained copy to primary, resulting in less data loss than allocating an empty shard.

allocate_empty_primary permanently deletes the lost shard's data. Use this command only when no stale copy exists and recovery from a snapshot is impossible or not required.

Node Load​

Check node load: JVM heap, disk, and CPU usage:

GET _cat/nodes?v&h=name,heap.percent,disk.used_percent,cpu,load_1m

Example response during normal operation:

name           heap.percent disk.used_percent cpu load_1m
smos-datawarm1           45                52   8    1.20
smos-datawarm2           48                51  11    1.35

Example under overload: high JVM heap and disk usage with sustained CPU load:

name           heap.percent disk.used_percent cpu load_1m
smos-datawarm1           92                94  87    7.60
smos-datawarm2           49                51  12    1.40

CPU​

What to Do​

  1. Identify threads causing CPU load with the Hot Threads API

    GET _nodes/smos-datawarm1/hot_threads?threads=5&interval=1s

    The response shows the busiest threads and their call stacks (search, indexing, segment merges, and so on)

  2. If long-running search queries cause the load, find them with the Tasks API

    GET _tasks?actions=*search&detailed

    If necessary, cancel a stuck or resource-intensive query without waiting for it to finish

    POST _tasks/<task_id>/_cancel

  3. To limit similar queries in the future, use the cluster settings search.max_buckets (the aggregation bucket limit) and search.allow_expensive_queries: false, as well as SME Circuit Breaker and search limits

    Note!

    search.allow_expensive_queries: false disallows wildcard, regexp, and script queries, along with some other resource-intensive query types. Before applying it on a production cluster, check in a test environment which SAFL queries and dashboards use these types: they will stop working after the setting is enabled.

  4. If indexing rather than search causes the load, check Data Collector (see Throughput)

  5. If node resources are consistently insufficient, increase available CPU cores or add a data node to the cluster

Heap​

What to Do​

See Circuit Breaker: high JVM heap usage and circuit breaker trips have related causes.

Thread Pool​

Check thread pool queues and rejection counts. Growth in these values indicates that the node cannot keep up with write or search requests:

GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected

Example response during normal operation (rejected = 0):

node_name       name   active queue rejected
smos-datawarm1  write       2     0        0
smos-datawarm1  search      4     0        0

Example under overload: rejected grows and write requests are rejected. On the Data Collector side, this appears as repeated write attempts (see Diagnosing Data Collector Issues):

node_name       name   active queue rejected
smos-datawarm1  write       8   124      347
smos-datawarm1  search     13    89       52

What to Do When rejected Is Growing​

  1. Reduce the incoming load at its source. For the write pool, this is the Data Collector stream (see Throughput and Backpressure); you may need to slow it down. For the search pool, reduce the number or resource usage of concurrent search queries
  2. Do not treat an increase in the relevant thread pool's queue_size as a permanent fix: it delays resource exhaustion rather than resolving it
  3. If the load is expected, for example because data volume is growing, add data nodes to the cluster

Circuit Breaker​

Check circuit breaker trips: the node forcibly stops a request when memory is at risk of exhaustion:

GET _nodes/stats/breaker

Example response during normal operation (tripped = 0):

{
  "nodes": {
    "a1b2c3": {
      "name": "smos-datawarm1",
      "breakers": {
        "parent": {
          "limit_size_in_bytes": 1037252096,
          "estimated_size_in_bytes": 123456789,
          "overhead": 1.0,
          "tripped": 0
        }
      }
    }
  }
}

Example under overload: estimated_size_in_bytes is close to limit_size_in_bytes, and tripped increases when requests are rejected. A circuit_breaking_exception error appears in the log and response:

{
  "nodes": {
    "a1b2c3": {
      "name": "smos-datawarm1",
      "breakers": {
        "parent": {
          "limit_size_in_bytes": 1037252096,
          "estimated_size_in_bytes": 1021345678,
          "overhead": 1.0,
          "tripped": 14
        }
      }
    }
  }
}

If the circuit breaker trips regularly, check sme.breaker.* (see Using the Circuit Breaker in SME).

To assess JVM heap pressure, check the percentage of heap memory used on the node:

GET _nodes/stats/jvm?filter_path=nodes.*.jvm.mem.heap_used_percent
note

If the node uses the G1GC garbage collector, also check old-generation pool usage: GET _nodes/stats?filter_path=nodes.*.jvm.mem.pools.old.

If heap_used_percent remains close to the parent circuit breaker threshold (95% by default), the node is regularly short on JVM heap memory. Check the node log: frequent garbage collection (GC) entries consuming a significant share of time confirm the shortage.

What to Do​

The circuit breaker is a protective mechanism, not the cause of the problem. Occasional trips during resource-intensive one-off queries are acceptable.

Regular growth in tripped together with persistently high heap_used_percent indicates insufficient JVM heap memory for the current load. Apply the same measures as for high CPU load: find and limit resource-intensive queries using the Hot Threads API, Tasks API, search.max_buckets, and search.allow_expensive_queries. If the load is caused by data volume, increase JVM heap memory or add nodes.

Lowering sme.breaker.limit to resolve a memory shortage is not recommended: it only increases the number of interrupted requests.

Problem Indices​

Get the list of indices with a yellow or red status:

GET _cat/indices?v&health=yellow
GET _cat/indices?v&health=red

During normal operation, both requests return an empty list. Example of a failure:

health status index            uuid       pri rep docs.count store.size
yellow open   winlog-000123    k3F8s...     1   1    1245833    512.4mb
red    open   syslog-000045    p9A2x...     1   1     845213    301.1mb

Typical causes and actions:

  • not enough nodes for the configured number of replicas (yellow) - add a data node to the cluster or reduce number_of_replicas for the index if replica redundancy is not required
  • disk threshold reached (red or yellow) - see Shard Allocation and Disk Space
  • field mapping conflict when writing data from Data Collector - convert the field type in the Data Collector pipeline before sending the data (see Dead Letter Queue) or adjust the index mapping template. An index with an incorrect field type already configured must be reindexed (see Reindex)
  • uncontrolled growth in the number of mapping fields (mapping explosion when parsing heterogeneous logs with dynamic mapping) - limit the number of fields using index.mapping.total_fields.limit in the index template and adjust Data Collector parsing so it does not create a separate field for every unique value. One possible cause is JSON with arbitrary keys parsed as separate fields instead of a nested object

If indices keep growing without rollover, configure an index lifecycle management policy (see Index Management).

Search Anywhere Framework Master Node​

Identify the current master node and check quorum:

GET _cat/cluster_manager
GET _cluster/state/master_node

Example during normal operation: a master node is elected, and its ID matches in both responses:

id                     host         ip           node
ebTq31e_RMi9lpmnsdBl0w 192.0.2.12 192.0.2.12 smos-master1

If no master node is elected because quorum is lost, GET _cat/cluster_manager returns an empty response. Requests that change cluster state, such as creating an index or updating settings, may hang or time out.

Check cluster tasks waiting for execution, for example when quorum is lost or the master node is overloaded:

GET _cluster/pending_tasks

Example during normal operation: the task list is empty:

{
  "tasks": []
}

Example of a failure: a task has remained in the queue for a long time (time_in_queue_millis is measured in milliseconds):

{
  "tasks": [
    {
      "insert_order": 101,
      "priority": "URGENT",
      "source": "shard-failed",
      "time_in_queue_millis": 184320
    }
  ]
}

An unavailable master node or lost quorum blocks cluster state changes, including index creation and shard reallocation.

What to Do​

Next steps depend on whether quorum exists. Check it with GET _cat/cluster_manager from the previous section.

Quorum exists (a master node is elected, and another node must be removed):

  1. If a node is temporarily unavailable because of a network issue or reboot, restore it. Quorum remains as long as a majority of nodes is available

  2. If a node is permanently lost or is being removed as planned, exclude it from the voting configuration. This changes cluster state and requires an active master node, so it can only be done while quorum exists

    POST _cluster/voting_config_exclusions?node_names=<NODE_NAME>

    After stopping and removing the node, clear the exclusion; it is not removed automatically

    DELETE _cluster/voting_config_exclusions?wait_for_removal=false

  3. For a planned removal of a node with the master role, use the Excluding a Node with the master Role procedure, which includes both actions above and additional configuration steps

No quorum (GET _cat/cluster_manager returns an empty response, and no master node is elected):

  1. Do not run POST _cluster/voting_config_exclusions: without an active master node, the command will hang. Restore enough nodes with the master role to reach a majority of the current voting configuration. If they are only temporarily unavailable because of a network issue or reboot, wait for them to return; quorum will recover automatically
  2. Do not try to restore quorum by adding a new node with the master role. Changing the voting configuration itself requires quorum and is impossible until quorum is restored; forcing a change may make recovery harder

If a majority of master nodes is lost permanently and no snapshot exists, use the offline opensearch-node tool to restore quorum.

Last resort

The opensearch-node unsafe-bootstrap and detach-cluster commands change cluster metadata without consensus and can cause data loss. Use them only if a majority of master nodes is permanently lost and no snapshot exists. Where possible, restore the cluster from a snapshot first.

Run all commands on stopped nodes. The tool is in the Data Storage installation directory, usually /app/opensearch/bin/opensearch-node:

  1. Select a surviving node with the master role and the most recent state. The tool prints the Current node cluster state (term, version) pair; if several master nodes survive, choose the node with the highest (term, version) pair

  2. Stop the Data Storage service on the selected node (systemctl stop opensearch). The tool only works on a stopped node; otherwise, the data directory is locked (node.lock)

  3. Bootstrap the cluster on that node and confirm the prompt

    /app/opensearch/bin/opensearch-node unsafe-bootstrap

    The Cluster-manager node was successfully bootstrapped message indicates success

  4. On every other node returning to the cluster (both master-eligible and data), stop the service and detach the node from the old cluster. Otherwise, it cannot join because the cluster UUID does not match

    /app/opensearch/bin/opensearch-node detach-cluster

  5. Start the Data Storage service on all nodes (systemctl start opensearch). The node from step 3 becomes the master node, and the others join it

  6. Verify recovery with GET _cat/cluster_manager (a master node is elected), GET _cluster/health (green/yellow), and GET _cat/nodes

Logs​

Data Storage node errors, such as circuit_breaking_exception, OutOfMemoryError, and CorruptIndexException, are recorded in the cluster log on each node:

${OS_LOGS}/<cluster_name>.log