Configuring Cluster Snapshotting in SAF

Snapshots are used for backing up indices and restoring data in case of failures.

Information

Terminology used:

• OS_IP - the IP address of one of the OpenSearch cluster servers
• OS_HOME - the home directory of SA Data Storage, typically /app/opensearch/
• BACKUP_DIR - the directory where snapshots will be stored
• REPO_NAME - the name of the snapshot repository
• SNAPSHOT_POLICY_NAME - the name of the snapshot policy
• SNAPSHOT_NAME - the name of the snapshot

Preparing Cluster Nodes

Pay attention!

Preparation must be performed on all nodes with the data role.

Pay attention!

For multi-node clusters, it is recommended to disable allocation before preparing the nodes. This can be done via the Developer Console (Main Menu - System Settings - Developer Console) executing the following command:

PUT _cluster/settings
{
 "persistent": {
 "cluster.routing.allocation.enable": "none"
 }
}

Alternatively, use the terminal with the following command:

curl -XPUT -k -u admin "https://$OS_IP:9200/_cluster/settings?pretty" -H "Content-Type: application/json" -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'

After all cluster nodes are prepared, re-enable allocation:

PUT _cluster/settings
{
 "persistent": {
 "cluster.routing.allocation.enable": "all"
 }
}

Or use the terminal:

curl -XPUT -k -u admin "https://$OS_IP:9200/_cluster/settings?pretty" -H "Content-Type: application/json" -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'

Before configuring snapshotting, you must create a directory where SAF will store backups. Open a terminal as the root user and execute the required commands to create the directory and set the appropriate permissions:

1. Create a directory for storing snapshots and grant read/write permissions to the opensearch:

mkdir -p $BACKUP_DIR
chown -R opensearch:opensearch $BACKUP_DIR

2. Edit the node configuration file $OS_HOME/opensearch.yml using any text editor, and add the path.repo parameter pointing to the created directory:

path.repo: ["{BACKUP_DIR}"]

Information

It is not recommended to place path.repo on the same disk as the node's data directory.

Pay attention!

The specified path must be identical across all nodes participating in snapshot creation. If even a single node lacks access to this path, the snapshot operation may fail.

3. Restart the node to apply the changes:

systemctl restart opensearch

Preparing the Snapshot Repository

A snapshot repository must be created with the necessary parameters. Supported storage types include FS, S3, HDFS, and others.

Pay attention!

This article provides a configuration example for a snapshot repository using FS (File System) as the storage type.

PUT /_snapshot/{REPO_NAME}
{
  "type": "fs",
  "settings": {
    "location": "{BACKUP_REPO}"
  }
}

List of FS Snapshot Repository Parameters:

Parameter	Description
location	Directory where snapshots will be stored.
chunk_size	Splits large files into smaller chunks during snapshot creation (e.g., 64MB, 1GB). Default: 1gb. (Optional)
compress	Boolean Whether to compress metadata files. Default: false. (Optional)
max_restore_bytes_per_sec	Maximum snapshot restore speed. Default: 40 MB/s. (Optional)
max_snapshot_bytes_per_sec	Maximum snapshot creation speed. Default: 40 MB/s. (Optional)
remote_store_index_shallow_copy	Boolean. Determines whether to store index snapshots as shallow copies. Default: false. (Optional)
shallow_snapshot_v2	Boolean. Enables second-generation shallow snapshotting. Default: false. (Optional)
readonly	Boolean. Whether the repository is read-only. Default: false. (Optional)

Configuring Automatic Snapshots

To automate cluster snapshotting, create a snapshot policy using the Developer Console (Main Menu - System Settings - Developer Console) by executing a command with the required parameters:

POST _plugins/_sm/policies/{SNAPSHOT_POLICY_NAME}
{
  "name": "snapshot-daily-{{date}}",
  "description": "Daily snapshot policy",
  "creation": {
    "schedule": {
      "cron": {
        "expression": "0 8 * * *",
        "timezone": "UTC"
      }
    },
    "time_limit": "1h"
  },
  "deletion": {
    "schedule": {
      "cron": {
        "expression": "0 8 * * *",
        "timezone": "UTC"
      }
    },
    "condition": {
      "max_age": "7d",
      "max_count": 50,
      "min_count": 30
    },
    "time_limit": "1h"
  },
  "snapshot_config": {
    "date_format": "yyyy-MM-dd-HH:mm",
    "timezone": "UTC",
    "indices": [".*"],
    "repository": "{REPO_NAME}",
    "ignore_unavailable": "true",
    "include_global_state": "false",
    "partial": "true",
    "metadata": {
      "any_key": "any_value"
    }
  }
}

Snapshot policy parameter table:

Parameter	Type	Description
description	String	Description of the snapshot policy. (Optional)
enabled	Boolean	Whether the policy should be enabled upon creation. (Optional)
snapshot_config	Object	Configuration settings for the snapshot. (Required)
snapshot_config.date_format	String	Snapshot names follow the forma {SNAPSHOT_POLICY_NAME}-<data>-<random number>. date_format defines the date format in the snapshot name. Default: yyyy-MM-dd’T’HH:mm:ss. (Optional)
snapshot_config.date_format_timezone	String	Snapshot names follow the forma {SNAPSHOT_POLICY_NAME}-<data>-<random number>. date_format_timezone Time zone used for the date in snapshot names. Default: UTC. (Optional)
snapshot_config.indices	String	Pattern for the indices to include in snapshots. Default: *. (all indexes)
snapshot_config.repository	String	Name of the repository where snapshots will be stored. (Required)
snapshot_config.ignore_unavailable	Boolean	Whether to ignore unavailable indices. Default: false. (Optional)
snapshot_config.include_global_state	Boolean	Whether to include cluster state in the snapshot. Default: true. (Optional)
snapshot_config.partial	Boolean	Allows creation of partial snapshots. Default: false. (Optional)
snapshot_config.metadata	Object	Key-value metadata associated with the snapshot. (Optional)
creation	Object	Settings for snapshot creation. (Required)
creation.schedule	String	Cron expression defining the snapshot schedule. (Required)
creation.time_limit	String	Maximum duration to wait for snapshot creation to complete. If time_limit exceeds the interval between scheduled snapshots, the next snapshot will not start until the previous one completes. (Optional)
deletion	Object	Settings for snapshot deletion. (Optional) By default, all snapshots are retained.
deletion.schedule	String	Cron expression for snapshot deletion. (Optional) Defaults to the creation.schedule setting.
deletion.time_limit	String	Maximum time allowed for completing snapshot deletion. (Optional)
deletion.delete_condition	Object	Conditions that trigger snapshot deletion. (Optional)
deletion.delete_condition.max_count	Integer	Maximum number of snapshots to retain. (Optional)
deletion.delete_condition.max_age	String	Maximum age for retained snapshots. (Optional)
deletion.delete_condition.min_count	Integer	Minimum number of snapshots to retain. Default: 1. (Optional)
notification	Object	Notification settings for snapshot policy events (requires a configured OpenSearch notification channel). (Optional)
notification.channel	Object	Defines the notification channel. (Required)
notification.channel.id	String	ID of the notification channel. (Required)
notification.conditions	Object	Events that trigger notifications — set to true to enable.
notification.conditions.creation	Boolean	Whether to notify when a snapshot is created. Default: true. (Optional)
notification.conditions.deletion	Boolean	Whether to notify when a snapshot is deleted. Default: false. (Optional)
notification.conditions.failure	Boolean	Whether to notify on snapshot creation or deletion failure. Default: false. (Optional)
notification.conditions.time_limit_exceeded	Boolean	Whether to notify when snapshot operations exceed time_limit. Default: false. (Optional)

Information

Snapshots are incremental — previously saved segments are not duplicated.

Manual Snapshot Creation

To create a one-time snapshot, execute the following command in the Dev Console:

PUT _snapshot/{REPO_NAME}/{SNAPSHOT_NAME}
{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": false
}

Restoring from a Snapshot

1. View available snapshots using the following command in the Dev Console:

GET _snapshot/{REPO_NAME}/_all

The output will list all created snapshots in the following format, where the snapshot field indicates the snapshot name (SNAPSHOT_NAME):

{
  "snapshots": [
    {
      "snapshot": "daily-snapshot-sm-policy-2025-04-21-14:30-bu2lfnek",
      "uuid": "iqHGMwR5T6yV-tInX3a5KQ",
      "version_id": 136397827,
      "version": "2.18.0",
      "remote_store_index_shallow_copy": false,
      "indices": [
        "index1",
        "index2"
      ],
      "data_streams": [],
      "include_global_state": true,
      "metadata": {
        "sm_policy": "daily-snapshot-sm-policy"
      },
      "state": "SUCCESS",
      "start_time": "2025-04-21T14:30:02.327Z",
      "start_time_in_millis": 1745245802327,
      "end_time": "2025-04-21T14:31:09.966Z",
      "end_time_in_millis": 1745245869966,
      "duration_in_millis": 67639,
      "failures": [],
      "shards": {
        "total": 2,
        "failed": 0,
        "successful": 2
      }
    }
  ]
}

2. Restore the desired snapshot using the following command in the Dev Console:

POST _snapshot/{REPO_NAME}/{SNAPSHOT_NAME}/_restore

Snapshot Deletion

1. View available snapshots using the following command in the Dev Console:

GET _snapshot/{REPO_NAME}/_all

This will return the list of snapshots in the same format as described in the Restoring from a Snapshot section.

2. Delete the desired snapshot using the following command:

DELETE _snapshot/{REPO_NAME}/{SNAPSHOT_NAME}