Policies

Description

Policies are index management configurations that describe:

• the states an index can be in, representing stages of its lifecycle (e.g., hot, warm, delete, etc.)
• the actions to be performed on indexes in a given state (e.g., relocating to cold storage nodes or deleting the index)
• the conditions that must be met for an index to transition from one state to another. For example, if the index age exceeds eight weeks, it should transition to a state where a delete action is defined

You can define any number of states, transitions between them, and actions to be performed in each state within a policy.

The following table lists the corresponding policy fields available when creating a policy via the interface (Main Menu - Index Management - Index Policies - Create policy):

Field Identifier	Field Name	Description	Type	Required	Editable
policy_id	Policy ID	The identifier (display name) of the policy	string	Yes	No
description	Description	A description of the policy	string	Yes	Yes
ism_template	ISM Templates	Index patterns to which the policy will apply	array<object>	No	Yes
states	States	States defined in the policy	array<object>	Yes	Yes

States

A state describes the current status of a managed index. A managed index can only be in one state at a time.

The following table describes the parameters that can be configured for a state:

Field Identifier	Field Name	Description	Type	Required
name	State Name	The name of the state	string	Yes
actions	Actions	The actions to perform upon entering the state. See the "Actions" section for more details	array<object>	Yes
Defined by order	Order	Position relative to other states. Selected via a dropdown containing the names of created states	string	Yes
transitions	Transitions	Next states and the conditions required to transition to them. If no transitions are defined, the policy is considered complete and stops managing the index. If multiple transitions exist, the first condition to match will be applied. See the "Transitions" section for details.	array<object>	Yes

Actions

Actions are the steps that a policy sequentially performs on an index when it enters a specific state.

ISM (Index State Management) executes actions in the order they are defined. For example, if actions [A, B, C, D] are specified, ISM performs action A and then goes into a sleep period, as configured by the cluster setting plugins.index_state_management.job_interval. After the sleep interval, ISM continues executing the remaining actions one by one. However, if ISM fails to successfully perform action A, the operation stops and actions B, C, and D are not executed. In such cases, a retry mechanism is available (described below).

Optionally, a timeout period can be defined for an action. If the timeout is exceeded, the operation is forcibly terminated. For example, if the timeout is set to 1d and the action is not completed within one day (even after retries), the action fails.

Each action can be configured with timeout and retry settings:

Field Identifier	Field Name	Description	Type	Required	Default
timeout	Timeout	Maximum allowed duration for the action	string (time unit, e.g., 30m, 4h, 7d)	No	Not set
count	Retry Count	Number of times the action should be retried upon failure	number	Yes	Not set
backoff	Retry Backoff Policy	Backoff function used between retry attempts	string (Exponential, Constant, Linear)	No	Exponential
delay	Retry Delay	Waiting period between retry attempts	string (time unit)	No	Not set

The following example configures an action with a timeout of one hour. The policy retries the action up to three times using exponential backoff, with an initial delay of 10 minutes:

Equivalent JSON configuration:

{
  "actions": {
    "timeout": "1h",
    "retry": {
      "count": 3,
      "backoff": "exponential",
      "delay": "10m"
    }
  }
}

Other parameters are defined by the specific action itself.

Available Actions

ISM supports the following actions:

• force_merge
• read_only
• read_write
• replica_count
• shrink
• close
• open
• delete
• rollover
• notification
• snapshot
• index_priority
• allocation
• rollup
• Move to ClickHouse

Force Merge

Reduces the number of index segments by merging shard segments. Before starting the merge, ISM attempts to set the index to read-only.

Parameter	Description	Type	Required
max_num_segments	The number of segments to reduce the shard to	number	Yes
wait_for_completion	Whether to wait for the action to complete or return immediately after start	boolean	No
task_execution_timeout	Timeout for task execution. Only applies if wait_for_completion is false	string (time unit)	No

Example configuration:

{
  "force_merge": {
    "max_num_segments": 1
  }
}

Read Only

Sets the managed index to read-only mode.

Example configuration:

{
  "read_only": {}
}

Read Write

Sets the index to read-write mode.

Example configuration:

{
  "read_write": {}
}

Replica Count

Sets the number of replicas for the index.

Parameter	Description	Type	Required
number_of_replicas	Defines the number of replicas to assign to the index.	number	Yes

Example configuration:

{
  "replica_count": {
    "number_of_replicas": 2
  }
}

Shrink

Allows reducing the number of primary shards in an index. This action supports the following parameters:

Parameter	Description	Type	Required
num_new_shards	The maximum number of primary shards in the shrunken index.	integer	Yes, but cannot be used with max_shard_size or percentage_of_source_shards
max_shard_size	The maximum shard size in bytes for the target index.	string (size unit, example values: 500mb, 1gb)	Yes, but cannot be used with num_new_shards or percentage_of_source_shards
percentage_of_source_shards	The percentage of the source primary shards to shrink. This parameter specifies the minimum percentage to use when shrinking the number of primary shards. Must be in the range from 0.0 to 1.0 (exclusive).	number	Yes, but cannot be used with max_shard_size or num_new_shards
target_index_name_template	The name of the shrunken index. Accepts strings and Mustache variables (e.g., {"source": "_shrunken"}).	string	No
aliases	Aliases to add to the new index.	array<object>	No
force_unsafe	Whether to perform the action even when no replicas are present.	boolean	No

Example configuration:

{
  "shrink": {
    "num_new_shards": 1,
    "target_index_name_template": {
      "source": "_shrunken"
    },
    "aliases": [
      {
        "my-alias": {}
      }
    ],
    "force_unsafe": false
  }
}

If aliases need to be assigned, the aliases parameter should contain an array of alias objects. For example:

"aliases": [
  {
    "my-alias": {}
  },
  {
    "my-second-alias": {
      "is_write_index": false,
      "filter": {
        "multi_match": {
          "query": "QUEEN",
          "fields": ["speaker", "text_entry"]
        }
      },
      "index_routing" : "1",
      "search_routing" : "1"
    }
  },
]

Delete

Deletes the index.

Example configuration:

"delete": {}

Rollover

Creates a new index when the current one exceeds specified limits (size, age, or document count).

Parameter	Description	Type	Required
min_index_age	Minimum index age	string (time unit)	No, only one of the parameters must be specified.
min_doc_count	Minimum document count	integer	No, only one of the parameters must be specified.
min_size	Minimum total index size	string (size unit)	No, only one of the parameters must be specified.
min_primary_shard_size	Minimum primary shard size	string (size unit)	No, only one of the parameters must be specified.

Example configuration:

"rollover": {
  "min_size": "100gb",
  "min_index_age": "1d",
  "min_primary_shard_size": "20gb",
  "copy_alias": false
}

Allocation

Updates the index allocation settings, causing it to move to nodes with specified attributes.

Parameter	Description	Type	Required
require	The index is allocated only to nodes whose attributes match all of the specified values.	object	No
include	The index may be allocated to nodes whose attributes match at least one of the specified values.	object	No
exclude	The index is not allocated to nodes whose attributes match any of the specified values.	object	No
wait_for	If set to true, the policy waits until all shards are relocated. If false (default), the action is async.	boolean	No

"allocation": {
  "require": {
    "routing_mode": "cold"
  },
  "include": {},
  "exclude": {},
  "wait_for": false
}

Please note!

When configuring this action, consider any existing index allocation rules applied to indices under this policy. For example, if the original index configuration includes include: hot, then you should only use the include field in the action settings (without require). Using both include and require may result in conflicting conditions, preventing shards from being allocated to any node.

Move to ClickHouse

Transfers data to ClickHouse.

Parameter	Description	Type	Required
table_name_template	A regular expression defining the table name in ClickHouse based on the index name. A named capture group in the regex will be used as the table name. The capture group must be named name.	string (e.g., (?<name>.*?)-\d+)	Yes
connection_id	The ClickHouse connection ID configured in the OpenSearch settings.	string	Yes
fetch_size	The number of events sent to ClickHouse in a single batch.	integer	No
data_scheme_parameters	Additional settings affecting interaction with the table in ClickHouse.	object	No

Schema data_scheme_parameters

Parameter	Description	Type	Required
ttl_interval	Document TTL interval in table in interval <count> <time unit> format Available intervals (set for entire table during creation).	string	No
ttl_column	Name of datetime type field used for TTL calculation.	string	No
cluster_name	ClickHouse cluster name (required when replication = true or distributed = true).	string	Yes, if replication or distributed is specified
replication	Flag for creating replicated table during data migration (considered if migration initiates new table creation).	boolean	No
distributed	Flag for creating distributed table during data migration (considered if migration initiates new table creation).	boolean	No
click_server_settings	A key-value map used to pass ClickHouse settings to the client that connects to ClickHouse during data migration.	object	No
override_field_types	A map used to override source index field types with ClickHouse types, for example, {"integer_field": "Int64"}. Nested fields are specified using dot notation, for example, human.age. If a parent nested field is overridden, its child fields are not created separately. Instead, the entire value is stored in the parent field as a String containing a JSON structure.	object	No
override_field_date_formats	A map used to specify date formats for fields whose values differ from the standard ISO 8601 format. One format can be specified per field, for example, "dd.MM.yyyy HH:mm:ss". After processing, the date is stored in the resulting ClickHouse table in the standard format.	object	No
view_json_as_string	Determines whether JSON columns are represented as String columns in a VIEW. Required for ClickHouse versions that do not support using JSON columns in a VIEW.	boolean	No
json_fields	A list of paths whose child fields are migrated to ClickHouse as a single JSON object. Required for migrating dynamically generated fields, for example, a1, a2, and a3.	string[]	No
blacklist	A list of paths that are ignored and not migrated. Path prefixes can be specified, for example, host.*.	string[]	No
whitelist	A list of paths that are migrated even if they match the blacklist. Path prefixes can be specified, for example, host.*.	string[]	No
timestamp_field_name	The name of the field that stores the event timestamp. Used when the field name differs from @timestamp.	string	No
codec	The name of the codec to use as the default codec, for example, ZSTD(3).	string	No
table_name_template	A template used to rename the target table. It can use the name extracted with index_name_pattern through the {{name}} token, for example, {{name}}-suffix.	string	No
source_read_mode	The source document read mode. Three modes are available: DEFAULT — the default mode, where the source document uses a standard JSON structure; FLAT — used when the source document was indexed in a flattened format, for example, {"user.firstname": "ivan", "user.lastname": "ivanov"}; MIXED — used when some fields were stored in a flattened format and others in a standard nested format.	string	No
cb_retries_max_retries	The maximum number of task retry attempts after the Circuit Breaker is triggered. After the limit is exceeded, the task fails.	long	No
cb_retries_initial_delay_millis	The initial delay before the first retry attempt after the Circuit Breaker is triggered. The value is specified in milliseconds.	long	No
cb_retries_max_delay_millis	The maximum delay between retry attempts. The delay increases with each attempt but does not exceed the specified value.	long	No

Allowed Mapping Changes

It is allowed to change the precision for date and numeric types to a higher or equal granularity.

From	To
date	date_nanos
date_nanos	date
byte	integer, long, short
short	integer, long
integer	long
float	half_float, scaled_float, double
half_float	float, scaled_float, double
scaled_float	float, half_float, double

Cluster Settings Related to the ClickHouse Data Migration Process

Setting	Default Value	Description
plugins.sm.move_to_click.circuit_breaker.limit	"30%"	The Circuit Breaker limit used to control the amount of memory consumed when transforming events. When the limit is reached, migration of the events that caused the limit to be exceeded is stopped.
plugins.sm.move_to_click.circuit_breaker.overhead	1.0	A constant by which event size estimates are multiplied to produce the final estimate.
plugins.sm.move_to_click.estimation.first_n_batch	10	The number of event batches used to calculate the coefficient for estimating the size of transformed events.
plugins.sm.move_to_click.estimation.each_n_batch	10	Determines how often the coefficient used to estimate the size of transformed events is adjusted. The coefficient is recalculated every N batches.

Delete by Max Size

This action deletes indices so that their total size in each ISM template group does not exceed the specified max_size value.

Each time the ISM policy is recalculated, the action processes ISM template groups separately. For each group, it creates a list of indices matching the specified patterns. The indices are sorted by creation date, from oldest to newest.

info

The Delete by Max Size action processes only open indices. Closed indices are excluded from the group’s total size calculation and are not added to the deletion list.

Parameter	Name	Description	Required
patterns_size	Index Groups	An array of settings for ISM template groups	Yes
patterns	Index Patterns	An array of ISM templates included in a single group	Yes
max_size	Maximum Total Size	The maximum total size of indices in the group	Yes
max_index_age	Maximum Index Age	The period after which an index is deleted based on its creation date	No

tip

For the patterns parameter, it is recommended to specify index patterns configured in ISM templates rather than individual index names.

Using patterns automatically includes new indices created within the ISM template in the group without requiring policy changes.

info

If the total size of the indices in a group exceeds max_size, the action adds the oldest indices to the deletion list one by one until the total size of the remaining indices falls below the specified limit.

If an index is older than the specified max_index_age, it is deleted regardless of whether the group’s total size exceeds max_size.

An example policy configuration and deletion calculation is provided in Configuring a Policy with Delete by Max Size.

{
  "delete_by_max_size": {
    "patterns_size": [
      {
        "patterns": [
          "patterns1*"
        ],
        "max_size": "5tb",
        "max_index_age": ""
      },
      {
        "patterns": [
          "patterns2*",
          "patterns3*"
        ],
        "max_size": "100gb",
        "max_index_age": "5d"
      }
    ]
  }
}

---

Policy Configuration Examples

General Information

To get started, create a new policy. Navigate to Index Policies (Main menu - System settings - Index management - Index policies) and click Create policy. In the popup window, select Visual editor and click Continue.

In the creation form, specify the following fields:

• Policy ID — a unique name for the policy
• Description — a brief explanation of its purpose

In the ISM Templates section, define the index patterns that this policy will automatically apply to when new indices are created.

Example configuration:

Configuring a Policy with Rollover

1. Add the init state

Go to the State section and click Add state. Name the state init, then click Add action and select the Rollover action type. Set the necessary parameters according to your configuration needs.

In this example, the following parameters are defined:

• minimum index age
• minimum primary shard size

See the detailed Rollover section for more on these parameters.

2. Configure Retry Settings

In the Timeout and Retry Settings block, it is recommended to use the following values:

• retry count: 144
• retry delay: 10m

These settings allow retrying for 24 hours with a 10-minute interval, which improves action reliability.

After entering the parameters, click Add action, then Save state.

Please note!

The Order and Transitions sections are not configured at this stage.

3. Add the delete state

Click Add state, name it delete, and configure it as shown below:

Add the delete action and specify the appropriate Timeout and retry settings:

Once configured, click Add action, then Save state.

4. Configure Order and Transitions

Return to the init state and in the Order section, click Add before and select the delete state.

Next, click Add transition and specify:

• destination state: delete
• condition: the condition under which the transition should occur

Click Add transition, then Update state to complete the setup.

5. Create the Policy

After completing all steps, click Create.

You have now successfully configured a rollover-based policy.

Configuring a Policy with Relocation

This setup demonstrates how to configure the allocation action in an existing policy.

1. Add a New State

Create a new state, for example named cold. In the Order section, set: Add after - select the init state.

2. Configure the Allocation Action

Add a new action of type - allocation. Fill in the parameters as follows:

{
  "require": {},
  "include": {
    "routing_mode": "cold"
  },
  "exclude": {},
  "wait_for": false
}

Please note!

When configuring this action, it's important to consider the current index allocation rules. See the allocation section for details.

In the Timeout and Retry Settings, you may apply the same recommended values used in the delete action (as described earlier in this guide).

3. Configure the Transition

Click Add transition and set the following:

• destination state — select the target state
• condition — define the condition under which the transition will occur

Example:

Click Add transition, then Save state.

4. Update State Order: init and delete

• open the init state and set its order: Add before - cold
• open the delete state and set: Add after - cold

This completes the policy configuration with relocation.

Configuring a Policy with a Wait State Before Rollover

To apply this configuration, you must already have a policy that uses rollover. For more information, see the corresponding section: Configuring a Policy with Rollover.

1. Create the State

Add a new state, for example named wait_events. Open it for editing and in the Order section, select: Add before - init.

No actions are added for the wait_events state.

2. Configure the Transition

Click Add transition and set the following parameters:

• destination state — select the target state
• condition — define the condition that triggers the transition

Example:

After completing the parameters, click Add transition, then Save state.

3. Update State Order

Open the init state and set its order: Add after - wait_events.

This completes the configuration of the policy with a wait state before the rollover countdown begins.

Configuring a Policy with Delete by Max Size

1. Add a state

Add a new state, for example, named init, and configure it according to your policy requirements.

2. Add the Delete by Max Size action

Click Add Action and select the Delete by Max Size action type. Then fill in the action parameters, click Add Action, and configure the required transitions.

Example action configuration:

In this example, two ISM template groups are configured:

• group 1: [index1, index3, index5]
• group 2: [index2, index4, index6]

Index	Size
index1	10gb
index2	50gb
index3	5gb
index4	5gb
index5	100gb
index6	62gb

Group 1

Group 1 has a size limit of 20gb and a maximum index age of 5d.

Assume that the following list of indices, sorted by creation date, is generated during policy recalculation:

[index5, index1, index3]

The total size of the group is 115gb, which exceeds the specified 20gb limit.

In this case, only index5 is deleted because, after its deletion, the total size of the remaining indices is 15gb.

This value does not exceed the 20gb limit, so index1 and index3 are not deleted during the current policy recalculation.

During subsequent policy recalculations, these indices may be deleted if either of the following conditions is met:

• the total size of the group exceeds 20gb again
• the age of the indices exceeds the max_index_age value of 5d

Group 2

Group 2 has a size limit of 100gb and a maximum index age of 1w.

Assume that the following list of indices, sorted by creation date, is generated during policy recalculation:

[index4, index2, index6]

The total size of the group is 117gb, which exceeds the specified 100gb limit.

First, the action deletes the oldest index, index4. After that, the total size of the remaining indices is 112gb.

Because 112gb still exceeds the 100gb limit, the action continues and deletes the next index, index2.

After index2 is deleted, the total size of the remaining indices is 62gb, which does not exceed the 100gb limit. Therefore, index6 is not deleted during the current policy recalculation.

warning

If the same ISM template matches multiple groups in patterns_size, the settings of the first matching group are applied.