Policies

Description

Policies are JSON documents that define:

States that an index can be in, including a default state for new indices. For example, you could name the states hot, warm, delete, etc.
Any actions the plugin should take when an index transitions into a certain state, such as performing a rollover.
Conditions that must be met for an index to transition to a new state, referred to as transitions. For example, if an index’s age is greater than eight weeks, it should move to the "delete" state.

In other words, a policy defines the states an index can be in, the actions to take in those states, and the conditions that must be met to transition from one state to another.

You have complete freedom when designing policies. You can create any state, transition to any other state, and specify any number of actions within each state.

This table lists the relevant policy fields:

Parameter	Description	Type	Required	Read-only
`policy_id`	The name of the policy	`string`	Yes	Yes
`description`	A human-readable description of the policy	`string`	Yes	No
`ism_template`	The specified ISM template pattern that matches the index for the policy to apply to	`nested list of objects`	No	No
`last_updated_time`	The time the policy was last updated	`timestamp`	Yes	Yes
`error_notification`	Message destination and template for error notifications	`object`	No	No
`default_state`	The default initial state for every index that uses the policy	`string`	Yes	No
`states`	The states that are defined in the policy	`nested list of objects`	Yes	No

States

A state is a description of the status that a managed index is currently in. A managed index can only be in one state at a time. Each state has actions associated with it that are executed sequentially upon entering the state and transitions that are checked after all actions have been executed.

This table lists the parameters you can configure for a state:

Field	Description	Type	Required
`name`	The name of the state	`string`	Yes
`actions`	The actions to take after entering the state. See the Actions section for further details.	`nested list of objects`	Yes
`transitions`	The next states and conditions required to transition to those states. If there are no transitions, the policy considers itself done and may stop managing the index. See the Transitions section for further details.	`nested list of objects`	Yes

Actions

Actions are the steps a policy takes sequentially when it transitions to a particular state.

ISM executes actions in the order they are defined. For example, if actions [A,B,C,D] are defined, ISM executes action A, and then goes into a sleep period, based on the cluster setting plugins.index_state_management.job_interval. After the sleep period, ISM continues to execute the remaining actions. However, if ISM can't successfully execute action A, the operation ends, and actions B, C, and D are not executed.

You can optionally specify an action timeout period, beyond which the operation is forcibly terminated. For example, if the timeout is set to 1d and ISM hasn’t executed the action within one day, even after retries, then the action fails.

This table lists the parameters you can configure for an action:

Parameter	Description	Type	Required	Default
`timeout`	The timeout period for the action. Accepts time units of minutes, hours, and days.	`time unit`	No	-
`retry`	Retry configuration for this action	`object`	No	Action dependent

The retry operation has the following parameters:

Parameter	Description	Type	Required	Default
`count`	The number of retries	`number`	Yes	-
`backoff`	The backoff policy type to use for retries. Valid values are Exponential, Constant, and Linear.	`string`	No	Exponential
`delay`	The wait time between retries. Accepts time units of minutes, hours, and days	`time unit`	No	1 minute

The following action example has a timeout period of one hour. The policy retries this action three times with an exponential backoff, with a delay of 10 minutes between each retry:

{
  "actions": {
    "timeout": "1h",
    "retry": {
      "count": 3,
      "backoff": "exponential",
      "delay": "10m"
    }
  }
}

Operations Supported by ISM

ISM supports the following operations:

Policies

force_merge

Reduces the number of Lucene segments by merging segments of individual shards. An attempt is made to set the index to read-only before starting the merge process.

Parameter	Description	Type	Required
`max_num_segments`	The number of segments to reduce the shard to	`number`	Yes
`wait_for_completion`	If set to false, the request returns immediately and doesn’t wait for the operation to complete. You can use the Tasks API with the task ID returned by the request to monitor the status of the operation. Defaults to true.	`boolean`	No
`task_execution_timeout`	Task execution timeout. Only applicable if wait_for_completion is false. Defaults to 1h.	`time unit`	No

{
  "force_merge": {
    "max_num_segments": 1
  }
}

read_only

Sets the managed index to read-only mode.

{
  "read_only": {}
}

Sets the index.blocks.write setting to true for the managed index.

Note

This block does not prevent updates to the index.

read_write

Sets the managed index to be writable.

{
  "read_write": {}
}

replica_count

Sets the number of replicas assigned to an index.

Parameter	Description	Type	Required
`number_of_replicas`	Defines the number of replicas to assign to the index.	`number`	Yes

{
  "replica_count": {
    "number_of_replicas": 2
  }
}

shrink

Allows you to shrink the number of primary shards in indices. Using this action, you can specify:

the number of primary shards the target index should have
the maximum shard size for primary shards in the target index
the percentage to reduce the primary shards to in the target index

Parameter	Description	Type	Example	Required
`num_new_shards`	The maximum number of primary shards in the shrunk index.	`integer`	`5`	Yes, however it can not be used with `max_shard_size` or `percentage_of_source_shards`
`max_shard_size`	The maximum shard size in bytes for the target index.	`keyword`	`5gb`	Yes, however it can not be used with num_new_shards or percentage_of_source_shards
`percentage_of_source_shards`	The percentage of the source primary shards to shrink to. This setting specifies the minimum percentage that should be used when shrinking the primary shards. Must be in the exclusive range of 0.0 to 1.0.	`percentage`	`0.5`	Yes, however it can not be used with max_shard_size or num_new_shards
`target_index_name_template`	The name of the shrunk index. Accepts strings and Mustache variables.	`string` or `Mustache` template	`{"source": "_shrunken"}`	No
`aliases`	Aliases to add to the new index.	`object`	`myalias`	No, but must be an array of alias objects
`force_unsafe`	If true, the shrink action proceeds even if there are no replicas.	`boolean`	`false`	No

{
  "shrink": {
    "num_new_shards": 1,
    "target_index_name_template": {
      "source": "_shrunken"
    },
    "aliases": [
      {
        "my-alias": {}
      }
    ],
    "force_unsafe": false
  }
}

If you need to add aliases to the action, the parameter must contain an array of alias objects. For example:

"aliases": [
  {
    "my-alias": {}
  },
  {
    "my-second-alias": {
      "is_write_index": false,
      "filter": {
        "multi_match": {
          "query": "QUEEN",
          "fields": ["speaker", "text_entry"]
        }
      },
      "index_routing" : "1",
      "search_routing" : "1"
    }
  },
]

Move to ClickHouse

The Move to ClickHouse action allows you to automatically move data from OpenSearch to ClickHouse, providing efficient management of large datasets and reducing the load on OpenSearch. This action can be configured as part of an ISM policy for specific indices.

Parameters for configuring the "Move to ClickHouse" action

Parameter	Description	Type	Example	Required
`table_name_template`	A regular expression that defines the table name in ClickHouse based on the index name. The named capturing group in the regex will be used as the table name. The capturing group must be named `name`.	`string`	`(?<name>.*?)-\d+`	Yes
`connection_id`	The ID of the ClickHouse connection configured in your OpenSearch configuration.	`string`	`cold_node`	Yes
`batch_size`	The number of events sent to ClickHouse in a single batch.	`integer`	`1000`	No

Description​

States​

Actions​

Operations Supported by ISM​

force_merge​

read_only​

read_write​

replica_count​

shrink​

Move to ClickHouse​

Parameters for configuring the "Move to ClickHouse" action​