Policies

Description

Policies are JSON documents that define:

• States that an index can be in, including a default state for new indices. For example, you could name the states hot, warm, delete, etc.
• Any actions the plugin should take when an index transitions into a certain state, such as performing a rollover
• Conditions that must be met for an index to transition to a new state, referred to as transitions. For example, if an index’s age is greater than eight weeks, it should move to the "delete" state

In other words, a policy defines the states an index can be in, the actions to take in those states, and the conditions that must be met to transition from one state to another.

You have complete freedom when designing policies. You can create any state, transition to any other state, and specify any number of actions within each state.

This table lists the relevant policy fields:

Parameter	Description	Type	Required	Read-only
policy_id	The name of the policy	string	Yes	Yes
description	A human-readable description of the policy	string	Yes	No
ism_template	The specified ISM template pattern that matches the index for the policy to apply to	nested list of objects	No	No
last_updated_time	The time the policy was last updated	timestamp	Yes	Yes
error_notification	Message destination and template for error notifications	object	No	No
default_state	The default initial state for every index that uses the policy	string	Yes	No
states	The states that are defined in the policy	nested list of objects	Yes	No

States

A state is a description of the status that a managed index is currently in. A managed index can only be in one state at a time. Each state has actions associated with it that are executed sequentially upon entering the state and transitions that are checked after all actions have been executed.

This table lists the parameters you can configure for a state:

Field	Description	Type	Required
name	The name of the state	string	Yes
actions	The actions to take after entering the state. See the Actions section for further details.	nested list of objects	Yes
transitions	The next states and conditions required to transition to those states. If there are no transitions, the policy considers itself done and may stop managing the index. See the Transitions section for further details.	nested list of objects	Yes

Actions

Actions are the steps a policy takes sequentially when it transitions to a particular state.

ISM executes actions in the order they are defined. For example, if actions A,B,C,D are defined, ISM executes action A, and then goes into a sleep period, based on the cluster setting plugins.index_state_management.job_interval. After the sleep period, ISM continues to execute the remaining actions. However, if ISM can't successfully execute action A, the operation ends, and actions B, C, and D are not executed.

You can optionally specify an action timeout period, beyond which the operation is forcibly terminated. For example, if the timeout is set to 1d and ISM hasn’t executed the action within one day, even after retries, then the action fails.

This table lists the parameters you can configure for an action:

Parameter	Description	Type	Required	Default
timeout	The timeout period for the action. Accepts time units of minutes, hours, and days.	time unit	No	-
retry	Retry configuration for this action	object	No	Action dependent

The retry operation has the following parameters:

Parameter	Description	Type	Required	Default
count	The number of retries	number	Yes	-
backoff	The backoff policy type to use for retries. Valid values are Exponential, Constant, and Linear.	string	No	Exponential
delay	The wait time between retries. Accepts time units of minutes, hours, and days	time unit	No	1 minute

The following action example has a timeout period of one hour. The policy retries this action three times with an exponential backoff, with a delay of 10 minutes between each retry:

{
  "actions": {
    "timeout": "1h",
    "retry": {
      "count": 3,
      "backoff": "exponential",
      "delay": "10m"
    }
  }
}

Operations Supported by ISM

ISM supports the following operations:

• Policies
  • Description
  • States
  • Actions
  • Operations Supported by ISM
    • force_merge
    • read_only
    • read_write
    • replica_count
    • shrink
    • Move to ClickHouse
      • Parameters for configuring the "Move to ClickHouse" action

force_merge

Reduces the number of Lucene segments by merging segments of individual shards. An attempt is made to set the index to read-only before starting the merge process.

Parameter	Description	Type	Required
max_num_segments	The number of segments to reduce the shard to	number	Yes
wait_for_completion	If set to false, the request returns immediately and doesn’t wait for the operation to complete. You can use the Tasks API with the task ID returned by the request to monitor the status of the operation. Defaults to true.	boolean	No
task_execution_timeout	Task execution timeout. Only applicable if wait_for_completion is false. Defaults to 1h.	time unit	No

{
  "force_merge": {
    "max_num_segments": 1
  }
}

read_only

Sets the managed index to read-only mode.

{
  "read_only": {}
}

Sets the index.blocks.write setting to true for the managed index.

Note

This block does not prevent updates to the index.

read_write

Sets the managed index to be writable.

{
  "read_write": {}
}

replica_count

Sets the number of replicas assigned to an index.

Parameter	Description	Type	Required
number_of_replicas	Defines the number of replicas to assign to the index.	number	Yes

{
  "replica_count": {
    "number_of_replicas": 2
  }
}

shrink

Allows you to shrink the number of primary shards in indices. Using this action, you can specify:

• the number of primary shards the target index should have
• the maximum shard size for primary shards in the target index
• the percentage to reduce the primary shards to in the target index

Parameter	Description	Type	Example	Required
num_new_shards	The maximum number of primary shards in the shrunk index.	integer	5	Yes, however it can not be used with max_shard_size or percentage_of_source_shards
max_shard_size	The maximum shard size in bytes for the target index.	keyword	5gb	Yes, however it can not be used with num_new_shards or percentage_of_source_shards
percentage_of_source_shards	The percentage of the source primary shards to shrink to. This setting specifies the minimum percentage that should be used when shrinking the primary shards. Must be in the exclusive range of 0.0 to 1.0.	percentage	0.5	Yes, however it can not be used with max_shard_size or num_new_shards
target_index_name_template	The name of the shrunk index. Accepts strings and Mustache variables.	string or Mustache template	{"source": "_shrunken"}	No
aliases	Aliases to add to the new index.	object	myalias	No, but must be an array of alias objects
force_unsafe	If true, the shrink action proceeds even if there are no replicas.	boolean	false	No

{
  "shrink": {
    "num_new_shards": 1,
    "target_index_name_template": {
      "source": "_shrunken"
    },
    "aliases": [
      {
        "my-alias": {}
      }
    ],
    "force_unsafe": false
  }
}

If you need to add aliases to the action, the parameter must contain an array of alias objects. For example:

"aliases": [
  {
    "my-alias": {}
  },
  {
    "my-second-alias": {
      "is_write_index": false,
      "filter": {
        "multi_match": {
          "query": "QUEEN",
          "fields": ["speaker", "text_entry"]
        }
      },
      "index_routing" : "1",
      "search_routing" : "1"
    }
  },
]

Move to ClickHouse

The Move to ClickHouse action allows you to automatically move data from OpenSearch to ClickHouse, providing efficient management of large datasets and reducing the load on OpenSearch. This action can be configured as part of an ISM policy for specific indices.

Parameters for configuring the "Move to ClickHouse" action

Parameter	Description	Type	Example	Required
table_name_template	A regular expression that defines the table name in ClickHouse based on the index name. The named capturing group in the regex will be used as the table name. The capturing group must be named name.	string	(?<name>.*?)-\d+	Yes
connection_id	The ID of the ClickHouse connection configured in your OpenSearch configuration.	string	cold_node	Yes
batch_size	The number of events sent to ClickHouse in a single batch.	integer	1000	No