Skip to main content

Index Management

This article is a translation from the official documentation of OpenSearch 2.6.X using SAF.

Description​

When analyzing data, you likely prioritize new data over old data. Periodically, you may perform certain operations on older indices, such as reducing the number of replicas or deleting them.

The Index State Management (ISM) plugin allows you to automate these operations based on changes in index age, index size, or document count. With the ISM plugin, you can define policies that automatically handle the migration or deletion of indices according to specific conditions.

For example, you can define a policy that transitions an index to a read_only state after 30 days and then deletes it after a specified period of 90 days. You can also configure the policy to send a notification when the index is deleted.

To improve search performance during peak load, you may need to perform an index rollover after a certain period or run a force_merge operation on the index during off-peak hours.

Note

To use the ISM plugin, the user role must be mapped to the all_access role, which grants full access to the cluster.

Getting Started with ISM​

To get started, select the "Index Management" option in the "System Settings" section of the main menu.

Configuring Policies​

A Policy is a set of rules that describe how an index should be managed. For information on creating a policy, see the Policies section.

You can create policies using either the visual editor or the JSON editor. Compared to the JSON editor, the visual editor offers a more structured way to define policies by breaking the process into creating error notifications, defining ISM templates, and adding states. It is recommended to use the visual editor if you want to see predefined fields, such as which actions can be assigned to a state or under what conditions a state can transition to a target state.

Visual Editor​

  1. In the Index Management menu, select the Index Policies tab.
  2. Click the Create Policy button.
  3. In the Configuration Method modal, select the Visual Editor option and click Next.
  4. In the Policy Information section, enter the policy ID and its general description.
  5. In the Error Notification section, configure notifications to be sent in case of policy execution failure. If the policy uses rollover, it is recommended to configure error notifications to warn about unexpectedly large indices during failed rollover operations.
  6. In the ISM Templates section, enter any ISM templates for automatically applying this policy to future indices. For example, if you specify a template sample-index*, the ISM plugin will automatically apply this policy to all indices whose names start with sample-index. The template cannot contain any of the following characters: :, ", +, /, \, |, ?, #, >, and <.
  7. In the States section, add all the states you want to include in the policy. Each state contains actions that the plugin performs when the index transitions to a specific state, and transitions that contain the conditions under which the index transitions to a target state. The first state created in the policy is automatically set as the initial state. Each policy must have at least one state, and actions and transitions are optional.
  8. Click the Create Script button.

JSON Editor​

  1. In the Index Management menu, select the Index Policies tab.
  2. Click the Create Policy button.
  3. In the Configuration Method modal, select the JSON Editor option and click Next.
  4. In the Policy Name section, enter the policy ID.
  5. In the Policy Definition section, fully configure the policy.
  6. Click the Create button.

After creating the policy, the next step is to apply it to an index or indices. You can set an ism_template in the policy so that when an index matching the ISM template is created, the plugin automatically applies the policy to the index.

The example below shows how to create a policy that automatically binds to all indices whose names start with index_name-.

PUT _plugins/_ism/policies/policy_id
{
  "policy": {
    "description": "Example policy.",
    "default_state": "...",
    "states": [...],
    "ism_template": {
      "index_patterns": ["index_name-*"],
      "priority": 100
    }
  }
}