Index Operations

Move to ClickHouse

API to initiate the move of a specified index: POST _ism/move_to_click/<index_name>.

Path Parameters

Name	Type	Description
index_name	string	The name of the index to be moved.

Request Body Parameters

Name	Type	Description
connection_id	string	The connection identifier for connecting to ClickHouse, through which the move will be performed.
index_name_pattern	string	A regular expression pattern for extracting the table name, must contain a named group (?<name>)
fetch_size	integer	The batch size for processed documents. The recommended value is between 10000 and 100000 , depending on the document size.
data_scheme_parameters	object	Additional settings affecting the interaction with the table in ClickHouse.

Scheme data_scheme_parameters

Name	Type	Description
ttl_interval	string	The Time-To-Live interval for documents in the table, in the format interval <count> <time unit> Available intervals (set for the entire table upon creation).
ttl_column	string	The name of the datetime-type field used as the basis for TTL calculation.
cluster_name	string	The ClickHouse cluster name (a required field when replication = true or distributed = true).
replication	boolean	Flag indicating whether to create a replicated table during the data transfer (considered if the move initiates the creation of a new table).
distributed	boolean	Flag indicating whether to create a distributed table during the data transfer (considered if the move initiates the creation of a new table).
click_server_settings	object	A key-value map used to pass ClickHouse settings to the client that connects to ClickHouse during data migration.
override_field_types	object	A map used to override source index field types with ClickHouse types, for example, {"integer_field": "Int64"}. Nested fields are specified using dot notation, for example, human.age. If a parent nested field is overridden, its child fields are not created separately. Instead, the entire value is stored in the parent field as a String containing a JSON structure.
override_field_date_formats	object	A map used to specify date formats for fields whose values differ from the standard ISO 8601 format. One format can be specified per field, for example, "dd.MM.yyyy HH:mm:ss". After processing, the date is stored in the resulting ClickHouse table in the standard format.
view_json_as_string	boolean	Determines whether JSON columns are represented as String columns in a VIEW. Required for ClickHouse versions that do not support using JSON columns in a VIEW.
json_fields	string[]	A list of paths whose child fields are migrated to ClickHouse as a single JSON object. Required for migrating dynamically generated fields, for example, a1, a2, and a3.
blacklist	string[]	A list of paths that are ignored and not migrated. Path prefixes can be specified, for example, host.*.
whitelist	string[]	A list of paths that are migrated even if they match the blacklist. Path prefixes can be specified, for example, host.*.
timestamp_field_name	string	The name of the field that stores the event timestamp. Used when the field name differs from @timestamp.
codec	string	The name of the codec to use as the default codec, for example, ZSTD(3).
table_name_template	string	A template used to rename the target table. It can use the name extracted with index_name_pattern through the {{name}} token, for example, {{name}}-suffix.
source_read_mode	string	The source document read mode. Three modes are available: DEFAULT — the default mode, where the source document uses a standard JSON structure; FLAT — used when the source document was indexed in a flattened format, for example, {"user.firstname": "ivan", "user.lastname": "ivanov"}; MIXED — used when some fields were stored in a flattened format and others in a standard nested format.
cb_retries_max_retries	long	The maximum number of task retry attempts after the Circuit Breaker is triggered. After the limit is exceeded, the task fails.
cb_retries_initial_delay_millis	long	The initial delay before the first retry attempt after the Circuit Breaker is triggered. The value is specified in milliseconds.
cb_retries_max_delay_millis	long	The maximum delay between retry attempts. The delay increases with each attempt but does not exceed the specified value.

Example Request

POST _ism/move_to_click/clickhouse_test-600006
{
  "connection_id": "TestClick",
  "index_name_pattern": "(?<name>.*?)-\\d+",
  "fetch_size": 100000,
  "data_scheme_parameters": {
    "ttl_interval": "interval 30 day",
    "cluster_name": "sm_cluster",
    "replication": true,
    "distributed": true,
    "click_server_settings": {
      "output_format_binary_write_json_as_string": 1,
      "input_format_binary_read_json_as_string": 1
    },
    "override_field_types": {
      "integer_field": "Int64",
      "object_field.nested_field1.nested_field2": "String"
    },
    "override_field_date_formats": {
      "date_field1": "dd.MM.yyyy HH:mm:ss",
      "date_field2": "MM/dd/yyyy hh:mm:ss a"
    },
    "view_json_as_string": false,
    "json_fields": ["EXECVE"],
    "blacklist": ["field1", "field2.*"],
    "whitelist": ["field2.subfield1"],
    "timestamp_field_name": "timestamp",
    "codec": "ZSTD(3)",
    "table_name_template": "{{name}}-suffix",
    "source_read_mode": "MIXED",
    "cb_retries_max_retries": 5,
    "cb_retries_initial_delay_millis": 3000,
    "cb_retries_max_delay_millis": 45000
  }
}

API for canceling an active data migration for the specified index: POST _ism/move_to_click/<index_name>/cancel.