Query Migration for Enabling Keyword Autocompletion

Important Note!

Migration guide for upgrading to version 5.1.

Migration API Description

The migration API allows checking queries for required migration before enabling Configuring Keyword Autocompletion.

Important Note!

Migration verification is required because .keyword autocompletion in the search command depends on index mappings in the current installation.

Checking Jobscheduler Queries

To verify jobscheduler queries, execute this command in the developer console (Main menu - Settings - Dev Console):

GET /_sme/migration/autokeyword/jobscheduler

This command returns an array of replacement objects in response.

Example response:

[
  {
    "job_id": "rnxSF5cBk65B9KQf1jWm",
    "index": ".sm_jsc_jobs",
    "doc_id": "r3xSF5cBk65B9KQf1jWy",
    "query": {
      "original": """
source internal_audit* | search audit_transport_headers.X-Opaque-Id="1b7adabf"
""",
      "replacement": """
source internal_audit* | search text audit_transport_headers.X-Opaque-Id="1b7adabf"
"""
    }
  }
]

Description of replacement object fields:

• job_id - job identifier (_meta.id field)
• index - index where the job document is stored
• doc_id - job document identifier (_id field)
• query - original query and replacement query

Checking Dashboard Queries

To verify dashboard queries, execute this command in the developer console (Main menu - Settings - Dev Console):

GET /_sme/migration/autokeyword/dashboards

This command returns an array of replacement objects in response.

Example response:

[
  {
    "pathInJson": "rows[3].panels[0].viz.options.sme.query",
    "dashboard_id": "incident_statistics",
    "index": ".sm_dashboards",
    "doc_id": "A8fyQ5cB8NGVbw6n_KHP",
    "query": {
      "original": ...,
      "replacement": ...
    }
  }
]

Description of replacement object fields:

• dashboard_id - dashboard identifier (_meta.id field)
• index - index where the job document is stored
• doc_id - job document identifier (_id field)
• query - original query and replacement query
• pathInJson - path to locate the query in the dashboard's json field

Checking a Single Query

To verify a single query, execute this command in the developer console (Main menu - Settings - Dev Console):

POST /_sme/migration/autokeyword/explain
{
  "query": """
  <SAFL request>
  """
}

A specific query might look like this:

POST /_sme/migration/autokeyword/explain
{
  "query": """
  source internal_audit* | search audit_transport_headers.X-Opaque-Id="1b7adabf"
  """
}

The command will return a structure containing information about the required query replacement (replacement field):

{
	"original": """
  source internal_audit* | search audit_transport_headers.X-Opaque-Id="1b7adabf"
  """,
	"replacement": """
  source internal_audit* | search text audit_transport_headers.X-Opaque-Id="1b7adabf"
  """,
	"errors": [],
	"search_fields": [
		{
			"position": 35,
			"original": "audit_transport_headers.X-Opaque-Id",
			"name": "audit_transport_headers.X-Opaque-Id",
			"type": "text",
			"hasTextOp": false,
			"message": "The 'text' operator was added to the field"
		}
	]
}

In this example, the API response indicates that:

source internal_audit* | search audit_transport_headers.X-Opaque-Id="1b7adabf"

The query needs to be replaced with:

source internal_audit* | search text audit_transport_headers.X-Opaque-Id="1b7adabf"

That is, in the replacement query, the text operator is applied to the audit_transport_headers.X-Opaque-Id field. The text operator disables the .keyword autocompletion mechanism. If the replacement isn't made, the text operator won't be applied, and in this case the audit_transport_headers.X-Opaque-Id field will be converted to audit_transport_headers.X-Opaque-Id.keyword.

Autocompletion would occur because the text field mapping includes a keyword. For text fields without keyword, the text operator isn't required as the autocompletion mechanism won't trigger for them.

The mapping of the audit_transport_headers.X-Opaque-Id field in this example:

{
    "audit_transport_headers.X-Opaque-Id": {
        "full_name": "audit_transport_headers.X-Opaque-Id",
        "mapping": {
            "X-Opaque-Id": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                    }
                }
            }
        }
    }
}

SAFL Query Replacement Description Structure

All fields that may appear in the replacement explanation structure are listed in the table below.

Field	Description
original	Original query
expansion	Result of expanding all macros
macros	Macros expansion trees (consists of macros expansion nodes). Describes first-level macros expansions
replacement	Replacement query
errors	Errors encountered during query processing
search_fields	Array of found fields and their attributes

Macros Expansion Tree Node Structure

The macros expansion tree node structure describes the expansion of a single macros.

Field	Description
expansion_position	Position of macros expansion in the original query
original	Original text (unexpanded macros)
expansion	Result of macros expansion
name	Macros name
macros	Expansion result of child (nested) macros
search_fields	Array of fields generated by macros expansion

Field Structure and Attributes

The field structure describes found fields and their attributes. May appear both in the general replacement explanation and in macro expansion nodes.

Important Note!

If fields are found within macro expansion trees, this case requires detailed consideration and manual meaningful replacement.

Field	Description
position	Field position (after all macro expansions)
original	Original field text
name	Field name
type	Field type
hasTextOp	Whether the text operator is applied to the field
message	Status information about adding the text operator to the field