Version: 6.2

SA Engine RE Configuration

SME-RE accepts requests on two routes:

Route	Called by	Authentication
`/sme_remote_exec/*`	The `script` command in search, the `script` type of the Job Scheduler	Basic: `sme.core.remote_script.user` and the password from the keystore
`/sme_scripts/v1/*`	Active Actions based on catalog scripts (the `sm-actions-api` plugin)	Bearer token from the `SME-RE` token store

The settings of the first route are described in the sections General SA Engine RE Connection Settings, Information for Running Scripts in Search and Information for Running Scripts in the Job Scheduler. The settings of the second route are described in the section Running Active Actions via the /sme_scripts/v1 Route.

SA Engine RE Configuration Check

Verifying SA Engine RE Utility Availability

The SA Engine RE utility is automatically installed during every system deployment or update. It is present on all OpenSearch nodes where node.attr.node_with_sme: true is set in the opensearch.yml file.

Note

Ensure that the configuration file /app/opensearch/utils/sme-re/application.properties exists and contains the correct settings. If the file is missing, the SA Engine RE utility cannot connect to OpenSearch. Also verify that node.attr.node_with_sme in opensearch.yml is set to true. If it is set to false, this node will not receive search script execution tasks.

Configuration Verification

To check the current settings, execute the following command in the Dev Tools console (Main Menu - Settings - Dev Tools):

GET _cluster/settings

Find the following block in the response:

...
    "sme": {
        "core": {
            "remote_script": {
                "base_path": "/app/opensearch/utils/scripts/",
                "enable_ssl": "true",
                "port": "18080",
                "base_interpreter_name": "bash",
                "url": "https://127.0.0.1",
                "interpreters": [
                    "python3::/app/opensearch/utils/python/bin/python3",
                    "bash::bash"
                ]
            },
            "enable_field_caps": "false",
            "enable_limits": "true"
        },
        "overwrite": {
            "scroll_timeout": "30000"
        }
    },
...

If the sme.core section is missing, add it and adjust the parameter values as needed.

Execute the following command in the Dev Tools console:

PUT _cluster/settings
{
  "persistent": {
    "sme": {
      "core": {
        "remote_script": {
          "base_path": "/app/opensearch/utils/scripts/",
          "enable_ssl": true,
          "port": "18080",
          "base_interpreter_name": "bash",
          "url": "https://127.0.0.1",
          "interpreters": [
            "python3::/app/opensearch/utils/python/bin/python3",
            "bash::bash"
          ]
        }
      }
    }
  }
}

Note

A script from the script search command runs on a random OpenSearch node where node.attr.node_with_sme: true is set. Therefore, deploy such scripts on all nodes with this setting.

This does not apply to Active Actions: the plugin sends the action script code to SME-RE itself, so it does not need to be placed on the nodes.

To check the list of nodes and their parameters, execute the following command in the Dev Tools console (Main Menu - Settings - Dev Tools):

GET _cat/nodeattrs

In the command output, look for the node_with_sme parameter.

SA Engine RE Startup Parameters

SME-RE startup parameters are specified in the application.properties configuration file, located in the same directory as the SME-RE utility.

General Parameters

Parameter	Description	Default Value
`server.port`	The port on which `SME-RE` will accept incoming connections	18080
`spring.servlet.multipart.max-request-size`	Maximum file upload size in bytes	100000000
`server.tomcat.max-http-form-post-size`	Maximum size for `multipart/form-data` requests in bytes	104857600
`sme.FileRotatePeriod`	Request data file rotation time in seconds	10000
`sme.baseData`	Directory for storing request data files	./data
`sme.directoryPollerInterval`	Interval for checking the request data file storage directory in seconds	50
`sme.scriptConfigPath`	Path to the `scripts.yaml` configuration file	./scripts.yaml
`sme.LogStringSize`	Log string size	300
`sme.RapidDelete`	Whether to delete request data files	True
`sme.redis.url`	Address for connecting to `Redis`	"127.0.0.1"
`sme.redis.port`	Port for connecting to `Redis`	6379
`sme.redis.pass`	Password for connecting to `Redis`	""
`logging.file.path`	Path for storing logs	logs
`server.ssl.enabled`	Whether to use SSL/TLS for incoming connections	true
`server.ssl.certificate`	Path to the public key (certificate) of the current `SME-RE` server in PKCS8 format	/app/opensearch/config/node-cert.pem
`server.ssl.certificate-private-key`	Path to the private key of the current `SME-RE` server in PKCS8 format	/app/opensearch/config/node-key.pem
`server.ssl.trust-certificate`	Path to the Certificate Authority (CA) chain in PKCS8 format	/app/opensearch/config/ca-cert.pem
`server.ssl.client-auth`	Certificate validation mode, can be `need`, `none`, `want`	need

Parameters of the Active Actions Route `/sme_scripts/v1`

The parameters are read at SME-RE startup. The exception is the token store, which is re-read without a restart.

Parameter	Description	Default Value
`sme.scripts.interpreters`	Comma-separated list of allowed interpreter paths. A run with a path not in the list is rejected. An empty list rejects all runs	(empty)
`sme.scripts.loadersDir`	Directory where the script loader and the `sm_actions` package are unpacked at startup	./loaders
`sme.scripts.cacheDir`	Script code cache directory	./scripts-cache
`sme.scripts.cacheMaxBytes`	Code cache size. When it overflows, the least recently used files are removed	1GB
`sme.scripts.workRoot`	Root of the per-call working directories	./runs
`sme.scripts.maxProcesses`	Total number of script processes running at the same time	16
`sme.scripts.maxPerScript`	Number of processes of one script running at the same time	4
`sme.scripts.queueSize`	How many calls may wait for a free slot. Beyond that, `429 busy` is returned. A free slot is taken without queueing, so `0` means "no queue", not "reject everyone"	32
`sme.scripts.queueWaitMs`	How many milliseconds a call waits for a slot in the queue	2000
`sme.scripts.maxSourceBytes`	Script code size limit on upload. Beyond it, `413 source.too_large` is returned	20MB
`sme.scripts.maxOutputBytes`	Script result size limit	10MB
`sme.scripts.logTailBytes`	Size of the stdout and stderr tail of the script that is returned in the response and written to the run diagnostics	64KB
`sme.scripts.interruptGrace`	How long before the call deadline the script receives SIGTERM, so that `run()` gets `DeadlineExceeded` and releases resources. At most 10% of the deadline	1s
`sme.scripts.pool.enabled`	Keep script processes running between calls. When disabled, every call starts a new process	false
`sme.scripts.pool.idleTtl`	How long an idle process remains running before it is stopped	5m
`sme.scripts.pool.maxCallsPerWorker`	How many calls one process serves before it is restarted	1000
`sme.scripts.pool.setupBackoff`	Pause before retrying to start a process after a failed `setup()`	30s
`sme.scripts.pool.maxMessageBytes`	Message size limit between `SME-RE` and a running process	64MB
`sme.scripts.pool.teardownTimeout`	How long to wait for `teardown()` when stopping a process	5s
`sme.scripts.auth.keystorePath`	Path to the PKCS12 store of call tokens. If empty, there are no tokens and every call gets `401`	(empty)
`sme.scripts.auth.keystorePassword`	Token store password	The value of the `SME_RE_KEYSTORE_PASSWORD` environment variable
`sme.scripts.auth.reloadCheckInterval`	How often the store file's modification time is checked. A changed file is re-read without a restart; if it cannot be read, the previous set of tokens is kept	5s

Note

a path in sme.scripts.interpreters must point to an executable interpreter (a virtual environment's bin/python3 is suitable), not a shim or wrapper that starts the interpreter as a child process. Otherwise, child script processes keep running when a call is interrupted. Use separate virtual environments and paths for different library sets
the script process group is terminated with the kill command. It must be on the PATH of the SME-RE process (minimal images get it from the procps package). If kill is missing, an error is written to the log at startup

General SA Engine RE Connection Settings

The settings below apply to the /sme_remote_exec route (the script command in search and the script type of the Job Scheduler). Active Actions on the /sme_scripts/v1 route use only sme.core.remote_script.enable_ssl from them, and a token instead of the user and password (see Running Active Actions).

Setting	Description	Default Value	Setting Type
`sme.core.remote_script.user`	Username used to make requests to `SME-RE`	sme_re_user	Cluster Setting
`sme.core.remote_script.password`	Password of the user used to make requests to `SME-RE`	—	Node Keystore Setting
`sme.core.remote_script.enable_ssl`	Flag indicating whether to use SSL for connecting to `SME-RE` from the `script` command in search, from the `script` type of the Job Scheduler and from Active Actions	true	Cluster Setting

Information for Running Scripts in Search

Connecting to SA Engine RE from the script Command in Search

The table lists the cluster settings used by the script command to connect to SME-RE:

Setting	Description	Default Value
`sme.core.remote_script.base_path`	Path to the directory where scripts are uploaded	"/app/opensearch/utils/scripts/"
`sme.core.remote_script.port`	Port on which `SME-RE` is running	18080
`sme.core.remote_script.url`	List of URLs for nodes running `SME-RE`	["https://127.0.0.1"]
`sme.core.remote_script.base_interpreter_name`	Name of the default interpreter, used to run the `script` command without the `intr` parameter	"bash"
`sme.core.remote_script.interpreters`	Array of strings containing available interpreters. Entry format: "interpreter name"::"interpreter path". Used by the `script` command; for Active Actions, see Interpreters	["python3::/app/opensearch/utils/python/bin/python3","bash::bash"]

info

Requests are evenly distributed among hosts with SME-RE using the Round Robin algorithm.

Authorization

Authorization of requests to SME-RE on the /sme_remote_exec route is always performed on the host where SME-RE is running.

Creating a Script

To use a script in search, place it on all cluster servers in the directory specified by sme.core.remote_script.base_path. Ensure that the file is executable and owned by the opensearch:opensearch user and group.

To return data from the script for further processing in search, output the result in JSON format.

Example for Python:

print(scriptResultJson)

If other commands are executed before the script command, their result is saved to a temporary file in the folder specified in sme.baseData. The name of this file will be passed to the script as a command-line argument.

Example for Python:

import sys

if __name__ == "__main__":
    # open the file with data from the request
    inputDataFile = open(sys.argv[1], "r")
    # read the file with data from the request
    inputData = inputDataFile.read()
    # close the file after reading
    inputDataFile.close()

Running Active Actions via the `/sme_scripts/v1` Route

Active Actions based on scripts from the core catalog (_core/adhoc_scripts) are executed by SME-RE at the request of the sm-actions-api plugin. The SME-RE host is chosen from the sme.core.remote_script.url list, and the plugin sends the script code to SME-RE.

Connection Settings on the OpenSearch Side

The SME-RE addresses are taken from the same cluster settings as for the script command. Changes to them are picked up without restarting the nodes.

Setting	Description	Default Value	Setting Type
`sme.core.remote_script.url`	List of URLs for `SME-RE` hosts. The scheme is taken from the entry; if it is not specified, it is determined by `enable_ssl`. The port is taken from the entry; if it is not specified, the common port is used	["https://127.0.0.1"]	Cluster Setting
`sme.core.remote_script.port`	`SME-RE` port for entries without a port	18080	Cluster Setting
`sme.core.remote_script.enable_ssl`	Whether to use SSL for entries without a scheme	true	Cluster Setting
`sm.actions.script.sme_re.token`	Script call token. Without it, runs are rejected and a warning is written to the node log at startup	—	Node Keystore Setting
`sm.actions.script.deadline_margin`	Margin between the `SME-RE` call deadline and the action execution deadline, so that the response has time to reach the plugin	2s	Node Setting (`opensearch.yml`)

Script Call Token

Calls on the /sme_scripts/v1 route are accepted with the Authorization: Bearer <token> header. SME-RE verifies the token against its own store: a PKCS12 file whose path is set in sme.scripts.auth.keystorePath. Every secret entry with an alias starting with scripts-token and a length of at least 32 bytes is accepted. There may be several entries.

Token creation procedure:

Generate a token:
```
openssl rand -base64 32
```
On every SME-RE host, add the token to the store (the store password is set by the SME_RE_KEYSTORE_PASSWORD environment variable):
```
keytool -importpass -storetype PKCS12 -keystore <path from sme.scripts.auth.keystorePath> -alias scripts-token-<label>
```
No SME-RE restart is needed: the file is re-read within sme.scripts.auth.reloadCheckInterval.
On every OpenSearch node, add the same token to the keystore. In the Dev Tools console (the setting is applied immediately):
```
POST _core/keystore/sm.actions.script.sme_re.token
{
  "value": "<token>"
}
```
Or use the utility on the node, then reload the secure settings:
```
opensearch-keystore add sm.actions.script.sme_re.token
```
```
POST _nodes/reload_secure_settings
```

To rotate a token without downtime, add a new entry to the store of every SME-RE, then replace the token on the OpenSearch nodes. Once the old alias no longer appears in the SME-RE log, delete it:

keytool -delete -storetype PKCS12 -keystore <path> -alias <old alias>

Interpreters

The interpreter of an action script is taken from the core interpreter registry, not from sme.core.remote_script.interpreters. The interpreter path must be the same on all SME-RE hosts and must be listed in the sme.scripts.interpreters parameter of the application.properties file on each of them, otherwise SME-RE rejects the run.

Information for Running Scripts in the Job Scheduler (the `script` Type)

The script type of the Job Scheduler runs an arbitrary command on the SME-RE host via the /sme_remote_exec/command route. The sm-actions-api plugin must be installed on the nodes for it to work.

Connecting to SA Engine RE from ScriptAction in the Job Scheduler

Settings can be retrieved via the following path:

GET _core/settings/job-scheduler

The table lists the settings used by ScriptAction to connect to SME-RE:

Setting	Description	Default Value
`sme-re_port`	Port on which `SME-RE` is running	18080
`sme-re_host`	List of URLs for nodes running `SME-RE`	["localhost"]

The username and password are taken from the sme.core.remote_script.user cluster setting and the sme.core.remote_script.password keystore setting.

Creating a Script

You can run any script in the Job Scheduler.

To pass secrets to a script, use the environment variables mechanism. It lets you store secrets in the keystore so that SME-RE adds them to the environment when the script runs.

To configure environment variables:

On all cluster servers, store the values to be passed to the script in the keystore under keys of the form jobscheduler.script.pass.<variable name>.
In the Job Scheduler settings, specify which keystore variables are available to each job. To do this, execute the following command:
```
PUT _core/settings/job-scheduler
{
  "script_environments": """{
    "<task id>": ["<variable name>"]
  }"""
}
```
The Job Scheduler now passes these variables to SME-RE when starting the script.

Add code to read the variable from the environment.

Example for Python:

import sys
from os import getenv

if __name__ == "__main__":
    # get the variable from the environment
    try:
        envParamData = getenv("<variable name>")
    except Exception as e:
        envParamData = e

SA Engine RE Logging Changes

Starting with version 4.2.1, the SME-RE log includes the following changes:

The following new fields have been added:
- Log ID — a unique identifier for each entry
- Thread ID — the identifier of the thread where the entry was created
- Error Code (for error entries only) — a code corresponding to a specific issue in the application
- Username — the user associated with the event. For calls on the /sme_scripts/v1 route, the token alias is shown
- Global Type — a type that identifies the system to which the entry belongs
Each entry now includes the date and time.

Log format before the update:

13:40:11.957 INFO [main] c.s.s.r.JavaClass#info:1 - Custom message

Log format after the update:

[2024-11-27 13:40:11,957][INFO ][c.s.s.r.r.JavaClass] [re][user_name][id][id_thread]  Custom message

SA Engine RE Configuration Check​

Verifying SA Engine RE Utility Availability​

Configuration Verification​

SA Engine RE Startup Parameters​

General Parameters​

Parameters of the Active Actions Route /sme_scripts/v1​

General SA Engine RE Connection Settings​

Information for Running Scripts in Search​

Connecting to SA Engine RE from the script Command in Search​

Creating a Script​

Running Active Actions via the /sme_scripts/v1 Route​

Connection Settings on the OpenSearch Side​

Script Call Token​

Interpreters​

Information for Running Scripts in the Job Scheduler (the script Type)​

Connecting to SA Engine RE from ScriptAction in the Job Scheduler​

Creating a Script​

SA Engine RE Logging Changes​

SA Engine RE Configuration Check

Verifying SA Engine RE Utility Availability

Configuration Verification

SA Engine RE Startup Parameters

General Parameters

Parameters of the Active Actions Route `/sme_scripts/v1`

General SA Engine RE Connection Settings

Information for Running Scripts in Search

Connecting to SA Engine RE from the script Command in Search

Creating a Script

Running Active Actions via the `/sme_scripts/v1` Route

Connection Settings on the OpenSearch Side

Script Call Token

Interpreters

Information for Running Scripts in the Job Scheduler (the `script` Type)

Connecting to SA Engine RE from ScriptAction in the Job Scheduler

Creating a Script

SA Engine RE Logging Changes