Version: 5.0

SA Engine RE Configuration

SA Engine RE Configuration Check

Verifying SA Engine RE Utility Availability

The SA Engine RE utility is automatically installed during every system deployment or update. It is present on all OpenSearch nodes where the node.attr.node_with_sme: true parameter is set in the opensearch.yml file.

warning

Ensure that the configuration file /app/opensearch/utils/sme-re/application.properties exists and contains the correct settings. If this file is missing, the SA Engine RE utility will not be able to connect to OpenSearch. Additionally, verify that the node.attr.node_with_sme parameter in opensearch.yml is set to true. If it is set to false, this node will not receive script execution tasks.

Configuration Verification

To check the current settings, execute the following command in the Developer Console (Main Menu - System Settings - Developer Console):

GET _cluster/settings

The response should include the following block:

...

    "sme": {
        "core": {
            "remote_script": {
                "base_path": "/app/opensearch/utils/scripts/",
                "enable_ssl": "true",
                "port": "18080",
                "base_interpreter_name": "bash",
                "url": "https://127.0.0.1",
                "interpreters": [
                    "python3::/app/opensearch/utils/python/bin/python3",
                    "bash::bash"
                ]
            },
            "enable_field_caps": "false",
            "enable_limits": "true"
        },
        "overwrite": {
            "scroll_timeout": "30000"
        }
    },

...    		

If the sme.core parameter is missing, add it by adjusting the settings as needed.

Execute the following command in the Developer Console to update the configuration:

PUT _cluster/settings
{
  "persistent": {
    "sme": {
      "core": {
        "remote_script": {
          "base_path": "/app/opensearch/utils/scripts/",
          "enable_ssl": true,
          "port": "18080",
          "base_interpreter_name": "bash",
          "url": "https://127.0.0.1",
          "interpreters": [
            "python3::/app/opensearch/utils/python/bin/python3",
            "bash::bash"
          ]
        }
      }
    }
  }
}

warning

Script execution occurs on a random OpenSearch node where node.attr.node_with_sme: true is set. Therefore, scripts must be deployed on all such nodes.

To check the list of nodes and their parameters, execute the following command in the Developer Console (Main Menu - System Settings - Developer Console):

GET _cat/nodeattrs

In the command output, look for the node_with_sme parameter.

SA Engine RE Startup Parameters

SA Engine RE startup parameters are specified in the application.properties configuration file, located in the same directory as the SA Engine RE utility.

Table of startup parameters:

Parameter	Description	Default Value
`server.port`	The port on which `SA Engine RE` will accept incoming connections.	18080
`spring.servlet.multipart.max-request-size`	Maximum file upload size in bytes.	100000000
`server.tomcat.max-http-form-post-size`	Maximum size for `multipart/form-data` requests in bytes.	104857600
`sme.FileRotatePeriod`	Request data file rotation time in seconds.	10000
`sme.baseData`	Directory for storing request data files.	./data
`sme.directoryPollerInterval`	Interval for checking the request data file storage directory in seconds.	50
`sme.scriptConfigPath`	Path to the `scripts.yaml` configuration file.	./scripts.yaml
`sme.LogStringSize`	Log string size.	300
`sme.RapidDelete`	Whether to delete request data files.	True
`sme.redis.url`	Address for connecting to `Redis`.	"127.0.0.1"
`sme.redis.port`	Port for connecting to `Redis`.	6379
`sme.redis.pass`	Password for connecting to `Redis`.	""
`logging.file.path`	Path for storing logs.	logs
`server.ssl.enabled`	Whether to use SSL/TLS for incoming connections.	true
`server.ssl.certificate`	Path to the public key (certificate) of the current `SA Engine RE` server in PKCS8 format.	/app/opensearch/config/node-cert.pem
`server.ssl.certificate-private-key`	Path to the private key of the current `SA Engine RE` server in PKCS8 format.	/app/opensearch/config/node-key.pem
`server.ssl.trust-certificate`	Path to the Certificate Authority (CA) chain in PKCS8 format.	/app/opensearch/config/ca-cert.pem
`server.ssl.client-auth`	Certificate validation mode, can be `need`, `none`, `want`.	need

General SA Engine RE Connection Settings

Table describing the general settings for connecting to SA Engine RE:

Setting	Description	Default Value	Setting Type
`sme.core.remote_script.user`	Username used to make requests to `SA Engine RE`.	sme_re_user	Cluster Setting
`sme.core.remote_script.password`	Password of the user used to make requests to `SA Engine RE`.		Keystore Setting
`sme.core.remote_script.enable_ssl`	Flag indicating whether to use SSL for connecting to `SA Engine RE` from the `script` command in search and in `ScriptAction` in the task scheduler.	true	Cluster Setting

Information for Running Scripts in Search

Connecting to SA Engine RE from the script Command in Search

Table describing cluster settings for connecting to SA Engine RE from the script command:

Setting	Description	Default Value
`sme.core.remote_script.base_path`	Path to the folder where scripts will be uploaded.	"/app/opensearch/utils/scripts/"
`sme.core.remote_script.port`	Port on which `SA Engine RE` is running.	18080
`sme.core.remote_script.url`	URL where `SA Engine RE` is running.	"https://127.0.0.1"
`sme.core.remote_script.interpreters`	List of available interpreters, as an array of strings. String format: "interpreter name"::"path to interpreter".	["python3::/app/opensearch/utils/python/bin/python3","bash::bash"]

info

Requests are evenly distributed among hosts with SA Engine RE using the Round Robin algorithm.

Authorization

Request authorization for SA Engine RE in SA Engine is always performed on the host where SA Engine RE is running.

Creating a Script

To use a script in search, you must place it in the folder specified in the sme.core.remote_script.base_path setting on all servers in the cluster, ensure the file is executable, and has opensearch:opensearch permissions.

To return data from the script for further processing in the search, output the result in JSON format.

Example for Python:

print(scriptResultJson)

If other commands are executed before the script command, their result is saved to a temporary file in the folder specified in sme.baseData. The name of this file will be passed to the script as a command-line argument.

Example for Python:

import sys

if __name__ == "__main__":
    # open the file with data from the request
    inputDataFile = open(sys.argv[1], "r")

    # read the file with data from the request
    inputData = inputDataFile.read()

    inputDataFile.close() # close the file after reading

Information for Running Scripts in the Task Scheduler

Connecting to SA Engine RE from ScriptAction in the Task Scheduler

Settings can be retrieved via the following path:

GET _core/settings/job-scheduler

Table describing the settings for connecting to SA Engine RE from ScriptAction:

Setting	Description	Default Value
`sme-re_port`	Port on which `SA Engine RE` is running.	`18080`
`sme-re_host`	Host on which `SA Engine RE` is running.	`localhost`

Creating a Script

You can run any script in the task scheduler.

If you need to pass sensitive data to the script, you can use the environment variables feature. This feature allows you to store sensitive data in the keystore so that SA Engine RE adds them to environment variables when the script is run.

Algorithm for configuring environment variables:

Store the necessary variables to be passed to the script in the keystore on all cluster servers, using the variable jobscheduler.script.pass.<variable_name>
Specify in the task scheduler settings which task can use which variables from the keystore To do this, execute the following command:

PUT _core/settings/job-scheduler
{
  "script_environments": """{
    "<task_id>": ["<variable_name>"]
  }"""
}

Now the scheduler will instruct SA Engine RE to run the script with the variables in the environment.

In the script, include code to retrieve the variable from the environment

Example for Python:

import os

if __name__ == "__main__":
    # get the variable from the environment
    try:
        envParamData = os.getenv("<variable_name>")
    except Exception as e:
        envParamData = e

Logging Changes in the Utility

The logging system in SA Engine RE has been updated:

The following new fields have been added:

Log ID – a unique identifier for each log entry
Thread ID – the identifier of the thread where the log entry was created
Error Code (only for error logs) – a numeric code corresponding to a specific issue in the application's operation
Username – the user associated with the event
Global Type – a type that indicates the system to which the log entry belongs

Timestamp – each entry now includes the date and time

Log format before the update:

13:40:11.957 INFO [main] c.s.s.r.JavaClass#info:1 - Custom message

Log format after the update:

[2024-11-27 13:40:11,957][INFO ][c.s.s.r.r.JavaClass] [re][user_name][id][id_thread]  Custom message