script_mc

Description

Executes a script and adds results to the query.

warning

The script_mc source requires a configured SA Engine Remote Executor (SA Engine RE) and a Memcached server. It is optimal to install the Memcached server with SA Remote Executor (or intermediate), not on the Search Anywhere Framework server.

SME-RE setup article

Syntax

...| script_mc "<path_to_script_engine> <path_to_script>"

---

Required Arguments

Parameter	Syntax	Description
path_to_script_engine	<string>	Path to executable file.
path_to_script	<string>	Path to script.

---

Source Configuration

info

All source settings are stored in _cluster/settings.

"sme" : {
    "core" : {
        "memcached": {
            "port": "11211",
            "ip": "127.0.0.1"
        }
    }
}

---

Query Examples

Example 1

In this example, script_mc runs a script by explicitly specifying the interpreter and file path in one line.

| script_mc "python /home/user/dev/test-script/test-script.py"

Examples with Argument Passing

To correctly execute a query regardless of where the script_mc command is used (at the beginning, middle, or end of the query), the following features must be considered:

1. Preparing data for the script: Before calling the script, the query result that will be processed is formed. Data transfer does not require mandatory limitation of the selection by required fields, but to improve performance, the data volume can be reduced using fields, table, or aggregating functions on the required fields. The query result in JSON format is passed to the script through a temporary file, the path to which is automatically specified as the last implicit argument.

2. Script data retrieval: The script must support receiving arguments and further working with them. Events can be the result of aggregation functions or simply the result of a search query. The main condition for successful transmission is the presence of the specified fields in the query result. To pass arguments, it is necessary to list the field names in the script_mc command call line. Input data is extracted from the resp.body.hits.hits block of the temporary file, where they are presented as an array of events (JSON objects). Each array element corresponds to a separate event, which can then be processed using the capabilities of the used interpreter

3. Sending data from the script: To send one event, it is sufficient to output the JSON result to stdout. To output multiple events, it is necessary to form a payload and similarly send it to stdout.

Example 1

This example demonstrates how to pass arguments to a script when calling script_mc. It is necessary to list the names of the required fields and implement the correct argument processing logic on the script side. The example also shows how to form data for outputting multiple events.

source temperature_sensors
| script_mc "python /home/user/test-script/test-script.py sensor_name location error_description"

Python script code

    import sys
    import json

    def main():

        # Reading arguments and checking their number
        args = sys.argv[1:]
        if len(args) != 6:
            raise Exception(f"Expected 5 fields + payload file, got {len(args)-1}")

        # Last argument - path to temporary file
        timestamp_field, sensor_name_field, location_field, error_description_field, connect_status_field, payload_filepath = args


        # Loading JSON payload from temporary SA Engine file
        with open(payload_filepath) as f:
            payload = json.load(f)

        # Extracting hits list - data block
        hits = payload.get("resp", {}).get("body", payload).get("hits", {}).get("hits", [])
        result_hits = []


        # Processing each document from hits
        for i, hit in enumerate(hits, 1):
            source = hit.get("_source", {})

            error_description = source.get(error_description_field)
            connect_status = source.get(connect_status_field)

            # Converting error_description to human-readable format
            error_description = (
                "No data" if error_description == -1 else
                "No error" if 1 <= error_description <= 3 else
                "Medium error" if 4 <= error_description <= 7 else
                "Critical error" if 8 <= error_description <= 10 else
                "No data"
            )

            # Converting connect_status to human-readable format
            connect_status = (
                "Connected" if connect_status == 1 else
                "Not connected" if connect_status == 0 else
                "No data"
            )

            # Forming the final result document
            result_hits.append({
                "_index": hit.get("_index", "custom"),
                "_type": hit.get("_type", "custom"),
                "_id": hit.get("_id", str(i)),
                "_score": hit.get("_score", 1),
                "_source": {
                    "@timestamp": source.get(timestamp_field, "No data"),
                    "sensor_name": source.get(sensor_name_field, "No data"),
                    "location": source.get(location_field, "No data"),
                    "error_description": error_description,
                    "connect_status": connect_status,
                }
            })


        # Forming payload for sending
        result = {
            "hits": {
                "total": {
                    "value": len(result_hits),
                    "relation": "eq"
                },
                "max_score": 1,
                "hits": result_hits
            }
        }

        # Outputting payload to stdout
        print(json.dumps(result))

    if __name__ == "__main__":
        main()

Example 2

Data is passed to a bash script in a similar way. The following example demonstrates executing the aggregation_value.sh script, which sends one event. Before using the script_mc function, data is selected using table. Then unique values for the passed fields are obtained, after which the unique_location field is transformed using the eval function, and the result is displayed through table.

source temperature_sensors
| table  sensor_name,location,error_description
| script "bash /home/user/test-script/aggregation_value.sh sensor_name location error_description"
| eval unique_location = upper(unique_location)
| table *

Bash script code

#!/bin/bash
# Getting the temporary query result file - the last implicit argument
payload_filepath="${!#}"

# Getting the field list 
fields=("${@:1:$(($#-1))}")

# Reading full JSON from file
payload=$(cat "$payload_filepath")

# Extracting document array located in hits
hits=$(echo "$payload" | jq '.resp.body.hits.hits // .hits.hits // []')

# Processing each passed field
jq_filter='{'

first=1

for field in "${fields[@]}"; do
  if [ $first -eq 1 ]; then
    first=0
  else
    jq_filter+=","
  fi

  jq_filter+="
    \"unique_$field\": (
      [ .[]?._source.\"$field\" ]
      | map(select(. != null))
      | unique
    )"
done

jq_filter+='
}'

# Outputting the result
echo "$hits" | jq "$jq_filter"