source

Description

Retrieves data from sources.

Syntax

source <source-name> [source-options]

Required Arguments

Parameter	Syntax	Description
source-name	<string>	For the OpenSearch source, simply specify the index pattern name. To fetch data from Hadoop or Clickhouse sources, use the prefix hdh: or clk: respectively. Optionally, specify the sample size (qsize) after the prefix and source name with :.

Examples:

source sysmon_operational-*

source hdh:win_events

source clk:nix_events:1000

Multiple Sources Query

You can query multiple sources by listing them separated by commas. By default, the result is combined using the append command. To use internal storage mechanisms to combine results, see the append optional argument.

Example:

source cisco_asa-*, clk:nix_events:1000

Optional Arguments

• source-options - optional parameters that provide additional search settings

Parameter	Syntax	Default	Description
qsize	qsize=<int>	1000000, when requested via web interface 1000	Maximum number of records to fetch.
timefield	timefield=<field>	@timestamp	Name of the field containing the timestamp.
append	append=<bool>	true	Data merging mode when querying multiple sources. With append=true, the append command algorithm is used; with append=false, merging is performed using the storage engine's internal mechanisms. Append Notes.
earliest	earliest=<string>		Start timestamp for the search.
latest	latest=<string>		End timestamp for the search.
timeout	timeout=<int>	See External Query Timeout	External query timeout in milliseconds.

For earliest and latest, the following time format is allowed: (+|-)<int>(s|m|h|d|w|month) | timestamp | unix-time:

• s/sec/secs/second/seconds - seconds
• m/min/mins/minute/minutes - minutes
• h/hr/hrs/hour/hours - hours
• d/day/days - days
• w/week/weeks - weeks
• mon/month/months - months

Local Search Parameters

Each source can use its own local search parameters that will override the global parameters. The source and its parameters should be enclosed in parentheses. See Example 4.

Append Notes

• The append = false option is only possible when querying OpenSearch
• The append parameter must be global
• When append = false, only global search parameters are considered

External Query Timeout

Timeout values for external sources are specified in _cluster/settings.

Name	Description	Default Value
sme.sa.timeout.os	Timeout for OpenSearch queries.	60000
sme.sa.timeout.es	Timeout for ElasticSearch queries.	60000
sme.sa.timeout.click	Timeout for ClickHouse queries.	60000
sme.sa.timeout.hadoop	Timeout for Hadoop Hive queries.	60000

Timeout for Hadoop Hive

The Hadoop Hive source only supports integer timeouts in seconds, so during query execution the timeout is converted to seconds with upward rounding.

Query Examples

Example 1

source ldap_users-* timefield=event_time

---

Example 2

source hdh:nix_events, ldap_computers-* qsize=1000 earliest=-2d latest=1649344240

---

Example 3

In this example, data is combined using OpenSearch.

source sysmon_operational-*, zabbix-* qsize = 150 append = false

---

Example 4

In this example, when querying data from the server_warnings index, local parameters are used, so qsize and the time boundaries differ from the queries to winevent and zabbix. Since timefield is not specified in the local parameters, the global parameter value will be used for server_warnings.

source (server_warnings:10 earliest="now-10m" latest="now-5m"),
 winevent, zabbix timefield=@timestamp earliest="now-1d" latest="now" qsize = 100