source

Description

Retrieves data from sources.

Syntax

source <source-name> [exclude] [source-options]

---

Required Arguments

Parameter	Syntax	Description
source-name	<string>	An index pattern in internal storage, a Search Anywhere source with a prefix (such as hdh: or clk:), or a Data Unit or Search Space name when the corresponding setting is enabled. For Search Anywhere sources, specify the sample size (qsize) after the source name, separated by :.

Examples:

source sysmon_operational-*

source hdh:win_events

source clk:nix_events:1000

Data Units and Search Spaces

When the sme.core.data_units.search_enabled setting is enabled, a Data Unit or Search Space can be used as a source:

source du:audit_events

source ss:security_events earliest="now-1d" latest="now"

A short name is resolved in this order: Search Space, Data Unit, index pattern. The explicit prefixes du: and ss: select the source type; an unknown name with one of these prefixes results in an error. Names are case-sensitive. A wildcard in source remains an index pattern and does not enumerate Data Units. To enumerate Data Units, specify a pattern in the Search Space definition.

A Data Unit or Search Space cannot be listed with other sources separated by commas, and exclude(...) cannot be applied to them. For time field, merging, permissions, and limitations, see Searching Data Units and Search Spaces.

Multiple Sources Query

Indexes and Search Anywhere sources can be listed separated by commas. By default, the results are combined using the append command algorithm. To use internal storage mechanisms to combine results, see the append optional argument.

Example:

source cisco_asa-*, clk:nix_events:1000

---

Optional Arguments

• exclude - optional parameter that accepts sources/patterns of sources that will be excluded from the search
• source-options - optional parameters that provide additional search settings

Parameter	Syntax	Default	Description
qsize	qsize=<int>	1000000, when requested via web interface 1000	Maximum number of records to fetch.
timefield	timefield=<field>	@timestamp	Name of the field that contains the timestamp.
append	append = <boolean>	true	Data merging mode when querying multiple sources. With append = true, the append command algorithm is used; with append = false, merging is performed using the storage engine's internal mechanisms. Append Notes.
earliest	earliest=<string>		Start timestamp for the search.
latest	latest=<string>		End timestamp for the search.
timeout	timeout=<int>	See External Query Timeout	External query timeout in milliseconds.
exclude	exclude(<source-name> [, <source-name> ...])		List of excluded sources.

For earliest and latest, the following time format is allowed: (+|-)<int>(s|m|h|d|w|month) | timestamp | unix-time:

• s/sec/secs/second/seconds - seconds
• m/min/mins/minute/minutes - minutes
• h/hr/hrs/hour/hours - hours
• d/day/days - days
• w/week/weeks - weeks
• mon/month/months - months

Local Search Parameters

Each source can use its own local search parameters that will override the global parameters. The source and its parameters should be enclosed in parentheses. See Example 4.

Append Notes

• the append = false option is only possible when querying OpenSearch
• the append parameter must be global
• when append = false, only global search parameters are considered

---

Default Time Parameters

If one or several time parameters are not specified for a source, default values for these parameters will be substituted during query compilation.

info

Default time parameters are stored in _cluster/settings in the sme.core.global_dt_params block.

Warning About Setting Default Time Parameters

In case default time parameters are substituted, a separate warning will be generated for each source, containing information about which values were substituted.

For example, executing a query to the index sysmon_operational-* without specifying time fields will result in a response containing a warning with the following message:

Default values were set for dtFieldName='@timestamp' from cluster settings, dtStart='now-24h' from cluster settings, dtEnd='now' from cluster settings for index 'sysmon_operational-*'

---

Time Parameter Priority

Local source parameters have the highest priority: timefield, earliest, and latest. If they are not specified, global search parameters are used: dtFieldName, dtStart, and dtEnd. If a global time parameter is also not specified, its value is taken from the cluster settings.

For a Data Unit, a non-empty time_field from its configuration takes precedence over the query's dtFieldName, but has lower priority than an explicitly specified timefield in source. When searching a Search Space, each Data Unit uses its own field unless timefield is specified for the entire source. The earliest and latest parameters also override the time boundaries when searching a Search Space.

Time Field Priority for Search Anywhere Configs

Special rules apply when determining the time field in a Search Anywhere query. Local search parameters have the highest priority. If timefield is not explicitly specified for a source, the value of the Default time field field in the Search Anywhere configuration is used. If the field is empty, the general order applies: dtFieldName from the query or the value from the cluster settings.

---

Ignoring Time Parameters

If an empty string is passed to the dtStart or dtEnd parameter, the corresponding time boundary is ignored.

In this example, all data up to the current time will be requested.

POST _sme
{
  "query": "source sysmon_operational-* | aggs count ",
  "dtStart": ""
}

tip

If an empty string is passed to the dtFieldName parameter, time boundaries and time-based sorting are disabled.

---

External Query Timeout

Timeout values for external sources are specified in _cluster/settings.

Name	Description	Default Value
sme.sa.timeout.os	Timeout for OpenSearch queries.	60000
sme.sa.timeout.es	Timeout for ElasticSearch queries.	60000
sme.sa.timeout.click	Timeout for ClickHouse queries.	60000
sme.sa.timeout.hadoop	Timeout for Hadoop Hive queries.	60000

Timeout for Hadoop Hive

The Hadoop Hive source only supports integer timeouts in seconds, so during query execution the timeout is converted to seconds with upward rounding.

---

Query Examples

Example 1

In this example, data from indexes ldap_users_* will be found, except for ldap_users_logs* and ldap_users_errors.

source ldap_users_* exclude(ldap_users_logs*, ldap_users_errors) timefield=event_time

---

Example 2

source hdh:nix_events, ldap_computers-* qsize=1000 earliest=-2d latest=1649344240

---

Example 3

In this example, data is combined using OpenSearch.

source sysmon_operational-*, zabbix-* qsize = 150 append = false

---

Example 4

In this example, when querying data from the server_warnings index, local parameters are used, so qsize and the time boundaries differ from the queries to winevent and zabbix. Since timefield is not specified in the local parameters, the global parameter value will be used for server_warnings.

source (server_warnings:10 earliest="now-10m" latest="now-5m"),
 winevent, zabbix timefield=@timestamp earliest="now-1d" latest="now" qsize = 100