lookup

Description

Allows obtaining data from a predefined lookup.

Syntax

lookup dedup=<boolean> <execution-type> <lookup-name> ( <lookup-field> [AS <event-field>] )... [ OUTPUT | OUTPUTNEW (<lookup-destfield> [AS <event-destfield>] )... ]

Required Arguments

Parameter	Syntax	Description
lookup-name	<lookup-name>	The name of the predefined lookup.

Optional Arguments

Parameter	Syntax	Default	Description
dedup	dedup=<boolean>	false	Indicates whether duplicates should be removed from the data compared to the lookup.
<execution-type>	type=ELK [packsize=<integer>] | FAST	DEFAULT	Specifies the execution type of the command. The ELK type indicates that the command is executed by Elasticsearch/OpenSearch, with a default packsize of 10,000. The FAST type represents an optimized, faster execution version of the command in DEFAULT mode.
lookup-field	<string>		Several fields can be specified, separated by spaces.
event-field	<string>		The name of the field in the main result for comparison with the lookup-field. This is used when lookup-field and event-field have different names.
lookup-destfield	<string>		The name of the resulting field with obtained data.
OUTPUT | OUTPUTNEW	OUTPUT | OUTPUTNEW	OUTPUT	The OUTPUT indicator means that existing data in the source query field will be overwritten by the lookup data. The OUTPUTNEW indicator means that data will be overwritten only if the source query field is empty but has corresponding data in the lookup.

Configuration Recommendations

1. Calling the command without specifying the execution type is recommended for small volumes of data in the lookup (up to 50,000 events). It is also not recommended to run the command this way when sme settings have max_initial_query_len greater than 10,000.
2. Calling the command with the FAST type is not recommended for lookups with more than 500,000 events.
3. For the ELK execution type, it's recommended not to set a high max_initial_query_len. The packsize parameter indicates how many events are processed in each iteration. For example, if max_initial_query_len = 10,000, and packsize = 5,000, then two requests are sent for a query of 10,000 events (max_initial_query_len/packsize). If max_initial_query_len is more than 20,000, it's recommended to set packsize to half or a third of max_initial_query_len.

Query Examples

Example #1

source tables
| search database = "hr"
| lookup my_click_db name as database OUTPUT engine

Example #2

source tables
| search database = "hr"
| lookup type=ELK packsize=15000 my_click_db name as database OUTPUTNEW engine