Skip to main content
Version: 6.1

Agent Data Collection in Search Anywhere Framework

Agent data collection is a way to receive logs, metrics, and other events directly from endpoint nodes in Search Anywhere Framework. For this purpose, the SAF Beat agent is installed on each node. SAF Beat is an aggregator of Elastic Beats applications that collect data from different sources. Agents, their configurations, and applications are managed centrally through SAF Beat Manager.

This approach allows different data types to be collected from one node at the same time, such as system metrics, application logs, and Windows security events. The set of applications and collection parameters can be assigned separately to each agent or to an entire agent group.

Agent Data Collection Architecture​

The agent collection process consists of the following steps:

  1. SAF Beat Manager assigns Elastic Beats application distributions and their configurations to an endpoint node.

  2. SAF Beat receives the assignment, installs the applications, and controls their startup.

  3. Each application collects its data type and sends events to SA Data Collector or directly to SA Data Storage.

SAF Beat Manager coordinates the agents but does not participate in transmitting collected events.

Agent data collection architecture

SAF Beat Manager​

SAF Beat Manager is a system component that coordinates SAF Beat agents. It provides centralized management of configurations and applications, as well as information about agent status, including activity, installed applications, and assigned groups.

For more information, see SAF Beat Management.

SAF Beat​

SAF Beat is installed on an endpoint node and receives assigned applications and configurations from SAF Beat Manager. The agent then installs, starts, and controls the corresponding Elastic Beats.

Several Beats can run on one node at the same time. For example, Filebeat can collect application logs, while Metricbeat can collect operating system metrics. The set of applications is determined by the collection tasks for a specific node.

Information

For information about installing and configuring the agent, see SAF Beat for Linux and SAF Beat for Windows.

Elastic Beats Overview​

Elastic Beats are lightweight specialized applications for collecting data from various sources. Each Beat type serves a specific purpose: collecting logs, system metrics, security events, availability check results, or network traffic.

Elastic Beats are installed and run inside SAF Beat. Because data types are separated, only the applications required for a specific collection scenario need to be assigned to an endpoint node.

Elastic Beats applications in SAF Beat

Filebeat​

Filebeat is designed to collect data from files. It monitors new entries and sends them to the data processing and storage system, so logs do not need to be uploaded manually.

The application supports text and structured logs, including JSON and CSV, and can combine multiline entries into a single event. Filebeat is commonly used to collect operating system and application logs.

Metricbeat​

Metricbeat collects system metrics and application performance indicators. It can provide information about CPU load, memory, disk, and network usage, as well as running processes and service status.

Collected data is used to monitor infrastructure status and analyze node performance. The metric set is determined by enabled modules and metric sets (metricsets).

Winlogbeat​

Winlogbeat is designed to collect events from Windows logs. It can transmit system errors, warnings, logon events, and other information required to monitor node operation and security.

The configuration can specify the standard Application, System, and Security channels, service and custom logs, as well as events forwarded from other nodes through ForwardedEvents.

Heartbeat​

Heartbeat regularly checks the availability of network nodes and services over ICMP, TCP, and HTTP. The results contain information about service status and response time, which helps detect unavailable or slow monitored resources.

Auditbeat​

Auditbeat collects information about user actions, system events, and file system changes. It can monitor process launches, file access, and changes to file attributes, as well as check the integrity of selected files and directories.

On Linux, Auditbeat can integrate with the Linux Audit Framework and receive kernel-level audit events. Filters can limit collection to the required event types, users, and file paths.

Packetbeat​

Packetbeat collects and analyzes network traffic at the level of supported application protocols, converting packets into events. The application groups related requests and responses into transactions and records response time, the amount of transferred data, and response codes.

This information can be used to analyze interactions between services, find delays, and detect network anomalies.

Elastic Beats Configuration​

Elastic Beats distributions and configurations are uploaded to SAF Beat Manager and assigned to individual agents or groups. SAF Beat receives the assignment, installs the corresponding Beat, and starts it with the specified configuration.

A configuration consists of parameters in the key: value format. Depending on the Beat type, it defines data sources and polling intervals, enabled modules, event filtering and processing, and the destination for collected data. For example, Filebeat configurations specify log file paths, while Metricbeat configurations specify modules and metric sets.

For more information, see Configuring Standard Data Collection Applications.