Skip to main content

Installing SA Data Collector

Conventions:

  • SAF_INSTALLER - the directory where the Search Anywhere Framework installation package is extracted.

General Information​

Unpack the installation package into a directory, for example, /app/distr and specify the unpacked directory as a variable:

tar xzf saf-installer-4.2-full.tar.gz -C /app/distr/
SAF_INSTALLER=/app/distr/saf_4.2/

SA Data Collector is based on Logstash, which may be used as an alternative name throughout this document.

Run the SA Data Collector installer as the root user, specifying the path to it:

${SAF_INSTALLER}/logstash/install.sh

The installer supports the following parameters:

  • -c, --config <path_to_config_file_yaml> - use a configuration file for installation
  • -h, --help - display help information

The installer supports the use of a configuration file. All parameters in the file correspond to the questions and parameters during manual installation.

Example configuration file ${SAF_INSTALLER}/logstash/example_config_logstash.yaml
Logstash:
  logstashLocation:
    # User for install logstash
    LS_INSTALL_USER: logstash
    # Group for install logstash
    LS_INSTALL_GROUP: logstash
    # Home path to install directory
    LS_PATH_HOME: "/app/logstash"
    # Directory logstash configuration
    LS_PATH_CONFIG: "/app/logstash/config"
    # Directory logstash data storage
    LS_PATH_DATA: "/app/data/logstash"
    # Directory logstash logs storage
    LS_PATH_LOGS: "/app/logs/logstash"
  baseLogstashParameters:
    # Name of the current node
    NODE_NAME: saf-logstash
    # JVM heap for logstash instanse
    JVM_HEAP_SIZE: 4
  openDistroSecurity:
    # Using existing certificates and keys
    USE_EXISTING_CERTS: y
    # Using existing CA certificate and private key to generate node certificate and private key
    EXISTING_CA_CERT: y
    # Path to CA cert
    CA_CERT_PATH: "/app/opensearch/config/ca-cert.pem"
    # Path to CA key
    CA_KEY_PATH: "/app/opensearch/config/ca-key.pem"
    # Path to existing node certificate
    EXISTING_NODE_CERT: "./certs/node-cert.pem"
    # Path to existing node private key
    EXISTING_NODE_PRKEY: "./certs/node-key.pem"
    # CN for node certificate
    CERT_NODE_NAME: saf-logstash
    # County code
    CERT_COUNTRY: AE
    # Region
    CERT_STATE: Dubai
    # City
    CERT_CITY: Dubai
    # Name organization
    CERT_ORG: SAF Test Org
    # IP to connect current node
    CERT_NODE_IP: 127.0.0.1
  installationProceed:
    # Kill processes of logstash if they started
    kill_current_instance: "y"
    # Clear existed folders
    clear_existed_folders: "y"
    # Making changes to the operating system
    continue_installation: "y"

If SA Data Collector is used as a collection point for metrics before sending them to SA Data Storage, we recommend using a separate server for installing SA Data Collector. By default, the logstash user and logstash group own the directories and files for SA Data Collector.

Our installer already includes the logstash-output-opensearch plugin, which is needed to connect to OpenSearch.

Installing the Main Directories​

The installer checks for a running logstash.service and, if found, prompts whether to continue the installation:

================================================================================
SEARCH ANYWHERE FRAMEWORK INSTALL SCRIPT - LOGSTASH
================================================================================

Current working directory: /app/distr/saf_4.2/logstash

Current name of install's archive: logstash-oss-with-opensearch-output-plugin-8.9.0-linux-x64.tar.gz

Current version Logstash: 8.9.0

================================================================================
-- STEP 1. INSTALLATION DIRECTORIES


!!! You have running instance of Logstash. Installer support only one running instance per node.
Do you want to continue? [y/N]:

This output also shows the detected installation archive and the version of Logstash to be installed. By default, the installation will be aborted. To continue, enter "y". If the installation continues, the running service will be stopped.

The first step sets the main installation directories. We recommend storing all files related to SAF in a single directory, such as /app, to simplify maintenance.

If you are performing a test installation and plan to install all Search Anywhere Framework components on the same server, you must specify separate directories for each component. Enter the following directories:

  • Home directory — the installation directory for SA Data Collector
  • Conf directory — the location of configuration files
  • Data directory — the directory for storing various files used by SA Data Collector
  • Logs directory — the directory for storing logs
-- STEP 1. INSTALLATION DIRECTORIES

Logstash home directory [/app/logstash]:
Logstash conf directory [/app/logstash/config]:
Logstash data directory [/app/data]: /app/data/logstash/
Logstash logs directory [/app/logs]: /app/logs/logstash/

Final Logstash home directory: /app/logstash
Final Logstash conf directory: /app/logstash/config
Final Logstash data directory: /app/data/logstash/
Final Logstash logs directory: /app/logs/logstash/
Is this correct? [y/n]:

After entering the information, confirm its correctness. The installer will analyze the existing directories, and if they are not empty, it will issue another warning:

!!! You've chosen non-empty directories. All files will be deleted.
Do you want to continue? [y/N]:
danger

If you continue, all content in the specified directories will be deleted without the possibility of recovery!

To continue the installation, enter y again and press Enter.

Setting Logstash Parameters​

In this step, set the basic parameters for Logstash, such as:

  • Node name — the name of the current Logstash server
  • JVM Heap Size — the amount of memory in GB allocated to the Logstash JVM. This parameter should not exceed the amount of free memory on the server. The installer will automatically suggest half of the available RAM on the server.
-- STEP 2. BASE LOGSTASH PARAMETERS

Node name [smos-node-00]:
JVM heap size in GB [4]:

Final node name: smos-node-00
Final JVM heap size in GB: 4
Is this correct? [y/n]:

Confirm the entered data and continue the installation.

Setting Certificate Parameters​

note

When creating the private key, the RSA cryptographic algorithm with a length of 2048 bits is used, and the SHA-256 algorithm is used to create the digital fingerprint. The administrator and current server certificates are created for 1095 days.

This step configures the key and certificate parameters for the current SA Data Collector server. If you previously created a self-signed CA, you must copy the private key and certificate of this CA to generate the current node's certificate automatically.

  • Do you want to use your existing certificates and keys? — You can specify your own files or use the built-in algorithm.
  • Do you want to use an existing CA certificate and private key to generate certificates and keys for this node? — If you choose no, you will need to manually provide all the necessary certificates for the node to work. Otherwise, by selecting yes, the remaining certificates will be automatically generated based on the CA certificate and key.
  • Path to CA certificate — the path to the root certificate. The certificate must be the same on all Search Anywhere Framework components.
  • Path to CA key — the path to the private key. The key must be the same on all Search Anywhere Framework components.
  • CN for node certificate — the common name for the current server.
  • Country for node certificate — the country name for the current server.
  • State for node certificate — the state/region name for the current server.
  • Locality for node certificate — the city name for the current server.
  • Organization for node certificate — the organization name for the current server.
  • External IP — the external IP address of the current server, which will be used for interaction with other components. The script will automatically determine the external addresses on the current server and suggest them as the default value.

Below is an example with specifying existing CA private key and certificate:

-- STEP 3. OPENDISTRO SSL/TLS SETTINGS

Do you want to use your existing certificates and keys? [Y/n]:
Do you want to use an existing CA certificate and private key to generate certificates and keys for this node? [Y/n]
Path to CA certificate [./certs/ca-cert.pem]: /app/opensearch/config/ca-cert.pem
Path to CA key [./certs/ca-key.pem]: /app/opensearch/config/ca-key.pem
CN for node certificate [saf-node-00]:
Country for node certificate [AE]:
State for node certificate [Dubai]:
Locality (city) for node certificate [Dubai]:
Organization for node certificate [SAF Test Org]:
External IP for alt_names extension [172.16.0.22]:
Final path to CA certificate: /app/opensearch/config/ca-cert.pem
Final path to CA key: /app/opensearch/config/ca-key.pem
Final CN for node certificate: saf-node-00
Final country for node certificate: AE
Final state for node certificate: Dubai
Final locality (city) for node certificate: Dubai
Final organization for node certificate: SAF Test Org
External IP for alt_names extension: 172.16.0.22
Is this correct? [y/n]:

Confirm the correctness of the input and continue the installation.

Generating Configuration Files​

No action is required from the administrator at this stage. This stage is conditionally divided into two parts:

  • Preparing temporary directories, performing checks, preparing templates
  • Generating configuration files from the specified parameters

Creating Certificates​

No action is required from the administrator at this stage. The private key and certificate of the current server are created from the specified parameters.

Unpacking the Archive and Modifying Configuration Files​

No action is required from the administrator at this stage. The archive containing Logstash is unpacked, the extracted configuration files are modified, and preparatory processes are performed before making system changes.

Starting System Modifications​

All operations up to this point are performed strictly within the installer directory and do not affect the OS. You can view the set of files ready for installation in the staging directory.

The administrator will be asked for permission to continue the installation:

!!! AT THIS POINT WE START TO MAKE CHANGES IN OPERATING SYSTEM !!!
Do you want to continue? [y/N]:

Confirm the changes to continue the installation by pressing y and then Enter.

Installation​

No action is required from the administrator at this stage. Several actions take place here:

  • Creating groups and users
  • Creating directories
  • Installing Logstash
  • Installing plugins
  • Changing file and directory ownership and setting SELinux rules

Preparatory Work for Starting SA Data Collector​

No action is required from the administrator at this stage. The logstash.service auto-start is enabled.

Completing the Installation​

Upon successful installation, a message about configuring the firewall using firewalld and ufw as examples should appear, as well as a message about successful installation, as shown in the image below. Any other type of installation completion indicates an error, and additional corrective actions are required.

================================================================================
-- Additional firewall configuration example (if needed)
-- For RHEL/CentOS with firewalld
    firewall-cmd --permanent --zone=public --add-port=9600/tcp
    firewall-cmd --reload
-- For AstraLinux/Debian with ufw
    ufw allow 9600/tcp
================================================================================

================================================================================
-- SEARCH ANYWHERE FRAMEWORK SUCCESSFULLY INSTALLED!
================================================================================

Upon successful completion of the installer, the node is deployed but requires configuration files (pipelines) for data collection. The logstash.service will automatically start after the OS reboots. Note that you need to configure the firewall for the ports on which data will be received.

To verify functionality after adding the configuration and enabling the logstash.service, you can enter the following in the command line:

curl -X GET http://localhost:9600?pretty

Example output:

{
  "host": "srv-saf",
  "version": "8.9.0",
  "http_address": "127.0.0.1:9600",
  "id": "0d6b3458-147c-4b81-b3bb-210ddafdc440",
  "name": "saf-logstash",
  "ephemeral_id": "bae83d99-e076-42f3-9fc8-546e06ba1591",
  "status": "green",
  "snapshot": false,
  "pipeline": {
    "workers": 4,
    "batch_size": 125,
    "batch_delay": 50
  },
  "build_date": "2023-07-18T22:17:29+00:00",
  "build_sha": "375f47bd852411a88d350503847f399e8f652104",
  "build_snapshot": false
}