Skip to main content
Version: 5.1

Installing SA Data Collector

Conventions:

  • SAF_INSTALLER - the directory where the Search Anywhere Framework installation package is extracted.

General Information​

Unpack the installation package into a directory, for example, /app/distr and specify the unpacked directory as a variable:

tar xzf saf-installer-*.tar.gz -C /app/distr/
SAF_INSTALLER=/app/distr/saf_5.1/

SA Data Collector is based on Logstash, which may be used as an alternative name throughout this document.

It is recommended to run the SA Data Collector installer as the root user. To do this, execute in the server terminal:

${SAF_INSTALLER}/logstash/install.sh

The installation script can be run by the logstash user with limited permissions (in rootless mode), but some commands must be executed before and after running the script by a user with elevated privileges. To start the installation without privilege escalation, use the command:

sudo -u logstash ${SAF_INSTALLER}/logstash/install.sh
Important Note!

A complete update cannot be performed without root privileges - some operations must be executed by a user with elevated permissions. The installer will display the list of required operations at the appropriate step. If you don't execute these commands before updating, data loss may occur.

The installer supports the following parameters:

  • -c, --config <path_to_config_file_yaml> - use a configuration file for installation
  • -h, --help - display help information

The installer supports the use of a configuration file. All parameters in the file correspond to the questions and parameters during manual installation.

Example configuration file ${SAF_INSTALLER}/logstash/example_config_logstash.yaml
Logstash:
  logstashLocation:
    # User for install logstash
    LS_USER: logstash
    # Group for install logstash
    LS_GROUP: logstash
    # Home path to install directory
    LS_PATH_HOME: "/app/logstash"
    # Directory logstash configuration
    LS_PATH_CONFIG: "/app/logstash/config"
    # Directory logstash data storage
    LS_PATH_DATA: "/app/data/logstash"
    # Directory logstash logs storage
    LS_PATH_LOGS: "/app/logs/logstash"
  baseLogstashParameters:
    # Name of the current node
    NODE_NAME: saf-logstash
    CALCULATE_JVM_HEAP_SIZE: y
    # JVM heap for logstash instanse
    JVM_HEAP_SIZE: 4
  openDistroSecurity:
    # Using existing certificates and keys
    USE_EXISTING_CERTS: y
    # Using existing CA certificate and private key to generate node certificate and private key
    EXISTING_CA_CERT: y
    # Path to CA cert
    CA_CERT_PATH: "/app/opensearch/config/ca-cert.pem"
    # Path to CA key
    CA_KEY_PATH: "/app/opensearch/config/ca-key.pem"
    # Path to existing node certificate
    EXISTING_NODE_CERT: "./certs/node-cert.pem"
    # Path to existing node private key
    EXISTING_NODE_PRKEY: "./certs/node-key.pem"
    # CN for node certificate
    CERT_NODE_NAME: saf-logstash
    # County code
    CERT_COUNTRY: AE
    # Region
    CERT_STATE: Dubai
    # City
    CERT_CITY: Dubai
    # Name organization
    CERT_ORG: SAF Test Org
    # IP or DNS names for connecting to current node
    CERT_NODE_ALTNAMES: 127.0.0.1,localhost
  installationProceed:
    # Kill processes of logstash if they started
    kill_current_instance: "y"
    # Clear existed folders
    clear_existed_folders: "y"
    # Making changes to the operating system
    continue_installation: "y"

If SA Data Collector is used as a collection point for metrics before sending them to SA Data Storage, we recommend using a separate server for installing SA Data Collector. By default, the logstash user and logstash group own the directories and files for SA Data Collector.

Our installer already includes the logstash-output-opensearch plugin, which is needed to connect to OpenSearch.

Installing the Main Directories​

The installer checks for a running logstash.service and, if found, prompts whether to continue the installation:

================================================================================
SEARCH ANYWHERE FRAMEWORK INSTALL SCRIPT - LOGSTASH
================================================================================

Current working directory: /opt/saf_5.1/logstash

Current name of install's archive: logstash-oss-8.13.4-linux-x86_64.tar.gz

Current version Logstash: 8.13.4

================================================================================
-- STEP 1. INSTALLATION DIRECTORIES


!!! You have running instance of Logstash. Installer support only one running instance per node.
Do you want to continue? [y/N]:

This output also shows the detected installation archive and the version of Logstash to be installed. By default, the installation will be aborted. To continue, enter "y". If the installation continues, the running service will be stopped.

The first step sets the main installation directories. We recommend storing all files related to SAF in a single directory, such as /app, to simplify maintenance.

If you are performing a test installation and plan to install all Search Anywhere Framework components on the same server, you must specify separate directories for each component. Enter the following directories:

  • Home directory — the installation directory for SA Data Collector
  • Conf directory — the location of configuration files
  • Data directory — the directory for storing various files used by SA Data Collector
  • Logs directory — the directory for storing logs
-- STEP 1. INSTALLATION DIRECTORIES

Logstash home directory [/app/logstash]:
Logstash conf directory [/app/logstash/config]:
Logstash data directory [/app/data/logstash]: /app/data/logstash/
Logstash logs directory [/app/logs/logstash]: /app/logs/logstash/

Final Logstash home directory: /app/logstash
Final Logstash conf directory: /app/logstash/config
Final Logstash data directory: /app/data/logstash/
Final Logstash logs directory: /app/logs/logstash/
Is this correct? [y/n]:

After entering the information, confirm its correctness. The installer will analyze the existing directories, and if they are not empty, it will issue another warning:

!!! You've chosen non-empty directories. All files will be deleted.
Do you want to continue? [y/N]:
danger

If you continue, all content in the specified directories will be deleted without the possibility of recovery!

To continue the installation, enter y again and press Enter.

Rootless Installation​

At this stage, you can disable the use of commands that require elevated root privileges by choosing the rootless mode.

-- STEP 2. DISABLE OPERATIONS WHERE ROOT RIGHTS ARE REQUIRED

Want to install SEARCH ANYWHERE FRAMEWORK without root privileges? [y/N]:

To continue installation in rootless mode, type y and press Enter. If you want to proceed with installation using the root user, type n or simply press Enter.

Linux username for SEARCH ANYWHERE FRAMEWORK [logstash]:
Linux groupname for SEARCH ANYWHERE FRAMEWORK [logstash]:


Final SEARCH ANYWHERE FRAMEWORK username: logstash
Final SEARCH ANYWHERE FRAMEWORK groupname: logstash
Is this correct? [Y/n]:

Next, provide the system (Linux) user and group under which the SA Data Collector service will run. By default, this is logstash. Press Enter to accept the default values.

Then confirm the entered user and group by pressing Enter.

If you chose the rootless mode, you will see a notice with a list of preparatory commands that must be executed as a user with elevated privileges (e.g., root).

 ATTENTION!
Before installing SEARCH ANYWHERE FRAMEWORK, you must run the following commands as a user with root privileges.
Change user, group and directories as needed

/bin/dnf install zip unzip openssl curl
systemctl stop logstash
groupadd --system logstash
useradd --system --no-create-home -g logstash logstash
mkdir -p /app/logstash
mkdir -p /app/logstash/config
mkdir -p /app/data/logstash
mkdir -p /app/logs/logstash
chown -R logstash:logstash /app/logstash /app/logstash/config /app/data/logstash /app/logs/logstash /app/distr/saf_5.1/logstash

!!! Are you sure you have done everything from the list above? [y/N]:

If you've completed all the required steps for installation, type y and press Enter. If you still need to perform any actions or want to cancel the installation, press Enter.

The installer includes several checks. If any discrepancies are found, the script will automatically stop and display a message like:

You need to chown all directories, login as logstash and run the script again.

Setting Logstash Parameters​

In this step, set the basic parameters for Logstash, such as:

  • Node name — the name of the current Logstash server
  • JVM Heap Size — the amount of memory in GB allocated to the Logstash JVM. This parameter should not exceed the amount of free memory on the server. The installer will automatically suggest half of the available RAM on the serve
-- STEP 3. BASE LOGSTASH PARAMETERS

Node name [saf-node-00]:
JVM heap size in GB [4]:

Final node name: saf-node-00
Final JVM heap size in GB: 4
Is this correct? [y/n]:

Confirm the entered data and continue the installation.

Setting Certificate Parameters​

note

When creating the private key, the RSA cryptographic algorithm with a length of 2048 bits is used, and the SHA-256 algorithm is used to create the digital fingerprint. The administrator and current server certificates are created for 1095 days.

This step configures the key and certificate parameters for the current SA Data Collector server. If you previously created a self-signed CA, you must copy the private key and certificate of this CA to generate the current node's certificate automatically.

  • Do you want to use your existing certificates and keys? — You can specify your own files or use the built-in algorithm
  • Do you want to use an existing CA certificate and private key to generate certificates and keys for this node? — If you choose no, you will need to manually provide all the necessary certificates for the node to work. Otherwise, by selecting yes, the remaining certificates will be automatically generated based on the CA certificate and key
  • Path to CA certificate — the path to the root certificate. The certificate must be the same on all Search Anywhere Framework components
  • Path to CA key — the path to the private key. The key must be the same on all Search Anywhere Framework components
  • CN for node certificate — the common name for the current server
  • Country for node certificate — the country name for the current server
  • State for node certificate — the state/region name for the current server
  • Locality for node certificate — the city name for the current server
  • Organization for node certificate — the organization name for the current server
  • External IP — the external IP address of the current server, which will be used for interaction with other components. The script will automatically determine the external addresses on the current server and suggest them as the default value

Below is an example with specifying existing CA private key and certificate:

-- STEP 4. OPENDISTRO SSL/TLS SETTINGS

Do you want to use your existing certificates and keys? [Y/n]:
Do you want to use an existing CA certificate and private key to generate certificates and keys for this node? [Y/n]
Path to CA certificate [./certs/ca-cert.pem]: /app/opensearch/config/ca-cert.pem
Path to CA key [./certs/ca-key.pem]: /app/opensearch/config/ca-key.pem
CN for node certificate [saf-node-00]:
Country for node certificate [AE]:
State for node certificate [Dubai]:
Locality (city) for node certificate [Dubai]:
Organization for node certificate [SAF Test Org]:
External IP for alt_names extension [172.16.0.22]:
Final path to CA certificate: /app/opensearch/config/ca-cert.pem
Final path to CA key: /app/opensearch/config/ca-key.pem
Final CN for node certificate: saf-node-00
Final country for node certificate: AE
Final state for node certificate: Dubai
Final locality (city) for node certificate: Dubai
Final organization for node certificate: SAF Test Org
External IP for alt_names extension: 172.16.0.22
Is this correct? [y/n]:

Confirm the correctness of the input and continue the installation.

Generating Configuration Files​

No action is required from the administrator at this stage. This stage is conditionally divided into two parts:

  • Preparing temporary directories, performing checks, preparing templates
  • Generating configuration files from the specified parameters

Creating Certificates​

No action is required from the administrator at this stage. The private key and certificate of the current server are created from the specified parameters.

Unpacking the Archive and Modifying Configuration Files​

No action is required from the administrator at this stage. The archive containing Logstash is unpacked, the extracted configuration files are modified, and preparatory processes are performed before making system changes.

Starting System Modifications​

All operations up to this point are performed strictly within the installer directory and do not affect the OS. You can view the set of files ready for installation in the staging directory.

The administrator will be asked for permission to continue the installation:

!!! AT THIS POINT WE START TO MAKE CHANGES IN OPERATING SYSTEM !!!
Do you want to continue? [y/N]:

Confirm the changes to continue the installation by pressing y and then Enter.

Installation​

No action is required from the administrator at this stage. Several actions take place here:

  • Creating groups and users
  • Creating directories
  • Installing Logstash
  • Installing plugins
  • Changing file and directory ownership and setting SELinux rules
info

If the rootless installation mode has been selected, some operations will not be performed.

Preparatory Work for Starting SA Data Collector​

No action is required from the administrator at this stage. The logstash.service auto-start is enabled.

info

This step will not be performed if the installation option was selected in the rootless mode.

Completing the Installation​

If the rootless installation mode was selected, it will be. the message is displayed:

 ATTENTION!
After installing SEARCH ANYWHERE FRAMEWORK, you must run the following commands as a user with root privileges.
Change user, group and directories as needed.

cp /app/distr/saf_5.1/logstash/staging/services/logstash.service /etc/systemd/system/
chown root:root /etc/systemd/system/logstash.service
chcon -Rv -u system_u -r object_r -t bin_t /app/logstasht/bin
chcon -Rv -u system_u -r object_r -t bin_t /app/logstash/jdk/bin
chown -R logstash:logstash /app/logstash /app/logstash/config /app/data/logstash /app/logs/logstash
systemctl daemon-reload
systemctl enable logstash

The specified commands must be executed from a user with elevated privileges.

Upon completion of the installation, you should see a message showing an example of firewall configuration using the example of firewalld and ufw, as well as a message about successful installation as shown in the image below. With any other type of installation completion, an error has occurred and additional correction actions are required.

================================================================================
-- In order to connect external data source it is required to open connection ports by configuring a firewall.
-- A connection port can be opened by running the following commands (port 10050 is provided as an example and should be replaced with the required one):
-- For RHEL/CentOS with firewalld
    firewall-cmd --permanent --zone=public --add-port=10050/tcp
    firewall-cmd --reload
-- For AstraLinux/Debian with ufw
    ufw allow 10050/tcp
================================================================================

================================================================================
-- SEARCH ANYWHERE FRAMEWORK SUCCESSFULLY INSTALLED!
-- For further instructions, please refer to the documentation: https://docs.saf-systems.com/
================================================================================
Warning!

Please note that before starting the SA Data Collector, you need to configure the pipeline.
Without it, the service will continuously restart.

Upon successful completion of the installer, the node is deployed but requires configuration files (pipelines) for data collection. The logstash.service will automatically start after the OS reboots. Note that you need to configure the firewall for the ports on which data will be received.

To verify functionality after adding the configuration and enabling the logstash.service, you can enter the following in the command line:

curl -X GET http://localhost:9600?pretty

Example output:

{
  "host": "srv-test",
  "version": "8.13.4",
  "http_address": "127.0.0.1:9600",
  "id": "0d6b3458-147c-4b81-b3bb-210ddafdc440",
  "name": "saf-logstash",
  "ephemeral_id": "bae83d99-e076-42f3-9fc8-546e06ba1591",
  "status": "green",
  "snapshot": false,
  "pipeline": {
    "workers": 4,
    "batch_size": 125,
    "batch_delay": 50
  },
  "build_date": "2023-07-18T22:17:29+00:00",
  "build_sha": "375f47bd852411a88d350503847f399e8f652104",
  "build_snapshot": false
}