SA Data Collector Upgrade

Pre-Upgrade Preparation

Legend:

• SAF_INSTALLER - directory where Search Anywhere Framework installation package is extracted
• LS_HOME - Logstash installation directory (typically /app/logstash)
• LS_CONFIG - Logstash configuration directory (typically /app/logstash/config)
• LS_DATA - Logstash data directory (typically /app/data/logstash)

SA Data Collector is based on Logstash and may be referred to as such in this document.

Important Note!

If uninterrupted data collection is critical, migrate current pipeline configurations to another Logstash instance and redirect all data collection to it.

In high-load installations, SA Data Collector might not stop via the update script. Manually stop it before upgrading:

sudo systemctl stop logstash

Create backups of these directories:

• LS_CONFIG
• LS_DATA

mkdir -p /app/backup/logstash/
cp -r ${LS_CONFIG} /app/backup/logstash/config_$(date +%Y%m%d)/
cp -r ${LS_DATA} /app/backup/logstash/data_$(date +%Y%m%d)/

Upgrade Process

The script requires these pre-installed packages:

• curl
• zip
• unzip

The script automates the update process and is located at $SAF_INSTALLER/logstash/setup.sh. To update, either pass the update action when running the script, or run the script without arguments and select the appropriate option from the interactive menu:

The update script supports the following runtime parameters:

Select action [default: update]:
  1) install
  2) update
  3) uninstall

Action [update]:

The script supports the following runtime parameters:

• <action> - action to perform: install, update, or uninstall
• -c, --config <path_to_config_file_yaml> - specify upgrade configuration file
• -h, --help - display command help

Example configuration file: $SAF_INSTALLER/logstash/example_config_logstash.yaml. The YAML file format matches the configuration file used during installation.

The script creates backup copies of the systemd service file, logstash.yml, and the configuration directory in the temporary $SAF_INSTALLER/logstash/staging/ directory. The script does not modify the data or logs directories.

warning

The staging directory contents will be cleared if the script is rerun. If the script is run again, the staging directory is cleared and the previous backup is lost. If you need the backup, copy it from this directory before rerunning the script.

To perform the upgrade, execute:

$SAF_INSTALLER/logstash/setup.sh update

The script automatically detects these key server paths:

• Logstash Home Directory - Logstash installation path (typically /app/logstash)
• Logstash Conf Directory - Configuration files location (typically /app/logstash/config/)
• Logstash Data Directory - Data storage location (typically /app/data/logstash)
• Logstash Logs Directory - Log files location (typically /app/logs/logstash)

================================================================================
SEARCH ANYWHERE FRAMEWORK INSTALL SCRIPT - LOGSTASH
================================================================================

Current working directory: /app/distr/saf_5.2/logstash

Current name of install's archive: logstash-oss-8.13.4-linux-x86_64.tar.gz

Current version of Logstash: 8.13.4

================================================================================
-- STEP 1. INSTALLATION DIRECTORIES

logstash.service file found. Will get necessary paths from there

Final Logstash home directory: /app/logstash
Final Logstash conf directory: /app/logstash/config
Final Logstash data directory: /app/data/logstash
Final Logstash logs directory: /app/logs/logstash
Is this correct? [y/n]:

After entering the directory paths, confirm your entries by pressing y, or manually re-enter them by pressing n.

The next step allows disabling all operations requiring privilege escalation to root during the update.

Important Note!

A complete update cannot be performed without root privileges. You must:

1. Execute some operations as the root user
2. Run the installer as the SA Data Collector runtime user
3. Perform final operations again as root after script completion

The installer will display the required operation list.

Failure to execute these commands before updating may result in data loss.

To launch the installer as the logstash user:

sudo -u logstash $SAF_INSTALLER/logstash/setup.sh update

The first prompt will ask whether to disable commands requiring root privilege escalation.

Next, you'll need to specify:

• The user
• The group
  under which SA Data Collector operates.

Finally, the system will display a checklist of operations that must be completed before executing the update script.

-- STEP 2. DISABLE OPERATIONS WHERE ROOT RIGHTS ARE REQUIRED

Want to install SEARCH ANYWHERE FRAMEWORK without root privileges? [y/N]: y

Final you set the non-root flag to: y

Linux username for SEARCH ANYWHERE FRAMEWORK [logstash]:
Linux groupname for SEARCH ANYWHERE FRAMEWORK [logstash]:

Final SEARCH ANYWHERE FRAMEWORK username: logstash
Final SEARCH ANYWHERE FRAMEWORK groupname: logstash
Is this correct? [Y/n]:

ATTENTION!
Before installing SEARCH ANYWHERE FRAMEWORK, you must run the following commands as a user with root privileges.
Change user, group and directories as needed

systemctl stop logstash
chown -R logstash:logstash /app/logstash /app/logstash/config /app/data/logstash /app/logs/logstash /app/distr/saf_5.2/logstash

!!! Are you sure you have done everything from the list above? [y/N]:

At the final step, you must confirm that all root user commands were executed.
Enter y to proceed - otherwise, the update will be aborted.

The system will then perform pre-upgrade preparations. Before applying updates:

1. You'll receive a final confirmation prompt
2. No system-impacting changes will occur before this confirmation

!!! AT THIS POINT WE START TO MAKE CHANGES IN OPERATING SYSTEM !!!
Do you want to continue? [y/N]: 

If you selected the update option without commands requiring root privileges, the system will display an information block listing all commands that must be executed as the root user.

================================================================================
ATTENTION!
After installing SEARCH ANYWHERE FRAMEWORK, you must run the following commands as a user with root privileges.
Change user, group and directories as needed.

cp /app/distr/saf_5.2/logstash/staging/services/logstash.service /etc/systemd/system/
chown root:root /etc/systemd/system/logstash.service
chcon -Rv -u system_u -r object_r -t bin_t /app/logstash/bin
chcon -Rv -u system_u -r object_r -t bin_t /app/logstash/jdk/bin
chown -R logstash:logstash /app/logstash /app/logstash/config /app/data/logstash /app/logs/logstash
systemctl daemon-reload
systemctl enable logstash
systemctl start logstash
systemctl status logstash

================================================================================
-- SEARCH ANYWHERE FRAMEWORK SUCCESSFULLY UPDATED!
-- For further instructions, please refer to the documentation: https://docs.saf-systems.com/
================================================================================

And this too: Upon successful completion of the update script, the corresponding text SEARCH ANYWHERE FRAMEWORK SUCCESSFULLY UPDATED will be displayed.

Before running Logstash, it's recommended to check all pipeline configurations using the following command:

sudo -u logstash $LS_HOME/bin/logstash --config.test_and_exit -f <pipeline-config>