Operating System Settings
All settings and recommendations are written and tested for use in the recommended operating systems.
For greater stability and speed of the Search Anywhere Framework, it is recommended to make preliminary settings in the operating system. The installer has built-in system settings and no administrator action is required except for optional parameters. This section provides all the modification parameters used for informational purposes.
Setting process parameters
All used parameters require rereading the systemd parameters and restarting the modified process. To do this, you need to execute two commands in the command line as the root user:
$ systemctl daemon-reload
$ systemctl restart opensearch.service
In this example, we will restart the opensearch.service process.
All parameters presented in this section are added to the SERVICE block of the configuration file for the systemd process.
Open File Descriptors
By default, in modern systems, particularly in RHEL, the number of open files for a single process is limited to 1024. For stable operation, it is necessary to increase this number to 65536:
LimitNOFILE=65535
Sometimes ulimits is specified, which overrides the parameter above. In this case, you need to add the following lines for the corresponding users (for example, the opensearch user) in the /etc/security/limits.conf file:
opensearch soft nofile 65535
opensearch hard nofile 65535
Number of Threads
During operation, OpenSearch uses thread pools for different types of operations. It is important to be able to create new threads when necessary, so the number of threads should be set to no less than 4096.
LimitNPROC=4096
Virtual Memory Address Space
The virtual address space sets the range allocated by the operating system for the process. For stable operation, services should not have any limitations. This is set by the parameter:
LimitAS=infinity
File Size
The operating system does not allow creating or copying files larger than a specified size. This is usually used if you need to limit the disk space occupied by a process. For our processes, this should be unlimited:
LimitFSIZE=infinity
Max Locked Memory
The maximum amount of RAM used by the process that can be locked before being saved to the swap file. This needs to be set to an unlimited value:
LimitMEMLOCK=infinity
Java Settings
We make minor changes to the Java settings. When using the installer, these settings are added automatically.
The amount of memory allocated to the JVM Heap has a strong impact on system performance.
To avoid dynamic heap resizing, which is a resource-intensive runtime operation, ensure that the -Xms and -Xmx parameters have the same values in the /app/opensearch/config/jvm.options file.
It is recommended to allocate 50% to 75% of the server's available RAM, but no more than 32GB to preserve pointer optimization.
To ensure that optimized pointers are used, when starting a node, check the main cluster log for the following message with the status true:
... heap size [31gb], compressed ordinary object pointers [true]
Parameter -Xms
Sets the minimum amount of memory allocated for the Java application. The value is specified in megabytes or gigabytes. For a server with 16GB of RAM, the parameter configuration will be as follows:
-Xms8g
Parameter -Xmx
Sets the maximum amount of RAM for the Java application. The value is specified in megabytes or gigabytes.
For a server with 16GB of RAM, the parameter configuration will be as follows:
-Xmx8g
Global System Settings
Some parameters cannot be applied to just one process and are applied to the entire system. In this section, all parameters are set for the operating system kernel, and the sysctl utility is used to change these parameters.
These parameters can be changed until the next reboot by simply changing the specific parameter:
sysctl -w vm.swappiness=1
To save changes after a reboot, you need to modify the /etc/sysctl.conf configuration file. You can use the sysctl.d directory to create separate configuration files for easier administration.
After changing the sysctl configuration file, to apply the changes, you need to either reboot the server or execute the following command in the command line as the root user:
sysctl -p
Disabling SWAP
Most operating systems try to store as much cached data as possible in RAM and save unused data to disk in the so-called SWAP. This process can negatively affect the speed of the Search Anywhere Framework.
We recommend not disabling SWAP completely, but only reducing the swappiness value to a minimum. Swappiness is responsible for the percentage of remaining free RAM, after which the OS will start writing operational data to SWAP. This will allow the use of SWAP under critical and non-standard server loads and will...
To change the parameter, you need to create the file /etc/sysctl.d/00-opensearch.conf and add the line:
vm.swappiness = 1
If you need to disable SWAP completely, then you need to do the following. Execute the command to disable swap usage until the next reboot:
swapoff -a
Then, in the /etc/fstab file, comment out all swap partitions:
#/dev/mapper/ol-swap none swap defaults 0 0
Increasing the number of indices for the process
OpenSearch uses the mmapfs file system type by default to store index segments. The operating system has a very low limit on this parameter, which can lead to memory limit errors.
To change the parameter, you need to create the file /etc/sysctl.d/00-opensearch.conf and add the line:
vm.max_map_count=262144
Reducing the number of TCP retransmissions
In OpenSearch and Logstash, TCP connections are opened to interact with various systems. When a connection is broken, packets are expected to be retransmitted. In Linux, this number is 15, and the wait for a response increases exponentially, which can lead to a long delay (about 900 seconds) in determining that the device is no longer available.
We recommend reducing this value to 5 attempts, which will be approximately equal to a 6-second delay in determining device availability.
To change the parameter, create the file /etc/sysctl.d/00-opensearch.conf and add the line:
net.ipv4.tcp_retries2=5
Access Rights
Firewall settings are not configured by the installer. At the end of the installation, the option "available to all" will be offered.
Configuring Access Rights
The opensearch user and opensearch group are used for OpenSearch and OpenSearch Dashboards. The logstash user and logstash group are used for Logstash.
Configuring SELinux
Changes to the settings will not be made when SELinux is in disabled mode. In other operating modes, changes will be made for the following directories:
- bin: system_u:object_r:bin_t:s0;
- run: system_u:object_r:var_run_t:s0.
Optional Parameters
The following parameters are not automatically modified by the installer. Changing them is usually not required and should be done with caution and after performance testing.
Disabling Transparent Huge Pages
Transparent Huge Pages is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on computers with large amounts of memory by using larger memory pages. In most cases, disabling is not required, especially if only OpenSearch is installed on the server, because we use the Xms and Xmx parameters in the Java settings and practically do not use swap.
According to private research (issue 26551), it is not recommended to set Transparent Huge Pages to the madvise value, which has been observed to degrade OpenSearch performance.
Original documentation on the Red Hat page: How to disable THP.
To disable the Transparent Huge Pages mechanism, you need to add the following line to the /etc/default/grub file in the GRUB_CMDLINE_LINUX parameter:
transparent_hugepage=never
Before applying the configuration, you need to save a copy of the /boot/grub2/grub.cfg file.
To apply the configuration, you need to execute the following command, for BIOS-based systems:
grub2-mkconfig -o /boot/grub2/grub.cfg
For UEFI-based systems:
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
After these steps, you should reboot the server. Create a tuned profile to override all configurations affecting THP.
Create the folder and profile file:
mkdir /etc/tuned/myprofile-nothp
touch /etc/tuned/myprofile-nothp/tuned.conf
Add the following text to the file:
[main]
include= throughput-performance
[vm]
transparent_hugepages=never
Then make the file executable and apply it:
chmod +x /etc/tuned/myprofile-nothp/tuned.conf
tuned-adm profile myprofile-nothp
A system reboot is not required.