Version: 6.2

Dynamically Changing Log Levels for Search Anywhere Framework Components

Important!

Changing the log level from INFO/WARN to DEBUG/TRACE can increase log volume and disk space usage. After testing, restore the previous or default log level.

General Information

This article explains how to change log levels for Search Anywhere Framework components without restarting them. For components that do not support dynamic log level changes, an alternative method is provided.

Summary Table of Dynamic Log Level Change Support

Component	Supported	Method
SA Data Storage/Master Node	Yes	API
SA Web	No	Configuration change and restart
SA Data Collector	Yes	API
PostgreSQL	Yes	SQL
Apache Kafka	Yes	`.sh` script
ClickHouse	Yes	Change the `.xml` file

List of Variables

${OSD_HOME} - SA Web home directory
${LS_ADDRESS} - SA Data Collector host address; commands use the default API port (9600)
${KAFKA_HOME} - Kafka home directory
${KAFKA_ADDRESS} - Kafka broker address; commands use the default port (9092)
${KAFKA_NODE_ID} - Kafka broker ID
${CLICK_CONFIG_HOME} - directory containing ClickHouse configuration files

SA Data Storage/Master Node

Prerequisites

all SA Data Storage logging parameters start with logger.
queries are run in the Dev Tools console (Main Menu - Settings - Dev Tools)
settings in the persistent block survive a full cluster restart, while settings in the transient block revert to the values from log4j2.properties; restarting a single node while the cluster is running does not affect transient settings. To roll back, use the same block (persistent or transient) in which the setting was configured

Check the Current Level

GET _cluster/settings?flat_settings

Important Note!

The query shows only explicitly configured overrides. If the required component setting is absent from the output, check log4j2.properties.

Change the Level

The query sets the log level using the Job Scheduler logger as an example:

PUT _cluster/settings
{
  "transient": {
    "logger.com.sa.jobscheduler": "DEBUG"
  }
}

The setting takes effect automatically after the query runs.

Verify the Result

Example of a DEBUG-level log for Job Scheduler:

[2026-09-18T12:57:00,397][INFO ][JobRunner      ] [sm-node-01] [JOB][][6SqF0Z8BadR_g8ZYhv9S] Job was successfully finished
[2026-09-18T12:57:00,397][DEBUG][InternalLoggingManager] [sm-node-01] [LOGGING_MANAGER][][6SqF0Z8BadR_g8ZYhv9S] got info log
[2026-09-18T12:57:00,397][DEBUG][InternalLoggingManager] [sm-node-01] [LOGGING_MANAGER][][6SqF0Z8BadR_g8ZYhv9S] configuring source json
[2026-09-18T12:57:00,397][DEBUG][InternalLoggingManager] [sm-node-01] [LOGGING_MANAGER][][6SqF0Z8BadR_g8ZYhv9S] configuring index request

Roll Back

Specify the previous log level (WARN in this example), or remove the setting by assigning null:

PUT _cluster/settings
{
  "transient": {
    "logger.com.sa.jobscheduler": "WARN"
  }
}

To remove the setting:

PUT _cluster/settings
{
  "transient": {
    "logger.com.sa.jobscheduler": null
  }
}

SA Web

SA Web does not support dynamic log level changes: edit its configuration and restart the service.

Prerequisites

Current logging settings are located in the ${OSD_HOME}/config/opensearch_dashboards.yml file. Logging is controlled by the following parameters:

logging.verbose - verbose logging
logging.silent - logging disabled
logging.quiet - error messages only

The log output destination is set by the logging.dest parameter (default - stdout).

Check the Current Level

Open ${OSD_HOME}/config/opensearch_dashboards.yml and check the current values of logging.verbose, logging.silent, and logging.quiet.

Change the Level

Change the parameter values in ${OSD_HOME}/config/opensearch_dashboards.yml and restart the service.

Verify the Result

Example of a log with logging.verbose: true:

{"type":"log","@timestamp":"2026-09-21T12:20:58Z","tags":["debug","security-service"],"pid":3575111,"message":"Starting plugin"}
{"type":"log","@timestamp":"2026-09-21T12:21:01Z","tags":["debug","metrics"],"pid":3575111,"message":"Refreshing metrics"}
{"type":"ops","@timestamp":"2026-09-21T12:21:03Z","tags":[],"pid":3575111,"os":{"load":[6.15,5.35,5.65],"mem":{"total":16770101248,"free":764551168},"uptime":3571085.67},"proc":{"uptime":18.813830455,"mem":{"rss":413589504,"heapTotal":313999360,"heapUsed":252867752,"external":10108280,"arrayBuffers":6361691},"delay":9.309514045715332},"load":{"requests":{},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 241.2MB uptime: 0:00:19 load: [6.15 5.35 5.65] delay: 9.310"}
{"type":"log","@timestamp":"2026-09-21T12:21:06Z","tags":["debug","metrics"],"pid":3575111,"message":"Refreshing metrics"}

Roll Back

Restore the original parameter values in ${OSD_HOME}/config/opensearch_dashboards.yml and restart the service.

SA Data Collector

Prerequisites

Important Note!

After updating the log level through the API, it is not retained after the service restarts and reverts to the value from log4j2.properties.

Important Note!

The DEBUG and TRACE levels can log the contents of processed events, including source data. Before enabling them on a production cluster, ensure that personal data, credentials, or other sensitive information will not be written to the logs.

The commands below use http. If SSL is enabled for the SA Data Collector API (the api.ssl.enabled: true parameter in logstash.yml), change the scheme to https. If basic authentication is configured (api.auth.type: basic), add -u <username>:<password> to curl.

Check the Current Level

curl -XGET "http://${LS_ADDRESS}:9600/_node/logging?pretty"

The following example shows part of the loggers section; other loggers and node metadata are omitted:

Example output

{
  "loggers" : {
    "logstash.agent" : "INFO",
    "logstash.api.service" : "INFO",
    "logstash.codecs.json" : "INFO",
    "logstash.codecs.jsonlines" : "INFO",
    "logstash.codecs.plain" : "INFO",
    "logstash.codecs.rubydebug" : "INFO",
    "logstash.config.source.local.configpathloader" : "INFO",
    "logstash.config.sourceloader" : "INFO",
    "logstash.filters.clone" : "INFO",
    "logstash.filters.csv" : "INFO",
    "logstash.filters.date" : "INFO",
    "logstash.filters.dissect" : "INFO",
    "logstash.filters.drop" : "INFO",
    "logstash.filters.grok" : "INFO",
    "logstash.filters.json" : "INFO",
    "logstash.filters.kv" : "INFO",
    "logstash.filters.mutate" : "INFO",
    "logstash.filters.ruby" : "INFO",
    "logstash.filters.smcompliance" : "INFO",
    "logstash.filters.split" : "INFO",
    "logstash.filters.translate" : "INFO",
    "logstash.filters.urldecode" : "INFO",
    "logstash.filters.xml" : "INFO",
    "logstash.inputs.beats" : "INFO",
    "logstash.inputs.exec" : "INFO",
    "logstash.inputs.file" : "INFO",
    "logstash.inputs.jdbc" : "INFO",
    "logstash.inputs.snmp" : "INFO",
    "logstash.inputs.syslog" : "INFO",
    "logstash.instrument.periodicpoller.cgroup" : "INFO",
    "logstash.instrument.periodicpoller.deadletterqueue" : "INFO",
    "logstash.instrument.periodicpoller.flowrate" : "INFO",
    "logstash.instrument.periodicpoller.jvm" : "INFO",
    "logstash.instrument.periodicpoller.os" : "INFO",
    "logstash.instrument.periodicpoller.persistentqueue" : "INFO",
    "logstash.javapipeline" : "INFO",
    "logstash.modules.scaffold" : "INFO",
    "logstash.outputs.file" : "INFO",
    "logstash.outputs.opensearch" : "INFO",
    "logstash.pluginmetadata" : "INFO",
    "logstash.plugins.registry" : "INFO",
    "logstash.runner" : "INFO",
    "logstash.snmpmib" : "INFO",
    "logstash.snmpmib.tree" : "INFO",
    "logstash.util.decorators" : "INFO",
    "slowlog.logstash.codecs.json" : "WARN",
    "slowlog.logstash.codecs.jsonlines" : "WARN",
    "slowlog.logstash.codecs.plain" : "WARN",
    "slowlog.logstash.codecs.rubydebug" : "WARN",
    "slowlog.logstash.filters.clone" : "WARN",
    "slowlog.logstash.filters.csv" : "WARN",
    "slowlog.logstash.filters.date" : "WARN",
    "slowlog.logstash.filters.dissect" : "WARN",
    "slowlog.logstash.filters.drop" : "WARN",
    "slowlog.logstash.filters.grok" : "WARN",
    "slowlog.logstash.filters.json" : "WARN",
    "slowlog.logstash.filters.kv" : "WARN",
    "slowlog.logstash.filters.mutate" : "WARN",
    "slowlog.logstash.filters.ruby" : "WARN",
    "slowlog.logstash.filters.smcompliance" : "WARN",
    "slowlog.logstash.filters.split" : "WARN",
    "slowlog.logstash.filters.translate" : "WARN",
    "slowlog.logstash.filters.urldecode" : "WARN",
    "slowlog.logstash.filters.xml" : "WARN",
    "slowlog.logstash.inputs.beats" : "WARN",
    "slowlog.logstash.inputs.exec" : "WARN",
    "slowlog.logstash.inputs.file" : "WARN",
    "slowlog.logstash.inputs.jdbc" : "WARN",
    "slowlog.logstash.inputs.snmp" : "WARN",
    "slowlog.logstash.inputs.syslog" : "WARN",
    "slowlog.logstash.outputs.file" : "WARN",
    "slowlog.logstash.outputs.opensearch" : "WARN"
  }
}

Change the Level

In the request body, select the required parameter from the list, prefix its name with logger., and specify the log level. The following request enables DEBUG for all integrations that use the opensearch output plugin:

curl -XPUT "http://${LS_ADDRESS}:9600/_node/logging?pretty" -H 'Content-Type: application/json' -d'
{
    "logger.logstash.outputs.opensearch" : "DEBUG"
}
'

Verify the Result

After the change, entries like the following appear in the integration logs:

[2026-09-16T14:19:37,673][WARN ][logstash.outputs.opensearch] Restored connection to OpenSearch instance {:url=>"https://logstash:xxxxxx@127.0.0.1:9200/"}
[2026-09-18T10:35:01,081][DEBUG][logstash.outputs.opensearch] Sending final bulk request for batch. {:action_count=>1, :payload_size=>3661, :content_length=>3661, :batch_offset=>0}
[2026-09-18T10:35:54,757][DEBUG][logstash.outputs.opensearch] Sending final bulk request for batch. {:action_count=>1, :payload_size=>2659, :content_length=>2659, :batch_offset=>0}
[2026-09-18T10:36:00,960][DEBUG][logstash.outputs.opensearch] Sending final bulk request for batch. {:action_count=>1, :payload_size=>2659, :content_length=>2659, :batch_offset=>0}
[2026-09-18T10:36:54,756][DEBUG][logstash.outputs.opensearch] Sending final bulk request for batch. {:action_count=>1, :payload_size=>1670, :content_length=>1670, :batch_offset=>0}
[2026-09-18T10:37:00,954][DEBUG][logstash.outputs.opensearch] Sending final bulk request for batch. {:action_count=>1, :payload_size=>1670, :content_length=>1670, :batch_offset=>0}

Roll Back

Reset the log levels to the values from log4j2.properties:

curl -XPUT "http://${LS_ADDRESS}:9600/_node/logging/reset?pretty"

PostgreSQL

Prerequisites

Important Note!

The changed log level is written to postgresql.auto.conf and persists after a restart. Restore the original value after testing.

Important Note!

PostgreSQL provides detail levels from debug1 through debug5; debug5 is the most detailed.

Valid values for the log_min_messages parameter are: debug5, debug4, debug3, debug2, debug1, info, notice, warning, error, log, fatal, panic. The default is warning.

Changing the level requires superuser privileges.

Check the Current Level

Run the query in the database:

SHOW log_min_messages;

Change the Level

ALTER SYSTEM SET log_min_messages = 'debug1';

After changing the setting, reload the configuration:

SELECT pg_reload_conf();

Verify the Result

Example of DEBUG-level messages:

2026-09-18 11:38:25.382 UTC [2446670] DEBUG:  autovacuum: processing database "template1"
2026-09-18 11:38:45.379 UTC [2446804] DEBUG:  autovacuum: processing database "postgres"
2026-09-18 11:39:05.405 UTC [2447241] DEBUG:  autovacuum: processing database "Inventory"

Roll Back

ALTER SYSTEM RESET log_min_messages;

After rolling back, reload the configuration (SELECT pg_reload_conf();).

Kafka

Prerequisites

Important Note!

The log level does not persist after a broker restart. Change the setting on each broker whose logs you need to inspect.

Important Note!

At the DEBUG level, kafka.request.logger records the contents of processed requests, including event data. Before enabling it on a production broker, ensure that sensitive data will not be written to the log.

Important Note!

If a secure connection (SSL/SASL) is used to connect to the broker, add --command-config <client_config_file> to all kafka-configs.sh commands, where <client_config_file> is the path to the client connection settings file.

Apache Kafka version 2.4+ supports dynamic log level changes. The request logger (the kafka.request.logger component, WARN by default in the standard configuration) is used as an example.

Check the Current Level

${KAFKA_HOME}/bin/kafka-configs.sh --bootstrap-server ${KAFKA_ADDRESS}:9092 --entity-type broker-loggers --entity-name ${KAFKA_NODE_ID} --describe

Change the Level

${KAFKA_HOME}/bin/kafka-configs.sh --bootstrap-server ${KAFKA_ADDRESS}:9092 --entity-type broker-loggers --entity-name ${KAFKA_NODE_ID} --alter --add-config kafka.request.logger=DEBUG

Verify the Result

kafka.request.logger entries are saved in the separate kafka-request.log file.

Example of a DEBUG-level log:

[2026-09-21 12:26:07,626] DEBUG Completed request:{"isForwarded":false,"requestHeader":{"requestApiKey":0,"requestApiVersion":11,"correlationId":6,"clientId":"console-producer","requestApiKeyName":"PRODUCE"},"request":{"transactionalId":null,"acks":-1,"timeoutMs":1500,"topicData":[{"name":"test-logs","partitionData":[{"index":0,"recordsSizeInBytes":72}]}]},"response":{"responses":[{"name":"test-logs","partitionResponses":[{"index":0,"errorCode":0,"baseOffset":3,"logAppendTimeMs":-1,"logStartOffset":0,"recordErrors":[],"errorMessage":null}]}],"throttleTimeMs":0},"connection":"127.0.0.1:9092-127.0.0.1:47299-13","totalTimeMs":1.142,"requestQueueTimeMs":0.144,"localTimeMs":0.907,"remoteTimeMs":0.0,"throttleTimeMs":0,"responseQueueTimeMs":0.034,"sendTimeMs":0.055,"securityProtocol":"PLAINTEXT","principal":"User:ANONYMOUS","listener":"PLAINTEXT","clientInformation":{"softwareName":"apache-kafka-java","softwareVersion":"3.9.0"}} (kafka.request.logger)
[2026-09-21 12:26:08,805] DEBUG Completed request:{"isForwarded":false,"requestHeader":{"requestApiKey":0,"requestApiVersion":11,"correlationId":7,"clientId":"console-producer","requestApiKeyName":"PRODUCE"},"request":{"transactionalId":null,"acks":-1,"timeoutMs":1500,"topicData":[{"name":"test-logs","partitionData":[{"index":0,"recordsSizeInBytes":73}]}]},"response":{"responses":[{"name":"test-logs","partitionResponses":[{"index":0,"errorCode":0,"baseOffset":4,"logAppendTimeMs":-1,"logStartOffset":0,"recordErrors":[],"errorMessage":null}]}],"throttleTimeMs":0},"connection":"127.0.0.1:9092-127.0.0.1:47299-13","totalTimeMs":2.231,"requestQueueTimeMs":0.55,"localTimeMs":1.406,"remoteTimeMs":0.0,"throttleTimeMs":0,"responseQueueTimeMs":0.192,"sendTimeMs":0.082,"securityProtocol":"PLAINTEXT","principal":"User:ANONYMOUS","listener":"PLAINTEXT","clientInformation":{"softwareName":"apache-kafka-java","softwareVersion":"3.9.0"}} (kafka.request.logger)

Roll Back

Important Note!

The command below resets the log level to the parent logger's level, not to the previous value.

${KAFKA_HOME}/bin/kafka-configs.sh --bootstrap-server ${KAFKA_ADDRESS}:9092 --entity-type broker-loggers --entity-name ${KAFKA_NODE_ID} --alter --delete-config kafka.request.logger

To explicitly set WARN instead of inheriting the parent logger's level:

${KAFKA_HOME}/bin/kafka-configs.sh --bootstrap-server ${KAFKA_ADDRESS}:9092 --entity-type broker-loggers --entity-name ${KAFKA_NODE_ID} --alter --add-config kafka.request.logger=WARN

ClickHouse

Prerequisites

Important Note!

The changed log level is stored in logger.xml and is not reset on restart until the file's original contents are restored.

Supported levels:

none (disables logging)
fatal
critical
error
warning
notice
information
debug
trace

Check the Current Level

SELECT name, value FROM system.server_settings WHERE name LIKE 'logger.%' FORMAT Pretty;

The query returns a table similar to the following:

	name	value
1.	logger.level	debug
2.	logger.log	/var/log/clickhouse-server/clickhouse-server.log
3.	logger.errorlog	/var/log/clickhouse-server/clickhouse-server.err.log
4.	logger.size	1000M
5.	logger.rotation	100M
6.	logger.count	10
7.	logger.stream_compress	0
8.	logger.console	0
9.	logger.console_log_level	trace
10.	logger.formatting.type	json
11.	logger.use_syslog	0
12.	logger.syslog_level	trace
13.	logger.async	1
14.	logger.async_queye_max_size	65536
15.	logger.startup_level
16.	logger.shutdown_level

note

This example was obtained with ClickHouse version 26.10.1.326. The parameter name in row 14 is preserved as it appears in this version. Due to a typo in the code of some versions, the parameter may be named async_queye_max_size instead of the documented async_queue_max_size.

Change the Level

Edit ${CLICK_CONFIG_HOME}/config.d/logger.xml: add <level>debug</level> inside the <logger> block, or change the existing level parameter. The configuration reloads automatically. If it does not, run SYSTEM RELOAD CONFIG.

Verify the Result

Example of a log at the debug level:

09.21 15:01:35.931617 [ 12745 ] {dcb47c51-fc04-4aee-8e39-0144f77af4fe::202609_1_934_375} <Debug> MergeTreeSequentialSource: Reading 3 marks from part 202609_1_933_374, total 13838 rows starting from the beginning of the part
09.21 15:01:35.931718 [ 12745 ] {dcb47c51-fc04-4aee-8e39-0144f77af4fe::202609_1_934_375} <Debug> MergeTreeSequentialSource: Reading 2 marks from part 202609_934_934_0, total 8 rows starting from the beginning of the part
09.21 15:01:35.945633 [ 12738 ] {dcb47c51-fc04-4aee-8e39-0144f77af4fe::202609_1_934_375} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 13846 rows, containing 34 columns (34 merged, 0 gathered) in 0.014 sec., 955011.944 rows/sec., 411.31 MiB/sec.
09.21 15:01:35.946683 [ 12738 ] {dcb47c51-fc04-4aee-8e39-0144f77af4fe::202609_1_934_375} <Debug> system.part_log (dcb47c51-fc04-4aee-8e39-0144f77af4fe) (DataPartWriter): Spent 0 ms calculating 3 skip indices for the part 202609_1_934_375: [event_time_index: 0 ms, event_time_microseconds_index: 0 ms, query_id_index: 0 ms]

Roll Back

Remove the <level>...</level> line, or change the level to the default value (trace).

General Information​

Summary Table of Dynamic Log Level Change Support​

List of Variables​

SA Data Storage/Master Node​

Prerequisites​

Check the Current Level​

Change the Level​

Verify the Result​

Roll Back​

SA Web​

Prerequisites​

Check the Current Level​

Change the Level​

Verify the Result​

Roll Back​

SA Data Collector​

Prerequisites​

Check the Current Level​

Change the Level​

Verify the Result​

Roll Back​

PostgreSQL​

Prerequisites​

Check the Current Level​

Change the Level​

Verify the Result​

Roll Back​

Kafka​

Prerequisites​

Check the Current Level​

Change the Level​

Verify the Result​

Roll Back​

ClickHouse​

Prerequisites​

Check the Current Level​

Change the Level​

Verify the Result​

Roll Back​

General Information

Summary Table of Dynamic Log Level Change Support

List of Variables

SA Data Storage/Master Node

Prerequisites

Check the Current Level

Change the Level

Verify the Result

Roll Back

SA Web

Prerequisites

Check the Current Level

Change the Level

Verify the Result

Roll Back

SA Data Collector

Prerequisites

Check the Current Level

Change the Level

Verify the Result

Roll Back

PostgreSQL

Prerequisites

Check the Current Level

Change the Level

Verify the Result

Roll Back

Kafka

Prerequisites

Check the Current Level

Change the Level

Verify the Result

Roll Back

ClickHouse

Prerequisites

Check the Current Level

Change the Level

Verify the Result

Roll Back