Logstash Configuration Management

Description

The Logstash Configuration Management role does not restart the Logstash service, modify parameters in the jvm.options file, or edit the heap size.

This role configures the following files:

logstash.yml
pipelines.yml
Pipeline configuration files

When changes are made to configuration files, backups are created. The role also searches for existing backups and deletes backups older than one day.

When a pipeline is disabled on a host, the configuration file is renamed with the extension .bkp and is not deleted.

Role Parameters

Default values are based on SAF recommendations.

var_logstash_pipelines_list

A YAML object describing the Logstash pipeline parameters. Typically used for host groups (affects merging priorities and order). Supports merging with parameters starting with var_logstash_pipelines_list_, for example: var_logstash_pipelines_list_syslog var_logstash_pipelines_list_win_log

host_var_logstash_pipelines_list

A YAML object describing the Logstash pipeline parameters. Typically used for individual host parameters (affects merging priorities and order). Supports merging with parameters starting with host_var_logstash_pipelines_list_, for example: host_var_logstash_pipelines_list_syslog host_var_logstash_pipelines_list_win_log

var_logstash_files

A YAML object with the main Logstash parameters:

Variable	Default Value	Description
`var_logstash_files.path_conf`	Logstash configuration directory, by default	`/app/logstash/config`
`var_logstash_files.path_log`	Logstash log directory, by default	`/app/logs/logstash/logstash-plain.log`
`var_logstash_files.dir_subcfg`	Name of the pipeline configuration files directory, usually	`conf.d`
`var_logstash_files.owner`	File owner, by default	`"logstash"`
`var_logstash_files.group`	File group owner, by default	`"logstash"`
`var_logstash_files.mode`	Permissions for created files, by default	`"0644"`
`var_logstash_files.folder_mode`	Permissions for created directories, by default	`"0755"`

`logstash.yml` File Parameters

The following parameters (YAML objects) relate to global Logstash parameters and are written to the logstash.yml file. They are divided into 5 objects solely for grouping parameters by "common features" and corresponding comments in the logstash.yml file.

var_logstash_node_settings

A YAML object with parameters related to the current node, typically includes:

Variable	Default Value	Description
`node.name`	Node name, default value	`smosed-logstash`
`config.reload.automatic`	Whether to automatically reload the Logstash pipeline configuration	`false`
`config.reload.interval`	Time interval to check for configuration changes for automatic application	`3s`

var_logstash_path_settings

A YAML object with parameters related to the current node and directory specifications, typically includes:

Variable	Default Value	Description
`path.data`	Data directory location	`/app/data/logstash`
`path.logs`	Log directory location	`/app/logs/logstash`

var_logstash_pipeline_settings

A YAML object with parameters related to global pipeline settings, typically includes:

Variable	Default Value	Description
`pipeline.ordered`	Pipeline event order management	`auto`
`pipeline.separate_logs`	Save pipeline logs to separate files	`true`

var_logstash_queue_settings

A YAML object with parameters related to queue operations, typically includes:

Variable	Default Value	Description
`queue.type`	`memory`	Type of buffer for storing events
`queue.page_capacity`	`64mb`	Page size for file-based event buffer
`queue.max_events`	`0`	Number of unprocessed events in the queue to enable the file-based buffer
`queue.max_bytes`	`1024mb`	Maximum size of the file-based buffer in bytes
`queue.checkpoint.acks`	`1024`	Maximum number of acknowledged events before forcing a checkpoint for file-based queues
`queue.checkpoint.writes`	`1024`	Maximum number of written events before forcing a checkpoint for file-based queues
`queue.checkpoint.interval`	`1000`	Interval in milliseconds for forced checkpointing for file-based queues

var_logstash_dead_letter_queue_settings

A YAML object with parameters related to dead-letter queue operations, typically includes:

Variable	Default Value	Description
`dead_letter_queue.enable`	`false`	Enables the dead-letter queue
`dead_letter_queue.max_bytes`	`1024mb`	Size of the dead-letter queue in bytes
`dead_letter_queue.flush_interval`	`5000`	Period for writing the queue to a file in milliseconds

Global Logstash Parameters

Global Logstash parameters are located in the <PATH_TO_LOGSTASH_CONFIG>/logstash.yml file and are visually divided into 5 groups for convenience:

Node settings (var_logstash_node_settings)
Directory settings (var_logstash_path_settings)
Pipeline settings (var_logstash_pipeline_settings)
Queue settings (var_logstash_queue_settings)
Dead-letter queue settings (var_logstash_dead_letter_queue_settings)

If you plan to change any parameter in a group, you will need to specify all of the default values and modify the necessary ones. Parameters must be specified according to YAML rules. Some parameters have default values within the Logstash logic; more details can be found in the official documentation.

For example:

var_logstash_queue_settings:
  queue.type: persistent
  queue.page_capacity: 64mb
  queue.max_events: 0
  queue.max_bytes: 1024mb
  queue.checkpoint.acks: 1024
  queue.checkpoint.writes: 1024
  queue.checkpoint.interval: 1000

This will use a different queue type with the listed parameters. If no variables in the group are specified, the logstash.yml configuration will be ignored.

Recommended File Structure

According to Ansible rules, you can place parameters for host groups in the group_vars directory and for specific hosts in the host_vars directory. Inside these directories, there can be a file with the same name as the group or host, respectively.

Instead of a file, there can be a directory with the same name, inside which files can be located, which, according to Ansible rules, will be collected into a single file when the playbook is executed.

For example, there is a hosts inventory file; it describes the _example_ group, as well as __example_. In the second case, all parameters are specified in one file; in the first, they are split into files within a directory of the same name. In the example, directories are not used for hosts, only files with parameters related to a specific host.

.
├── group_vars
│   ├── __example_
│   └── _example_
│       ├── connect
│       ├── logstash_base
│       ├── pipeline__default
│       ├── pipeline_apache
│       ├── pipeline_cisco_ios
│       └── pipeline_os
├── host_vars
│   ├── test1
│   ├── test2
│   ├── test3
│   └── test4
└── hosts

Note!

We recommend specifying additional parameters for groups and hosts in directories and splitting parameters into files.

It is recommended to place the settings for each pipeline in a separate file, as shown in the example above. It is recommended to name the files starting with the keyword pipeline_, followed by the pipeline name (id in pipelines.yml).

Common pipeline parameters are placed in a separate file pipeline__default; a double underscore is used specifically to make the file stand out in the list. More details on common parameters are in the next section.

You can describe a specific pipeline in the parameters (more details in the next section) or create a template in JINJA2 format in the templates/conf.d directory. For example, in the example below, when specifying the wineventlog pipeline, the wineventlog.conf.j2 file template will be used. There is also a template for the test pipeline; for all other pipelines, the default_pipeline.j2 template will be used.

.
└── templates
    ├── conf.d
    │   ├── common
    │   ├── default_pipeline.j2
    │   ├── test.conf.j2
    │   └── wineventlog.conf.j2
    ├── logstash.yml.j2
    └── pipelines.yml.j2

Example of a pipeline template file content:

input {
{%      include 'common/input.j2' %}
}

filter {
  json{
    source => "message"
  }

  date {
    match => [ "timestamp" , "ISO8601"]
  }

  mutate{
    remove_field => ["path"]
  }
}

output {
{%      include 'common/output.j2' %}
}

In this example, the input and output blocks will be populated from the parameters described in the configurations, and the filter block will be ignored and used from this template.

Pipeline Configuration

There are several levels of pipeline configuration, listed in descending order of priority:

Host level for a specific pipeline
Host level default
Group level for a specific pipeline
Group level default

Note!

When running the playbook, all objects (parameters) are merged in the specified priority. If a parameter was a list, it is completely replaced by the list of a higher priority level.

Two objects (variables) are used to describe pipeline parameters:

var_logstash_pipelines_list - used in host groups
host_var_logstash_pipelines_list - used in hosts

When separating group and host parameters into files according to our recommendation, the above objects are replaced by multiple objects; _<id_pipeline> is added to their name, where <id_pipeline> should be replaced with a unique id, for example: var_logstash_pipelines_list_test var_logstash_pipelines_list_apache host_var_logstash_pipelines_list_win_log

The following objects are reserved for default settings: var_logstash_pipelines_list_default host_var_logstash_pipelines_list_default

Note!

It is recommended to place objects in corresponding files named pipeline_<id_pipeline>.

Inside the corresponding object, there should be an object with the name <id_pipeline>. For example, to describe default, the pipeline__default file should be created, and it should contain the following objects:

var_logstash_pipelines_list_default:
  default:

For the apache pipeline, the pipeline_apache file should be created, containing the following objects:

var_logstash_pipelines_list_apache:
  apache:

The following objects can be inside these objects:

copy_files
configs
input
filter
output

Object `default`

The default object is used to specify the same parameters for all pipelines. Matching parameters or plugin objects by name will be replaced by data from higher priority levels. Inside the object, you can specify all common parameters for different plugins at once. When describing plugin blocks input, output, parameters from default will be applied only when specifying the corresponding plugin.

For example:

input:
  file:
    add_field:
      '[logstash][fil1]': 'fsdbfhsdbfgdfsfgbdfbg'
      '[logstash][fil2]': 'fsdbfhsdbfgdfsfgbdfb2'
      '[logstash][fil3]': 3423
    codec:
      cef:
        ecs_compatibility: 'disabled'
  relp:
    codec: 'cef {}'

In this example, plugins for the input file and relp blocks are specified; in all pipelines where one of these plugins is specified, the corresponding parameters will be added.

Object `copy_files`

This object is specifically designed for cases where you need to copy certain files as is to the server. It has the following syntax:

copy_files:
  - source: 'vbtest/apache.sql'
    destination: '/app/logstash/config/conf.d/apache/'
  - source: 'vbtest/apache2.sql'
    destination: '/app/logstash/config/conf.d/apache/'

Directories are created automatically if they do not exist.

Object `configs`

This object describes the parameters for the pipeline, which are written to the pipelines.yml file. Parameters are specified according to YAML syntax and use parameters from the Logstash documentation.

Example:

configs:
  pipeline:
    workers: 5
    batch:
      size: 125
  modules: |
    - name: MODULE_NAME1
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY1: VALUE
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY2: VALUE
      var.PLUGIN_TYPE2.PLUGIN_NAME2.KEY1: VALUE
      var.PLUGIN_TYPE3.PLUGIN_NAME3.KEY1: VALUE
    - name: MODULE_NAME2
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY1: VALUE
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY2: VALUE

The result will be:

- pipeline.id: apache
  path.config: '/app/logstash/config/conf.d/apache.conf'
  pipeline.workers: 5
  pipeline.batch.size: 125
  modules:
    - name: MODULE_NAME1
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY1: VALUE
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY2: VALUE
      var.PLUGIN_TYPE2.PLUGIN_NAME2.KEY1: VALUE
      var.PLUGIN_TYPE3.PLUGIN_NAME3.KEY1: VALUE
    - name: MODULE_NAME2
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY1: VALUE
      var.PLUGIN_TYPE1.PLUGIN_NAME1.KEY2: VALUE

Object `input`

This object can contain the following objects:

clear_text
LDAPSearch
exec
kafka
snmp
udp
file
opensearch
syslog
beats
elasticsearch
jdbc
relp
tcp

clear_text is a special object, described in more detail in the filter object.

For other objects, you can use the parameters of the corresponding plugins from the ElasticSearch Logstash input plugin documentation.

If a pipeline uses a single input plugin, you can specify the plugin parameters in YAML syntax, for example:

input:
  file:
    path: '/app/generator/data/generated/apc_snmp_generated.log'
    start_position: 'beginning'
    sincedb_path: '/dev/null'

If a pipeline uses multiple identical input plugins, you can specify them as a list, for example:

input:
  relp:
    - id: 'cef_erpv1'
      port: 10343
      codec: 'cef { reverse_mapping => false }'
    - id: 'cef_kredan'
      port: 10804
      codec:
        json:
          ecs_compatibility: 'v8'
    - id: 'cef_sasid'
      port: 11655

Additional parameters from the default.relp object will be added to each plugin in the example above.

Object `filter`

This object can only have the clear_text parameter. The text within the parameter will be added as is. Example:

clear_text: |
  grok {
    match => { "message" => "%{IP_OR_HOST:clientip} %{DATA:ident}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  mutate{
    remove_field => ["path"]
  }

Pay attention to the vertical bar "|" and the indentation within the clear_text parameter according to YAML rules.

Object `output`

This object can contain the following objects:

clear_text
http
stdout
udp
elasticsearch
opensearch
tcp

You can specify multiple different plugins, for example:

input:
  file:
    path: ['/app/generator/data/generated/apache_generated.log', '/var/log/test.log']
    check_archive_validity: true
    start_position: 'beginning'
    exclude:
      - test1
      - test2
    sincedb_path: '/dev/null'
    add_field:
      '[logstash][fil1]': 'yyyyy'
      '[logstash][fil5]': 1923
  kafka:
    topics: ['top1', 'top2']

In the example above, there will be two plugins, file and kafka, with the described parameters.

Object `codec`

Some plugins have a codec parameter, which can be specified as a string:

input:
  relp:
    - id: 'cef_erpv1'
      port: 10343
      codec: 'cef { reverse_mapping => false }'

You can also specify parameters within an object:

input:
  relp:
    id: 'cef_erpv1'
    codec:
      json:
        ecs_compatibility: 'v8'

Specifying codec parameters as objects is only possible for the following objects:

cef
csv
json
json_lines
plain

Examples

Ansible Role Configuration Files

File pipeline__default:

var_logstash_pipelines_list_default:
  default:
    output:
      opensearch:
        hosts:
          - 'https://172.16.0.174:9200'
          - 'https://172.16.0.175:9200'
        index_region: 'testregion'
        user: 'logstash'
        password: '{ES_PWD}'
        ssl: true
        cacert: '/app/logstash/config/ca-cert.pem'
        ilm_enabled: false
    input:
      relp:
        codec: 'cef {}'

File pipeline_test:

var_logstash_pipelines_list_test:
  test:
    output:
      opensearch:
    input:
      file:
        path: '/app/generator/data/generated/cisco_ios_generated.log'
        start_position: 'beginning'
        sincedb_path: '/dev/null'
    filter:
      clear_text: |
        grok {
          match => ["message", "%{DATA:timestamp} %{IPV4:ip} %{INT:number}:"]
        }

File pipeline_test2:

var_logstash_pipelines_list_test2:
  test2:
    output:
      opensearch:
        hosts:
          - 'https://172.16.0.174:9200'
        user: 'admin'
    input:
      relp:
        id: 'cef_erpv1'
        port: 10343

Results after running the Ansible playbook

File pipelines.yml:

- pipeline.id: test
  path.config: '/app/logstash/config/conf.d/test.conf'

- pipeline.id: test2
  path.config: '/app/logstash/config/conf.d/test2.conf'

File conf.d/test.conf

input {
    file {
        path => "/app/generator/data/generated/cisco_ios_generated.log"
        sincedb_path => "/dev/null"
        start_position => "beginning"

    }
}

filter {
    grok {
      match => ["message", "%{DATA:timestamp} %{IPV4:ip} %{INT:number}:"]
    }
}

output {
    opensearch {
        cacert => "/app/logstash/config/ca-cert.pem"
        hosts => ["https://172.16.0.174:9200", "https://172.16.0.175:9200"]
        ilm_enabled => "false"
        index => "test_testregion"
        password => "{ES_PWD}"
        ssl => true
        user => "logstash"

    }
}

File conf.d/test2.conf

input {
    relp {
        port => 10343
        codec => cef {}
        id => "cef_erpv1"

    }
}


output {
    opensearch {
        cacert => "/app/logstash/config/ca-cert.pem"
        hosts => ["https://172.16.0.174:9200"]
        ilm_enabled => "false"
        index => "test2_testregion"
        password => "{ES_PWD}"
        ssl => true
        user => "admin"

    }
}

Description​

Role Parameters​

var_logstash_pipelines_list​

host_var_logstash_pipelines_list​

var_logstash_files​

logstash.yml File Parameters​

var_logstash_node_settings​

var_logstash_path_settings​

var_logstash_pipeline_settings​

var_logstash_queue_settings​

var_logstash_dead_letter_queue_settings​

Global Logstash Parameters​

Recommended File Structure​

Pipeline Configuration​

Object default​

Object copy_files​

Object configs​

Object input​

Object filter​

Object output​

Object codec​

Examples​

Ansible Role Configuration Files​

Results after running the Ansible playbook​

Description

Role Parameters

var_logstash_pipelines_list

host_var_logstash_pipelines_list

var_logstash_files

`logstash.yml` File Parameters

var_logstash_node_settings

var_logstash_path_settings

var_logstash_pipeline_settings

var_logstash_queue_settings

var_logstash_dead_letter_queue_settings

Global Logstash Parameters

Recommended File Structure

Pipeline Configuration

Object `default`

Object `copy_files`

Object `configs`

Object `input`

Object `filter`

Object `output`

Object `codec`

Examples

Ansible Role Configuration Files

Results after running the Ansible playbook