Skip to main content

Data Loading into the System

Introduction​

Data analysis is crucial for ensuring the security of enterprise information systems. By analyzing information, it's possible to track user activity, identify potential security threats, and prevent incidents.

The use of monitoring tools, such as Search Anywhere Framework, facilitates this process by providing automated data analysis and visualization capabilities.

The functionality of the Search Anywhere Framework is an important tool in this process, providing comprehensive analysis and monitoring capabilities, as well as the ability to create customized reports.

About Data Import​

The Search Anywhere Framework platform supports various methods of information collection.

One of the most common methods is where data is ingested into the system directly from log sources.

The simplest method, recommended for familiarizing yourself with the capabilities of Search Anywhere Framework, is direct data loading into the system through a specialized interface.

As an example for familiarization, we recommend using prepared data (jollymeal_wineventlog.csv).

What's Included in the Data​

The data provided for familiarization contains information from the security audit log, which includes details about login attempts, changes in system settings, file access, and other actions that may pose a security risk to the system.

The example below represents a typical event presented in the prepared data sample.

JSON Example
{
  "agent": {
    "name": "jollymeal-demo",
    "id": "e13410f4-896d-4140-a4ba-4ed54ce58149",
    "type": "winlogbeat",
    "ephemeral_id": "02e29f56-c819-4371-ab81-ce9eb68c8b15",
    "version": "8.0.0"
  },
  "winlog": {
    "computer_name": "JM-MAN-014",
    "process": {
      "pid": 88463,
      "thread": {
        "id": 5651
      }
    },
    "keywords": [
      "Audit Failure"
    ],
    "level": "information",
    "channel": "Security",
    "event_data": {
      "TargetLogonId": "0x12345678",
      "WorkstationName": "JM-MAN-014",
      "TargetUserName": "SanchezThomas",
      "TargetDomainName": "JMCORP"
    },
    "opcode": 0,
    "record_id": "123456789",
    "task": "Logon",
    "event_id": 4625,
    "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
    "time_created": "2024-03-06T07:15:09Z",
    "provider_name": "Microsoft-Windows-Security-Auditing",
    "outcome": "failure"
  },
  "log": {
    "file": {
      "path": "/app/auth-events/output/auth_events-2024-03-06.json"
    }
  },
  "destination": {
    "address": "TERM-SERV-JMCORP",
    "domain": "JMCORP",
    "ip": "192.168.16.220"
  },
  "source": {
    "address": "JM-MAN-014",
    "ip": "192.168.16.17",
    "domain": "JMCORP"
  },
  "@timestamp": "2024-03-06T07:15:09Z",
  "related": {
    "ip": [
      "192.168.16.17",
      "192.168.16.220"
    ],
    "user": [
      "SanchezThomas"
    ]
  },
  "ecs": {
    "version": "8.9.0"
  },
  "host": {
    "name": "JM-MAN-014"
  },
  "@version": "1",
  "event": {
    "original": "{\"@timestamp\": \"2024-03-06T07:15:09Z\", \"event\": {\"kind\": \"event\", \"category\": [\"authentication\"], \"type\": [\"start\"], \"outcome\": \"failure\", \"action\": \"logon-failed\", \"code\": 4625, \"provider\": \"Microsoft-Windows-Security-Auditing\", \"module\": \"security\"}, \"agent\": {\"name\": \"jollymeal-demo\", \"id\": \"e13410f4-896d-4140-a4ba-4ed54ce58149\", \"type\": \"winlogbeat\", \"ephemeral_id\": \"02e29f56-c819-4371-ab81-ce9eb68c8b15\", \"version\": \"8.0.0\"}, \"winlog\": {\"computer_name\": \"JM-MAN-014\", \"process\": {\"pid\": 88463, \"thread\": {\"id\": 5651}}, \"keywords\": [\"Audit Failure\"], \"level\": \"information\", \"channel\": \"Security\", \"event_data\": {\"WorkstationName\": \"JM-MAN-014\", \"TargetUserName\": \"SanchezThomas\", \"TargetDomainName\": \"JMCORP\", \"TargetLogonId\": \"0x12345678\"}, \"opcode\": 0, \"record_id\": \"123456789\", \"task\": \"Logon\", \"event_id\": 4625, \"provider_guid\": \"{54849625-5478-4994-a5ba-3e3b0328c30d}\", \"time_created\": \"2024-03-06T07:15:09Z\", \"provider_name\": \"Microsoft-Windows-Security-Auditing\", \"outcome\": \"failure\"}, \"source\": {\"address\": \"JM-MAN-014\", \"ip\": \"192.168.16.17\", \"domain\": \"JMCORP\"}, \"destination\": {\"address\": \"TERM-SERV-JMCORP\", \"ip\": \"192.168.16.220\", \"domain\": \"JMCORP\"}, \"related\": {\"ip\": [\"192.168.16.17\", \"192.168.16.220\"], \"user\": [\"SanchezThomas\"]}, \"user\": {\"domain\": \"JMCORP\", \"name\": \"SanchezThomas\", \"id\": \"0005\"}, \"host\": {\"name\": \"JM-MAN-014\"}, \"ecs\": {\"version\": \"8.9.0\"}, \"outcome\": \"failure\"}",
    "code": 4625,
    "provider": "Microsoft-Windows-Security-Auditing",
    "kind": "event",
    "module": "security",
    "action": "logon-failed",
    "category": [
      "authentication"
    ],
    "type": [
      "start"
    ],
    "outcome": "failure"
  },
  "user": {
    "domain": "JMCORP",
    "name": "SanchezThomas",
    "id": "0005"
  },
  "outcome": "failure"
}

Data Loading​

Search Anywhere Framework provides a simple interface for importing data, designed to be user-friendly without requiring any special technical knowledge.

To upload data into Search Anywhere Framework, follow these steps:

  1. Navigate to the web interface of the Search Anywhere Framework.

  2. Click on the Upload Data button in the top right corner.

Main Page

  1. The following interface will be presented, allowing you to import data into the system.
Note!

Only files in the formats .xlsx, .csv, .json are supported for import. Additionally, the file size should not exceed 100 MB.

  1. Select the file jollymeal_wineventlog.csv for import.

File Import 2

  1. Click on Next to proceed to the next import step.

  2. In the dropdown menu Select options for index, choose New index, and specify a name for it in the corresponding field. It's recommended to use jollymeal_wineventlog as the index name.

  3. Perform the data index schema setup.

This interface allows for customizing the data types for imported fields without requiring any special technical knowledge. Users can easily select the data type for each imported field, such as text, integer, date and time, etc. This enables the accurate interpretation and analysis of data according to its actual content, ensuring more precise and useful results when analyzing data in Search Anywhere Framework.

It's necessary to change the type of the following fields:

  • event.code: integer
  • winlog.event_id: integer
  • winlog.opcode: integer
  • winlog.process.pid: integer
  • winlog.process.thread.id: integer
  • @timestamp: date

After that, click the "Next" button.

Mapping

  1. A message indicating successful import will be displayed.

Next, you have the following options:

  • Create a template
  • Open in search
  • Load more

Searching for information about events is already available, but to proceed further, it's necessary to create an index template, so let's choose this option.

  1. In the opened window, click on the Create Index Template button.

  2. In the Index Template Name field, enter jollymeal_wineventlog.

Note!

The template name must match the index name. Remove the * symbol at the end of the template name.

  1. In the Time Field row, select @timestamp, and then complete the template creation by clicking the Create Index Template button.

  2. Data loading is complete. Now the data is available for search and analysis. To verify this, simply go to the Main - Search section.

In the field, you can enter the query:

source jollymeal_wineventlog