Updating Search Anywhere Framework

Caution!

This instruction describes the process of updating Search Anywhere Framework from version 6.0.* to 6.1.*.

Important Note!

For proper operation of notes and other features, you must:

1. Create the $OS_HOME/uploads directory on SA Data Storage nodes
2. Set ownership to the user running SA Data Storage
3. Mount a shared network folder to this directory

Refer to the configuration guide for setup instructions.

Information

Conventions:

• SAF_INSTALLER - the directory where the Search Anywhere Framework version 6.1 installation package is unpacked
• USER - a system user with administrator rights, usually admin
• OS_HOME - the OpenSearch home directory, usually /app/opensearch/
• OS_DATA - the directory where indexed data is stored, usually /app/data/opensearch
• OS_IP - IP address of one of the OpenSearch cluster servers
• OSD_HOME - the OpenSearch Dashboards home directory, usually /app/opensearch-dashboards/
• PATH_SSL - the location of the certificate, the admin private key, and the ca-cert, usually /app/opensearch/config/

The first step in updating is to determine the currently installed version of Search Anywhere Framework. This can be done by viewing the module versions on the main page or by running the following command in the command line:

curl https://$OS_IP:9200/_cat/plugins -k -u $USER

After entering this command, you will need to enter the password for the $USER account. It is recommended to use the admin user.

A detailed list of new features can be found in the article What's New in SAF 6.1.

Let's consider the update procedure for each component. The 6.1 installer needs to be unpacked into a directory, for example, /app/distr/.

Caution!

Before starting work, it is strongly recommended to back up the main configuration files and Security settings.

Recommended Actions

It is recommended to create a directory, for example, /app/backup, where you should save:

• The config directory, usually $OS_HOME/config or $OSD_HOME/config
• The systemd files, usually /etc/systemd/system/opensearch.service and /etc/systemd/system/opensearch-dashboards.service, /etc/systemd/system/sme-re.service
• The file /etc/sysctl.d/00-opensearch.conf
• A copy of the Security settings. This needs to be done once, and requires the certificate and private key of the admin user. (The command below will create a directory with the current date containing the OpenSearch security settings.)

  chmod +x $OS_HOME/plugins/opensearch-security/tools/securityadmin.sh
  JAVA_HOME=$OS_HOME/jdk/ $OS_HOME/plugins/opensearch-security/tools/securityadmin.sh -backup /app/backup/security_$(date +%Y%m%d) \
  -icl \
  -nhnv \
  -cacert $OS_HOME/config/ca-cert.pem \
  -cert $OS_HOME/config/admin-cert.pem \
  -key $OS_HOME/config/admin-key.pem

Updating OpenSearch

The Search Anywhere Framework 6.1 installer needs to be unpacked into a directory, for example, /app/distr/. The location where you unpack the archive contents will be referred to as $SAF_INSTALLER.

tar xzf saf-installer-*.tar.gz -C /app/distr/
SAF_INSTALLER=/app/distr/saf_6.1

Note!

For clusters consisting of multiple nodes, it is recommended to disable allocation before upgrading. This can be done through the developer console (Main Menu - System Settings - Dev Console) by executing the following command:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}

The same can be done from the terminal with the following command:

curl -XPUT -k -u admin "https://$OS_IP:9200/_cluster/settings?pretty" -H "Content-Type: application/json" -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'

When upgrading cluster nodes, do not use the update script to disable allocation. After upgrading all cluster nodes, enable allocation:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}

The same can be done from the terminal with the following command:

curl -XPUT -k -u admin "https://$OS_IP:9200/_cluster/settings?pretty" -H "Content-Type: application/json" -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'

Automatic Mode

The script requires the following pre-installed packages:

• curl
• zip
• unzip

Note!

If you do not see the message indicating that Search Anywhere Framework has been updated at the end, do not rerun the update script. Take a screenshot of where the script stopped and contact technical support.

The script automates the update process and is located at $SAF_INSTALLER/opensearch/setup.sh. To update, either pass the update action when running the script, or run the script without arguments and select the appropriate option from the interactive menu:

Select action [default: update]:
  1) install
  2) update
  3) uninstall

Action [update]:

The script supports the following startup parameters:

• <action> - action to perform: install, update, or uninstall
• -c, --config <path_to_config_file_yaml> - specify the configuration file for the update
• -h, --help - displays help information about available commands

Example configuration file: $SAF_INSTALLER/opensearch/example_config_opensearch.yaml. The YAML file format is similar to the configuration file used during installation.

Note!

Start the upgrade with nodes that do not have the master role. Data nodes can connect to older versions of master nodes, but not vice versa.

To start the update, run the script:

$SAF_INSTALLER/opensearch/setup.sh update

After launching, the script automatically finds the paths to the main directories:

• OpenSearch Home Directory - the OpenSearch installation directory, usually /app/opensearch
• OpenSearch Conf Directory - the OpenSearch configuration files directory, usually /app/opensearch/config/
• OpenSearch Data Directory - the data directory, usually /app/data/
• OpenSearch Logs Directory - the logs directory, usually /app/logs/

The update script does not perform any actions with the data and logs directories. The configuration files directory and systemd files will be saved to a temporary directory $SAF_INSTALLER/opensearch/staging/.

Note!

If you run the script again, the staging directory will be cleared.

================================================================================
SEARCH ANYWHERE FRAMEWORK UPDATE SCRIPT - OPENSEARCH
================================================================================

Current working directory: /app/distr/saf_6.1/opensearch

Current name of install's archive: opensearch-3.5.0-linux-x64.tar.gz

New version OpenSearch: 3.5.0

================================================================================
-- STEP 1. INSTALLATION DIRECTORIES

opensearch.service file found. Will get necessary paths from there

Final Opensearch home directory: /app/opensearch
Final Opensearch conf directory: /app/opensearch/config
Final Opensearch data directory: /app/data/opensearch
Final Opensearch logs directory: /app/logs/opensearch
Is this correct? [y/n]:

After entering the directories, you need to confirm the automatically found data by pressing y, or enter your directories manually by pressing n.

At the second step, you need to answer the question about user usage during the update. If no information output is required, enter n; if additional information output is required, enter y.

The user can be used either for information output or for allocation management, consequently requiring different permission sets. For simple information output, the permissions of the sa-monitoringrole are sufficient.

-- STEP 2. GET USER

Do you want to see cluster information or manage allocation? [Y/n] :

Your account must have role "sa-monitoring"
Your account must have permission "cluster:admin/settings/update" for manage allocation
================================================================================

Enter username: admin
Enter password for user "admin":
Is this correct? [y/n]:

At the next step, you can disable all operations requiring privilege escalation to root user rights during the update.

Important Note!

It is impossible to completely perform the update without root privileges, part of operations need to be performed as root user, then run the installer as the user under which OpenSearch operates, after the script finishes, some operations need to be performed again as root user. The list of operations will be displayed in the installer.

If commands are not executed before updating, data loss is possible.

You can run the installer as opensearch user with the following command:

sudo -u opensearch $SAF_INSTALLER/opensearch/setup.sh update

The first prompt will ask about disabling commands requiring root privilege escalation.

Next, you'll need to enter the user and group under which OpenSearch operates.

Following this, information will be displayed listing operations that must be performed before running the update script.

-- STEP 3. DISABLE OPERATIONS WHERE ROOT RIGHTS ARE REQUIRED

Want to update Search Anywhere Framework without root privileges? [y/N]: y

Final you set the non-root flag to: y

Linux username for Search Anywhere Framework [opensearch]:
Linux groupname for Search Anywhere Framework [opensearch]:

Final Search Anywhere Framework username: opensearch
Final Search Anywhere Framework groupname: opensearch
Is this correct? [Y/n]:

ATTENTION!
Before updating Search Anywhere Framework, you must run the following commands as a user with root privileges.
If yo need to disable allocation, run the following command:

curl -XPUT -k -u YOUR_USER_OPENSEARCH:YOUR_USER_OPENSEARCH_PASSWORD_HERE "https://YOUR_IP_OPENSEARCH:9200/_cluster/settings?pretty" -H "Content-Type: application/json" -d '{"persistent":{"cluster.routing.allocation.enable": "none"}}'

Change user, group and directories as needed

systemctl stop sme-re
systemctl stop opensearch
chown -R opensearch:opensearch /app/opensearch /app/opensearch/config /app/data/opensearch /app/logs/opensearch /app/distr/saf_6.1/opensearch

!!! Are you sure you have done everything from the list above? [y/N]:

At the final step, you must confirm that all commands were executed as the root user. Enter y to confirm, otherwise the update will be aborted.

At the next step, if you chose to use a user during the update, you need to answer the allocation question. If you enter y, the script will:

1. Disable allocation before updating
2. Re-enable it after completing the script operations

-- STEP 4. CONFIGURE ALLOCATION

Do you want to disable allocation during update? [y/N]: n

You don't want to disable allocation: n
Is this correct? [y/n]:

Then preparatory actions will be performed before the update. Before applying the update, a confirmation prompt will appear - until this point, no changes are made to the system. Additionally, some information about the current node and the cluster as a whole will be displayed, if you selected to use a user during the update at step two.

get current list of plugins
sm-core
sm-im
sm-inventory
sm-ism-action-clickhouse
sm-job-scheduler
sm-job-scheduler-actions-incident
sm-job-scheduler-actions-mitre
sm-knowledge-center
sm-mitre
sm-mssp
sm-rsm
sm-uba
sme
opensearch-security
Information about current node OpenSearch:
{
  "name" : "smos-node-00",
  "cluster_name" : "smos-cluster",
  "cluster_uuid" : "yKPPDCHGSA6rHQT948jokQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.5.0",
    "build_type" : "tar",
    "build_hash" : "99a9a81da366173b0c2b963b26ea92e15ef34547",
    "build_date" : "2025-10-29T17:05:01.906258Z",
    "build_snapshot" : false,
    "lucene_version" : "10.2.2",
    "minimum_wire_compatibility_version" : "2.19.0",
    "minimum_index_compatibility_version" : "2.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

!!! AT THIS POINT WE START TO MAKE CHANGES IN OPERATING SYSTEM !!!
Do you want to continue? [y/N]:

Pressing Enter will interrupt the update; press y to continue.

Upon successful completion of the update, you should see the message SEARCH ANYWHERE FRAMEWORK SUCCESSFULLY UPDATED!. Preliminary information about the cluster and current node will be displayed, if you selected to use a user during the update at step two.

-- STEP 10. PRINT INFORMATION
current state of cluster
{
  "cluster_name" : "smos-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 50,
  "active_shards" : 50,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 15,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 76.92307692307693
}
current state clusters nodes
172.16.0.27 14 99 8 1.47 0.63 0.26 dim data,ingest,master * smos-node-00
Information about current node OpenSearch:
{
  "name" : "smos-node-00",
  "cluster_name" : "smos-cluster",
  "cluster_uuid" : "5V2rIp1sRj-M-ANnGfF0cA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.5.0",
    "build_type" : "tar",
    "build_hash" : "99a9a81da366173b0c2b963b26ea92e15ef34547",
    "build_date" : "2025-10-29T17:05:01.906258Z",
    "build_snapshot" : false,
    "lucene_version" : "10.2.2",
    "minimum_wire_compatibility_version" : "2.19.0",
    "minimum_index_compatibility_version" : "2.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

================================================================================
ATTENTION!
After updating Search Anywhere Framework, you must run the following commands as a user with root privileges.
Change user, group and directories as needed.

cp /app/distr/saf_6.1/opensearch/staging/services/sme-re.service /etc/systemd/system/
chown root:root /etc/systemd/system/sme-re.service
chcon -Rv -u system_u -r object_r -t bin_t /app/opensearch/bin
chcon -Rv -u system_u -r object_r -t bin_t /app/opensearch/jdk/bin
chcon -Rv -u system_u -r object_r -t var_run_t /app/opensearch/run
chown -R opensearch:opensearch /app/opensearch /app/opensearch/config /app/data/opensearch /app/logs/opensearch
systemctl daemon-reload
systemctl enable sme-re opensearch
systemctl start sme-re
systemctl start opensearch
systemctl status sme-re opensearch

If you disable allocation before, wait 60 seconds and run the following commands as a "opensearch" or user with root privileges:

curl -XPUT -k -u YOUR_USER_OPENSEARCH:YOUR_ADMIN_PASSWORD_HERE "https://127.0.0.1:9200/_cluster/settings?pretty" -H "Content-Type: application/json" -d '{"persistent":{"cluster.routing.allocation.enable": "all"}}'

The following plugins cannot be installed:
-- sm-job-scheduler-actions-incident
-- sm-job-scheduler-actions-mitre

================================================================================
-- Search Anywhere Framework SUCCESSFULLY UPDATED!
-- For further instructions, please refer to the documentation: https://docs.saf-systems.com/
================================================================================

If you chose to update without using commands requiring privilege escalation to the root user, an information block will be displayed listing commands that need to be executed as the root user.

If for some reason the update script fails to update certain plugins, it will additionally display information about these plugins at the end (text reading The following plugins cannot be installed).

Important Note!

The update script takes into account the current list of installed plugins on OpenSearch nodes. If you need to install any additional plugins, this should be done manually after completing the node update.

Updating SA Web

This script automates the OpenSearch Dashboards update process. It requires the following pre-installed packages:

• curl
• zip
• unzip

The script automates the update process and is located at $SAF_INSTALLER/opensearch-dashboards/setup.sh. To update, either pass the update action when running the script, or run the script without arguments and select the appropriate option from the interactive menu:

Select action [default: update]:
  1) install
  2) update
  3) uninstall

Action [update]:

The update script supports the following parameters:

• <action> - action to perform: install, update, or uninstall
• -c, --config <path_to_config_file_yaml> - Specifies the configuration file for the update
• -h, --help - Displays help information about available commands

Example configuration file: $SAF_INSTALLER/opensearch-dashboards/example_config_dashboards.yaml. The YAML file format is similar to the configuration file used during installation.

The script creates backup copies of the systemd service file, opensearch-dashboards.yml, and the configuration directory in the temporary $SAF_INSTALLER/opensearch-dashboards/staging/ directory. The script does not modify the data or logs directories.

Important!

If the script is run again, the staging directory is cleared and the previous backup is lost. If you need the backup, copy it from this directory before rerunning the script.

To update, run the script:

$SAF_INSTALLER/opensearch-dashboards/setup.sh update

The script automatically detects the main paths on the current server for the following directories:

• OpenSearch Dashboards Home Directory - The OpenSearch Dashboards installation directory, typically /app/opensearch-dashboards
• OpenSearch Dashboards Conf Directory - The OpenSearch Dashboards configuration directory, typically /app/opensearch-dashboards/config/
• OpenSearch Dashboards Data Directory - The data directory, typically /app/data/
• OpenSearch Dashboards Logs Directory - The logs directory, typically /app/logs/

Example output:

================================================================================
SEARCH ANYWHERE FRAMEWORK INSTALL SCRIPT - OPENSEARCH DASHBOARDS
================================================================================

Current working directory: /opt/saf_6.1/opensearch-dashboards

Current name of install's archive: opensearch-dashboards-3.5.0-linux-x64.tar.gz

Current version of OpenSearch-Dashboards: 3.5.0

================================================================================
-- STEP 1. INSTALLATION DIRECTORIES

opensearch-dashboards.service file found. Will get necessary paths from there

Final Opensearch Dashboards home directory: /app/opensearch-dashboards
Final Opensearch Dashboards conf directory: /app/opensearch-dashboards/config
Final Opensearch Dashboards data directory: /app/data/opensearch-dashboards
Final Opensearch Dashboards logs directory: /app/logs/opensearch-dashboards
Is this correct? [y/n]:

After the directories are displayed, confirm the information by pressing y, or enter your directories manually by pressing n.

At the next step, you can disable all operations requiring privilege escalation to root user rights during the update.

Important Note!

A complete update cannot be performed without root privileges. You must:

1. Execute some operations as the root user
2. Run the installer as the SA Web runtime user
3. After script completion, perform additional operations again as root The operation list will be displayed in the installer

Failure to execute these commands before updating may cause data loss.

To launch the installer as the opensearch user:

sudo -u opensearch $SAF_INSTALLER/opensearch-dashboards/setup.sh update

The first prompt will ask about disabling commands requiring privilege escalation to the root user.

Next, you'll need to enter the user and group under which SA Web operates.

Finally, information will be displayed listing operations that must be performed before running the update script.

-- STEP 2. DISABLE OPERATIONS WHERE ROOT RIGHTS ARE REQUIRED

Want to update SEARCH ANYWHERE FRAMEWORK without root privileges? [y/N]: y

Final you set the non-root flag to: y

Linux username for SEARCH ANYWHERE FRAMEWORK [opensearch]:
Linux groupname for SEARCH ANYWHERE FRAMEWORK [opensearch]:

Final SEARCH ANYWHERE FRAMEWORK username: opensearch
Final SEARCH ANYWHERE FRAMEWORK groupname: opensearch
Is this correct? [Y/n]:

ATTENTION!
Before updating SEARCH ANYWHERE FRAMEWORK, you must run the following commands as a user with root privileges.
Change user, group and directories as needed

systemctl stop opensearch-dashboards
chown -R opensearch:opensearch /app/opensearch-dashboards /app/opensearch-dashboards/config /app/data/opensearch-dashboards /app/logs/opensearch-dashboards /app/distr/saf_6.1/opensearch-dashboards

!!! Are you sure you have done everything from the list above? [y/N]:

At the final step, you must confirm that all commands were executed as the root user. Enter y to confirm; otherwise, the update will be terminated.

Next, preparatory actions will be performed before the update. Before applying the update, you'll be prompted for confirmation - until this point, no system-impacting operations are performed. Additional information about the current node will also be displayed.

Current list of plugins:
-- smartMonitor
-- smartMonitorColumnChart
-- smartMonitorCyberSecurity
-- smartMonitorDrawio
-- smartMonitorHeatmapChart
-- smartMonitorHtmlChart
-- smartMonitorIncidentManager
-- smartMonitorInventory
-- smartMonitorKnowledgeCenter
-- smartMonitorLineChart
-- smartMonitorLookupManager
-- smartMonitorMitreAttack
-- smartMonitorPDFExport
-- smartMonitorPieChart
-- smartMonitorSingleValue
-- smartMonitorTable
-- smartMonitorUserBehaviorAnalytics

Current version of OpenSearch-Dashboards: 3.5.0

!!! AT THIS POINT WE START TO MAKE CHANGES IN OPERATING SYSTEM !!!
Do you want to continue? [y/N]:

If the update was selected without using commands requiring privilege escalation to the root user, an information block will be displayed containing the list of commands that must be executed as the root user.

================================================================================
ATTENTION!
After updating SEARCH ANYWHERE FRAMEWORK, you must run the following commands as a user with root privileges.
Change user, group and directories as needed.

cp /app/distr/saf_6.1/opensearch-dashboards/staging/services/opensearch-dashboards.service /etc/systemd/system/
chown root:root /etc/systemd/system/opensearch-dashboards.service
chcon -Rv -u system_u -r object_r -t bin_t /app/opensearch-dashboards/bin
chown -R opensearch:opensearch /app/opensearch-dashboards /app/opensearch-dashboards/config /app/data/opensearch-dashboards /app/logs/opensearch-dashboards
systemctl daemon-reload
systemctl enable opensearch-dashboards
systemctl start opensearch-dashboards
systemctl status opensearch-dashboards

================================================================================
-- SEARCH ANYWHERE FRAMEWORK DASHBOARDS SUCCESSFULLY UPDATED!
-- For further instructions, please refer to the documentation: https://docs.saf-systems.com/
================================================================================

Upon successful completion of the update script, the following message is displayed: SEARCH ANYWHERE FRAMEWORK DASHBOARDS SUCCESSFULLY UPDATED.

Post-Update Actions

Lookup Manager

Important

After the update, manually remove the obsolete lookup creation page from the main menu.

1. Go to Main Menu - System Settings - Module Settings - Main - Menu Settings
2. Expand the Lookup Manager group with the system name lookup-manager
3. Click the trash icon next to the Lookup Creation page
4. Click Save Changes

After removal, only the Lookup List page should remain in the Lookup Manager group.

Updating the peval Function

To update the peval random function, run the following request in the Dev Console (Main Menu - System Settings - Dev Console):

POST _core/migration/6.1.0/peval_random_migration

Incident Manager Migration

Important

If the Incident Manager module is not installed, proceed to the next step.

Migrating the fields Field from object to flat_object for .smos_incident-* Indexes

Important

After migrating the fields field from object to flat_object, the fields.* fields in the .smos_incident-* index will no longer be available for aggregations.

The migration changes the fields field type from object to flat_object in all indexes whose names match the .smos_incident-* pattern. The operation runs asynchronously. After it starts, the request returns a migration_id, which is used to track the status and perform subsequent operations.

The migration also:

• stores service records in the .sm_fields_flat_object_migration index
• creates the sm_incidents_template OpenSearch index template, which sets the fields field type to flat_object in new indexes

tip

To view the template, go to System Settings - Index Management - Templates.

Prerequisites

Before starting the migration, make sure that:

1. Sufficient shard capacity is available. During migration, the source and backup indexes exist at the same time, so the number of shards associated with the migrated indexes temporarily increases approximately twofold. If there are many indexes, the cluster.max_shards_per_node setting or another cluster shard limit may be exceeded

Check the current settings and shard status:

GET /_cluster/settings?include_defaults=true&filter_path=**.max_shards*
GET /_cat/shards?v&h=index,state,docs,store

For instructions on resolving shard limit errors, see Errors While Processing Specific Indexes.

2. The cluster is operational. The cluster status must be green or yellow. Do not start the migration if the status is red

GET _cluster/health

3. No other migration is active. Only one migration can run at a time. An attempt to start a second migration returns an error

4. Sufficient disk space is available. During migration, the source and backup indexes exist at the same time, so their peak disk usage may be approximately twice the original size. The available disk space must be at least equal to the current size of the data being migrated. Allow additional space for segments, service data, and concurrent write operations

5. No indexes are blocked. Make sure none of the indexes being migrated have the read-only, read_only_allow_delete, or write block enabled. Run the following request to check:

GET .smos_incident-*/_settings/index.blocks*

6. No backup indexes from a previous migration remain. Before starting, check for indexes with the .backup-.smos_incident- prefix:

GET /_cat/indices/.backup-.smos_incident-*?v

warning

Do not delete the indexes until you have confirmed that they are not associated with an active migration and are not the only remaining copy of the data.

7. The user has sufficient permissions. The user on whose behalf the migration runs must have permissions to:
   • read and write documents
   • create and delete indexes
   • read and modify index mappings and settings
   • create or update an index template

Migration Procedure

The migration must be performed on an index that is not being modified. Therefore, you can run the migration in one of two ways:

1. Stop all writes to the .smos_incident-* indexes and run the migration according to the instructions below. In this case, the index template is loaded automatically.
2. Manually run the following request to load the index template:

POST /_core/migration/6.1.0/im_reload_template

Wait until the beginning of the next week. New indexes will then be created with the correct mapping, after which the old indexes can be migrated. Run the migration according to the instructions below.

Migration Instructions

1. Start the migration

POST /_core/migration/6.1.0/im_fields_to_flat_object

The response returns the migration identifier. Save the migration_id value because it is required for monitoring and subsequent operations.

{
   "migration_id": "Pte3ZJ8BL4fIwRYMyNYo",
   "status": "ACCEPTED"
}

2. Monitor the migration

GET /_core/migration/6.1.0/im_fields_to_flat_object/{migration_id}

The status field in the response can have the following values:

Status	Description
ACCEPTED	The migration has been accepted but has not started
RUNNING	Indexes are being processed
COMPLETED	All indexes have been migrated successfully
COMPLETED_WITH_ERRORS	Some indexes were migrated, while others failed
FAILED	The migration could not start

The indices field contains the status of each index:

Index Status	Description
RUNNING	The index is being processed
COMPLETED	The index was migrated successfully
SKIPPED	The index was skipped
FAILED	An error occurred while processing it

The errors_by_index field contains a detailed error description for each index.

Wait for one of the following final statuses:

• for COMPLETED, proceed to verify the results and clean up
• for COMPLETED_WITH_ERRORS, review errors_by_index, restore or reprocess the affected indexes, and then perform cleanup
• for FAILED, resolve the cause of the error before restarting the migration

3. Clean up after a successful migration

info

After receiving the COMPLETED status, check the mapping and document count for each migrated index:

GET .smos_incident-*/_mapping?filter_path=*.mappings.properties.fields
GET .backup-.smos_incident-*/_mapping?filter_path=*.mappings.properties.fields

GET /<index_name>/_count
GET /.backup-<index_name>/_count

After the migration is complete, delete the backup copies after confirming that they do not contain the only remaining copy of the data:

DELETE /_core/migration/6.1.0/im_fields_to_flat_object/{migration_id}/backups

Delete the service records associated with this migration from the .sm_fields_flat_object_migration index. Run this request after deleting the backup copies. Backup copies of indexes whose migration failed are not deleted.

DELETE /_core/migration/6.1.0/im_fields_to_flat_object/{migration_id}/cleanup

Potential Errors and Solutions

Errors Before Index Processing Starts

1. Error creating or updating the template

Cause. The user has insufficient permissions, index creation is blocked at the cluster level, or the existing template conflicts with the template being created.

Solution. Check the existing template and the user's permissions. Do not delete the existing template until you have confirmed that it is not used by other indexes or system components. After resolving the cause, restart the migration.

2. Another migration is already running

Cause. A migration with a different migration_id is already running.

Solution. Check the status of the active migration:

GET /_core/migration/6.1.0/im_fields_to_flat_object/{migration_id}

Errors While Processing Specific Indexes

When an error occurs, the index receives the FAILED status and the migration continues processing the remaining indexes.

1. Reindexing errors

Possible causes include insufficient resources during reindexing or a document count mismatch after reindexing is complete. Depending on the migration stage, the data is located in either the source or backup index. Review the error details in errors_by_index, and restart the migration.

2. Error while creating the migrated index

This error occurs if the source index was deleted but the new index was not created or reindexing into it was not completed. All data is stored in the .backup-<index_name> backup index.

Required actions:

• Make sure that the backup index exists and contains data:

  GET .backup-<index_name>/_count

• Manually create the source index with the required mapping and reindex the data from the backup:

  POST _reindex
  {
    "source": { "index": ".backup-<index_name>" },
    "dest":   { "index": "<index_name>" }
  }

If the index being migrated contains a large amount of data, create an index with the original mapping and restart the migration after resolving the cause of the error.

3. Shard count limit exceeded

Cause. The cluster shard limit described in Prerequisites has been exceeded.

Solution. Free unused shards or temporarily increase the corresponding limit. If cluster operating policies do not allow the limit to be increased, restart the migration until all indexes have been processed. Before each restart, delete the backup copies of successfully migrated indexes.

4. Scroll context expired

Cause. Reindexing is performed in batches using the Scroll API. If processing the next batch takes longer than the Scroll context lifetime, OpenSearch deletes the context and the operation fails.

Solution. Reduce the cluster load. After the current migration stops, delete the incomplete backup index and restart the migration.

5. Index skipped

The cause is specified in the message field. An index is usually skipped if the fields field is missing or already has the flat_object type.

6. Backup index already exists

Cause. A previous migration failed after creating the backup index but before deleting it.

Solution.

• make sure that no other migration is running
• check the existence and contents of the corresponding backup index
• delete the backup index only if it does not contain the only remaining copy of the data
• restart the migration

7. Index is blocked

Solution. If necessary, remove the write or metadata modification blocks configured for the index.

Errors After Migration Completion

1. Backup copies cannot be deleted.

If the deletion request returns errors in the errors field, resolve them or delete the backup indexes manually:

DELETE /.backup-<index_name>

2. The service index cannot be cleaned up.

If the request fails, you can delete the records manually. Before deleting them, make sure that the migration completed successfully:

POST /.sm_fields_flat_object_migration/_delete_by_query
{
  "query": {
    "term": { "migration_id.keyword": "<migration_id>" }
  }
}

Configuring SLA Scripts

To run the selected scripts in SLA policies, specify the Smart Monitor URL in the cluster settings:

PUT _cluster/settings
{
  "persistent": {
    "sm.im.adhoc.url": "https://example.com:8080"
  }
}

IM: Content Configuration

Version 6.1.0 adds an SLA statistics dashboard and updates the incident statistics dashboard.

To install the updates:

1. Go to Main Menu - Module Settings - Content Management
2. Select the Incident Manager module
3. Select version sm_incident_manager_system_6.1.0
4. Follow the installation instructions

To view the SLA statistics dashboard, the user must have the search permission for the .sm_im_sla_states-* index.

ASM

Important

If you have purchased a Service Monitor Toolkit license, proceed to the next step.

Starting with this release, the ASM 1.0 module is no longer supported. In the remainder of this documentation, ASM refers to ASM 2.0. Before migrating and updating, make sure that the data has been migrated successfully and is no longer associated with the ASM 1.0 module.

If you have not previously created a menu for ASM, create a navigation menu item to add the new module. Go to Main Menu - System Settings - Module Settings - Main - Menu Settings, and click Add Module. If the ASM module already exists, rename it according to the settings below.

Complete the module fields as follows:

Field Name	Value
Item Type	Group
Title	ASM
System Name	rsm
Enable Display	Yes, the checkbox must be selected

If you have a menu item associated with the first version of ASM, delete it.

Service Monitor Toolkit

Service Monitor Toolkit Migration: Adaptive Metrics

Important

If the Service Monitor Toolkit module is not installed or you do not have a license, skip this step.

To update the data storage schema, run the following requests in the Developer Console (Main Menu - System Settings - Developer Console):

POST _core/migration/6.1.0/base_metric_permissions
{
  "backup_index": ".sm_rsm_v2_metrics_permissions_migration_backup",
  "owner": "admin",
  "read": {"roles": [], "users": ["*"]},
  "write": {"roles": ["*"], "users": []},
  "overwrite_permissions": false
}


POST _core/migration/6.1.0/adaptive_models_v1_to_v2
{
  "backup_index": ".sm_rsm_v2_adaptive_parameters_migration_backup",
  "delete_existing_backup": true
}

Updating the Main Menu

If you have not previously created a menu for Service Monitor Toolkit, create a navigation menu item to add the new module. Go to Main Menu - System Settings - Module Settings - Main - Menu Settings, and click Add Module. If the Service Monitor Toolkit module already exists, rename it according to the settings below.

Complete the module fields as follows:

Field Name	Value
Item Type	Group
Title	Service Monitor Toolkit
System Name	rsm
Enable Display	Yes, the checkbox must be selected

Starting with version 6.1.0, two new sections are available: Maintenance Windows and Adaptive Models.

In the Service Monitor Toolkit module, click Add Section, and complete the fields for Maintenance Windows as follows:

Field Name	Value
Item Type	Page
Title	Maintenance Windows
System Name	maintenance
Enable Display	Yes, the checkbox must be selected

In the Service Monitor Toolkit module, click Add Section, and complete the fields for Adaptive Models as follows:

Field Name	Value
Item Type	Page
Title	Adaptive Models
System Name	adaptive-models
Enable Display	Yes, the checkbox must be selected