Configuring Cross Cluster Search

Description

Cross Cluster Search allows you to perform searches on data from a central cluster across one or more subordinate clusters. The article uses the following definitions and variables:

• Central Cluster, Coordinating Cluster: The cluster from which requests are made to subordinate clusters
• Subordinate Cluster, Remote Cluster: A remote cluster to which requests are made from the central cluster
• OPENSEARCH_HOME: The home directory for the OpenSearch installation, typically /app/opensearch
• ALIAS - A conditional name for a certificate; each new trusted certificate requires a unique name
• PATH_TO_CERT - The path to the certificate
• TRUSTSTORE_PASSWORD - The password for the certificate store, with a minimum length of 6 characters
• REMOTE_CLUSTER_NAME - A conditional name for the subordinate cluster, which can be named arbitrarily. It is used to specify where to perform the search on the central cluster in queries
• REMOTE_CLUSTER_ADDRESS - The IP address of the node in the subordinate cluster
• TRANSPORT_PORT - The communication port between cluster nodes, typically 9300/tcp, set by the transport.port parameter

Configuration process

warning

Configuration is performed on all nodes of all clusters participating in the Cross Cluster Search process.

Nodes of subordinate clusters

Before connecting a subordinate cluster, it is necessary to add the CA certificate of the other cluster, or the certificates of all nodes participating in Cross Cluster Search, to the trusted certificates:

$OPENSEARCH_HOME/jdk/bin/keytool -import -noprompt -trustcacerts -alias <ALIAS> -file <PATH_TO_CERT> -keystore $OPENSEARCH_HOME/config/cacerts.jks -storepass <TRUSTSTORE_PASSWORD>
chown opensearch:opensearch $OPENSEARCH_HOME/config/cacerts.jks

warning

These actions must be performed on the nodes participating in Cross Cluster Search. For example, if the central OpenSearch Dashboards cluster is connected only to nodes with the data role, it is sufficient to perform these actions only on those nodes. The same applies to the nodes of the subordinate cluster.

In other words, you need to copy the CA certificate of the subordinate cluster and paste it onto the nodes of the central cluster, then execute the commands mentioned above on the corresponding cluster nodes. A similar procedure must be performed on the subordinate cluster: copy the CA certificate from the central cluster and execute the commands mentioned above.

You can add not the CA certificate, but the certificates of the nodes that will be connected. However, you will need to execute the first command from the list below for each node of the other cluster, and then proceed with the subsequent commands:

openssl s_client -showcerts -connect <NODE_CLUSTER>:9200 </dev/null 2>/dev/null|openssl x509 -outform PEM > <PATH_TO_CERT>
$OPENSEARCH_HOME/jdk/bin/keytool -import -noprompt -trustcacerts -alias <ALIAS> -file <PATH_TO_CERT> -keystore $OPENSEARCH_HOME/config/cacerts.jks -storepass <TRUSTSTORE_PASSWORD>
chown opensearch:opensearch $OPENSEARCH_HOME/config/cacerts.jks

After completing the above steps, the OpenSearch configuration on all nodes where the commands were executed must be updated to include information about the new trusted certificate store. The following lines need to be added to the $OPENSEARCH_HOME/config/opensearch.yml file:

plugins.security.ssl.transport.external_truststore_filepath: ./cacerts.jks
plugins.security.ssl.transport.external_truststore_password: '<TRUSTSTORE_PASSWORD>'

To avoid storing the certificate store password in plain text, it can be specified in the keystore. To do this, first remove the relevant line from the configuration above:

$OPENSEARCH_HOME/bin/opensearch-keystore add plugins.security.ssl.transport.external_truststore_password;
chown opensearch:opensearch $OPENSEARCH_HOME/config/opensearch.keystore;

Additionally, you need to add the DN of all nodes from the other cluster to the plugins.security.nodes_dn parameter. This means that on all nodes of the central cluster participating in Cross Cluster Search, you must specify the DN of the certificates of all nodes from the subordinate cluster. For the nodes of the subordinate cluster, you need to perform a similar procedure. To obtain the correct DN entry of a certificate, you can run the following command:

openssl x509 -subject -nameopt RFC2253 -noout -in <PATH_TO_CERTIFICATE>

In the output, you need to remove the initial characters "subject=".

To simplify the process, you can simply copy the contents of the plugins.security.nodes_dn parameter from the subordinate cluster for the nodes of the central cluster, and vice versa. In the DN, you can use the "*" symbol to replace any number of arbitrary characters. For example, as shown in the screenshot below:

After completing these steps, it is essential to restart the OpenSearch service:

systemctl restart opensearch

Nodes of the central cluster

n the nodes of the central cluster, it is necessary to add the cluster role remote_cluster_client to the node.roles parameter in the $OPENSEARCH_HOME/config/opensearch.yml file:

node.roles: [data, remote_cluster_client]

Configuring the Remote Cluster

On the central cluster, you need to execute a request to add the subordinate cluster in Menu -> System Settings -> Developer Console:

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote": {
      "<REMOTE_CLUSTER_NAME>": {
        "seeds": ["<REMOTE_CLUSTER_ADDRESS:TRANSPORT_PORT>"]
      }
    }
  }
}

In the seeds field, you can specify a list of addresses of the nodes in the subordinate cluster, ensuring to include the port (usually 9300).

Verification

To verify the connection status, you need to execute the command:

GET _remote/info

The connected field should have the value true.

To execute a query in the subordinate cluster, you need to run a command of the form:

GET <REMOTE_CLUSTER_NAME>:<REQUEST>

For example, a query to search in the books index:

GET 27:books/_search