Skip to main content
Version: 5.0

Sharding and Replication in ClickHouse

Basic Concepts:​

Shard (Shard):​

A shard is a subset of data in a database that is stored on a single server. In the context of a cluster, shards are distributed across different servers for parallel query processing. Data sharding is used when the total volume of data is too large for a single server or when the system needs to be scaled to improve performance. Data is distributed across shards based on a specific key, such as user_id or timestamp.

Replica (Replica):​

A replica is a copy of data from one or more shards that is stored on another server. Replication enhances the fault tolerance of the system, as data is duplicated, and in the event of one server failing, another can continue to operate. Replication is necessary to ensure high data availability and to protect against data loss in the event of hardware failure. The more replicas there are, the higher the availability, but this also requires more resources.

Defining Cluster Topology:​

Determine the number of replicas and shards in your cluster. Ensure that you have suitable servers to host the shards and replicas.

Determining the Number of Shards:​

This is determined by the volume of data and the required performance. More shards allow the load to be distributed across more servers, increasing performance.

Determining the Number of Replicas:​

This is determined by the required fault tolerance and data availability. The more replicas, the higher the reliability of the system, but also the higher the resource costs.

Configuring ClickHouse Keeper:​

ClickHouse Keeper is used for coordination between cluster nodes. Ensure that ClickHouse Keeper is installed and properly configured on all nodes.

Introduction to Sharding​

Sharding is a technique for dividing large volumes of data into smaller parts called shards, which can be distributed across different servers (nodes) in a cluster. Sharding is used to increase the performance and scalability of databases, including ClickHouse.

Recommendations for using sharding:

  • Large Data Volumes:

When the volume of data becomes too large to store on a single server, sharding helps distribute the data across multiple servers, reducing the load on each one.

  • Improved Performance:

Dividing data into sharded fragments allows queries to be processed in parallel on multiple servers, speeding up the execution of complex analytical queries and improving overall system performance.

  • Scalability:

New servers can be added to the cluster, and sharded data can be distributed among them without significant changes to the architecture.

  • Reduced Load on Individual Nodes:

When all data is on a single server, it can lead to overloading of the CPU, disk system, or network. Sharding helps distribute the load, increasing the stability and fault tolerance of the system.