Introduction to Klustron Cluster RCR Functionality and Usage Examples

Background

Klustron Distributed Database supports high availability for cluster deployments across the same city or across different cities. It allows users to establish a Remote Cluster Replication (RCR) relationship between a replica Klustron cluster (replica cluster) in a secondary data center and the primary Klustron cluster (primary cluster) in the main data center. This combined setup of primary and replica clusters is referred to as the Primary-Replica cluster. The RCR ensures real-time synchronization of data updates at the Klustron cluster level.

1 Functionality Overview

In the event of an entire main data center becoming unavailable, the user's business systems can automatically switch to the replica cluster in a different data center and continue providing services. If the secondary data center is in the same city, the DBA can configure synchronous replication between the primary and replica clusters, ensuring that no data is lost during the switch. In cases where the replica cluster is in a remote data center with higher network latency, it's common to set up asynchronous replication for the Primary-Replica clusters. This means that when switching to the Klustron replica cluster after the main data center fails, there might be a possibility of losing some of the most recent data updates.

2 Deployment Architecture

The Klustron Enterprise Edition's cluster_mgr component is responsible for creating and maintaining the Remote Cluster Replication (RCR) relationship. The cluster_mgr in Klustron provides API interfaces, and the XPanel offers a GUI for users to manage and maintain the RCR replication relationship. DBAs need to create a replica cluster in a secondary data center and configure the RCR replication relationship using the RCR configuration management functionality. The user's business system needs to monitor the availability of the primary cluster or data center and automatically switch to the replica cluster when the primary cluster becomes unavailable.

The primary and replica clusters must have their own metadata clusters and cluster_mgr components; they cannot be shared. This is mainly due to potential network latency between the primary and replica clusters, which could lead to performance degradation if shared.

Klustron RCR requires that the primary and replica clusters have the same number of storage shards (referred to as "shards"). The RCR component of cluster_mgr establishes replication relationships for each pair of shards between the primary and replica clusters. This ensures that each shard in the replica cluster can replicate data updates from the corresponding shard in the primary cluster.

The replica cluster can be initialized with either the full physical backup data or the full logical backup data from the primary cluster.

Klustron RCR functionality ensures that in the event of storage node failures in either the primary or replica cluster, leading to partial or complete loss of RCR connections, the connections are automatically restored, and data updates are not lost.

3 Implementation Principles and Function Usage

3.1 Klustron Cluster Data Synchronization Involves Two Parts.

User Data Synchronization Between Shards

This is achieved using binlog replication for primary-replica synchronization. Fullsync technology, a Klustron-specific feature, is used for synchronous replication.

Metadata Synchronization for Klustron Clusters

Metadata synchronization is necessary because computing nodes need to replicate and execute DDL statements to have the same metadata information as the source cluster. The distribution information of databases and tables cannot be directly used and needs to be transformed and mapped based on the target cluster's configuration.

This synchronization is accomplished using the binlog_sync tool. Binlog_sync connects to the metadata cluster where the primary cluster resides through binlog_dump. It captures updates to the DDL log system table in the target cluster. When changes in the binlog are captured, binlog_sync converts the DDL SQL statements from the binlog records and writes them into the metadata table of the replica cluster after necessary transformations.

Currently, the necessary transformations include mapping of shard ID information. Mapping is required because the shard IDs in the target cluster might not necessarily match the corresponding shard IDs in the source cluster.

3.2 Implementation and Usage

3.2.1 Establishing RCR Data Synchronization

The cluster_mgr checks whether the number of shard nodes, configuration parameters (parameters include the RCR cluster_id for both primary and replica clusters, and the IP:port of the corresponding metadata cluster), etc., are consistent between the primary and replica clusters. If they are not consistent, it returns a failure. If they are consistent, binlog synchronization links are established between the shards of the primary and replica clusters. To ensure data consistency between the primary and replica RCR shards, cluster_mgr cleans the data in each node of the replica RCR shard and resynchronizes the data from the corresponding primary RCR shard node. When establishing the RCR relationship, cluster_mgr automatically matches each shard of the primary and replica RCR clusters. Cluster_mgr establishes data synchronization by pairing the primary node of each shard in the primary RCR cluster with a low-latency replica node in the corresponding replica RCR shard. After establishing data-level synchronization, cluster_mgr automatically starts the binlog_sync tool to synchronize real-time metadata updates from the primary Klustron cluster. After transformation, these updates are synchronized to the DDL log table in the metadata cluster of the replica cluster. The computing nodes of the replica cluster then execute the necessary DDL statements.

To ensure data consistency, Klustron RCR sets each primary node of the replica cluster's shards to read-only, preventing accidental data writes that could lead to inconsistency between the replica and primary clusters.

Special note: When establishing RCR, cluster_mgr automatically clears the data in the replica RCR shards.

3.2.2 Removing RCR Data Synchronization

3.2.3 Manual Switching of RCR Relationship

cluster_mgr verifies the input parameters (parameters include the RCR cluster_id for both primary and replica clusters, and the IP:port of the corresponding metadata cluster) to check if they are valid. If the input parameters do not match the RCR cluster_id and metadata cluster information recorded in cluster_mgr, the task fails. Once the validation is successful, cluster_mgr checks the synchronization delay between the primary and replica RCR clusters. If the delay exceeds the configured value (5 seconds), the task fails. cluster_mgr sets the primary nodes of the primary RCR cluster's shards to read-only to prevent further data writes. It then connects to the primary nodes of the replica RCR cluster's shards and checks if the primary and replica data synchronization is completed (checks GTID). This phase has a timeout of 10 seconds, and if it times out, the task rolls back and fails. The Klustron cluster metadata synchronization relationship is disconnected. The original data synchronization links between the primary and replica RCR shards are broken, and the data synchronization links are reversed—originally primary becomes replica, and original replica becomes primary. The primary nodes of the original replica RCR cluster's shards are set to read-write, while the primary nodes of the original primary RCR cluster's shards remain read-only. Based on the new synchronization relationship between the shards, cluster_mgr restarts the binlog_sync tool to establish Klustron cluster metadata synchronization.

Manual switching scenarios:

Changes in business deployment, switching from the primary RCR to the replica RCR location. Manual RCR switching can be initiated directly. Since manual RCR switching ensures data consistency, data loss is not a concern.
Disaster recovery drills.

3.2.4 Configuring RCR Synchronization Relationship

Currently, pausing RCR primary-replica data synchronization, resuming RCR primary-replica data synchronization, and setting data delay replication between the primary and replica RCR clusters are supported.

3.2.5 Cluster Topology Changes

If any cluster in the RCR relationship adds/deletes a shard, cluster_mgr automatically adds/deletes the shard in the peer cluster_mgr based on the RCR relationship. Currently, cluster_mgr automatically creates an add/delete shard task and sends it to the peer cluster_mgr, waiting for the execution status of the task. If the execution fails, the user-initiated add/delete shard task fails.

Adding a shard: After cluster_mgr has established shards for both the primary and replica RCR, it initiates an RCR relationship for the newly added shard.
Deleting a shard: cluster_mgr first breaks the RCR synchronization relationship for the shard being deleted and then initiates the deletion task for the primary and replica RCR shards.

4 Configuration and Usage

4.1 Using the cluster_mgr API

4.1.1 Establishing an RCR Relationship

curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"create_rcr",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "master_info": {
            "meta_db":"127.0.0.1:1001, 127.0.0.2:1001, 127.0.0.3:1001",
            "cluster_id":"2"
        },
        "cluster_id":"3"
    }
}
' -X POST http://127.0.0.1:58000/HttpService/Emit

4.1.2 Deleting an RCR Relationship

curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"delete_rcr",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "master_info": {
            "meta_db":"127.0.0.1:1001, 127.0.0.2:1001, 127.0.0.3:1001",
            "cluster_id":"2"
        },
        "cluster_id":"3"
    }
}
' -X POST http://127.0.0.1:58000/HttpService/Emit

4.1.3 Manual RCR Failover, Supported from the cluster_mgr in the Primary and Replica RCR

Initiated from the cluster_mgr in the replica RCR

curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"manualsw_rcr",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "master_info": {
            "meta_db":"127.0.0.1:1001, 127.0.0.2:1001, 127.0.0.3:1001",
            "cluster_id":"2"
        },
        "cluster_id":"3"
    }
}
' -X POST http://127.0.0.1:58000/HttpService/Emit

Initiated from the cluster_mgr in the Primary RCR

curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"manualsw_rcr",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "slave_info": {
            "meta_db":"127.0.0.1:1001, 127.0.0.2:1001, 127.0.0.3:1001",
            "cluster_id":"2"
        },
        "cluster_id":"3"
    }
}
' -X POST http://127.0.0.1:58000/HttpService/Emit

4.1.4 Configuring Primary and Replica RCR Synchronization, Supported from the cluster_mgr in the Primary and Replica RCR

Initiated from the cluster_mgr in the replica RCR

curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"manualsw_rcr",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "master_info": {
            "meta_db":"127.0.0.1:1001, 127.0.0.2:1001, 127.0.0.3:1001",
            "cluster_id":"2"
        },
        "cluster_id":"3",
        "work_mode":"modify_sync_delay|stop_rcr|start_rcr",
        "sync_delay":"10" //如果是modify_sync_delay，则需要配置sync_delay
    }
}
' -X POST http://127.0.0.1:58000/HttpService/Emit

Initiated from the cluster_mgr in the Primary RCR

curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"manualsw_rcr",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "slave_info": {
            "meta_db":"127.0.0.1:1001, 127.0.0.2:1001, 127.0.0.3:1001",
            "cluster_id":"2"
        },
        "cluster_id":"3"
         "work_mode":"modify_sync_delay|stop_rcr|start_rcr",
        "sync_delay":"10" //如果是modify_sync_delay，则需要配置sync_delay
    }
}
' -X POST http://127.0.0.1:58000/HttpService/Emit

4.2 Using the XPanel Interface

As XPanel supports managing multiple Klustron clusters, you should first report the metadata information of the Klustron cluster for which you want to establish an RCR relationship, as shown in the following figures.