Klustron cluster_mgr Cluster Management

1 Background

Klustron's cluster_mgr serves as the cluster management system for Klustron, providing high availability guarantee for the Klustron cluster (i.e. fullsync HA), offering cluster management services in the form of APIs (cluster installation and uninstallation, node startup and shutdown, adding and removing computing nodes and storage nodes, adding and removing storage shards, physical and logical backup and restore, scaling, repartitioning, and monitoring and restarting cluster nodes), as well as automatic background services that handle partial fault recovery when node failures occur during cluster transaction processing.

These functions require cluster_mgr to have real-time and high availability characteristics, ensuring that cluster_mgr continuously provides cluster management services to ensure that Klustron cluster continuously provides services to the outside world without interruption.

To achieve this, the Klustron team has developed the cluster_mgr cluster high availability feature based on braft. This feature not only ensures that the cluster_mgr cluster automatically selects a new master node to provide all cluster_mgr services when the cluster_mgr cluster master node goes down, but also allows users (DBAs) to replace failed machines and adjust the number of cluster_mgr cluster nodes without stopping the cluster_mgr service.

Currently, the cluster_mgr cluster provides manual master-slave switch, adding & deleting cluster_mgr nodes, etc.

2 Implementation Principles

2.1 Manual leader-follower switch of cluster_mgr cluster

When a business sends an HTTP request to cluster_mgr, the request parameters are parsed and verified, and then the underlying braft API interface is called to initiate a transfer_leader command. The request is forwarded to the current leader node through the braft rpc interface, and the leader node performs operations according to the request parameters.

2.1.1 If the target node to be switched to is the current leader node, return success directly

2.2.2 If the target node to be switched to is not the current leader node, send an rpc command to the target node to load the braft log at the recorded braft log loading position in the current braft state machine, and set the timeout. If the specified position is loaded, return success. The current leader node is automatically degraded to a follower and the target node is promoted to a leader.

2.2 Adding nodes to the cluster_mgr cluster

When a business sends an HTTP request to cluster_mgr, the request parameters are parsed and verified, and then the underlying raft API interface is called to initiate an add_peer command. The request is forwarded to the current leader node through the braft rpc interface, and the leader node performs operations according to the request parameters.

2.2.1 Connect to the braft port of the node to be added through RPC.

2.2.2 Add the IP:port information of the node to be added to the braft configuration.

2.2.3 Synchronize data with the node to be added through snapshot+ incremental raft log

2.3 Deleting nodes from the cluster_mgr cluster

When a business sends an HTTP request to cluster_mgr, the request parameters are parsed and verified, and then the underlying braft API interface is called to initiate a remove_peer command. The request is forwarded to the current leader node through the braft rpc interface, and the leader node performs operations according to the request parameters.

2.3.1 Connect to the braft port of the node to be deleted through rpc

2.3.2 Delete the ip:port information of the node to be deleted from the braft configuration

3 Configuration and Usage

In order to facilitate business operations, cluster_mgr API interface is used for cluster management.

3.1 Manual leader-follower switch

curl -d '

{

  "version":"1.0",

  "job_id":"",

  "job_type":"raft_mission",

  "timestamp":"1435749309",

  "user_name":"kunlun_test",

  "paras":{

    "task_type":"transfer_leader",

    "target_leader":"127.0.0.1:59001"

  }

}

' -X POST http://127.0.0.1:58000/HttpService/Emit

This interface is an asynchronous interface, returning a job_id, and querying the task execution status according to the returned job_id.

curl -d '

{

  "version":"1.0",

  "job_id":"10",

  "job_type":"get_status",

  "timestamp":"1435749309",

  "user_name":"kunlun_test",

  "paras":{

  }

}

' -X POST http://127.0.0.1:58000/HttpService/Emit

3.2 Adding nodes

To add a new node to the Klustron cluster, the user needs to first configure the cluster_mgr.cnf file on the new node machine. The raft_group_member_init_config needs to include the ip:port of each machine in the current cluster as well as the ip:port of the new machine. Then, start the cluster_mgr process on the new node. The new node's cluster_mgr will keep trying to become the leader, but since the other nodes in the running cluster do not have information about the new node's cluster_mgr, they will reject its requests. The user needs to call the cluster_mgr add node API to add the new node to the running cluster.

curl -d '

{

  "version":"1.0",

  "job_id":"",

  "job_type":"raft_mission",

  "timestamp":"1435749309",

  "user_name":"kunlun_test",

  "paras":{

    "task_type":"add_peer",

    "peer":"127.0.0.2:59001"

  }

}

' -X POST http://127.0.0.1:58000/HttpService/Emit

This interface is an asynchronous interface, returning job_id, and querying the task execution status according to the returned job_id.

curl -d '

{

  "version":"1.0",

  "job_id":"10",

  "job_type":"get_status",

  "timestamp":"1435749309",

  "user_name":"kunlun_test",

  "paras":{

  }

}

' -X POST http://127.0.0.1:58000/HttpService/Emit

3.3 Deleting nodes

After the user initiates the deletion of the node, the deleted node is no longer in the current cluster, and the cluster_mgr process can be directly stopped to take the machine offline.

curl -d '

{

  "version":"1.0",

  "job_id":"",

  "job_type":"raft_mission",

  "timestamp":"1435749309",

  "user_name":"kunlun_test",

  "paras":{

    "task_type":"remove_peer",

    "peer":"127.0.0.2:59001"

  }

}

' -X POST http://127.0.0.1:58000/HttpService/Emit

This interface is an asynchronous interface, returning job_id, and querying the task execution status according to the returned job_id.

curl -d '

{

  "version":"1.0",

  "job_id":"10",

  "job_type":"get_status",

  "timestamp":"1435749309",

  "user_name":"kunlun_test",

  "paras":{

  }

}

' -X POST http://127.0.0.1:58000/HttpService/Emit

Special Emphasis:

Adding/deleting nodes, cluster_mgr only involves the update of braft memory configuration. The raft_group_member_init_config configuration item in the cluster_mgr configuration file is not updated, and needs to be manually modified by logging in to the machine. It is not necessary to restart the cluster_mgr process. When the cluster_mgr is restarted, it will automatically load the modified configuration file.