Skip to main content

Klustron ClusterManager API

KlustronAbout 16 min

Klustron ClusterManager API

0. Introduction to use

KunlunClusterManager accepts HTTP protocol requests and performs corresponding tasks according to the request type and parameters. Interface response modes are divided into synchronous and asynchronous modes. In the synchronous mode, after the client sends a request, the response received includes the execution result of the current interface (if the cluster information is obtained, etc.). Asynchronous mode is usually for tasks that execute for a long time. After the client sends the request, it needs to obtain the execution status of the task through the get_status interface, and guide the task until it succeeds or fails.

0.1 Issue tasks

Interface Description

Deliver cluster management tasks.

request parameters

parameter nameParameter TypeIs it required?Remark
versionstring1Custom protocol version, default '1.0'
job_idstring0To deliver a task, you don't need to fill in the job_id, which will be automatically generated by the clustermanager and returned through the response.
job_typestring1Task type, related to specific task content
timestampstring1The timestamp when the task was initiated
bestjson0The custom parameter section is related to the task type.

response parameter

parameter nameParameter TypeIs it required?Remark
versionstring1Custom protocol version, default '1.0'
error_codestring1error code
error_infostring1error message
statusstring1Task status, possible values are accepted, started, ongoing, done, failed
job_idstring0The job_id automatically generated by clustermanager, if the task type is an asynchronous task, the job_id can be used to query the task progress and status.

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"expand_cluster",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "dst_shard_id":"2",
        "src_shard_id":"1",
        "table_list":[
            "sbtest.public.test1"
        ]
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

0.2 Get task execution status

Interface Description

Get the execution status of the specified job_id task.

request parameters

parameter nameParameter TypeIs it required?Remark
versionstring1Custom protocol version, default '1.0'
job_idstring1The id of the task to be queried
job_typestring1task type, current interface 'get_status'
timestampstring1The timestamp when the task was initiated
bestjson0In the custom parameter part, the get_status type interface does not need to be filled in, and other task types may need to be filled in.

response parameter

parameter nameParameter TypeIs it required?Remark
versionstring1Custom protocol version, default '1.0'
error_codestring1error code
error_infostring1error message
statusstring1Task status, possible values are started, ongoing, done, failed

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"expand_cluster",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "dst_shard_id":"2",
        "src_shard_id":"1",
        "table_list":[
            "sbtest.public.test1"
        ]
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

1. Device management

1.1 Reporting machinery and equipment

Interface Description

Register the initialized server device into the system.

answer mode

asynchronous. The status is queried through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1create_machine
host addrstring1Server IP
machine_typestring1Equipment type. Device types include two types of storage and computer. Represents storage type and computer type respectively. The storage model indicates that this device can be used to deploy Klustron-storage. Similarly, the computer type means that the device can be used to deploy Klustron-server. The same physical device can be a storage device or a computing device, or both a storage device and a computing device. In this case, the port range needs to be clearly distinguished to ensure that computing nodes and storage nodes installed on the same device Node will not conflict with ports.
port_rangestring1Port allocation range. That is, the newly created power port is allocated within the range specified by this parameter
rack_idstring1rack information
datadirstring1Instance data directory storage location
logdirstring1Log directory storage location
wal_log_dirstring1wal log storage location
comp_datadirstring1If the current device is a computer type, this field identifies the data directory of the computing node
total_memstring1The total available memory of the model is in MB
total_cpu_coresstring1The total number of logical cores available for this model

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"create_machine",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "hostaddr":"192.168.0.135",
        "machine_type":"computer",
        "port_range":"45000-48000",
        "rack_id":"1",
        "datadir":"/home/play_ground/run_data",
        "logdir":"/home/play_ground/run_data",
        "wal_log_dir":"/home/play_ground/run_data",
        "comp_datadir":"/home/play_ground/run_data",
        "total_mem":"8192",
        "total_cpu_cores":"8"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回

{
    "version":"1.0",
    "error_code":"0",
    "error_info":"create machine successfully",
    "status":"done",
    "attachment":{

    }
}

1.2 Delete device

Interface Description

Perform the delete operation on the servers that have been registered to the system.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1delete_machine
host addrstring1machine ip
machine_typestring1The model information of the device storage or computer

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"delete_machine",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "hostaddr":"127.0.0.1",
        "machine_type":"storage"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"delete machine successfully",
    "status":"done",
    "attachment":{

    }
}

1.3 Set server status

Interface Description

Set the status of the specified server, such as dead and running, which respectively indicate that the current device is offline and online. Only online devices can be divided into resource pools by the system.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1delete_machine
host addrstring1machine ip
node_statsstring1running, dead

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"set_machine",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "hostaddr":"127.0.0.1",
        "node_stats":"running"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"set machine success",
    "status":"done",
    "attachment":{

    }
}

2. Cluster purchase and deletion

2.1 Purchase a new cluster

Interface Description

Create a new Klustron cluster according to the specified parameter configuration.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1create_cluster
nick_namestring1Business name, such as 'reconciliation interest settlement database'
ha_modestring1rbr, mgr. Identifies the high availability mode. The former is strong synchronous mode, and the latter is mgr mode.
shardsstring1number of fragments
nodesstring1The number of nodes in each shard, at least 3 nodes, forming a high-availability cluster with one master and two slaves
compsstring1Calculate the number of nodes
max_storage_sizestring1Maximum storage space
max_connectionsstring1Maximum number of connections
cpu_coresstring1The maximum number of cores that can be used
cpu_limit_nodestring1share identifies the soft isolation mode; quota identifies the hard isolation mode;
innodb_sizestring1The innodb_buffer size used by Klustron-storage (the unit of this field innodb_size: M)
rocksdb_block_cache_size_Mstring1The block cache size for rocksdb used by Klustron-storage (the unit of this field: M), version>=1.1
dbcfgstring0Set the small memory mode, 0 and no field are normal mode, 1 is small memory mode, the value is 0 or 1.
fullsync_levelstring1The number of strong synchronization standby machines in the shard should be less than or equal to the number of standby machine nodes
storage_iplistsJson array1Install the storage node machine for the specified machine, if not specified, the existing machine is selected by default
computer_iplistsJson array1Install the computing node machine for the specified, if not specified, the existing machine is selected by default
install_proxysqlstring0Whether to install proxysql when installing the cluster, if proxysql is installed, the cluster is in single shard mode, the default is 0

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"create_cluster",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "nick_name":"my_nick_name",
        "ha_mode":"mgr",
        "shards":"1",
        "nodes":"3",
        "comps":"1",
        "max_storage_size":"20",
        "max_connections":"6",
        "cpu_cores":"8",
        "innodb_size":"1024",
	"rocksdb_block_cache_size_M":"1024",
        "dbcfg":"1",
        "fullsync_level":"1",
        "install_proxysql":"0",
        "storage_iplists":[
            "192.168.0.2",
            "192.168.0.3"
        ],
        "computer_iplists":[
            "192.168.0.1"
        ]
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"create cluster successfully",
    "status":"done",
    "attachment":{
        "cluster_id":"3",
        "cluster_name":"cluster_1651903646_000001",
        "shards":"1",
        "comps":"1"
    }
}

2.2 New fragmentation in the cluster

Interface Description

According to the specified parameter configuration, add new fragments on the existing cluster, and you can add more than one.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1add_shards
cluster_idstring1cluster id
shardsstring1Number of new shards
nodesstring1In the new shard, the number of nodes in each shard
storage_iplistsJson array1Install the storage node machine for the specified

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"add_shards",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "shards":"2",
  "nodes":"3",

        "storage_iplists":[
            "192.168.0.2",
            "192.168.0.3"
        ]
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"add shards successfully",
    "status":"done",
    "attachment":{
        "cluster_id":"2",
        "cluster_name":"cluster_1651836141_000001",
        "shards":"2",
        "comps":"1",
        "list_shard":[
            {
                "shard_id":"3",
                "shard_name":"shard2"
            }
        ]
    }
}

2.3 Delete existing shards in the cluster

Interface Description

According to the specified parameter configuration, on the existing cluster, the fragments can only be deleted one by one.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1delete_shards
cluster_idstring1cluster id
shard_idstring1Shard id to be deleted

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"delete_shard",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "shard_id":"1"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"delete shard successfully",
    "status":"done",
    "attachment":{

    }
}

2.4 Adding new computing nodes in the cluster

Interface Description

According to the specified parameter configuration, add new fragments on the existing cluster, and you can add more than one.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1add_comps
cluster_idstring1cluster id
compsstring1The number of newly added computing nodes
computer_iplistsJson array1Specify the machine where the compute node is installed

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"add_comps",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "comps":"2",
        "computer_iplists":[
            "192.168.0.2",
            "192.168.0.3"
        ]
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"add comps successfully",
    "status":"done",
    "attachment":{
        "cluster_id":"3",
        "cluster_name":"cluster_1651903646_000001",
        "shards":"1",
        "comps":"2",
        "list_comp":[
            {
                "comp_id":"3",
                "comp_name":"comp2"
            }
        ]
    }
}

2.5 Deleting computing nodes in the cluster

Interface Description

According to the specified parameter configuration, on the existing cluster, computing nodes can only be deleted one by one.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1delete_shards
cluster_idstring1cluster id
comp_idstring1Compute node id to be deleted

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"delete_comp",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "comp_id":"1"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"delete comp successfully",
    "status":"done",
    "attachment":{

    }
}

2.6 Add node nodes to the specified cluster

Interface Description

According to the specified parameter configuration, add nodes for all fragments of the cluster, asynchronously, and need to be queried.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1add_nodes
cluster_idstring1cluster id
shard_idstring0If this field is specified, only the node will be added on the shard
nodesstring1increase the number of nodes
storage_iplistsJson array1Install the storage node machine for the specified

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"add_nodes",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "shard_id":"1",
        "nodes":"2",
        "storage_iplists":[
            "192.168.0.2",
            "192.168.0.3"
        ]
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"add new nodes successfully",
    "status":"done",
    "attachment":{
        "cluster_id":"1",
        "cluster_name":"cluster_1651907758_000001",
        "shards":"1",
        "comps":"1",
        "list_shard":[
            {
                "shard_id":"1",
                "shard_name":"shard1",
                "nodes":"5",
                "list_node":[
                    {
                        "hostaddr":"192.168.0.127",
                        "node_id":"4",
                        "port":"58509"
                    },
                    {
                        "hostaddr":"192.168.0.127",
                        "node_id":"5",
                        "port":"58512"
                    }
                ]
            }
        ]
    }
}

2.7 Specify the cluster to delete the node node

Interface Description

According to the specified parameter configuration, delete the specified node for the specified fragment of the cluster, asynchronously, and need to be queried.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1delete_nodes
cluster_idstring1cluster id
shard_idstring0If this field is specified, only the node will be added on the shard
host addrstring1The IP of the machine where the node to be deleted is located
portstring1The port number

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"delete_node",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "shard_id":"1",
        "hostaddr":"127.0.0.1",
        "port":"57338"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"delete node successfully",
    "status":"done",
    "attachment":{

    }
}

2.8 Redo Standby

Interface Description

Specifies to redo the standby machine of a shard.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1delete_nodes
cluster_idstring1cluster id
shard_idstring0If this field is specified, only the node will be added on the shard
host addrstring1The IP of the machine where the node to be deleted is located
portstring1The port number
need_backupstring1Do you need to make a backup before redoing
hdfs_hoststring1Standby location
pv_limitstring1The speed limit unit for cold standby transmission is MB
allow_pull_from_masterstring1Whether to use the master's data to redo the standby machine when the standby machine is unavailable
allow_replica_delaystring1The maximum value of the standby machine delay. If the standby machine delay exceeds this value, the standby machine is considered unsuitable as a data source

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"rebuild_node",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "shard_id":"1",
        "cluster_id":"1",
        "rb_nodes":[
            {
                "hostaddr":"192.168.0.125",
                "port":"57001",
                "need_backup":"0",
                "hdfs_host":"hdfs",
                "pv_limit":"10"
            }
        ],
        "allow_pull_from_master":"1",
        "allow_replica_delay":"15"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
    "version":"1.0",
    "error_code":"0",
    "error_info":"delete node successfully",
    "status":"done",
    "attachment":{

    }
}

2.9 Delete a specified cluster

Interface Description

According to the specified parameter configuration, delete the specified node for the specified fragment of the cluster, asynchronously, and need to be queried.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1delete_cluster
cluster_idstring1cluster id

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"delete_cluster",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

#查询接口get_status返回
{
"attachment":  {
"cluster_id": "43" ,
"cluster_name": "cluster_1665971623_000043" ,
"computer_id": "776" ,
"computer_state": "done" ,
"computer_step":  [
 {
"cluster_id": "43" ,
"computer_hosts": "192.168.0.132_53704;" ,
"computer_id": "778" ,
"computer_state": "done" ,
"error_code": 0 ,
"error_info": "OK" ,
"rollback_flag": "1"
}
],
"job_steps": "shard,computer" ,
"shard_id": "775" ,
"shard_step":  [
 {
"cluster_id": "43" ,
"error_code": 0 ,
"error_info": "OK" ,
"rollback_flag": "1" ,
"shard_ids": "83," ,
"storage_hosts": "192.168.0.132_53407;192.168.0.132_53411;" ,
"storage_id": "777" ,
"storage_state": "done"
}
],
"storage_state": "done"
},
"error_code": "0" ,
"error_info": "OK" ,
"job_id": "" ,
"job_type": "" ,
"status": "done" ,
"version": "1.0"
}

2.10 Modify instance resource isolation parameters

Interface Description

According to the specified parameter configuration, modify the resource isolation parameters of the MySQL or PostgreSQL instance. Currently, only cpu isolation is supported.

answer mode

asynchronous. The status is obtained through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1update_instance_cgroup
ipstring1instance ip
Portstring1instance listening port
typestring1mysql identifies the storage node instance, pg identifies the computing node instance
cpu_coresstring1The number of allocated cpu cores
cgroup_modestring1The mode of cpu resource limitation, the default is quota, you can set quota or share, where share indicates soft isolation, and quota indicates hard isolation.

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"update_instance_cgroup",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "ip":"192.168.0.135",
        "port":"57001",
        "type":"mysql",
        "cpu_cores":"5",
        "cgroup_mode":"quota"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

3. Cluster backup

3.1 Manually initiate cluster backup

Interface Description

Call this interface to manually initiate a full backup. When cluster_id is -1, it means backup metadata cluster.

answer mode

synchronous interface.

request parameters

parameter nameParameter TypeIs it required?Remark
cluster_idstring1Cluster ID

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"manual_backup_cluster",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

4. Cluster recovery

4.1 Cluster rollback (restore cluster by point in time)

Interface Description

Roll back the specified cluster by time point.

answer mode

asynchronous. Obtain task progress through the get_status interface.

request parameters

parameter nameParameter TypeIs it required?Remark
src_cluster_idstring1The source cluster ID of the rollback operation
dst_cluster_idstring1The target cluster ID of the rollback operation. The cluster must be a newly purchased empty cluster
restore_timestring1Rollback time

example

#请求
{
    "version":"1.0",
    "job_id":"",
    "job_type":"cluster_restore",
    "timestamp":"1435749309",
    "paras":{
      "src_cluster_id":"1",
        "dst_cluster_id":"2",
        "restore_time":"restore_time"
    }
} -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

5. Cluster expansion

5.1 Automatically obtain table relocation list

Interface Description

Obtain the list of tables to be relocated automatically selected by the system.

answer mode

Synchronize.

request parameters

parameter nameParameter TypeIs it required?Remark
cluster_idstring1Cluster ID
shard_idstring1The source shard id of the table to be migrated out in the current cluster
policystring1top_hit: Select the table with the highest access frequency. top_size: Select the table with the largest amount of data

response parameter

parameter nameParameter TypeIs it required?Remark
attachment->table_liststringList of tables to be migrated

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"get_expand_table_list",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"3",
        "shard_id":"2",
        "policy":"top_size"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":{
        "table_list":"kunlundb_super_dba.kunlundb_super_dba_$$_public.test1"
    },
    "error_code":"0",
    "error_info":"",
    "job_id":"",
    "job_type":"",
    "status":"done",
    "version":"1.0"
}

5.2 Initiate table relocation task

Interface Description

Initiate the table relocation task.

answer mode

asynchronous.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1expand_cluster
cluster_idstring1Cluster ID
src_shard_idstring1The source shard id of the table to be migrated out in the current cluster
dst_shard_idstring1The target shard id of the table to be migrated out in the current cluster
table_listjson_array1List of tables to be migrated
drop_old_tablestring0Whether to keep the source table

example

#请求
curl -d '
{
  "version": "1.0",
  "job_id":"",
  "job_type": "expand_cluster",
  "timestamp" : "1435749309",
  "user_name":"kunlun_test",
  "paras": {
    "cluster_id": "1",
    "dst_shard_id": "5",
    "src_shard_id": "6",
    "table_list": [
      "postgres.public.test2"
    ]
  }
} ' -X POST http://192.168.0.135:8010/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

6. Metadata Management

6.1 Get metadata cluster synchronization mode

Interface Description

Query metadata cluster synchronization mode.

answer mode

Synchronize.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1get_meta_summary

response parameter

parameter nameParameter TypeIs it required?Remark
attachment->modestringMetadata cluster synchronization mode no_rep, mgr, rbr three types of results

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"get_meta_mode",
    "timestamp":"1435749309",
    "paras":{

    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":{
        "mode":"mgr"
    },
    "error_code":"0",
    "error_info":"",
    "job_id":"",
    "job_type":"",
    "status":"done",
    "version":"1.0"
}

6.2 Obtain cold standby storage service information

Interface Description

Query the details of the cold standby storage service.

answer mode

Synchronize.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1get_backup_storage

response parameter

parameter nameParameter TypeIs it required?Remark
host addrstringBackup service IP
namestringbackup service name
portstringBackup service port
supportstringbackup service type

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"get_backup_storage",
    "timestamp":"1435749309",
    "paras":{

    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":{
        "list_backup_storage":[
            {
                "hostaddr":"192.168.0.135",
                "name":"hdfs_backup1",
                "port":"9000",
                "stype":"HDFS"
            }
        ]
    },
    "error_code":"0",
    "error_info":"",
    "job_id":"",
    "job_type":"",
    "status":"done",
    "version":"1.0"
}

6.3 Get machine status information

Interface Description

Query detailed information on the status of all machines.

answer mode

Synchronize.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1get_machine_summary

response parameter

parameter nameParameter TypeIs it required?Remark
host addrstringserver address
statusstringserver status

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"get_machine_summary",
    "timestamp":"1435749309",
    "paras":{

    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":{
        "list_machine":[
            {
                "hostaddr":"192.168.0.135",
                "status":"online"
            },
            {
                "hostaddr":"192.168.0.125",
                "status":"online"
            }
        ]
    },
    "error_code":"0",
    "error_info":"",
    "job_id":"",
    "job_type":"",
    "status":"done",
    "version":"1.0"
}

6.4 Get the cluster list

Interface Description

Query the list of all Klustron clusters.

answer mode

Synchronize.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1get_cluster_summary

response parameter

parameter nameParameter TypeIs it required?Remark
namestringcluster name
nick_namestringbusiness name
shardsstringNumber of fragments
compsstringCalculate the number of nodes

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"get_machine_summary",
    "timestamp":"1435749309",
    "paras":{

    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":{
        "list_machine":[
            {
                "hostaddr":"192.168.0.135",
                "status":"online"
            },
            {
                "hostaddr":"192.168.0.125",
                "status":"online"
            }
        ]
    },
    "error_code":"0",
    "error_info":"",
    "job_id":"",
    "job_type":"",
    "status":"done",
    "version":"1.0"
}

6.5 Get cluster details

Interface Description

Get detailed information about the specified cluster.

answer mode

Synchronize.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1get_cluster_detail
cluster_namestring1cluster name

response parameter

parameter nameParameter TypeIs it required?Remark
namestringcluster name
nick_namestringbusiness name
shardsstringNumber of fragments
compsstringCalculate the number of nodes
comp_namestringcompute node name
host addrstringCompute node or storage node IP
portstringCompute node or storage node port
statusstringcurrent node status
masterstringWhether the current storage node is the master node

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"get_cluster_detail",
    "timestamp":"1435749309",
    "paras":{
        "cluster_name":"cluster_1659333382_000001"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":{
        "cluster_name":"cluster_1659333382_000001",
        "comps":"1",
        "list_comp":[
            {
                "comp_name":"comp_1",
                "hostaddr":"192.168.0.135",
                "port":"45001",
                "status":"online"
            }
        ],
        "list_shard":[
            {
                "list_node":[
                    {
                        "hostaddr":"192.168.0.135",
                        "master":"true",
                        "port":"55001",
                        "status":"online"
                    },
                    {
                        "hostaddr":"192.168.0.135",
                        "master":"false",
                        "port":"55002",
                        "status":"online"
                    },
                    {
                        "hostaddr":"192.168.0.135",
                        "master":"false",
                        "port":"55003",
                        "status":"online"
                    }
                ],
                "nodes":"3",
                "shard_name":"shard_1"
            }
        ],
        "nick_name":"my_nick_name",
        "shards":"1"
    },
    "error_code":"0",
    "error_info":"",
    "job_id":"",
    "job_type":"",
    "status":"done",
    "version":"1.0"
}

6.6 Get storage node system variable information

Interface Description

Query a variable value of Klustron-storage (such as innodb_buffer_pool_size).

answer mode

Synchronize.

request parameters

parameter nameParameter TypeIs it required?Remark
job_typestring1get_variable
host addrstring1Klustron-storage ip
portstring1kunlun Storage port
variablestring1variable name

response parameter

parameter nameParameter TypeIs it required?Remark
resultstringWhether the query was successful
valuestringsearch result

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"get_variable",
    "timestamp":"1435749309",
    "paras":{
        "hostaddr":"192.168.0.135",
        "port":"55001",
        "variable":"innodb_buffer_pool_size"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":{
        "result":"true",
        "value":"5242880"
    },
    "error_code":"0",
    "error_info":"",
    "job_id":"",
    "job_type":"",
    "status":"done",
    "version":"1.0"
}

7. Table weight distribution

Interface Description

According to the user input table, dump the data in the table and import it into the new table

answer mode

asynchronous.

request parameters

parameter nameParameter TypeIs it required?Remark
src_cluster_idstring1The cluster_id where the original table is located
dst_shard_idstring1The target table is redistributed to the cluster_id of the target
distribution_tablesstring1The target table is redistributed to the cluster_id of the target

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"repartition_tables",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "src_cluster_id":"3",
        "dst_cluster_id":"2",
        "repartition_tables":"test_$$_public.t=>test1_$$_private.t2,test_$$_priv.ta=>test1_$$_priv1.tb"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

Special Note:

  1. The database, schema, and table need to be established under the target cluster in advance, and the original and target table structures are consistent. This cluster_mgr will automatically verify
  2. The tables required for one redistribution table task must be placed in the same database, and the original and target databases can be different

8. Table logical backup

Interface Description

The user initiates a logical backup of a table and sets a scheduled logical backup

answer mode

asynchronous.

request parameters

parameter nameParameter TypeIs it required?Remark
cluster_idstring1The cluster_id where the logical backup table is located
db_tablestring1Tables to be logically backed up
backup_timestring1Scheduled backup period

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"logical_backup",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "cluster_id":"1",
        "backup":[
          {
            "db_table":"postgres_$$_public.t1",
            "backup_time":"01:00:00-02:00:00"
          },
          {
            "db_table":"postgres_$$_public.t2",
            "backup_time":"01:00:00-02:00:00"
          }
        ]

    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

9. Table logic recovery

Interface Description

The user initiates the recovery of a logical backup table to a specified historical point in time

answer mode

asynchronous.

request parameters

parameter nameParameter TypeIs it required?Remark
src_cluster_idstring1The cluster_id where the backup table is located
dst_cluster_idstring1The cluster_id to which the logical recovery table should be
db_tablestring1the table to restore logically
restore_timestring1The historical point in time to restore to

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"logical_restore",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "src_cluster_id":"1",
        "dst_cluster_id":"2",
    "db_table":"postgres_$$_public.t1",
    "restore_time":"2022-10-20 19:20:15"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

10. Manually trigger cluster_mgr active/standby switchover

Interface Description

The user initiates an active-standby switchover operation for the current cluster_mgr cluster

answer mode

asynchronous.

request parameters

parameter nameParameter TypeIs it required?Remark
task_typestring1Task type, this place is transfer_leader
target_leaderstring1target new master to switch

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"raft_mission",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "task_type":"transfer_leader",
        "target_leader":"127.0.0.1:25001"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

11. Add nodes to cluster_mgr

Interface Description

The user initiates the operation of adding nodes to the current cluster_mgr cluster

answer mode

asynchronous.

request parameters

parameter nameParameter TypeIs it required?Remark
task_typestring1Task type, this place is add_peer
peerstring1To add cluster_mgr node

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"raft_mission",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "task_type":"add_peer",
        "peer":"127.0.0.1:25001"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

12. Delete node for cluster_mgr

Interface Description

The user initiates a node deletion operation on the current cluster_mgr cluster

answer mode

asynchronous.

request parameters

parameter nameParameter TypeIs it required?Remark
task_typestring1Task type, this place is remove_peer
peerstring1The cluster_mgr node to delete

example

#请求
curl -d '
{
    "version":"1.0",
    "job_id":"",
    "job_type":"raft_mission",
    "timestamp":"1435749309",
    "user_name":"kunlun_test",
    "paras":{
        "task_type":"remove_peer",
        "peer":"127.0.0.1:25001"
    }
}
' -X POST http://192.168.0.135:35001/HttpService/Emit

#应答
{
    "attachment":null,
    "error_code":"0",
    "error_info":"",
    "job_id":"35",
    "status":"accept",
    "version":"1.0"
}

With particular emphasis on

To add/delete nodes for the cluster_mgr cluster, you need to manually update the raft_group_member_init_config configuration item in the cluster_mgr configuration file, and update the configuration of adding/deleting nodes

END

Copyright © Klustron
Copyright © 2025 Klustron