KunlunBase FAQ

1. Is distributed transaction processing always enabled in KunlunBase? Is there a toggle to disable it?

• Distributed transactions in KunlunBase are automatically managed without any toggle to turn them on or off. A two-phase commit is only performed if a transaction writes to multiple shards; otherwise, it proceeds with a single-phase commit. This means there's no need for manual toggling.

2. How can I view standard metrics such as TPS (including rollback rates), QPS, RT, etc., in KunlunBase? Are these available in system views or on XPanel?

• In version 1.3, these metrics can be viewed on XPanel.

3. What is the locking mechanism in KunlunBase? The documentation only discusses MySQL's locking mechanisms. Similarly, what is the deadlock detection mechanism and what parameters are involved?

• On compute nodes, table locks are acquired for DML statements to obtain row-level intention locks; for DDL statements, exclusive table-level locks are acquired. On storage nodes, the mechanism is identical to MySQL: DML operations like INSERT/DELETE/UPDATE acquire table-level intention write locks at the server layer and row locks at the InnoDB layer; SELECT operations acquire global and local snapshots without transaction locks. DDL operations acquire exclusive table-level locks at both the server and engine layers.
• KunlunBase Global Deadlock Detection Technology | Klustron

4. Does KunlunBase support the standard isolation levels RR and RC?

• KunlunBase supports multiple transaction isolation levels: Read Committed, Repeatable Read, and Serializable.

5. Regarding sequences, I see they are recorded in a table in the metadata database. Could this become a performance bottleneck? Also, if caching is specified during creation, will there be issues like sequence skipping when sequences are cached locally on different compute nodes?

• When creating sequences, you can specify caching multiple values to avoid performance bottlenecks. If the cache is set to 1, each value retrieval requires modifying the sequence record in the metadata shard, which can significantly impact performance.
• If multiple compute nodes, CN1 and CN2, use the same sequence simultaneously, and each sequence caches 20 values, then when CN1 is using sequence values 100-119, CN2 will use sequence values 120-139.

6. Regarding table groups, I understand that this is a manual setting of the storage location of tables; this differs from conventional distributed databases' ER tables. Does KunlunBase support ER tables?

• Table groups ensure that multiple tables always reside on the same shard, as if they are bound together in a box. ER tables can indeed be grouped into the same table group, ensuring that these tables stay on the same shard, even during scaling operations.

7. Does KunlunBase support creating various indexes supported by PostgreSQL, such as non-BTree indexes?

• No. KunlunBase storage nodes are based on MySQL to avoid performance issues with PostgreSQL's storage engine, which also means it does not use PostgreSQL’s index types. However, MySQL supports more than just B+tree indexes; it also supports spatial and full-text indexes, so there are no data types that cannot be indexed.

8. Is there any plan to support global secondary indexes in the future?

• Global secondary indexes can severely degrade the performance of data modifications. If only used for querying, they are somewhat useless. KunlunBase's compute nodes know the location of each table's shards, so even if the query condition does not specify a shard key, the system only needs to read from the shards that store the partitions of the table, not all shards.
• However, we will decide when to implement this feature based on user demand.

9. Do views support cross-shard statements? If so, are the views decomposed and pushed down during query execution?

• Views support cross-shard statements. During the query processing transformation phase, the query statements of all views are merged into the syntax tree of the top-level query that uses the view and executed uniformly. Hence, at the query execution level, it is not apparent whether a query used a view, so there is no 'pushing down' of views as such.

10. As mentioned earlier, does composite sharding (sharding + sharding) support partitioning methods?

• KunlunBase supports multi-level partitioning, where the method and parameters of each level of partitioning are independent and do not affect each other. The only requirement is that the set of shard key columns for each level of partitioning must include the shard key columns of all the preceding levels. For more details, refer to this documentation.

11. Are cursors and custom functions supported?

• KunlunBase supports both cursors and custom functions. It supports the definition of stored procedures using PL/SQL, as well as multiple programming languages such as C, Python, Java, JavaScript, Lua, etc.

12. Regarding character sets, are common MySQL character sets like UTF8MB4 and GB18030 supported?

• Client-side use of GBK/GB18030/GB2312 is supported.
• Internally, KunlunBase supports various Unicode encodings including UTF8 for storing data, but not GBK/GB18030. However, KunlunBase can automatically convert text input in Chinese GB encoding from the client into an internally supported character encoding.

13. In cases where the source and target databases have different but compatible character sets, can migration and synchronization be achieved using tools provided by KunlunBase?

• KunlunBase supports common client character sets. As long as the source database's character encoding is one of the client character sets supported by KunlunBase (the same ones supported by PostgreSQL), the data will be automatically converted to KunlunBase’s internal character set (such as UTF8, which can be specified when creating a database) upon being inserted via SQL statements.
• Data migration tools like CDC that import data from MySQL instance binlogs into KunlunBase can successfully import the data as long as the character encoding is supported by KunlunBase.

14. Regarding backups, I see that physical backup to HDFS is supported. Is backing up to local storage (or NFS) supported? Is point-in-time recovery supported? Is object flashback supported?

• Recovery to any point in time is supported, allowing users to retrieve various database objects by restoring to specified points.

15. Are the typical MySQL and PostgreSQL import and export commands supported? Can data be quickly imported directly to data nodes without going through compute nodes?

• KunlunBase supports PostgreSQL's data import commands but does not support MySQL's LOAD DATA command. Data must be imported through compute nodes.

16. Is version upgrading currently supported? What about rolling upgrades and downgrades?

• Work is ongoing for hot, non-disruptive rolling upgrades, to be released in version 1.3. Both upgrading and downgrading are supported.

17. Regarding scalability, can compute nodes be scaled online? Is there a corresponding Load Balancer (LB) solution provided? For storage node scaling, does data need to be manually redistributed?

• Compute nodes can be added at any time during the cluster operation through the XPanel GUI interface or by calling the cluster-mgr API, without affecting any ongoing operations. Once a compute node is initialized, it can begin serving externally.
• An LB solution must be configured by the user. After the compute node is initialized, it can be added to the pool of candidate nodes in the LB.

18. In terms of read-write separation, is delay awareness supported (i.e., removing replicas with high delays from serving reads)? For cross-shard scenarios, is read consistency supported?

• Compute nodes automatically select the standby with the smallest delay for reading; global MVCC mechanisms cannot operate normally due to unpredictable primary-standby delays and only work for data reads from primary nodes.

19. Are cluster_mgr and node_mgr designed to support high availability? What happens if node_mgr encounters an issue, and how is it recovered?

• Cluster_mgr supports high availability and can automatically elect a new primary if the cluster_mgr primary goes down; node_mgr runs in each server and is automatically restarted by cron jobs or Linux system services. Since cluster_mgr and node_mgr do not maintain local persistent states, no data recovery is necessary.

20. How does the system behave during a split-brain scenario, such as cluster_mgr being split from compute and storage nodes? Have chaos tests been conducted?

• Cluster_mgr and the storage cluster's high-availability mechanisms ensure that split-brain scenarios do not occur.
• There are no direct replication relationships between multiple compute nodes in a KunlunBase cluster. These compute nodes write DDL logs into the metadata cluster's ddl log and always replicate DDL logs executed by other compute nodes to maintain consistent metadata across the cluster.
• Extensive automated chaos tests have been conducted, with daily runs in Jenkins.

21. Do compute nodes support connection pooling?

• External connection pools are supported.

22. When adding new storage nodes, how are Mirror Tables handled? Are they automatically replicated with data?

• Yes, data is automatically replicated. For more details, refer to this documentation

23. Does Xpanel feature slow query capabilities?

• Version 1.3 will graphically display slow queries and related information. Currently, data is collected in Elasticsearch and can be accessed through Kibana.

24. Do storage nodes support the National Secret Algorithm?

• Version 1.3 implements a cryptographic plugin for the National Secret Algorithm.

25. In terms of security, are SQL auditing, firewall functions, and whitelisting supported?

• Supported. These features utilize PostgreSQL's built-in functionalities.

26. Does KunlunBase fully adhere to SQL92/SQL99 standards?

• Almost fully compliant, with only a few exceptions that are rarely used in modern software development. Detailed information can be found in PostgreSQL's documentation. Additionally, KunlunBase has made selective trade-offs in PostgreSQL syntax for distributed database scenarios, which do not affect application system development. For more details, refer to: PostgreSQL syntax and features not supported by KunlunBase。

27. Does KunlunBase support CTE syntax?

• CTE, window functions, and OLAP constructs such as cube, rollup, and grouping sets are all supported.

28. Can all functions registered in PostgreSQL be directly registered in KunlunBase without adjustments?

• All system functions in PostgreSQL are present in KunlunBase and can be directly called in SQL statements, including PostgreSQL's PL/SQL stored procedures. The CREATE EXTENSION statement can be used to add various plugins from the PostgreSQL ecosystem to KunlunBase.
• Most extensions from the PostgreSQL ecosystem work in KunlunBase without any modifications. Only a few heavy extensions like PostGIS and pgvector, which have their own data types or data storage methods, require further development work for special handling.

29. Does KunlunBase support API or JDBC connections? Is there anything special about JDBC?

• Generic JDBC libraries can be used without modification. Additionally, various programming language client libraries (connectors) can connect to KunlunBase. For more details, refer to: Summary of MySQL Connectors for Major Programming Languages in KunlunBase
• KunlunBase supports both MySQL and PostgreSQL connection protocols and SQL syntax, allowing applications originally using MySQL and PostgreSQL to connect and operate with KunlunBase without modifications or recompilation.

END