database ○ Schrodinger-test platform ○ Failpoint injection ○ Goroutine-leak detection ● Optimization ○ Chunk vs interface{} ○ Vectorized execution TiDB Overview TiDB TiDB TiDB Application via MySQL Protocol goroutines func TestT(t *testing.T) { testleak.BeforeTest() TestingT(t) testleak.AfterTestT(t)() } Chunk - Effective row format in Go Row format in TiDB CREATE TABLE `t` ( `a` int(11) DEFAULT NULL, buffer ● Values Array ● More details please see the doc Chunk ● Columnar layout ● Fixed length type can eliminate the offsets buffer. type Chunk struct { columns []*column } type column struct { length

Mongos Config Server Chunk (100, 200)  Shard1 Chunk (200, 300)  Shard2 Chunk (300, 400)  Shard3 …. 2016Postgres中国用户大会 Postgres Conference China 2016 中国用户大会 分片方式-范围 范围 所在分片 Chunk1 [minKey, ‐75) Shard2 Shard2 Chunk2 [‐75, 25) Shard1 Chunk3 [25, 175) Shard3 Chunk4 [175, MaxKey] Shard1 • 根据某个字段的值，顺序 划分为多个范围，每个范 围对应一个 Shard，能很 好的支持范围查询 2016Postgres中国用户大会 Postgres Conference China 2016 中国用户大会 分片方式-hash Shard Shard1 Client Client Shard2 Shard2 Shard3 Mongos Config Server Chunk (100, 200)  Shard1 Chunk (200, 300)  Shard2 Chunk (300, 400)  Shard3 …. Shard4 2016Postgres中国用户大会 Postgres Conference

threads are used to export the data. • -F 256MB: means a table is partitioned into chunks and one chunk is 256 MB. After executing this command, the full backup data is exported to the /data/ �→ my_database threads are used to export the data. • -F 256MB: means a table is partitioned into chunks and one chunk is 256 MB. If the data source consists of CSV files, see CSV support for configuration. 11.8.3.4 into �→ multiple chunks to process in ### parallel. max-region-size is the maximum size of each chunk after �→ splitting. ### max-region-size = "256MiB" # default value ### Only import tables if these

threads are used to export the data. • -F 256MB: means a table is partitioned into chunks and one chunk is 256 MB. After executing this command, the full backup data is exported to the /data/ �→ my_database threads are used to export the data. • -F 256MB: means a table is partitioned into chunks and one chunk is 256 MB. If the data source consists of CSV files, see CSV support for configuration. 11.8.3.4 into �→ multiple chunks to process in ### parallel. max-region-size is the maximum size of each chunk after �→ splitting. ### max-region-size = "256MiB" # default value ### Only import tables if these

速度， 取决于 每个表 的复杂 性 Import speed upload to tikv 从 TiKV Importer 上传 SST 文件到 所有 TiKV 副本的 总体速 度 Chunk process duration 完全编 码单个 数据文 件所需 的平均 时间 有时导入速度会降到 0，这是为了平衡其他部分的速度，属于正常现象。 11.7.6.2.2 第二行：进度面板 engines �→ 的设 置 11.7.6.2.5 第五行：读取速度面板 图 131: 第五行读取速度面板 面板名称 序列 描述 Chunk parser read block duration read block 读取一个字节块来准备解析时所消耗的时间 Chunk parser read block duration apply worker 等待 io-concurrency 空闲所消耗的时间 • tikv_import_write_chunk_bytes（直方图） 从 TiDB Lightning 接收的键值对区块大小（未压缩）的直方图。 • tikv_import_write_chunk_duration（直方图） 从 tidb-lightning 接收每个键值对区块所需时间的直方图。 • tikv_import_upload_chunk_bytes（直方图） 上传到 TiKV

速度， 取决于 每个表 的复杂 性 Import speed upload to tikv 从 TiKV Importer 上传 SST 文件到 所有 TiKV 副本的 总体速 度 Chunk process duration 完全编 码单个 数据文 件所需 的平均 时间 有时导入速度会降到 0，这是为了平衡其他部分的速度，属于正常现象。 11.7.6.2.2 第二行：进度面板 engines �→ 的设 置 11.7.6.2.5 第五行：读取速度面板 图 130: 第五行读取速度面板 面板名称 序列 描述 Chunk parser read block duration read block 读取一个字节块来准备解析时所消耗的时间 Chunk parser read block duration apply worker 等待 io-concurrency 空闲所消耗的时间 • tikv_import_write_chunk_bytes（直方图） 从 TiDB Lightning 接收的键值对区块大小（未压缩）的直方图。 • tikv_import_write_chunk_duration（直方图） 从 tidb-lightning 接收每个键值对区块所需时间的直方图。 • tikv_import_upload_chunk_bytes（直方图） 上传到 TiKV

old MySQL versions #5017 @lyzx2001 2.2.5 Bug fixes • TiDB • Fix the issue of memory chunk misuse for the chunk reuse feature that occurs in some cases #38917 @keeplearning20221 • Fix the issue that processing unit performs the precheck and �→ exports data from the upstream database (4 by default) chunk-filesize: 64 # The size of the files generated by the �→ dump processing unit (64 MB by default) table using multi-thread. The value of rows is the maximum number of rows contained in each exported chunk. After this option is enabled, DM selects a column as the split benchmark when the data of a MySQL

processing unit performs the precheck and �→ exports data from the upstream database (4 by default) chunk-filesize: 64 # The size of the files generated by the �→ dump processing unit (64 MB by default) table using multi-thread. The value of rows is the maximum number of rows contained in each exported chunk. After this option is enabled, DM selects a column as the split benchmark when the data of a MySQL this value as needed. chunk-filesize During full backup, DM splits the data of each table into multiple chunks according to the value of the chunk-filesize option. Each chunk is saved in a file with

threads are used to export the data. • -F 256MB: means a table is partitioned into chunks and one chunk is 256 MB. After executing this command, the full backup data is exported to the /data/ �→ my_database threads are used to export the data. • -F 256MB: means a table is partitioned into chunks and one chunk is 256 MB. If the data source consists of CSV files, see CSV support for configuration. 11.8.3.4 into �→ multiple chunks to process in ### parallel. max-region-size is the maximum size of each chunk after �→ splitting. ### max-region-size = "256MiB" # default value ### Only import tables if these

table expressions (CTEs) re- port runtime error: index out of range [32] with length 32 when tidb_max_chunk_size is set to a small value #48808 @guo-shaoge • Fix the issue of Goroutine leak when using AUTO_ID_CACHE=1 processing unit performs the precheck and �→ exports data from the upstream database (4 by default) chunk-filesize: 64 # The size of the files generated by the �→ dump processing unit (64 MB by default) table using multi-thread. The value of rows is the maximum number of rows contained in each exported chunk. After this option is enabled, DM selects a column as the split benchmark when the data of a MySQL