transparent huge pages ● GC Related ■ case study: GC sweep caused latency jitter ■ case study: Lock and NUMA aware Agenda Part I - Latency in scheduler ● The client consists of a goroutine and a channel ○ Go GC ● Unexpected latency jitter caused by GC may still exist Conclusion Case study: Lock and NUMA aware ● TiDB server CPU usage is around 60% ● Networking and IO are both OK ● Increase the benchmark columns ● The deployment does not consider NUMA ○ CPU usage is higher when binding CPUs to NUMA nodes Reproduce ● Context Switch, Minor Page Fault, and NUMA Hit Miss are related Reproduce Reproduce

automatic scheduling integration occurs, that is, Kubernetes is not aware of the underlying vSphere topology (sites, affinity groups, NUMA, etc.). This session will explain the options to gain better performance improve scheduling Using vSphere tags to define regions and zones – add cloud provider What is NUMA? How to solve potential issues with CPU and memory intensive workloads Kubernetes default resource the nodes themselves within vSphere failure domains. 11 What is NUMA? Non Uniform Memory Architecture 12 Why should you care about NUMA? Memory intensive workloads Nearly all database servers (e.g. Oracle

TiUP 1.5.0 及以上 注意： 中控机需要部署TiUP 软件来完成 TiDB 集群运维管理。 5.1.2.2 目标主机建议配置软件 软件 版本 sshpass 1.06 及以上 numa 2.0.12 及以上 tar 任意 5.1.3 服务器建议配置 TiDB 支持部署和运行在 Intel x86-64 架构的 64 位通用硬件服务器平台或者 ARM 架构的硬件服务器平台。对于开 少，设置得过小，可能会导致内存的申请速度超过后台的回收速度，进而导致内存 回收并引起内存分配延迟。 • 建议将 vm.min_free_kbytes 最小设置为 1048576 KiB（即 1 GiB）。如果安装了 NUMA， 建议设置为 NUMA 节点个数 * 1048576 KiB。 • 对于内存小于 16 GiB 的小规格服务器，保持 vm.min_free_kbytes 的默认值即可。 • tcp_tw_recycle 1 tidb]# 5.2.8 安装 numactl 工具 本段主要介绍如何安装 NUMA 工具。在生产环境中，因为硬件机器配置往往高于需求，为了更合理规划资源， 会考虑单机多实例部署 TiDB 或者 TiKV。NUMA 绑核工具的使用，主要为了防止 CPU 资源的争抢，引发性能衰退。 注意： • NUMA 绑核是用来隔离 CPU 资源的一种方法，适合高配置物理机环境部署多实例使用。 • 通过

1.5.0 及以上 注意： 中控机需要部署TiUP 软件来完成 TiDB 集群运维管理。 5.1.2.2 目标主机建议配置软件 522 软件 版本 sshpass 1.06 及以上 numa 2.0.12 及以上 tar 任意 5.1.3 服务器建议配置 TiDB 支持部署和运行在 Intel x86-64 架构的 64 位通用硬件服务器平台或者 ARM 架构的硬件服务器平台。对于开 后台的回收速度，进而导致内存 回收并引起内存分配延迟。 537 • 建议将 vm.min_free_kbytes 最小设置为 1048576 KiB（即 1 GiB）。如果安装了 NUMA， 建议设置为 NUMA 节点个数 * 1048576 KiB。 • 对于内存小于 16 GiB 的小规格服务器，保持 vm.min_free_kbytes 的默认值即可。 • tcp_tw_recycle 1 tidb]# 5.2.8 安装 numactl 工具 本段主要介绍如何安装 NUMA 工具。在生产环境中，因为硬件机器配置往往高于需求，为了更合理规划资源， 会考虑单机多实例部署 TiDB 或者 TiKV。NUMA 绑核工具的使用，主要为了防止 CPU 资源的争抢，引发性能衰退。 注意： • NUMA 绑核是用来隔离 CPU 资源的一种方法，适合高配置物理机环境部署多实例使用。 • 通过

TiUP 1.5.0 及以上 注意： 中控机需要部署TiUP 软件来完成 TiDB 集群运维管理。 5.1.2.2 目标主机建议配置软件 软件 版本 sshpass 1.06 及以上 numa 2.0.12 及以上 tar 任意 526 5.1.3 服务器建议配置 TiDB 支持部署和运行在 Intel x86-64 架构的 64 位通用硬件服务器平台或者 ARM 架构的硬件服务器平台。对于开 少，设置得过小，可能会导致内存的申请速度超过后台的回收速度，进而导致内存 回收并引起内存分配延迟。 • 建议将 vm.min_free_kbytes 最小设置为 1048576 KiB（即 1 GiB）。如果安装了 NUMA， 建议设置为 NUMA 节点个数 * 1048576 KiB。 • 对于内存小于 16 GiB 的小规格服务器，保持 vm.min_free_kbytes 的默认值即可。 • tcp_tw_recycle tidb]# 542 5.2.8 安装 numactl 工具 本段主要介绍如何安装 NUMA 工具。在生产环境中，因为硬件机器配置往往高于需求，为了更合理规划资源， 会考虑单机多实例部署 TiDB 或者 TiKV。NUMA 绑核工具的使用，主要为了防止 CPU 资源的争抢，引发性能衰退。 注意： • NUMA 绑核是用来隔离 CPU 资源的一种方法，适合高配置物理机环境部署多实例使用。 • 通过

N: the feature is not supported. • E: the feature is not GA yet (experimental) and you need to be aware of the usage limitations. Experimental features are subject to change or removal without prior notice index on columns that you do not query often. A useful secondary index can speed up queries, but be aware that it also has side effects. Each time you add an index, an additional Key-Value is added when you operate and manage TiDB clusters. 5.1.2.2 Target machines Software Version sshpass 1.06 or later numa 2.0.12 or later tar any 5.1.3 Server recommendations You can deploy and run TiDB on the 64-bit generic

N: the feature is not supported. • E: the feature is not GA yet (experimental) and you need to be aware of the usage limitations. Experimental features are subject to change or removal without prior notice index on columns that you do not query often. A useful secondary index can speed up queries, but be aware that it also has side effects. Each time you add an index, an additional Key-Value is added when you operate and manage TiDB clusters. 5.1.2.2 Target machines Software Version sshpass 1.06 or later numa 2.0.12 or later tar any 5.1.3 Server recommendations You can deploy and run TiDB on the 64-bit generic

N: the feature is not supported. • E: the feature is not GA yet (experimental) and you need to be aware of the usage limitations. Experimental features are subject to change or removal without prior notice index on columns that you do not query often. A useful secondary index can speed up queries, but be aware that it also has side effects. Each time you add an index, an additional Key-Value is added when you operate and manage TiDB clusters. 5.1.2.2 Target machines Software Version sshpass 1.06 or later numa 2.0.12 or later tar any 5.1.3 Server recommendations You can deploy and run TiDB on the 64-bit generic

N: the feature is not supported. • E: the feature is not GA yet (experimental) and you need to be aware of the usage limitations. Experimental features are subject to change or removal without prior notice index on columns that you do not query often. A useful secondary index can speed up queries, but be aware that it also has side effects. Each time you add an index, an additional Key-Value is added when you operate and manage TiDB clusters. 5.1.2.2 Target machines Software Version sshpass 1.06 or later numa 2.0.12 or later 700 Software Version tar any 5.1.3 Server recommendations You can deploy and

N: the feature is not supported. • E: the feature is not GA yet (experimental) and you need to be aware of the usage limitations. Experimental features are subject to change or removal without prior notice index on columns that you do not query often. A useful secondary index can speed up queries, but be aware that it also has side effects. Each time you add an index, an additional Key-Value is added when you operate and manage TiDB clusters. 695 5.1.2.2 Target machines Software Version sshpass 1.06 or later numa 2.0.12 or later tar any 5.1.3 Server recommendations You can deploy and run TiDB on the 64-bit generic