Go runtime related problems in TiDB production environment About me ● Arthur Mao(毛康力), Senior Engineer@PingCAP ● TiDB core developer (top3 contributor) ● GitBook about golang internals (@tiancaiamao) IO is ready => goroutine wake up == 4.3ms ○ Sometime even 10ms+ latency here! ○ The time spend on runtime schedule is not negligible ● When CPU is overload, which goroutine should be given priority? Analysis longer to be scheduled ● The runtime scheduling does not consider priority ● CPU dense workload could affect IO latency Conclusion Part II - Memory control ● Go Runtime ○ Allocated from OS (mmaped)

浅谈全链路监控： 从应用到数据库到 Runtime 黄东旭, Co-founder & CTO, PingCAP 关于我 黄东旭，联合创始人 & CTO @ PingCAP 做分布式数据库的程序员 ● 现在能写代码的时间是奢侈品 TiDB 的亲爹之一兼首席客服和新功能的第一个用户 ● 冤有头债有主，SQL 慢了来找我。。。 偶尔玩玩音乐 ● 摇滚乐->实验音乐 Go 的粉丝！！！！ tool trace go tool trace ● 优点：好用，好看（UI） ● 缺点：性能损耗太大，不能一直开着 Trace in Go runtime ● go tool trace 的原理是？ Trace 会 Go Runtime 的代码中打桩收集 CPU time，在 Goroutine 开始执行时记录 start_run_time, 在调度退出执行时记录 end_run_time，累加 goroutine 的 CPU time。 A little bit about Go runtime https://learnku.com/articles/41728 https://github.com/golang/go/blob/ma ster/src/runtime/trace.go hack runtime 的思路： follow the tracing event. PingCAP

Log................................................................................252 16.4.7. Runtime Statistics ....................................................................................254 ...................................................................1131 E.1.4.1. Performance Improvements........................................................1131 E.1.4.2. Server Changes ......... Operation Changes ..........................................................1154 E.8.3.2. Performance Improvements........................................................1155 E.8.3.3. Server Configuration Changes

...................................................................................266 16.4.7. Runtime Statistics ..................................................................................... .............................................................1209 xxii E.1.4.1. Performance Improvements...............................................................1209 E.1.4.2. Server Changes .. Changes .................................................................1233 E.8.3.2. Performance Improvements...............................................................1235 E.8.3.3. Server Configuration

.................................................................2468 E.171.3.1. Performance Improvements....................................................2468 E.171.3.2. Server Changes ........... .................................................................2509 E.195.3.1. Performance Improvements....................................................2509 E.195.3.2. Server Changes ........... .................................................................2548 E.222.4.1. Performance Improvements....................................................2548 E.222.4.2. Server Changes ...........

.................................................................2616 E.171.3.1. Performance Improvements...........................................................2617 E.171.3.2. Server Changes .... ............................................................2660 lii E.195.3.1. Performance Improvements...........................................................2660 E.195.3.2. Server Changes .... .................................................................2702 E.222.4.1. Performance Improvements...........................................................2702 E.222.4.2. Server Changes ....

.................................................................2310 E.145.3.1. Performance Improvements....................................................2310 E.145.3.2. Server Changes ........... .................................................................2351 E.169.3.1. Performance Improvements....................................................2351 E.169.3.2. Server Changes ........... .................................................................2390 E.196.4.1. Performance Improvements....................................................2390 E.196.4.2. Server Changes ...........

.................................................................2456 E.145.3.1. Performance Improvements...........................................................2457 E.145.3.2. Server Changes .... .................................................................2500 E.169.3.1. Performance Improvements...........................................................2500 E.169.3.2. Server Changes .... .............................................................2542 li E.196.4.1. Performance Improvements...........................................................2542 E.196.4.2. Server Changes ....

..................................................................1328 E.12.3.1. Performance Improvements.............................................................1328 E.12.3.2. Server Changes ... ..................................................................1359 E.28.4.1. Performance Improvements.............................................................1359 E.28.4.2. Server Changes ... Changes ...............................................................1393 E.48.3.2. Performance Improvements.............................................................1394 E.48.3.3. Server Configuration

...................................................................1307 E.1.3.1. Performance Improvements...............................................................1307 E.1.3.2. Server Changes .. ...................................................................1326 E.6.4.1. Performance Improvements...............................................................1327 E.6.4.2. Server Changes .. Changes ...............................................................1353 E.16.3.2. Performance Improvements.............................................................1354 E.16.3.3. Server Configuration