impact to p99/p999 RPC latency, HBase 2.x has made an offheap read and write path. The KV are allocated from the JVM offheap, and the memory area in offheap won’t be garbage collected by JVM, so upstreams explicitly by themself. On the write path, the request packet received from client will be allocated offheap and retained until those key values are successfully written to the WAL log and Memstore. The C reference to cells, which are encoded in multiple Chunks in MSLAB, this is easier to manage the offheap memory. Similarly, on the read path, we’ll try to read the BucketCache firstly, if the Cache misses

buffer. ● But JVM will still sweep the buffer and compact them. (old generation) BucketCache + OffHeap Before GC (G1) bucket-4KB (2MB) bucket-8KB (2MB) bucket-16KB (2MB) bucket-32KB (2MB) Memory allocated shorter STW time. End-to-end offheap on the read-path (HBASE-11425) BucketCache StoreFileScanner Copy the Block from BucketCache(offheap) to onheap. Rpc Handler offheap onheap BucketCache StoreFileScanner len) which reference to the offheap block. Rpc Handler offheap onheap Use the DirectByteBuffer in all the read path. branch-1.5 branch-2 ● No need to copy block from offheap to onheap. ● Less onheap

block.cache.size = 0.1 ○ hbase.hregion.memstore.flush.size = 256 MB ○ hbase.bucketcache.ioengine = offheap Test Environment RegionServer RegionServer RegionServer RegionServer RegionServer HMaster tuning ? ○ IHOP > MaxMemstoreSize%Heap + L1CacheSize%Heap + Delta(~ 10%) ○ Bucket Cache is offheap, need NO consideration when tuning IHOP ● Next Tuning ○ MemstoreSize = 45% , L1CacheSize ~ 10% of your survivorRatio. G1GC In XiaoMi Cluster 12G + 12G 12G + 12G 12G + 12G 50G heap + 50G offheap CMS to G1 12G + 12G ● Less Full GC，Better availability ● Less region servers. One Node

coding.net/p/gonggongbanji/files/all/DF46 CPU 使⽤太⾼了-GC 使⽤ CPU 过⾼ https://github.com/glycerine/offheap • 可以将变化较少的结构放在堆外 • 通过 cgo 来管理内存 • 让 GC 发现不了这些对象 • 也就不会扫描了 Off heap 也可以减少 Go 进程的内存占⽤和内存使 ⽤波动，但要⽤到 CPU 过⾼ • 减少堆上对象分配 • sync.Pool 进⾏堆对象重⽤ • Map -> slice • 指针 -> ⾮指针对象 • 多个⼩对象 -> 合并为⼀个⼤对象 • offheap • 降低 GC 频率 • 修改 GOGC • Make 全局⼤ slice • CPU 使⽤太⾼ • 调度相关的函数使⽤ CPU 过⾼ • 注意 automaxprocs • 尝试使⽤ 对象复⽤ • 为不同⼤⼩的对象提供不同⼤⼩ level 的 sync.Pool • 松散的对象调整为紧凑的对象，如：map -> slice，查找时间复杂度增加，但内存使⽤ 降低 • offheap • 邪道：动态调整 GOGC • Goroutine 栈占⽤过多内存 • 减少 goroutine 数量 • 如每个连接⼀读⼀写 -> 合并为⼀个连接⼀个 goroutine • Goroutine

性能调优-集群调优化 集群参数设置 1.memtable_allocation_type heap_buffers: on heap nio buffer offheap_buffers: off heap(direct) nio buffers offheap_objects: native memory 2.concurrent_write和concurrent_read 3.Sstable compression

into spark-defaults.conf: spark.plugins=io.glutenproject.GlutenPlugin spark.memory.offHeap.size=20g spark.memory.offHeap.enabled=true spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager

into spark-defaults.conf: spark.plugins=io.glutenproject.GlutenPlugin spark.memory.offHeap.size=20g spark.memory.offHeap.enabled=true spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleMana