impact to p99/p999 RPC latency, HBase 2.x has made an offheap read and write path. The KV are allocated from the JVM offheap, and the memory area in offheap won’t be garbage collected by JVM, so upstreams explicitly by themself. On the write path, the request packet received from client will be allocated offheap and retained until those key values are successfully written to the WAL log and Memstore. The C reference to cells, which are encoded in multiple Chunks in MSLAB, this is easier to manage the offheap memory. Similarly, on the read path, we’ll try to read the BucketCache firstly, if the Cache misses

buffer. ● But JVM will still sweep the buffer and compact them. (old generation) BucketCache + OffHeap Before GC (G1) bucket-4KB (2MB) bucket-8KB (2MB) bucket-16KB (2MB) bucket-32KB (2MB) Memory allocated shorter STW time. End-to-end offheap on the read-path (HBASE-11425) BucketCache StoreFileScanner Copy the Block from BucketCache(offheap) to onheap. Rpc Handler offheap onheap BucketCache StoreFileScanner len) which reference to the offheap block. Rpc Handler offheap onheap Use the DirectByteBuffer in all the read path. branch-1.5 branch-2 ● No need to copy block from offheap to onheap. ● Less onheap

block.cache.size = 0.1 ○ hbase.hregion.memstore.flush.size = 256 MB ○ hbase.bucketcache.ioengine = offheap Test Environment RegionServer RegionServer RegionServer RegionServer RegionServer HMaster tuning ? ○ IHOP > MaxMemstoreSize%Heap + L1CacheSize%Heap + Delta(~ 10%) ○ Bucket Cache is offheap, need NO consideration when tuning IHOP ● Next Tuning ○ MemstoreSize = 45% , L1CacheSize ~ 10% of your survivorRatio. G1GC In XiaoMi Cluster 12G + 12G 12G + 12G 12G + 12G 50G heap + 50G offheap CMS to G1 12G + 12G ● Less Full GC，Better availability ● Less region servers. One Node

into spark-defaults.conf: spark.plugins=io.glutenproject.GlutenPlugin spark.memory.offHeap.size=20g spark.memory.offHeap.enabled=true spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager

into spark-defaults.conf: spark.plugins=io.glutenproject.GlutenPlugin spark.memory.offHeap.size=20g spark.memory.offHeap.enabled=true spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleMana