impact to p99/p999 RPC latency, HBase 2.x has made an offheap read and write path. The KV are allocated from the JVM offheap, and the memory area in offheap won’t be garbage collected by JVM, so upstreams explicitly by themself. On the write path, the request packet received from client will be allocated offheap and retained until those key values are successfully written to the WAL log and Memstore. The C reference to cells, which are encoded in multiple Chunks in MSLAB, this is easier to manage the offheap memory. Similarly, on the read path, we’ll try to read the BucketCache firstly, if the Cache misses

buffer. ● But JVM will still sweep the buffer and compact them. (old generation) BucketCache + OffHeap Before GC (G1) bucket-4KB (2MB) bucket-8KB (2MB) bucket-16KB (2MB) bucket-32KB (2MB) Memory allocated shorter STW time. End-to-end offheap on the read-path (HBASE-11425) BucketCache StoreFileScanner Copy the Block from BucketCache(offheap) to onheap. Rpc Handler offheap onheap BucketCache StoreFileScanner len) which reference to the offheap block. Rpc Handler offheap onheap Use the DirectByteBuffer in all the read path. branch-1.5 branch-2 ● No need to copy block from offheap to onheap. ● Less onheap

block.cache.size = 0.1 ○ hbase.hregion.memstore.flush.size = 256 MB ○ hbase.bucketcache.ioengine = offheap Test Environment RegionServer RegionServer RegionServer RegionServer RegionServer HMaster tuning ? ○ IHOP > MaxMemstoreSize%Heap + L1CacheSize%Heap + Delta(~ 10%) ○ Bucket Cache is offheap, need NO consideration when tuning IHOP ● Next Tuning ○ MemstoreSize = 45% , L1CacheSize ~ 10% of your survivorRatio. G1GC In XiaoMi Cluster 12G + 12G 12G + 12G 12G + 12G 50G heap + 50G offheap CMS to G1 12G + 12G ● Less Full GC，Better availability ● Less region servers. One Node

性能调优-集群调优化 集群参数设置 1.memtable_allocation_type heap_buffers: on heap nio buffer offheap_buffers: off heap(direct) nio buffers offheap_objects: native memory 2.concurrent_write和concurrent_read 3.Sstable compression