Ozone on NVMe Wei-Chiu Chuang (jojochuang) Ritesh Shukla (kerneltime) Agenda • Overview of how Ozone and how it scales • Why NVME is important for Ozone for scaling • Benefits of using NVME • Impala Impala performance results from NVME clusters • Write path improvements results from NVME clusters • Summary • Questions Ozone Architecture Why does Ozone Scale? Separation of concerns Ozone Manager limitations, working set can be cached in memory and unused data can be destaged to disk • OM uses NVME to store RocksDBs • Future projects such as Snapshots leverage RocksDB to preserve simplicity of

bottlenecked by IO, meaning CPUs were constantly waiting on HDDs to respond. But when we replace HDDs with NVMe SSDs on a Dual Socket Best Practices for MySQL with SSDs Page 4 Server, the CPU becomes the Storage SAS HDD 2x SEAGATE ST600MP0005 15K rpm SATA SSD 2x Samsung 850 PRO NVMe SSD 2x Samsung XS1715 Quad‐socket (28 Core) Configuration Database Server (Client) 2x SEAGATE ST600MP0005 15K rpm SATA SSD 2x Samsung 850 Pro SAS SSD 2x Samsung PM1633 NVMe 2x Samsung PM1725 It is generally accepted that OLTP database applications have been I/O bound

1 基于SPDK的CurveBS PFS存储引擎10/17/22 2 Why ●为了减少使用cpu做内存copy，减少系统调用 ●发挥某些被操作系统屏蔽的功能，例如nvme write zero ●根据阿里《When Cloud Storage Meets RDMA》的说法 ●在100Gbps网络带宽时，内存带宽成为瓶颈 ●Intel Memory Latency Checker (MLC)测试得到的CPU内存带宽是 ●读写内存都由网卡进行offload ●应用程序不再通过系统调用在内核和用户态来回切换10/17/22 4 磁盘的读写 ●基于EXT4的存储引擎，依然需要通过系统调用来回切换 ●读写都需要CPU拷贝数据 ●不能发挥某些NVME的功能，例如write zero10/17/22 5 为什么用PFS ●对代码比较熟悉 ●找一个能管理裸盘，具有产品级可靠性的代码挺难的 ●PFS支持类POSIX文件的接口，与使用EXT4的存储引擎代码很像， ●直接DMA读写，要求的内存必须是DPDK的hugetlb内存 ●必须符合NVME 内存读写地址对齐要求 ●offset 512对齐 ●为零copy提供接口10/17/22 10 BRPC IOBuf DMA ●修改BRPC，允许使用dpdk内存作为IOBuf的内存分配器 ●BRPC接收到的数据在IOBuf中，IOBuf直接使用于NVME DMA传输 ●使用IOBuf内存读nvme，避免自己写PRP页面对齐内存分配代码10/17/22

0:87:00.0-nvme-1 - /dev/disk/by-path/pci-0000:88:00.0-nvme-1 optionalPaths: - /dev/disk/by-path/pci-0000:89:00.0-nvme-1 - /dev/disk/by-path/pci-0000:90:00.0-nvme-1 thinPoolConfig: 000:87:00.0-nvme-1 - /dev/disk/by-path/pci-0000:88:00.0-nvme-1 optionalPaths: - /dev/disk/by-path/pci-0000:89:00.0-nvme-1 - /dev/disk/by-path/pci-0000:90:00.0-nvme-1 thinPoolConfig: name: thin-pool-1 sizePercent: 90 overprovisionRatio:

主机之间的通信。 3.3.1. 通过 OpenStack 使用 PCI 透传向 etcd 节点提供存储 要为 etcd 节点提供快速存储以便 etcd 在大规模稳定，使用 PCI 透传将非易失性内存表达(NVMe)设备直 接传递给 etcd 节点。要使用 Red Hat OpenStack 11 或更高版本进行设置，请在存在 PCI 设备的 OpenStack 节点上完成以下内容。 1. 确定在 BIOS all_filters [pci] alias = { "vendor_id":"144d", "product_id":"a820", "device_type":"type-PCI", "name":"nvme" } 6. 在控制器上重启 nova-api 和 nova-scheduler。 7. 在 /etc/nova/nova.conf 中的计算节点上： [pci] passthrough_whitelist "product_id":"a820", "device_type":"type-PCI", "name":"nvme" } 要检索您想要透传的 NVMe 设备的 address、vendor_id 和 product_id 值，请运行： 8. 重启计算节点上的 nova-compute。 9. 将您正在运行的 OpenStack 版本配置为使用 NVMe 并启动 etcd 节点。 3.4. 使用 TUNED 配置集扩展主机

Acceleration Kit (HSAK) improves the I/O performance of NVMe devices. The software library implements an I/O software stack of high-performance NVMe devices. It is advanced for its user mode, asynchronization lock-free, and polling features. Compared with the NVMe device I/O software stack in the native Linux kernel, the HSAK greatly reduces the latency of NVMe commands and improves the I/O processing capability capability (IOPS) of a single CPU. Challenges Evolving storage media such as NVMe solid state drives (SSDs) and storage class memory (SCM) have resulted in decreasing access latencies caused by the media

. . . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

. . . 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe . . . . . . 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) . . . . . . . . . . . . . . . . . . . Logic and Bus- Logic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special: