As the distributed block storage component of Arcfra Enterprise Cloud Platform (AECP), Arcfra Block Storage (ABS) introduces Boost mode to enhance the I/O link and optimize storage performance. This article will explore this feature further and compare AECP’s performance with and without enabling Boost mode.

How Boost Mode Works

In general, ABS Boost mode leverages vhost-user protocol to share memory between Guest OS, QEMU, and ABS, simplifying the I/O link and accelerating the processing of I/O requests.

Why vHost Matters

Currently, there are two mainstream I/O device virtualization solutions:

• QEMU–pure software emulation: Provide pure-software-simulated I/O devices to VMs.
• Virtio-paravirtualization: The front-end and back-end are standardized to an official Virtio specification in this scheme. The front-end driver (Virtio driver) is used in the VM (Guest OS) while the back-end device (Virtio device) is used in the hypervisor (QEMU) to provide I/O capabilities. I/O performance is improved by reducing the number of memory copies and VM traps. This scheme requires the installation of the Virtio driver.

Benefiting from Vring’s I/O communication mechanism, Virtio effectively reduces I/O latency and performs better than QEMU pure software emulation. This is why Virtio has become a prevalent paravirtualization solution for I/O devices across vendors.

In QEMU, the Virtio device is a PCI/PCIe device emulated for the Guest OS, which complies with PCI standards and features configuration space and interrupt functions. Notably, Virtio registered the PCI vendor ID (0x1AF4) and device IDs. Each device ID represents a device type. For example, the device ID for storage Virtio-BLK is 0x1001 and Virtio-SCSI is 0x1004.

Virtio consists of three parts: 1) the front-end driver layer, which is integrated into the Guest OS; 2) Virtqueue as the middle part, which is responsible for data transmission and command interaction; and 3) the back-end device layer, which processes the requests sent by the Guest OS.

Normally, in AECP, the entire I/O link consists of seven steps:

1. Guest Initiates an I/O operation. The Virtio driver (Guest’s kernel) writes to the PCI configuration space to trigger VM EXIT and returns to the Host KVM (to notify the KVM).
2. The QEMU’s vCPU thread returns to QEMU from the KVM kernel mode. Then QEMU device processes the request from Virtio Vring.
3. QEMU initiates storage connection (iSCSI over TCP to localhost) through iSCSI Drive.
4. The request is connected to the iSCSI Target provided by the storage process through the Socket.
5. The storage engine receives the request and performs I/O processing.
6. The storage engine initiates the I/O processing to the local storage media.
7. After the I/O operation is complete, it is returned to the Virtio back-end device through the same path. Then QEMU sends an interrupt notification to the simulated PCI and Guest terminates the entire I/O process.

However, this process lacks efficiency. As QEMU connects the storage process through local Sockets, dataflows switch between user mode to kernel mode, causing data replication overhead. Besides, the iSCSI protocol layer also causes unnecessary processing consumption. If the storage process can directly receive and process local I/O, the performance reduction can be avoided, and the implementation of Virtio Offload to Storage Software can be achieved.

Vhost is a technology derived from Virtio standards to optimize I/O paths and improve the I/O performance of paravirtual Virtio devices in QEMU.

How vHost Works

As mentioned above, the Virtio back-end device is used to handle Guest requests and I/O responses. The solution that puts the I/O processing module outside the QEMU process is called vhost. Since we aim to optimize the performance between two user-mode processes (in AECP), we use vhost-user scheme for storage acceleration. As in the vhost-kernel scheme, I/O load is processed in the kernel, we do not discuss this scheme in this article.

The data plane processing of vhost-user is mainly divided into primary and secondary. Generally, QEMU is the primary which is supplier of Virtqueue, and storage software is the secondary which is responsible for consuming I/O requests in the Virtqueue.

Advantages of the vhost-user include:

• Eliminate the space for Guest kernel updating the PCI configuration, and the CPU context overhead caused by QEMU capturing the Guest’s VM traps (back-end processing threads poll all Virtqueues).
• User-mode memory sharing between processes optimizes data replication efficiency (zero-copy).

With vhost-user, the I/O link only takes five steps:

1. After the Guest initiates an I/O operation, the storage software detects new requests through the Polling mechanism and obtains data from the Virtqueue.
2. The storage engine receives requests and performs I/O processing.
3. The storage engine initiates the I/O processing to the local storage media.
4. After the I/O operation is complete, the vhost device sends an irqfd (eventfd) notification to KVM.
5. KVM interrupts data injection and notifies Guest OS to terminate I/O processing.

Note: Control information is transmitted between front-end vhost driver and back-end vhost device through the UNIX Domain Socket files.

AECP Performance: Boost Mode On vs. Off

We tested AECP’s benchmark performance with and without enabling Boost mode based on a three-node cluster (using two replicas).

Configuration

• CPU: 2 * 32 cores Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
• Memory: 187 GiB
• Cache: 2 * Intel SSDPF2KE032T1 (P5620), 2320 GiB
• Storage NIC: Mellanox Technologies MT27800 Family [ConnectX-5]

Test Result

* Higher IOPS/bandwidth and lower latency represent better performance.

As we can see from the test data, AECP clusters show higher performance and lower latency after enabling Boost mode in all performance tests, especially in read I/O scenarios.

Conclusion

Leveraging vhost-user, Boost mode optimizes I/O request processing and data transfer, improving AECP VM performance and reducing I/O latency. Users can enable Boost mode to support high-performance applications such as databases.

For more information on AECP, please visit our website.