Pain-Free Migration to SR-IOV with the IBM SR-IOV Migration Tool
James Chandler explains this new IBM Systems Lab Services tool.
By James Chandler07/14/2020
Have you heard about IBM’s exciting network virtualization technology, SR-IOV? If you’re a long-time IBM PowerVM client currently using Shared Ethernet Adapter(s) and are looking for more network bandwidth, reduced latency and lower CPU and memory overhead, SR-IOV might be the solution you’ve been waiting for. IBM Systems Lab Services can help you get there, with no downtime and minimal effort.
IBM first introduced support for Single-Root I/O Virtualization (SR-IOV) on IBM POWER8® systems in 2015. SR-IOV is an industry-standard extension to the PCI Express (PCIe) specification that allows multiple client partitions to share a physical adapter. SR-IOV is similar to previous IBM proprietary PowerVM virtualization technologies such as Integrated Virtual Ethernet (IVE) and Host Ethernet Adapter (HEA). SR-IOV is analogous to PowerVM processor micro-partitioning, in that each client LPAR is guaranteed a "slice" of the physical adapter port's capacity. The fact that multiple client partitions can share the same physical port allows for increased adapter consolidation, reducing the overall data center hardware footprint and lowering energy costs.
Prior to the introduction of SR-IOV, network virtualization on the PowerVM platform was chiefly implemented utilizing a Shared Ethernet Adapter (SEA) residing on a Virtual I/O (VIO) server. The SEA is responsible for bridging client network traffic using a virtual Ethernet adapter with one or more physical adapters that are connected to a network switch. While this design allows for flexibility in implementing different network configurations for client partitions, there is also overhead incurred as a result of multiple data copies being needed to transmit packets from the client's virtual Ethernet adapter through the SEA to the underlying physical adapter(s).
SR-IOV support was initially implemented through "logical" ports or Virtual Functions (VFs). Each logical port is mapped to an underlying SR-IOV physical adapter and is allocated a slice of the physical port's capacity. This is specified using the desired capacity setting and is a guaranteed minimum allocation. This provides a quality of service feature missing in the SEA, whereby each client adapter can be given a different bandwidth allocation.
SR-IOV logical ports take advantage of direct access I/O, which provides CPU and latency characteristics similar to a dedicated adapter. With direct access I/O, there is no additional hypervisor or VIOS overhead required to process client network traffic as compared to a traditional virtual Ethernet configuration backed by a VIOS SEA adapter. This can result in a significant reduction in CPU consumption, as well as reduced latency for a given workload. However, a partition with an assigned SR-IOV logical port is not capable of being migrated using Live Partition Mobility (LPM).
In late 2018, IBM introduced SR-IOV VNIC (Virtual Network Interface Controller), which supports LPM. Unlike SR-IOV logical ports, which have no dependency on a VIO server, VNIC utilizes a VF device that is automatically created and assigned to the VIOS when the VNIC client adapter is created. While VNIC does not support the direct access I/O model of SR-IOV logical ports, it does still provide a reduction in VIOS CPU and memory consumption, as well as lower latencies when compared to SEA configurations with virtual Ethernet.
Why Transition to SR-IOV?In addition to reducing overall CPU consumption, SR-IOV should be considered for environments that require a large number of transactions per second. For workloads that consist primarily of smaller packet sizes, SR-IOV can provide a substantial boost in network I/O performance versus traditional virtual Ethernet utilizing a SEA. Additionally, SR-IOV provides better out-of-the-box performance, with no additional tuning.
Converting From Virtual Ethernet to SR-IOV VNICWith a traditional virtual Ethernet configuration utilizing a Shared Ethernet Adapter, each client LPAR would be assigned an adapter through the client’s HMC partition profile. The client OS will have a number of logical devices associated with the virtual Ethernet adapter, possibly with their own interface-specific tunables.
Migrating from virtual Ethernet to SR-IOV would require (at a minimum):
- Creation of an SR-IOV device for each individual client partition, using either the HMC GUI or command-line interface (CLI)
- Saving any interface-specific tunable such as ‘tcp_sendspace’, ‘tcp_recvspace’, etc.
- Migration of the client IP address from the old virtual Ethernet device to the new SR-IOV device
- Manually setting any interface-specific tunables on the new SR-IOV device
- Removal of virtual Ethernet devices at the OS level
- Updating the client’s HMC partition profile to remove the old virtual Ethernet adapter
This can be a very time-consuming and error-prone process, especially for clients that have a large number of partitions to migrate to SR-IOV.
What is the IBM SR-IOV Migration Tool?
IBM developed the SR-IOV Migration Tool to assist in the migration process from virtual Ethernet to SR-IOV. With the IBM SR-IOV Migration Tool, clients can dynamically and non-disruptively migrate to SR-IOV VNIC with no downtime. This includes live-migrating the IP address from the old virtual Ethernet device to the new SR-IOV VNIC adapter (AIX only). The tool also allows for the optional removal of any OS-level devices, as well as removing the virtual Ethernet adapter from the client’s HMC partition profile.
The tool can also gather information about the client’s existing SR-IOV adapters and their configuration, such as the number of logical ports assigned to a given physical port, port labels and capacities, and report that information in an easily digestible format. This information can be invaluable when planning for the adoption of SR-IOV, as it allows the client to see how many logical ports are available, as well as their remaining capacity.
The tool currently supports IBM AIX versions 7.1 and 7.2, with SUSE Linux on Power support planned in the near future. Additionally, the tool allows for adjusting the maximum capacity setting for a given VNIC device, a feature that’s not available through the HMC GUI. The SR-IOV Migration Tool can be a very valuable asset, allowing clients to implement a repeatable, consistent methodology for the adoption of SR-IOV in their environment.
If you think this tool is right for you and want to learn more, please contact IBM Systems Lab Services.
James Chandler is a certified IT management consultant working for the IBM Systems Lab Services Power Systems Delivery Practice.