A Primer on Power Systems 10 Gb Ethernet
The use of 10 Gb Ethernet at the server level is on the rise. However, there’s much to consider when implementing 10 Gb Ethernet adapters for IBM POWER7 servers. When selecting adapters, you’ll want to examine:
- The availability and type of adapter slots for your server
- Architectural requirements for the use of 10 Gb Ethernet
- Your company’s standards for cabling and connector types
Here are a few areas to explore:
Four different types of adapter slots are available on POWER7 servers. It’s important to know the quantity of each type on your server in order to optimize the placement of 10 Gb Ethernet adapters.
1. Host Ethernet Adapter (HEA). HEAs are commonly referred to as the “integrated Ethernet” adapters for POWER7 servers. They’re also known as Integrated Virtual Ethernet (IVE) in IBM documentation. An HEA adapter can logically be divided into 32 separate ports, each of which can be assigned to a separate LPAR or Virtual I/O (VIO) server. This alleviates the need to purchase and cable separate Ethernet adapters for each LPAR. The HEA adapter slots offer high levels of performance. POWER7 server models 710, 720, 730, 740 and 750 have one HEA adapter slot. Server models 770 and 780 feature one HEA adapter slot per processor drawer. While the 795 server has none.
2. PCI Express (PCIe). The PCIe adapter slot is the follow-on technology to PCI-X. It operates with greater throughput and is designed to be scaled to higher levels of performance for future implementations. 2-port 10 Gb Ethernet cards first become available in the PCIe format due to the higher throughput as compared to PCI-X.
3. PCIe Low Profile (LP). PCIe LP adapters were introduced to the Power Systems family with the POWER7 models 710, 720, 730 and 740. These adapters feature the same throughput capabilities of standard PCIe adapters. PCIe LP adapters allow for thinner servers (server models 710/730) or more internal slots (server models 720/740). 2-port 10 Gb Ethernet cards are available in PCIe LP format. PCIe LP cards, however, are not supported in standard PCIe slots because the edge connector at the back of the server would not be the proper size.
4. PCI-eXtended (PCI-X). Older in design, PCI-X adapter card slots operate with lower throughput compared to PCIe adapter slots. As a result, 10 Gb Ethernet cards are only available in 1-port, fiber-based configurations. No 2-port or copper-based 10 Gb Ethernet adapters are available in PCI-X format.
Because 10 Gb Ethernet adapter cards, cabling and switch ports can be expensive, your server design should make the best use of configured ports and minimize the number of underutilized or hot standby ports. Using PowerVM virtualization can help improve utilization by aggregating LPAR network traffic across fewer 10 Gb Ethernet ports. With proper configuration, multiple PowerVM virtual switches can be used to maximize the value of your 10 Gb Ethernet adapters. (See Resources.)
Cabling and Connector Types
10 Gb Ethernet adapters use fiber or copper cabling, and the adapter configurations available for each slot type include:
- HEA—2-port copper or fiber
- PCIe—1-port copper, 1-port and 2-port* fiber
- PCIe LP—1-port copper, 1-port and 2-port* fiber
- PCI-X—1-port fiber
Those marked with an asterisk (*) are also offered with Fibre Channel over Ethernet (FCoE) functionality. The following connector types are available for the 10 Gb Ethernet adapters:
- 10GBASE-T RJ-45 (100 m)
- 10GBASE-CX4 (15 m)
- SFP+ copper twinaxial (5 m)
- LC (300 m and 10 km)
- SC (10 km)
The supported distances provided with each connector type are based on IBM documentation and may be less than published Institute of Electrical and Electronics Engineers (IEEE) standards. 10GBASE-T is the newest connector implementation for copper-based cabling. It offers improved distance capability over previous copper implementations. But keep in mind, the availability of cabling and connector types will vary for each POWER7 server model.
The POWER7 Models 720 and 740 now feature PCIe Generation2 (Gen2) slots, which offer double the throughput of PCIe Generation 1 (Gen1) adapter slots. Two 10 Gb Ethernet adapters currently operate at Gen2 data rates. They are features No. 5284 “PCIe Gen2 LP 2-port 10Gbt SR Adapter” and No. 5286 “PCIe Gen2 LP 2-port 10Gbt Ethernet (SFP+)”. With these adapters, you can run both ports at high levels of throughput. For other 2-port adapter implementations, you might not be able to get full throughput with both ports operating at the same time.
When designing your adapter card layout, the total quantity of high throughput adapters on each system bus should also be considered. The IO bus layout for each server model and associated IO drawers can be found in IBM Redbooks Technical Overview manuals. (See Resources.)
It is important to work closely with your networking department to ensure you select the correct cabling type (copper or fiber) and connector type when ordering 10 Gb Ethernet adapters. With 10 Gb Ethernet, there are more choices than the traditional RJ-45 copper based connector that we have become accustomed to over the past 20 years.