iWARP (Internet Wide Area RDMA Protocol)

Remote Direct Memory Access, or RDMA, allows a computer to access another computer's memory without interacting with either computer's operating system data buffers, thus increasing networking speed and throughput. Internet Wide Area RDMA Protocol (iWARP) is a protocol for implementing RDMA across Internet Protocol networks.

Microsoft* Windows* provides two forms of RDMA: Network Direct (ND) and Network Direct Kernel (NDK). ND allows user-mode applications to use iWARP features. NDK allows kernel mode Windows components (such as File Manager) to use iWARP features. NDK functionality is included in the Intel base networking drivers. ND functionality is a separate option available during Intel driver and networking software installation. If you plan to make use of iWARP features in applications you are developing, you will need to install the user-mode Network Direct (ND) feature when you install the drivers. (See Installation below.)

NOTES: Even though NDK functionality is included in the base drivers, if you want to allow NDK RDMA features across subnets, you will need to select "Enable iWARP routing across IP Subnets" on the iWARP Configuration Options screen during base driver installation (see Installation below). In order to enable RDMA capability on a virtual adapter connected to a virtual machine switch, *SRIOV (Single Root IO Virtualization) and *VMQ (Virtual Machine Queues) advanced properties must be enabled on each port driver. Use the Advanced tab of the adapter properties dialog or the following Windows PowerShell commands to set these options. Set-NetAdapterAdvancedProperty -Name <nic_name> -RegistryKeyword *SRIOV -RegistryValue 1 Set-NetAdapterAdvancedProperty -Name <nic_name> -RegistryKeyword *VMQ -RegistryValue 1

Requirements

The Intel® Ethernet User Mode iWARP Provider is supported on Linux* operating systems and Microsoft* Windows Server* 2012 R2 or later. For Windows installations, Microsoft* HPC Pack or Intel® MPI Library must be installed.

Installing User-Mode Network Direct (ND) iWARP Features

NOTE: For installing the Intel® Ethernet User Mode iWARP Provider on Windows Server 2016 Nano Server, see Installing on Nano Server below.

Network Direct Kernel (NDK) features are included in the Intel base drivers. Follow the steps below to install user-mode Network Direct (ND) iWARP features.

1. From the installation media, run Autorun.exe to launch the installer, then choose "Install Drivers and Software" and accept the license agreement.
2. On the Setup Options screen, select "Intel® Ethernet User Mode iWARP Provider".
3. On the iWARP Configuration Options screen, select "Enable iWARP routing across IP Subnets" if desired. Note that this option is displayed during base driver installation even if user mode iWARP was not selected, as this option is applicable to Network Direct Kernel functionality as well.
4. If Windows Firewall is installed and active, select "Create an Intel® Ethernet iWARP Port Mapping Service rule in Windows Firewall" and the networks to which to apply the rule. If Windows Firewall is disabled or you are using a third party firewall, you will need to manually add this rule.
5. Continue with driver and software installation.

Installing on Nano Server

Follow the steps below to install the Intel® Ethernet User Mode iWARP Provider on Microsoft Windows Server 2016 Nano Server.

1. Create a directory from which to install the iWARP files. For example, C:\Nano\iwarp.
2. Copy the following files into your new directory:
• \Disk\APPS\PROSETDX\Winx64\DRIVERS\i40wb.dll
• \Disk\APPS\PROSETDX\Winx64\DRIVERS\i40wbmsg.dll
• \Disk\APPS\PROSETDX\Winx64\DRIVERS\indv2.cat
• \Disk\APPS\PROSETDX\Winx64\DRIVERS\indv2.inf
• \Disk\APPS\PROSETDX\Winx64\DRIVERS\indv2.sys
3. Run the DISM command to inject the iWARP files into your Nano Server image, using the directory you created in step 1 for the AddDriver path parameter. For example, "DISM .../Add-Driver C:\Nano\iwarp"
4. Create an inbound firewall rule for UDP port 3935.
5. If desired, use the Windows PowerShell commands below to enable iWARP routing across IP Subnets.
• Set-NetOffloadGlobalSetting -NetworkDirectAcrossIPSubnets Allow
• Disable Adapter
• Enable Adapter

Configuring iWARP Guest Support (NDK Mode 3)

NDK Mode 3 allows kernel mode Windows components to use iWARP features inside Hyper-V guest partitions. To enable NDK mode 3 on an Intel Ethernet device, do the following:

1. Enable SR-IOV in your system's BIOS or uEFI.
2. Enable the SR-IOV advanced option on the device.
3. Configure the device with SR-IOV enabled on the switch. Perform this step for all functions on the same device:
   New-VMSwitch -Name <switch_name> -NetAdapterName <device_name>
-EnableIov $true
4. Configure the number of RDMA virtual functions (VFs) on the device by setting the "RdmaMaxVfsEnabled" advanced option. All physical functions must be set to the same value. The value is the maximum number of VFs that can be capable of RDMA at one time for the entire device. Enabling more VFs will restrict RDMA resources from physicals functions (PFs) and other VFs.
   Set-NetAdapterAdvancedProperty -Name <device_name> -RegistryKeyword RdmaMaxVfsEnabled -RegistryValue <Value: 0 – 32>
5. Disable all PF adapters on the host and re-enable them. This is required when the registry keyword “RdmaMaxVfsEnabled” is changed or when creating or destroying a VMSwitch.
   Get-NetAdapterRdma | Disable-NetAdapter
Get-NetAdapterRdma | Enable-NetAdapter
6. Create VM Network Adapters for VMs that require RDMA VF support.
   Add-VMNetworkAdapter -VMName <vm_name> -VMNetworkAdapterName <device_name> -SwitchName <switch_name>
7. If you plan to use Microsoft Windows 10 Creators Update (RS2) or later on a guest partition, set the RDMA weight on the VM Network Adapter by entering the following command on the host:
   Set-VMNetworkAdapterRdma -VMName <vm_name> -VMNetworkAdapterName <device_name> -RdmaWeight 100
8. Set SR-IOV weight on the VM Network Adapter (Note: SR-IOV weight must be set to 0 before setting the RdmaWeight to 0):
   Set-VMNetworkAdapter -VMName <vm_name> -VMNetworkAdapterName <device_name> -IovWeight 100
9. Install the VF network adapter with the PROSET Installer in the VM.
10. Enable RDMA on the VF driver and Hyper-V Network Adapter using PowerShell in the VM:
    Set-NetAdapterAdvancedProperty -Name <device_name> -RegistryKeyword RdmaVfEnabled -RegistryValue 1
Get-NetAdapterRdma | Enable-NetAdapterRdma

Verify that RDMA was successfully enabled by checking the "RDMA Capable" field. Run:
Get-SmbServerNetworkInterface

If the "RDMA Capable" field reports false, try disabling and re-enabling RDMA capabilities in the VM:
Get-NetAdapterRdma | Disable-NetAdapterRdma
Get-NetAdapterRdma | Enable-NetAdapterRdma

If the "RDMA Capable" field still reports false, try disabling and re-enabling the physical functions on the host: Get-NetAdapterRdma | Disable-NetAdapter
Get-NetAdapterRdma | Enable-NetAdapter