                             Mellanox Technologies
                             *********************

              InfiniBand OFED Driver for VMware(R) Infrastructure 5.5
                                 Release Notes
                            Version ESXi 1.8.3 Beta
                            Last Updated: June 2014
						  


===============================================================================
Table of Contents
===============================================================================
1. Overview
2. Changes in This Release
3. Contents of Mellanox OFED ESXi
4. Supported Platforms and Operating Systems
5. Supported HCAs
6. Switch and Gateway Systems and Storage
7. Change Log History
8. Known Issues
9. Bug Fixes
   
===============================================================================
1. Overview
===============================================================================
These are the release notes of "InfiniBand OFED Driver for VMware(R) vSphere
5.5". This document provides instructions on InfiniBand drivers for Mellanox 
Technologies ConnectX(R) based adapter cards with VMware
ESXi Server environment.

This release notes file and the user's manual are available from
http://www.mellanox.com/page/products_dyn?product_family=36&mtag=vmware_drivers

===============================================================================
2. Changes in This Release
===============================================================================
MLNX-OFED-ESX-1.8.3 provides the following changes and new features:

  - iSER

===============================================================================
3. Contents of Mellanox OFED ESXi
===============================================================================  
3.1 Package Contents
--------------------
MLNX-OFED-ESX package contains:
o MLNX-OFED-ESX-1.8.3-10EM-500.0.0.xxx.zip - Hypervisor bundle which 
  contains the following kernel modules:
   - mlx4_core (ConnectX family low-level PCI driver)
   - mlx4_ib (ConnectX family InfiniBand driver)
   - ib_core
   - ib_sa
   - ib_mad
   - ib_umad
   - ib_ipoib
   - ib_cm
   - ib_srp
   - ib_iser

The driver package is distributed as an offline bundle (.zip file). 

===============================================================================
4. Supported Platforms and Operating Systems
===============================================================================
 o CPU architectures:
    - x86_64
  
  o ESX Hypervisor:

    - ESXi 5.1UX/5.5UX

   
 Note: For the supported hardware compatibility list (HCL) and guest operating
       systems, please refer to VMware support documentation at 
       http://www.vmware.com/support/pubs 
       and follow the "Compatibility Guides" link

===============================================================================
5. Supported HCAs
===============================================================================
This release supports Mellanox Technologies InfiniBand (IB) HCAs:

    - ConnectX-3 Pro: 2.30.8000 and above
	- ConnectX-3: 2.30.8000 and above
	- ConnectX-2: 2.9.1200 and above


Please note that older firmware versions were not tested with this release. 

For the latest firmware versions, visit:
http://www.mellanox.com/page/vmware_matrix

===============================================================================
6. Switch and Gateway Systems and Storage
===============================================================================
o All production InfiniBand switches and gateways are supported.

o Tested platforms:
  - This release was tested with switches provided by Mellanox: IS5025, IS5035  
    SX6036, SX6518, SX6512, Grid Director 4036 and 4036E 
  
  - Storage:
    o NFS over IPoIB
    o GPFS over IPoIB
    o SRP	

===============================================================================
7. Change Log History
===============================================================================
MLNX-OFED-ESX-1.8.3 Main Changes (from 1.8.2.4) 
---------------------------------------------------
  - Added iSER support

MLNX-OFED-ESX-1.8.2.4 Main Changes (from 1.8.2) 
---------------------------------------------------
  - Added ESXi 5.5 support
  - Enabled module heap to expand into high memory
  - Bug Fixes (See section 9. Bug Fixes)

MLNX-OFED-ESX-1.8.2 Main Changes (from 1.8.0-672) 
---------------------------------------------------
  - Added SRP support
  - Bug Fixes (See section 9. Bug Fixes)

MLNX-OFED-ESX-1.8.0-672 Main Changes (from v1.8.0)
---------------------------------------------------
  - ESXi 5.1 support
  - GPFS support
  - InfiniBand resource cleanup tuning
  - Removed LID based MAC generation
  - Debug file system (PROCFS) is currently at beta level
  - SR-IOV is not supported in the current release 
  
MLNX-OFED-ESX-1.8.0 Main Changes and New Features
---------------------------------------------------
o This is the initial GA release for ESXi 5.0 driver,
  containing the following main features: 
  - IPoIB  Para Virtualization
  - ESXi 5.0 support
  - InfiniBand support only
  - FDR support
  - 4k MTU support

===============================================================================
8. Known Issues
===============================================================================

o TCP traffic may resume after a certain delay after the port is down/up 

o Number of MAC addresses per host is limited to 400 

o ibportstate utility (MLNX_OFED) usage for setting port down/up is not supported

o IPv6 is not supported in the current release. 
    
o The minimum supported MTU value is 1500  

o On ESXi5.0/ESXi5.0u1, the "esxcli network nic list" command 
  (a.k.a esxcfg-nics -l) displays the wrong link state. 
  We recommend running the ethtool instead.

o Using vSwitch nic teaming feature requires setting the network failure 
  detection to Link status Only 

o Packets might get dropped upon initial communication between interfaces 

o The "lspci" command displays an incorrect interface name

o Poor TCP performance may occur in Linux virtual machines with LRO enabled.
  To solve the issue, please refer to the VMware website:
  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1027511

o Unloading the driver is not supported by VMware

o The "esxcli network nic down -n <vmnic_ibX>" and  
  the "esxcli network nic up -n <vmnic_ibX>" commands are only 
  supported in ESXi 5.1 and above.


8.1  Migration/VMotion Known Issues
-------------------------------------------------------------------------------
o Frequent migration/VLAN changes may cause connectivity loss up to 1 minute

o When using VGT, connectivity may be lost for up to 40 seconds after vmotion. 
  To restore connectivity, send ping from the migrated VM.
  
Migration/VMotion Workaround:
o  To solve the issues above flash the ARP tables in all sources and targets


8.2  SRP Known Issues
-------------------------------------------------------------------------------
o Unloading mlx4 low-level drivers under heavy SRP traffic may cause system
  PSOD. 
  Workaround: 
  Unload the SRP module before mlx4 drivers.
  Note that running "/opt/mellanox/bin/openib.sh restart" does not make sure
  that SRP module is successfully unloaded before continuing to unload mlx4
  drivers, and therefore may cause PSOD under heavy traffic.

o Restart subnet manager with re-assign lid or perform subnet manager failover
  under heavy traffic from VMs and/or multiple concurrent storage migrations
  conditions may cause increasing in latency, or lose connectivity to storages,
  or datastor corruption.

o "/opt/mellanox/bin/mlnx-srp-config start/stop" does not save and revert
  VAAI features settting if VAAI's /DataMover/HardwareAcceleratedMove,
  /DataMover/HardwareAcceleratedInit, and /VMFS3/HardwareAcceleratedLocking
  options are set to "1" by default before executing the script.

o Multiple ESXi SRP hosts/initiators may flood embedded subnet manager
  running in InfiniBand switch with PortInfoRecord queries and get 
  "multiple matching records" back from switch. 
  This behavior has no effect on the system.

o Unloading SRP driver without traffic or storage connected may take up to
  60 seconds.

8.3 iSER Known Issues
-------------------------------------------------------------------------------
o Manually unloading and reloading mlx4 low-level drivers may stop 
  iSER from working. 

o Rescanning vmhbas while creating and deleting targets may freeze the scan.

o Restarting the Subnet Manager with a re-assign lid or performing Subnet Manager
  failover under heavy traffic from VMs and/or multiple concurrent storage 
  migrations may cause increase in latency, or lose of connectivity 
  to storages, or datastore corruption.

o ESXi will stop trying to reconnect to a target that is lost for a long time
  See VMWare KB:
  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004684

o ZFS appliance: connection establishment causes iSER to log a stack trace.

o Installing a new MLNX_OFED_ESX bundle requires removing all 
  old RDMA related vibs first.  

===============================================================================
9. Bug Fixes
===============================================================================
The following are ESXi 5.1 driver bug fixes:

o Fixed port stucking in INIT state

o Fixed SRP module dma_boundary reset to 32 bit

o Fixed an issue related to the unsupported packets formats in TX flow. 
  Currently, all unsupported packets formats are dropped in TX flow.
  
o Fixed a PSOD issue in the IPoIB driver occurred when dropping a packet

o Fixed stop / start TX queue mechnism in IPoIB driver to prevent timeout 
  in the TX flow

o Added a prefix with the card number to the MAD sysfs. 
  Now a multi HCA setup is supported 
