Interviews

posted on 27 June 2008 14:49

Brocade has just announced a big push to make Converged Enhanced Ethernet the converged network platform for server-to-server and server-to-storage networking in enterprise data centers. InfiniBand is also being presented as a converged network platform candidate. We were lucky enough to get a chance to interview Voltaire's Chief Technology Officer, Yaron Haviv, and ask him some questions about this. Here is what he said:-

B & F: What is InfiniBand’s role in connecting external storage systems to servers in supercomputing and HPC and enterprise data centers?

Yaron Haviv: InfiniBand is widely used as a high performance server to server interconnect in supercomputers and is rapidly increasing share as a cluster and grid interconnect in high performance data centers in the enterprise. In supercomputing, InfiniBand storage is typically used to improve performance of scalable file systems and file I/O. It is used to increase storage throughput and the number of nodes that can simultaneously access the external storage devices. In the enterprise, InfiniBand is typically used to connect large clusters to Fibre Channel SANs and can greatly improve the performance of storage intensive applications – often by 200% or more.

B & F: Are customers using InfiniBand as a converged server and storage network foundation in data centers today? If so then why, who, and how?

Yaron Haviv: Absolutely. We refer to the convergence of server and storage networks as “unified fabrics” and believe this will be the network architecture of choice for next generation data centers. Unified fabrics provide seamless, high performance networking services between InfiniBand fabrics, Fibre Channel SANs and Ethernet LANs over a single high performance fabric with multiple virtual interfaces replacing actual physical adapters. By consolidating multiple data center fabrics into one, enterprises save costs, reduce complexity and enable further consolidation and virtualization of the data center. JPMorgan Chase uses Voltaire InfiniBand switches for a unified data center fabric because it accelerates performance of their applications across different lines of business in an efficient manner while significantly reducing overall data center costs.

B & F: Are customers specifically using InfiniBand to carry Fibre Channel storage network data today? If so then who and why?

Yaron Haviv: There are a couple of options available today to use InfiniBand to connect to storage. Native InfiniBand storage solutions are available today from Voltaire and our partners such as LSI and DataDirect Networks. Using InfiniBand to directly attach to storage can improve storage performance by orders of magnitude at a very reasonable price point. There are also gateways available today which bridge InfiniBand fabrics to FC storage.

This means that customers can use a single gateway to connect 20 Gbps InfiniBand and 4 Gbps FC and get 1500 MB/second of storage throughput, which is a 200% performance improvement over a 4 Gbps FC connection. These gateways connect to a single InfiniBand adapter eliminating the need for multiple FC HBAs in each server, creating significant cost savings.

The High Performance Computing Center Stuttgart (HLRS) in Stuttgart, Germany is one customer using Voltaire’s InfiniBand-to-storage gateways to connect their InfiniBand-based scalable file systems with Fibre Channel storage. The result is simplified storage connectivity and better application performance for their manufacturing engineering applications.

Businesses that rely on storage intensive applications such as large databases, digital media and entertainment, financial services, biosciences and oil and gas exploration software have a lot to gain from InfiniBand storage.

In the future we expect to see significant use of InfiniBand for connecting SSD (Solid State Disks) due to their higher throughput and lower latencies. This just cannot be matched by any other technology.

B & F: Could you compare and contrast FCOE and FC over InfiniBand please?

Yaron Haviv: The wire formats for FCoE and FCoIB are quite similar and are a simple encapsulation of FC packets; the differences are in the underlying fabric capabilities.

B & F: Could you compare and contrast InfiniBand and data center-class Ethernet (DCE) as converged network foundation candidates?

Yaron Haviv: Today both Fibre Channel and InfiniBand provide a lossless fabric, Layer 2 mesh routing, and hardware-based congestion management, which are all required to deliver high-performance storage traffic. When using a single wire for multiple purposes it is also required to provide isolation in the form of virtual lanes and fabric partitioning. Those features have both been standard in InfiniBand from day one.

In contrast, Ethernet stills lacks those key features and is currently in the process of becoming standardized: this will continue throughout 2008 and 2009. New switches that will support the convergence features (Converged Enhanced Ethernet (CEE) or DCE switches) will take time to mature and interoperate. Users will be forced to rip and replace infrastructure because they’ll need different cables (ie: SFP+, Twinex), different adapters and different switches.

InfiniBand is here today and has supported all of the convergence features from its inception. Moreover, InfiniBand continues to evolve to much higher speeds. The latest introduction of 40 Gbps InfiniBand is a prime example of that.

B & F: How do DCE and InfiniBand compare on acquisition costs and total cost of ownership?

Yaron Haviv: 20 Gbit/s InfiniBand director-class switches cost about US$800 per port (list price). We’ve been seeing street pricing for DCE switches at costs between US$2600-3600 per port! In addition, these switches have a high degree of oversubscription (4:1) on their 10 Gig E ports, so when you compare price/performance of Voltaire InfiniBand vs. DCE, the Voltaire switch has 8 times the bandwidth at a quarter to one-third of the cost.

B & F: Why would it be worth adding InfiniBand to an existing Ethernet and Fibre Channel mix in an enterprise data center? Won't it increase complexity? What would be the benefit?

Yaron Haviv: InfiniBand makes a lot of sense for data centers that derive business value from the improved speed and efficiency of their applications. One of the great things about InfiniBand is that it works with legacy systems and doesn’t require a forklift approach to rearchitect the data center. Voltaire’s InfiniBand switches come with the option to add gateways to easily connect to 1 and 10 Gigabit Ethernet and Fibre Channel, so it is not disruptive to add InfiniBand to the mix.

Using InfiniBand for the data center fabric delivers 10 or 20 Gbit/s bandwidth (40 Gbit/s switches will be available later this year) and very low latencies to deliver as much as a 300% performance improvement to the application. Moreover, requiring less than 5 watts of power per 20 Gbit/s port, InfiniBand is a much more energy efficient technology than 10 Gigabit Ethernet. On average InfiniBand requires about 50% less power than 10 Gig E. This is an important benefit for CIOs and data center managers looking to green their data centers or simply lower the cost of their energy bill.

Also, InfiniBand can be integrated inside of a rack or a server farm in a seamless manner and is typically easier to manage than Ethernet or Fibre Channel due to its standardized and centralized configuration and management. Users can actually wire the InfiniBand fabric once and virtualize it to create multiple SAN and LAN fabrics on a single wire without the need to physically change the layout and connectivity.

B & F: So ... this is a view that says you can have a cost-effective converged data center network platform now and that platform is InfiniBand, not Ethernet. There are no 800lb gorillas pushing InfiniBand for the convergence platform role as there are, called Cisco and Brocade, for the Ethernet candidate. But as HPC offers InfiniBand a ride into enterprise data centers we can expect its virtues to become more widely known and utilised.

^{[Chris Mellor.]}