InfiniBand: The Battle for I/O Hill.
InfiniBand: The Battle for I/O Hill
Proponents of InfiniBand have been parachuting to the front lines of the
storage networking wars for months now, performing early attacks on
enemy technologies as good airborne troops are supposed to. They've led
sessions at conferences, hosted development summits, and shown
preliminary demos at trade shows.
If you've had the pleasure of attending any of these events, you may
have come away convinced that InfiniBand was The Next Big Thing, certain
to replace everything from Fibre Channel to motherhood and apple pie.
However, as with most new technologies, considerable confusion surrounds
what InfiniBand actually does and what its chances are in the market.
Many industry observers believe InfiniBand will coexist with Fibre
Channel, Ethernet, and future I/O technologies. And while there's no
doubt InfiniBand will have a presence in the server and storage arenas,
the questions are: How much? and how soon will it happen?
Before we jump into the trenches, let's do a high-altitude
reconnaissance of the battlefield. The InfiniBand Architecture, first of
all, is the result of a merger of efforts to develop new I/O
technologies for servers and workstations. Essentially, it's a
replacement for the aging and bandwidth-constrained PCI bus found in
today's PCs. Future I/O (FIO) and Next-Generation I/O (NGIO) were being
developed by different camps, which wisely decided to merge in August
1999.
At first, the name given to the merged effort was "System I/O." That
moniker obviously lacked marketing pizzazz, and was changed to
InfiniBand in October of that year. The InfiniBand Trade Association
(IBTA) was formed at that time to develop the specification and promote
the technology.
The seven founding members of the IBTA represent the Who's Who of
800-pound technology gorillas: Compaq Computer Corp. (NYSE: CPQ -
message board), Dell Computer Corp. (Nasdaq: DELL - message board),
Hewlett-Packard Co. (NYSE: HWP - message board), IBM Corp. (NYSE: IBM -
message board), Intel Corp. (Nasdaq: INTC - message board), Microsoft
Corp. (Nasdaq: MSFT - message board), and Sun Microsystems Inc. (Nasdaq:
SUNW - message board). Not only do these founding members of IBTA
include the leading server vendors, they also include major players in
the storage and operating systems spaces. If these folks could agree on
something, it had to be good. And, the thinking went, InfiniBand would
ultimately become a de facto industry standard.
The involvement of the technology superpowers in InfiniBand invited the
formation of new companies to build products around the technology and
venture capital firms to provide seed funding. As noted in Byte and
Switch's Venture Capital Survey, at least $200 million had been invested
in startups developing InfiniBand products through mid-2001. Since then,
more than $100 million has been pumped into the sector: See InfiniCon
Lands $26M, Cisco Muses on Voltaire, Lane15 Lines Up $12M, Exanet Lands
$17M, InfiniSwitch Bucks Boost InfiniBand, and VIEO Snags $4M More.
A massive infusion of VC money doesn't always ensure the success of a
technology, of course (you haven't forgotten about the implosion of
well-funded dotcoms already, have you?). But, combined with backing from
major industry leaders, the outlook for the InfiniBand market isn't too
bad. IDC projects that more than 3.4 million InfiniBand-capable servers
will ship annually by 2005.
Nevertheless, several questions remain to be answered when InfiniBand
products actually start hitting the streets, which analysts expect will
probably be mid-2002.
First, will InfiniBand actually deliver on the promise of
high-bandwidth, low-latency communications between servers in a cleanly
interoperable way? Next, will the IT community buy into the technology
as providing sufficient cost benefits to warrant yet another
interconnection technology shift? And finally, as products start to roll
out from various vendors, will we see the same sort of interoperability
problems that have plagued almost every other technology - including
Fibre Channel - thus slowing down the adoption rate?
Before delving into market dynamics and primary players, let's look at
the fundamentals of the InfiniBand technology and see if it has a
snowball's chance of making it into the data center. (We'll also note
here that, while the official name of the technology is "InfiniBand
Architecture," only the trademark lawyers at the IBTA will likely
continue to call it by its full name.)
InfiniBand is touted as a serial technology, one capable of being
implemented across either optical fiber or copper cabling. That's great,
because all things parallel have gone out of favor these days. Why?
Because the parallel bus architecture has an inherent latency - it needs
to wait for all the bits sent across a parallel link to arrive before it
can send more data - and that slows a system down. (Trust us: The
nanoseconds add up, especially at high bandwidths.) Also, any new
networking technology that even hopes to have a prayer needs to
accommodate optical fiber.
Diagrammatically, an InfiniBand switch fabric looks quite similar to
current Fibre Channel SANs. In this architecture, InfiniBand nodes (that
is, storage devices and servers) interconnect with one another over the
InfiniBand I/O fabric:
InfiniBand Fabric
The first release of the InfiniBand spec describes three initial
implementations - 1x, 4x, and 12x - with aggregate bandwidths of 500
megabytes per second, 2 GB/s, and 6 GB/s respectively, each with a
2.5-Gbit/s wire signaling rate. This is fairly heavy ordnance compared
with current network throughputs, but it's not orders of magnitude
greater than current 2-Gbit/s Fibre Channel or 1-Gbit/s Ethernet.
Jim Pappas, Intel's director of initiative marketing, says the company
has been demonstrating InfiniBand systems with four 1x connectors,
resulting in a bandwidth of 10 Gbit/s. "Any CPU made right now is
hard-pressed to generate 10 Gbits/s of I/O," he says. "The point is that
we have an architecture that scales over time. We could have gone faster
with the first implementation, but that would have been more expensive.
We think we have the right mix for the present."
Now, who are the generals driving the InfiniBand advance?
As noted earlier, the IBTA is the driving force behind both the
technology development and the marketing of InfiniBand, and the
organization was founded by heavyweights Compaq, Dell, HP, IBM, Intel,
Microsoft, and Sun. Since that time, more than 150 additional companies
have joined IBTA. Some of the new members are startups planning
InfiniBand-based products, while others are existing industry players
who will either compete with InfiniBand or need to implement it in their
existing products.
Here are the IBTA's members, divided by technology segment:
interconnectivity devices and adapters; servers and storage systems;
software; components; and other.
Table 1: Interconnectivity Devices and Adapters Company Product
Description Public or Private Adaptec Inc. Adapters, controllers Public
Advanced Digital Information Corp. Storage gateways, tape drives Public
ATTO Technology Inc. Adapters, bridges Private Broadband Storage Inc.
SAN/NAS bridge Private Brocade Communications Systems Inc. Storage
switches Public Catalyst Enterprises Adapters, analyzers Private Cisco
Systems Inc. Networking systems Public CMD Technology Inc. Adapters,
controllers Private Computer Network Technology Corp. (CNT) Storage
routers Public Crossroads Systems Inc. Storage routers and gateways
Public Cyclone Microsystems Adapters Private Desana Systems Inc. Data
center application switch Private DivergeNet Inc. Adapters, switches,
bridges Private Egenera Blade systems Private Emulex Corp. Adapters
Public Essential Communications Inc. Adapters, switches Private Force
Computers Adapters Private Gadzoox Networks Inc. Storage switches Public
Huawei Technologies Co. Ltd. Networking systems Private ICP Vortex, an
Intel subsidiary Controllers Private InfiniCon Systems Multiprotocol
storage switch Private InfiniSwitch Corp. InfiniBand switches Private
Inrange Technologies Corp. Storage switches Public Interphase Corp.
Adapters Public JNI Corp. Host bus adapters Public Juniper Networks Inc.
Optical networking systems Public Lucent Technologies Inc. Networking
systems Public Marconi PLC Optical networking systems Public McData
Corp. Storage switches Public Myricom Inc. Clustering switches Private
Nishan Systems Inc. Multiprotocol storage switch Private Nortel Networks
Corp. Networking systems Public OmegaBand Inc. InfiniBand gateways
Private Paceline Systems Corp. InfiniBand switches Private SANcastle
Technologies Inc. Multiprotocol storage switch Private Sanera Systems
Inc. Multiprotocol storage switch Private SysKonnect Adapters Private
3Com Corp. Adapters, networking systems Public QLogic Corp. Storage
switches, components Public Voltaire Inc. Multiprotocol storage router
Private
Table 2: Servers and Storage Systems Company Product Description Public
or Private American Megatrends Inc. Storage subsystems Private Auspex
Systems Inc. NAS systems Public Ciprico Inc. Storage arrays Public
Compaq Computer Corporation Servers, storage systems Public Dell
Computer Corp. Servers, storage systems Public Dot Hill Systems Corp.
Storage systems Public EMC Corp. Storage systems Public Exanet Inc.
Storage subsystems Private Fujitsu Computer Products of America Inc.
Storage arrays Private Fujitsu Ltd. Servers, storage systems Public
Fujitsu Siemens Computers Servers, storage systems Public
Hewlett-Packard Co. Servers, storage systems Public Hitachi Ltd.
Servers, storage systems Public IBM Corp. Servers, storage systems,
storage switches, components Public Ikadega Inc. Rich-media storage
appliance Private Maxtor Corp. NAS systems, disk drives Public NCR Corp.
Servers, storage arrays Public NEC Electronics Inc. Servers, components
Public Network Appliance Inc. NAS systems Public Network Engines Inc.
Server appliances Public Seagate Technology Inc. Storage subsystems
Private SGI Servers, storage arrays Public Storage Technology Corp.
(StorageTek) Storage systems, tape drives Public Sun Microsystems Inc.
Servers, storage systems Public Unisys Servers, storage systems Public
Western Digital Corp. Disk drives Public YottaYotta Inc. IP storage
switch/server Private
Table 3: Software Company Product Description Public or Private BMC
Software Inc. Enterprise management software Public GoAhead Software
Inc. High-availability software Private Lane15 Software InfiniBand
management software Private Lineo Inc. Embedded Linux products Private
Microsoft Corp. Operating system software Public MPI Software Technology
Inc. Clustering software Private Novell Inc. Network software Public
Oracle Corp. Database software Public Phoenix Technologies Ltd. Device
software Public PolyServe Inc. Clustering software Private Prisa
Networks Inc. SAN management software Private Topspin Communications
Inc. InfiniBand clustering software Private Turbolinux Inc. Linux
software Private Troika Networks Inc. Management software Private
Veritas Software Corp. Storage management software Public VIEO Inc.
InfiniBand management software Private
Table 4: Components Company Product Description Public or Private Actel
Corp. Components Public Agilent Technologies Components Public Altera
Corp. Components Public Alvesta Corp. Optical components Private
Amphenol Interconnect Products Corp. Cable components Public API
NetWorks Inc. Components Private Astek Corp. Components Private Astute
Networks Components Private Aurora Technologies Inc. Boards, controllers
Public Banderacom Inc. Components Private C&M Corp. Cable components
Private Corrent Corp. Components Private Cypress Semiconductor Corp.
Components Public DCM Technologies Components Private E2O Communications
Inc. Optical components Private Fairchild Semiconductor Semiconductors
Public FCI Fiber Optics Components Private Finisar Corp. Optical
components Public Foxconn Electronics Inc. Cable components Public
Fujikura America Inc. Optical components Private Fujitsu Takamisawa
Component Ltd. Components Private I-Bus/Phoenix Board products Private
Infineon Technologies AG Semiconductors Public Infortrend Technology
Inc. RAID controllers Public Intel Corp. Microprocessors, adapters
Public Intersil Corp. Wireless components Public Jasmine Networks
Optical components Private JST Sales America Inc. Cable components
Public LSI Logic Corp. Storage components Public Madison Cable Corp.
Cable components Private Marvell Technology Group Ltd. Components Public
Mellanox Technologies Ltd. InfiniBand I/O fabrics Private Mindspeed
Technologies Components Private Molex Inc. Components Public
Montrose/CDT Cable components Private Motorola Inc. Components Public
New C-Tek Inc. Optical components Private ON Semiconductor
Semiconductors Public Parama Networks Inc. Optical components Private
Parthus Technologies PLC Wireless components Public PMC-Sierra Inc.
Semiconductors Public Power X Networks Inc. Switch fabrics Public
Primarion Inc. Components Private Prolific Technology Inc. Components
Private QuickLogic Corp. Components Public Raytheon Co. Components
Public RedSwitch Inc. Switch components Private SBE Inc. Components
Public ServerWorks Corp. (a Broadcom company) Components Private Silicon
Image Inc. Components Public Sky Computers (an Analogic company)
Components, board products Private Stargen Inc. Component design Private
Stratos Lightwave LLC Optical components Public Systran Corp. Boards,
components Private Texas Instruments Inc. Semiconductors Public
TranSwitch Corp. Semiconductors Public Tundra Semiconductor Corp.
Components Public VIA Technologies Inc. Semiconductors Public W.L. Gore
& Associates Inc. Cable components Private Xicor Inc. Components Public
Xilinx Inc. Components Public Zarlink Semiconductor Inc. Semiconductors
Public ZNYX Networks Board products and software Private
Table 5: Other Company Product Description Public or Private 0-In Design
Automation Inc. Verification tools Private Advanced Micro Devices
Microprocessors Public Ancot Corp. Testing and measurement tools Private
Cadence Design Systems Inc. Electronic design products and services
Public Computer Access Technology Corp. (CATC) Testing and measurement
tools Public
Data Transit Corp. Testing and measurement tools Private DY4 Systems
Inc. Embedded computing systems Public I-TECH Corp. Testing and
measurement tools Private Mercury Computer Systems Inc. Embedded
computing systems Public MindShare Inc. Training Private Marathon
Technologies Corp. High-availability products Private Sandia National
Laboratories Research lab Public SBS Technologies Inc. Embedded
computing systems Public Stratus Technologies International
High-availability servers Private Synopsys Inc. Design automation tools
Public Sypris Data Systems Inc. Tape drives Private Tektronix Inc.
Testing and measurement tools Public Ochs Industries Inc. Contract
manufacturing Private Wavecrest Corp. Testing and measurement tools
Private
The spec for InfiniBand defines what IBTA calls a System Area Network
for connecting "multiple independent processor platforms" (they mean
"servers"), I/O platforms (the new InfiniBand products), and I/O devices
(the same old stuff we've been connecting to for ages). [Ed. note:
You'll also notice that the IBTA has unhelpfully given us another
acronym that spells "SAN," which should not be confused with the Fibre
Channel-based Storage Area Network. But we digress.]
The kicker is that InfiniBand's System Area Network is both a
communication and management infrastructure that supports conventional
I/O as well as interprocessor communications (IPC). The spec also
defines a switched communication fabric (borrowing another term from the
Fibre Channel realm) in which many devices can concurrently communicate.
The protocol provides protections to prevent the communications from
doing things it shouldn't and also provides for remote management.
One of the more interesting aspects of InfiniBand is the IPC element,
and the fact that InfiniBand includes a direct pipe into the memory
controllers of connected systems. That allows for "message passing," or
message queuing, between systems and processes without the usual wait
states associated with conventional bus-based I/O. In this diagram,
"HCA" is the host channel adapter (i.e., the server); and "TCA" is the
target channel adapter (i.e., the storage device).
Tom Bradicich, CTO for IBM's xSeries servers, describes the
message-passing process as follows: "Message passing is when data goes
into memory and is left there for later retrieval. For example, message
passing is like you mailing a letter - you go out to your mailbox and
leave it there. You don't wait at the mailbox for the postman to come to
get it. Similarly, when the postman brings you a letter, he puts it in
your mailbox, and goes on his way. He doesn't wait for you to come out
and get the mail, before he goes on to the next house."
[Ed. note: Great analogy, Tom, but one certainly hopes that InfiniBand
is more reliable than the U.S. Postal Service.]