Tuesday, August 24, 2010

XIV Is Cool, But Not Cool Enough

Now that I’ve gotten your attention, I have to say I love this box. It takes fairly inexpensive components and makes them sing. The real magic comes in two forms, grid computing and the algorithms that power it.

Back in the 90s, I used to install supercomputers across the USA. These were the deep blue variety that played Garry Kasparov in chess and eventually beat him. Supercomputers were being used to do many high performance computing tasks from analyzing road wear and tear to chemistry, data mining, nuclear simulation and weather forecasting. My wife would ask me if they wore capes.

The industry was going along fine, until some people decided to take inexpensive commodity Intel servers and network them together with a high-speed network, and distributing the workload amongst the different processing nodes. This is when grid computing took down the supercomputing heavyweights.

Grids can be powerful, but the real power comes in their scale-out architecture. Instead of building a really big, massively parallel box, with increasing demands on busses and interconnects – a scale-up system – they add performance and capacity by adding additional nodes to the grid – a scale-out design. XIV is not alone in grid architecture, Left-hand is another smaller competitor. In secondary storage, we have Exagrid, Avamar as well as others. While the grid does wonderful things to scaling performance and capacity, the other real ingredient is the software powering it.

Eventually I expect the entire industry to go to grid-based computing. We’re already starting to see this. If clusters are an active-passive pair in a relationship, grids are the active-active distributed variety. Think Web servers front-ended with a load balancer, Citrix, vSphere farms – ready or not, grids are taking over the world. They just make sense. After taking over your datacenter, like the cluster, they’ll go geo[graphic] and span more than one DC creating a meshed fault-tolerant interconnected computing center.

So Why Isn’t XIV Cool Enough?

If XIV can do this much with inexpensive commodity-based components, what would happen if we took the same approach, but decided to make one monster piece of storage?

If I set out to make such a beast, I’d take the overall design and tweak the components. First, I’d ditch the 1 TB SATA drives and put in solid state disk (SSD). SSD would have amazing throughput, scaled even further by distributing the IO throughout the grid architecture.

Next I’d replace the interconnects. I’d ditch the 1 Gb Ethernet backend for InfiniBand. IBM has extensive experience architecting InfiniBand as component interconnects, and I’d think they could do this fairly easily with little development effort.

For the front end, I’d offer a little variety. It would come with either 8 or 16 Gb FC (FibreChannel), 10 and 40 Gb FCoE (FibreChannel over Ethernet) and for the I need it now, not when 16 Gb FC or 40 Gb FCoE comes out, InfiniBand. The last might take some writing of code, but the FC and FCoE would be pretty easy to plug in right away. I would leave in 1/10 Gb Ethernet for replication and those pesky legacy iSCSI people. (Here’s a nickel kid, go buy yourself a real SAN.)

All of this SSD, InfiniBand and high-speed interconnects and interfaces might require a little more oomph in the processing. Today, the nodes run Linux. IBM has extensive experience porting Linux to AIX, so why not just uplift those nodes to inexpensive p7 710 or 720 nodes? A quick change of platform, one the already supports InfiniBand interconnects and high-speed FC, FCoE and 10 GbE interfaces would add the additional processing power that an all SSD system might require. Or it might not. It’s worth looking at.

Lastly, I’d allow the grid to be stretched within between two metro-distanced data centers, allowing servers connecting to it to see it as one interconnected grid. (More in my next post on active-active datacenters).

Off The Chart Performance

If XIV is producing tier one performance with tier 2 disks and inexpensive components today, imagine what it could do with upgrading all of these things? It’s already cool. But what if … what if you wanted to bury everything out there – with scalable performance and capacity to go even further?

The magic of XIV isn’t the boxes making up the grid, it’s the software making the grid perform. The components today are cheap (or inexpensive as people prefer to say), what if they weren’t? What if it was truly built for performance – performance beyond anything we have today?

It’s already cool – but it could be cooler. It could be the most impressive, most scalable system out there. It might cost a bit more, but there are people out there who would buy it. People that still buy supercomputers today.


  1. I'd be happy if they offered a parity disk per tray, or a 3rd mirror position for business critical data.

    The non 50% 50% distribution between interface and data nodes is a little bizarre too... would like to see it symmetric.

  2. Passion and technology are two things I hold much respect. It's nice change of pace -- In fairness to your opinion, Let's just say I don't share the same rose colored glasses. Food for thought - An alternative perspective. If XIV innovated as you designed - improving the data classes and interconnects to allow for varied workloads - MultiTenancy Storage would have a home - This innovation would be called Quality of Service and The Storage Array would be called Pillar. No offense to Moshe but XIV lacks one key ingredient for being a repeat Symetrix/EMC - Erez Ofer - The prodigy software engineer responsible for EMC open systems success. Software being the paramount weakness. Symmetrix was a true innovation of it's day - Unified Storage with MVS and Unix data on a single shared array. Enterprise Storage ---

    Having worked for EMC my role provided several opportunities to collaborate w/ Erez, Moche, Donatelli etc... All information is not created equal. Why then build storage that services requests FIFO ?

    Executive platinum air travel is the best analogy for Storage I can offer.
    The card jumps the queue and from last in line to the First in line. Based on Priority. 120.000 IOPS concurrently served in accordance to their QoS priority - FIFO has no place in SSD, SATA, FC intermixed storage systems. Software is a requirement.

    Priority of DATA based on the application and dynamic allocation of CPU, CACHE and resources is the future. Moshe will surely induct to the Storage Hall of fame someday. Just not for XIV.

    Larry Ellison has spend 10 years and a respectable sum of capital on Pillar. They are Private, Not for sale and growing at an aggressive clip. Quiet Giant that walks softly and yields a Big Stick!

    When you've had XIV for breakfast - It's a game changing meal.

  3. I would be most surprised if they wouldn't already be working on the things you mention. So let's wait and see. Regarding the SSD's you suggest, I don't think it'll be inexpensive anymore then.

  4. As a customer I hope that XIV doesn't abandon its commodity parts architecture. This is what makes XIV so attractive ... $/TB, stability, availability. All the improvements listed above may be nice as options for the subsets of corporate data that require the performance boost. XIV in it's current state handles general SAN Storage requirements quite nicely.

  5. Am I the only one who's noticed that if you turn the XIV logo upside down, it spells AIX?