HPC

I posted this at last year’s ComputerWorld Canada Blogging Idol contest, although a controversial position, it’s worth discussion:

A few months ago, I was chatting with a friend on the phone about HPC when he made the statement which is the title of this post.  I jotted it down, and it’s been rattling around in my brain ever since.  At first it seemed very odd, HPC is a neiche market, a small proportion of the overall IT market.  The more research I did the more it made sense, especially after I read a book called “The Fourth Paradigm: Data-Intensive Scientific Discovery”, free for download from Microsoft Research.  It’s not the typical marketing piece, but an excellent read if you enjoy Science and IT.  It speaks about four paradigms of Scientific enquiry and then goes on to expand on the new discipline of data-intensive computing.  The four paradigms of Scientific persuit are:

1. empirical – describing natural phenomena, performing experiments
2. theoretical – models, generalizations, developing ‘laws’ of science
3. computational – simulation of complex phenomenon
4. data exploration – eScience, discoveries are to be found in searching huge volumes of captured data from devices recording natural phenomena, experiments generating new results and simulations modelling problems in levels of detail beyond that acheiveable in paradigm 3.

In last year’s blog idol contest, I put some focus on the technology adoption curve and where I thought virtualization technology fell for 5 different classes of computing; Mainframes, Servers, Desktop,Embedded and High Performance Computing .  In the final category, HPC, I concluded that virtualization technology was in the earliest stages of adoption.  Compute virtualization is a pre-requisite technology to cloud computing (some might consider shared HPC resources using a batch scheduler without virtualization as cloud computing, but I think that’s stretching the definition of cloud too far).  But my experience in HPC to that point had only been up to paradigm 3.

Paradigm 4 is a poster child for cloud computing, and makes the case of HPC dominating cloud compute cycles an obvious conclusion the more you think about it.  HPC users are a small community, traditionally in the scientific and engineering disciplines.  With the advent of data-intensive science, this small community will consume a disproportionate amount of the computing capacity available in the world.  Not only that, but the HPC community is growing beyond science and engineering.  Business analytics are making their way into HPC to accelerate time to solution, exploring mountains of structured and unstructured data for patterns that provide intelligence towards competitive advantage.  I’ve also read about the potential for HPC in the Humanities.

We’ve got big data from science, engineering and now business too; open data from governments world wide as well as organizations like the World Bank.  So much data and so little time - HPC enables us to look at the data in different ways with thousands of scenarios, perhaps millions of scenarios being tested concurrently.

Is ‘Cloud Computing’ = HPC?

That depends on who you talk to.

According to the U.S. National Institute of Standards and Technology who have published the official definition for cloud computing, cloud computing uses loosely coupled resources and HPC uses tightly coupled resources.  Thus in this case, cloud computing ≠ HPC.

Some cloud service providers have begun to offer compute resources with HPC attributes, so in this case cloud = HPC.

But then some HPC programs are designed to take advantage of large numbers of loosely coupled processes, so in this case cloud = HPC.

HPC is about performance and scalability. Cloud computing is about service delivery and scalability.  They are complementary rather than competing technologies.

Copied here is an impressive chart on the projected growth of HPC computational requirements in the global defense industry.  It looks like about an order of magnitude increase in demand for HPC cycles in the next three years; that’s definitely ahead of Moore’s Law.  Makes me wonder what the forecast is for Canada, I have been searching for data on HPC use in the Canadian forces, but haven’t found very much.

The point of the article is the emphasis on use of commodity Commercial Off The Shelf (COTS) components, to assemble large scale systems that are application specific – ie. the system is carefully tuned for one specific parallel program.  What is really interesting is the price/performance argument made for building an HPC cluster from Sony Playstations.  The cell processor in the PS3 is a very powerful compute engine, much more powerful that the AMD|Intel chip in your laptop.  Because of the manufacturing volume of the games market, you get a very high performance compute node for the fraction of the price of an equivalent performance compute server; and they don’t even need the bluray drive or hard drive, which would make the unit more energy efficient, less noisy and even cheaper.  That reminds me, the PS3 I bought the boys is really noisy, you’d think Sony could have done a better job with that; conductive cooling and a solid state drive would have been really cool (which by the way is also a requirement for military embedded systems).

Curious then that IBM has announced that there will be no further development of this amazing processor chip.  Curious as well that the machine we buy our kids to simulate war games, when configured as an HPC compute cluster, is a powerful tool to manage the real thing.

Update – unfortunately the Sun Microsystems links are broken, but the link to the other blog post is still good.

Sun Microsystems has a nice write up on the new high performance computer system they built at the University of Laval for the CLUMEQ consortium.  There is also a nicely done interactive video tour presenting a clever re-use of an obsolete cyclotron silo for a server room. Not only is a landmark building preserved, but it actually turns out to be an efficient way to build a data center or supercomputer. This blog post has a nice collection of graphics to illustrate the new system.

The “cylindrical machine” has three floors of equipment racks, each configured in a circle, with the cooling infrastructure in the basement. This allows for highly optimized air flow, which they modelled with Computational Fluid Dynamics (CFD) simulations. The “hot aisle” being the centre of the building. One of the speakers on the video shows alot of enthusiasm for the building itself being the CRAC (Computer Room Air Conditioner).  For most of the year the system uses “free air”, or outside air cooling.  Although there are cold water chillers, I’m assuming that this is a simple heat pump configuration, as the coils are reversed in the winter to transfer waste heat from the system into other buildings on campus to further improve their Power Usage Efficiency (PUE).  It is said that the entire system “breathes naturally”, and also incorporates Intel’s Intelligent Power technology, so that power consumption can be scaled to compute load. The high efficiency fans also dynamically adapt to cooling needs, and there is enough heat buffer in the facility to operate safely even if the chillers fail.

From a networking perspective, with three circles of equipment racks on three floors, and the networking equipment placed on the second floor, the 75 racks of equipment to all be connected with cables no longer than 10m, something that could never be acheived in a standard server room.  Not only does this save cabling costs, but should save some power by minimizing the optical power budget (I’m assuming optical as 10m is a stretch for copper at QDR Infiniband rates).  The cluster networking is configured as a fat tree with full bisectional bandwidth. This means that the bandwidth between any two servers is constant, thus there are no bandwidth bottlenecks anywhere in the system. This is an important attribute of a high performance cluster, as any imbalance in the system will directly impact “time to solution”. That’s quite an accomplishment for a cluster with 960 compute nodes.

It is fascinating that a technology originally developed for performance and utilized for that purpose in the planned Gordon supercomputer currently being built at the San Diego supercomputer center, is being used as a power saving technology. So we can have faster performance and save energy at the same time; it sounds almost too good to be true!  This article shows how MySpace uses FusionIO solid state drives that mount in a PCIe slot, not primarily for performance, but as a power saving strategy.  In this case, the higher cost of solid state storage must be offset by the savings in electricity. With this technology increasing in density and decreasing in price, that intersection was inevitable. The figure given is that the power consumed by solid state storage is 1% of an equivalent solution using spinning disks; keep in mind that this is more that the elimination of the spinning disk power consumption, but also the power cost of cooling, which in many cases can be roughly equal to the cost of powering the servers.  Makes you wonder if this has the potential to be a threat to the shared storage industry; shared storage provided data centres with better performance and reliability, as well as an easier method for sharing data and having a common back up scheme.  If local solid state storage provides a better solution for performance, reliability and scalability than shared storage while saving a lot of electricity, then this sounds like a game changer for the storage industry, if you can find a solution for sharing backing up data.  You’d think this would make for a rather complex file system requirements, as you would need to share local disk data across all servers. Although an older article, it seems that MySpace has already created a home grown distributed file system solution for that purpose a while back.  Maybe it’s time to sell stock in network storage companies, and move that money into solid state storage companies, as if solid state storage wasn’t a disruptive technology already.