Cloud providers are too expensive an option for many high-end academic computing projects, the head of a London-based team crunching Large Hadron Collider data said this week.

Professor John Hays, head of the particle physics research centre at Queen Mary University London was speaking as the university unwrapped a project which uses the vast amounts of heat generated by the unit’s data centre to provide heat and hot water across the campus in East London.

The centre, which is part of the Grid for Particle Physics analysing data from CERN’s LHC project, had started running into problems with its infrastructure, based on air cooling systems that were past end of life.

“We were keeping it running with small cash injections to put emergency cooling In and whatever, which is way cheaper than refitting a whole room,” said Hays. “But clearly not a sustainable way to keep going.”

He added that the standard when the centre was first built around 20 years ago was a PUE of 1.8, well below the 1.2 and below levels new datacentres would aim for.

See also: Digital Realty: CIOs have "no clue" about data centre performance, energy KPIs

The project, developed with Schneider and APT, and based on the French power firm’s EcoStruxure Row Data Center Solution, uses liquid cooling to recover the heat from the datacentre, which is rated at 390KW.

The “hot aisle” of the data centre operates at 40 degrees C. That means that at 100% capacity, the centre would produce the equivalent of 700 tonnes of CO2 a year, or put another way, 2.8million 10-minute showers.

Under the new system, the captured heat is then passed through a heat exchanger and ultimately used to heat water and provide heat across the campus.

That means the University is, in effect, getting “free” heat as well helping it reach its broader sustainability goals by reducing CO2 emissions. For example, it can power down the four gas boilers in the science block housing the datacentre and new district heating system, as well as other heating systems across the campus.

It also helps offsets the problem of getting further electricity onto the campus, a challenge for datacentre operators and developers alike.

Hays added that the datacentre, which has 20PB of storage, was largely x86 based, with “a little bit of GPU”. Ultimately some of the workloads would run more efficiently on GPU, he said, but “Most of the people using our systems are not really ready to move to fully exploiting GPUs. There's a bit of a development gap on the software side there.”

ARM-based systems were being investigated by “some our collaborator sites, which is showing some promise” he said.

“Longer term, we need to look at new architectures to gain those sustainability goals. And it might not be GPUs.

And while NVIDIA is talking about the dawn of the 1MW rack in a few years, Hays was confident the new cooling setup would not become redundant anytime soon. “There's an upgrade path there, because we've got water coming to the racks, even though it's going to the chillers. There's an upgrade path to have water to the chip capability.”

An alternative would be turn to the cloud. However, Hays said this was not really an option for many in the scientific computing community.

“It's fairly rare we would use commercial cloud providers because our requirements are fairly specialized, and if you work out the total cost, it's substantially more expensive to use those services,” he said. “The flexibility of being able to adjust to demand comes at too high a price.”

Unsurprisingly, “It's always the data egress charges. We've got two exabytes of data. What's that going to cost? Well, that's the budget of a small country to move that around.”

On top of that, he said, the value of the data from the LHC was almost inestimable – and it would be a leap of faith to trust it to a third party. “We have a public requirement to maintain that data that we've paid for with, largely with taxpayer money, and to be able to conserve it as a resource for the future, for the public good.”

Were a third party to lose it, he said, “The cost of reproducing all of that is 40 years of people's lives and work and billions of pounds of investment.”

Join peers following The Stack on LinkedIn

The link has been copied!