At the apex of the computing world, super computers are edging toward “exascale”, looking to achieve ExaFlops (and also Exabyte) computations. Scaling at the high end is always tough engineering, but in this case it is harder than ever before for one really fundamental reason: power consumption.
The trick really isn’t to amass processors, memory, and wires, it is to do it with far less power consumption per unit than conventional computers. It’s simply too expensive to run super computers, and pretty much impossible to go bigger unless operating is much cheaper.
Rachel Courtland writes in IEEE Spectrum about the Chinese Sunway TaihuLight which is now ranked number one in computation, with a reported 93 petaflops on the Linpack Benchmark. This is an astonishing achievement, doubling the previous leader, Tianhe-2 (though it is only .093 exaflops).
Courtland reports that this speed is achieve by trade offs, that trade numerical computations (used by Linpack) against data movement and everything else (not used in Linpack). This is a very familiar tradeoff, dating back to the beginning of computing, and exacerbated in the 1980s by the de facto adoption of Linpack as the measure of “speed”.
Reflecting this tradeoff, it is reported that the more balanced High Performance Conjugate Gradients (HPCG) benchmark shows Sunway TaihuLight achieves 0.3 % of its theoretical peak. Other supercomputers usually achieve 1-5% peak on this benchmark. Basically Sunway TaihuLight is extremely specialized.
The report suggests that the system uses lower power memory, and a decentralized memory (close to the processors). This dramatically reduces power needed for moving stuff around memory, but man, oh, man, it is going to be hard to program!
This is not to say that this isn’t a great system. It’s an awesome bit of engineering, and, for some uses, it will deliver astonishing performance. Even if it is not a complete or perfect solution, it is still a fine piece of work. (Think how much fun it would be to write a program to run on 10 million mobile phones.)
Linpack didn’t make sense as the performance benchmark when I was a lad, but we had to score big on it for PR and marketing purposes. It makes no more sense now.