What’s the H100, the chip driving generative AI? (2024)

It’s rare that a computer component sets pulses racing beyond the tech industry. But when Nvidia Corp issued a blowout sales forecast in May to send its market value above US$1 trillion, the star of the show was its latest graphics processing unit, the H100. The new data centre chip is showing investors that the buzz around generative artificial intelligence (AI) – systems that can perform a wide range of tasks at superpowered speed – is translating into real revenue, at least for Nvidia. Demand for the H100 is so great that some customers are having to wait as long as six months to receive it.

What is the H100?

The H100, whose name is a nod to computer science pioneer Grace Hopper, is a graphics processor. It’s a type of chip that normally lives in PCs and helps gamers get the most realistic visual experience. Unlike its regular counterparts, though, the chip’s 80 billion transistors are arranged in cores that are tuned to process data at high speed, not generate images. Nvidia, founded in 1993, pioneered this market with investments in technology going back almost two decades, when it bet that the ability to do work in parallel would one day make its chips valuable in applications outside of gaming.

Why is the H100 so special?

Generative AI platforms learn to complete tasks such as translating text, summarising reports and writing computer code after being trained on vast quantities of pre-existing material. The more they see, the better they become at things like recognising human speech or writing job cover letters. They develop through trial and error, making billions of attempts to achieve proficiency and sucking up huge amounts of computing power in the process. Nvidia says the H100 is four times faster than the chip’s predecessor, the A100, at training these so-called large language models, or LLMs, and is 30 times faster when replying to user prompts. For companies racing to train their LLMs to perform new tasks, that performance edge can be critical.

How did Nvidia get pole position?

It’s the world leader in so-called graphics processing units (GPUs) – the bits of a computer that generate the images you see on the screen. The most powerful GPUs, which can produce realistic-looking scenery in fast-moving video games, have multiple processing cores that perform several simultaneous computations. Nvidia’s engineers realised in the early 2000s that GPUs could be retooled to become so-called accelerators for other applications, by dividing tasks up into smaller lumps and then working on them at the same time. Just over a decade ago, AI researchers discovered that their work could finally be made practical by using this type of chip.

What’s the state of the competition?

Nvidia controls about 80 per cent of the market for the accelerators in the AI data centres operated by Amazon.com’s AWS, Alphabet’s Google Cloud and Microsoft’s Azure. Those companies’ in-house efforts to build these chips, and rival products from chipmakers such as Advanced Micro Devices (AMD) and Intel, haven’t made much of an impression on the accelerator market so far.

Why is that?

Nvidia has rapidly updated its offerings, including software to support the hardware, at a pace that no competitor has been able to match up till now. Chips such as Intel’s Xeon processors have fewer processing cores. While they’re capable of more complex data crunching, they’re much slower at working through the mountains of information typically used to train AI software. Nvidia’s data centre division posted a 41 per cent increase in revenue to US$15 billion in 2022.

SEE ALSO

See Also

NVIDIA H100 Tensor Core GPU

If Nvidia keeps rising like this, it will be bigger than the global economy

Are others catching up?

AMD, the second-largest maker of computer graphics chips, unveiled a version of its Instinct line in June aimed at the market that Nvidia’s products dominate. The chip, called MI300X, has more memory to handle workloads for generative AI, AMD chief executive officer Lisa Su told the audience at an event in San Francisco. “We are still very, very early in the life cycle of AI,” she said. Intel is bringing specific chips for AI workloads to the market but acknowledged that, for now, demand for data centre graphics chips is growing faster than for the central processor units that were traditionally the company’s strength. Nvidia’s advantage isn’t just in the performance of its hardware. The company invented something called Cuda, a language for its graphics chips that allows them to be programmed for the type of work that underpins AI programs. BLOOMBERG