Followers | 13 |
Posts | 1802 |
Boards Moderated | 0 |
Alias Born | 06/15/2011 |
Thursday, December 14, 2023 3:09:51 PM
Karl Freund
Contributor
Founder and Principal Analyst, Cambrian-AI Research LLC
Dec 13, 2023,06:19pm EST
At the MI300 launch, AMD claimed it had significantly better performance than Nvidia. While the AMD chip does look good, and will probably run most AI just fine out of the box, the company did not use the fastest Nvidia software. The difference is enormous.
At a recent launch event, AMD talked about the inference performance of the H100 GPU compared to that of its MI300X chip. The results shared did not use optimized software, and the H100, if benchmarked properly, is 2x faster at a batch size of 1.
Nvidia has just released a blog that counters AMD's claim that its latest chip, the MI300X, is 40-60% faster in latency and throughput than Nvidia in inference processing for generative AI. Here is one of AMD’s slides at the MI300 launch event, which we covered here.
One of the incorrect slides AMD shared last week.
One of the incorrect slides AMD shared last week.AMD
Below is from Nvidia’s counterclaim. While this sort of tit-for-tat isn’t what anyone wants to hear, it is massively relevant this time; all the press and analyst reporting I’ve seen echo AMD’s claims, which are inaccurate and misleading.
The latest results, run on software available long before AMD prepared its presentation, doubled the performance claimed by AMD. And with batching for the 2.5-second latency AMD used, a standard in the industry, Nvidia beats the MI300 by an astonishing 14-fold.
The latest data from Nvidia leaves no doubt as to whose GPU is the fastest.
The latest data from Nvidia leaves no doubt as to [+]
NVIDIA
How Could This Happen?
It is simple. AMD did not use Nvidia’s software, which is optimized to improve performance on Nvidia hardware. “Though TensorRT-LLM is freely available on GitHub, recent comparisons by AMD used alternative software that does not yet support Hopper’s Transformer Engine and is missing these optimizations,” said the Nvidia blog post. Additionally, AMD did not take advantage of the TensorRT-LLM software that Nvidia released in September, doubling the inference performance on LLMs, nor the Triton inference engine. No TensorRT-LLM + no Transformer Engine + No Triton = non-optimal performance.
Since AMD has no equivalent software, it probably thought this was a better apples-t0-apples metric. These chips are expensive; I doubt anyone would not use the Nvidia software for production AI. It is free. “As LLM inference continues to grow in complexity, maximizing GPU performance on larger, increasingly sophisticated models using the latest inference software is critical to reducing cost and broadening adoption,” said Nvidia’s blog post.
What Does This Mean?
First, you can calm down if you are invested in Nvidia (stock or hardware). Nvidia remains the GPU leader. As Barron’s previously reported, “Investors Don’t Need to Worry.” And that was published before this latest news.
Second, if you are interested in the MI300X, we are not saying the new GPU is a bad AI platform. It appears to be the third fastest AI chip, behind Cerebras' massive WSE CS2 (for which there are no benchmarks) and the Nvidia H100. And that is probably good enough for those seeking a more available GPU that should be reasonably priced (whatever that means; AMD did not release pricing).
The AI hardware market is moving extremely fast, and the H100 will soon become old news. The H200 is coming more quickly than AMD probably hopes. We note that the MI300 FLOP specs are indeed better than Nvidia H100, and the MI300 also has more HBM memory. But it takes optimized software to make any AI chip sing and translate all those flops and bytes into customer value. The AMD ROCm software has made significant progress, but AMD still has much to do.
"AI is moving fast. NVIDIA’s CUDA ecosystem enables us to quickly and continuously optimize our stack. We look forward to continuing to improve AI performance with every update of our software,” said Nvidia.
Conclusions
While all this may seem like a tempest in a teapot to the uninitiated, all silicon vendors should work carefully to ensure accurate performance claims with actual data (not just normalized bar charts) and provide all the details necessary to reproduce those results. Handicapping a competitor's platform by not using the vendor's software isn't okay. That’s why MLCommons has published peer-reviewed MLPerf inference and training performance benchmarks every three months for several years.
Despite the kerfuffle, we stand by our earlier comments that AMD will sell every MI300 it can produce next year.
We asked AMD for a response and did not hear back.
When I asked Mark Papermaster, AMD CTO, if his company planned to run these benchmarks, he said they would publish MLPerf, but did not say when. We expect AMD will address the need for optimizations before they publish, and we can’t wait!
Recent NVDA News
- U.S. Stocks Close Mixed Following Lackluster Session • IH Market News • 06/21/2024 08:38:41 PM
- Honeywell Acquires CAES Systems for $1.9 Billion, Sarepta Therapeutics Surges 34%, Gilead Continues Gains • IH Market News • 06/21/2024 12:00:38 PM
- U.S. Futures Decline Ahead of Triple Witching, Oil Prices Edge Lower • IH Market News • 06/21/2024 12:00:27 PM
- Dow Closes Firmly Positive But Nasdaq, S&P 500 Give Back Ground • IH Market News • 06/20/2024 08:30:11 PM
- Nvidia May Lead Early Upward Move On Wall Street • IH Market News • 06/20/2024 01:10:49 PM
- U.S. Index Futures Surge; Oil Prices Increase • IH Market News • 06/20/2024 10:56:57 AM
- Trump Media Resells Stocks and Warrants; KB Home Exceeds Q2 Expectations, and More News • IH Market News • 06/20/2024 10:55:11 AM
- Nvidia Overtakes Microsoft To Become World’s Most Valuable Company • IH Market News • 06/19/2024 09:32:08 AM
- Form 144 - Report of proposed sale of securities • Edgar (US Regulatory) • 06/18/2024 09:16:15 PM
- Hewlett Packard Enterprise and NVIDIA Announce ‘NVIDIA AI Computing by HPE’ to Accelerate Generative AI Industrial Revolution • Business Wire • 06/18/2024 04:30:00 PM
- Fisker Files for Bankruptcy, Chegg Stocks Rise on 23% Workforce Reduction, and More • IH Market News • 06/18/2024 11:02:13 AM
- Form 144 - Report of proposed sale of securities • Edgar (US Regulatory) • 06/17/2024 09:06:07 PM
- Form 4 - Statement of changes in beneficial ownership of securities • Edgar (US Regulatory) • 06/17/2024 09:00:47 PM
- NVIDIA Announces Omniverse Microservices to Supercharge Physical AI • GlobeNewswire Inc. • 06/17/2024 01:00:00 PM
- Form 144 - Report of proposed sale of securities • Edgar (US Regulatory) • 06/14/2024 08:51:55 PM
- Form 144 - Report of proposed sale of securities • Edgar (US Regulatory) • 06/14/2024 08:29:57 PM
- Form 4 - Statement of changes in beneficial ownership of securities • Edgar (US Regulatory) • 06/13/2024 09:52:15 PM
- Form 144 - Report of proposed sale of securities • Edgar (US Regulatory) • 06/13/2024 08:39:28 PM
- NVIDIA Stockholder Meeting Set for June 26; Individuals Can Participate Online • GlobeNewswire Inc. • 06/12/2024 09:00:00 PM
- Form 4 - Statement of changes in beneficial ownership of securities • Edgar (US Regulatory) • 06/10/2024 08:55:57 PM
- Apple Showcases AI at WWDC 2024, Nvidia Stock Split Starts Today, and More News • IH Market News • 06/10/2024 11:29:44 AM
- Form 4 - Statement of changes in beneficial ownership of securities • Edgar (US Regulatory) • 06/07/2024 09:15:17 PM
- Form 4 - Statement of changes in beneficial ownership of securities • Edgar (US Regulatory) • 06/07/2024 09:14:22 PM
- Form 4 - Statement of changes in beneficial ownership of securities • Edgar (US Regulatory) • 06/07/2024 09:12:44 PM
- Form 8-K - Current report • Edgar (US Regulatory) • 06/07/2024 08:19:34 PM
Last Shot Hydration Drink Announced as Official Sponsor of Red River Athletic Conference • EQLB • Jun 20, 2024 2:38 PM
ATWEC Announces Major Acquisition and Lays Out Strategic Growth Plans • ATWT • Jun 20, 2024 7:09 AM
North Bay Resources Announces Composite Assays of 0.53 and 0.44 Troy Ounces per Ton Gold in Trenches B + C at Fran Gold, British Columbia • NBRI • Jun 18, 2024 9:18 AM
VAYK Assembling New Management Team for $64 Billion Domestic Market • VAYK • Jun 18, 2024 9:00 AM
Fifty 1 Labs, Inc Announces Acquisition of Drago Knives, LLC • CAFI • Jun 18, 2024 8:45 AM
Hydromer Announces Attainment of ISO 13485 Certification • HYDI • Jun 17, 2024 9:22 AM