Advanced Micro Devices Inc (AMD)

[Petz]

Have any of you looked at the disclosure document for BAPCO 's new Sysmark 2004, which consists of the Internet Content Creation benchmark and the Office Productivity Benchmark. Overall Sysmark score is (as it should be) the geometric mean of these two (square root of the product). I find the application weightings absolutely insane, and the benchmark still favors Intel processors over AMD. How AMD let them release this is beyond me.
http://www.bapco.com/techdocs/SYSmark2004WhitePaper.pdf

1. The document is sorely lacking in details. It lists the applications and types of documents, but doesn't list specific filters or document sizes.

2. Internet Content Creation has way too few applications (9) in by book, especially when you see how they are weighted in the overall score. Only 5 apps have significant contributions.
Discrete Software's 3D Studio Max at 32%
Windows Media Encoder 9 Series at 19%
Adobe Photoshop 7.0.1 at 14%
Adobe Premiere 6.5 at 13%
McAfee Viruscan 7.0 at 10% (also contributes 27% of Office Productivity benchmark)
That's 88% of the total score in 5 apps and over 50% in just two. (Data derived from table on page 25.)

3. Office Productivity similarly has too few applications (10) and the top four contribute 77% of the total score. They are:
McAfee Viruscan 7.0 at 27%
Microsoft Excel 2002 at 26% - Again >50% in just two apps!
Adobe Acrobat 5.0.5 at 13%
Dragon Naturally Speaking 6.0 at 12%

4. Why McAfee Viruscan? Isn't Symantec (Norton) far more common in the corporate environment? Viruscan 7.0 has the single highest influence on Sysmark 2004, followed by 3D Studio Max. Combined they determine 35% of the score.

5. Dragon Naturally Speaking is given way too much weight.

6. I can't complain about Excel's heavy weight, but when Sysmark 2002 was analyzed, they found that the only feature used in Word was a rarely used sorting macro, rather than lots of math in a huge spreadsheet. So if they are using the script from Sysmark 2002, its bogus.

7. Microsoft Word is given way too little weight - 3% of Office Productivity. Word can take quite a while just to scroll through a document with lots of embedded graphs, charts and spreadsheets. I don't believe they did anything like that in this benchmark.

8. See figures 4,5. The benchmark cleverly rewards the fact that hyperthreaded systems will complete the background task faster, at the expense of a slower foreground task. This gives a huge boost to HT! Here's how. While rendering in Premier in the background, the benchmark starts a bunch of short content creation tasks in the foreground. When these operations are done, it starts a long Photoshop operation in the foreground. Here's what happens:

On an HT system, the small foreground content creation tasks take longer to execute because the HT is giving more time to the background Premiere rendering. So, it just happens that by time the Photoshop operation is started, Premiere is done! Photoshop, which is extremely memory sensitive, gets to run without Premiere running.

On a non-HT system, the foreground content creation operations are indeed completed faster, because while they are executing, the background Premiere rendering doesn't get much execution. So, because the HT system gets the background operation done more quickly at the expense of many smaller foreground operations, the hypothetical "user" of the non-HT system winds up running Photoshop when it is starved for memory. If the sequence of short operations were 20% longer, or the Premiere rendering was 20% shorter, there would be no overlap between Premiere and Photoshop. BAPCO just invented this clever contrived method of penalizing systems with good foreground response time vs. background response time.

How did AMD, as a member of BAPCO allow this to happen, AGAIN? It will be interesting to see if there is a "fix" to the Centrino weakness in the ICC sub-benchmark.

Petz