Login to Account Create an Account
Frame Rating: the New Gold Standard in Graphics Card Performance Testing
UPDATE: PC Per have now interviewed NVIDIA about Frame Rating.
This system has served the gaming community well for over a decade now and gives a good indication of performance, but it doesn't tell the whole story, especially with multi-card setups. This is because it doesn't show how well individual frames are rendered and doesn't expose important problems such as runt frames and microstuttering, which can compromise the animation making it jerky, even when the raw GPU horsepower is there. Microstuttering is particularly relevant to multi-card setups too, SLI from NVIDIA and CrossFire from AMD. AMD in particular, has some questions to answer regarding runt frames and dropped frames in CrossFire setups, which we'll get to later.
Enter PC Perspective's Ryan Shrout with a new testing system and methodology designed to look at individual frames that he calls "Frame Rating" which takes a really close look at the true performance of any single graphics card or cards running in tandem. It's a financially expensive system to set up and a complex system to understand, so I'll just give a brief(ish) overview here and let Ryan explain it all fully in his article, which is well worth a read.
It turns out that the framerate in games which Fraps captures may not be the true framerate that's output to your monitor, which is due to the complex nature of creating frames when rendering a game's scene. The block diagram below shows the main steps required to achieve this:
PC Perspective's new Frame Rating system works by overlaying multi-coloured bars over a game frame by frame and then capturing actual live video frames directly off the digital video output of the graphics card using a special video capture card on another PC and then storing them in realtime on a very fast and high capacity 960GB external SSD RAID 0 array (4x Corsair Force GS 240GB SSD) using a high speed Thunderbolt data connection. The system under test also has its video output go to a standard monitor via a Gefen dual-link DVI splitter, allowing the tester to play the game. By watching the flickering coloured bars as they play a game, the tester can get a sense of how well the system is working, even without the full video capture and data analysis.
Datapath VisionDVI-DL capture card. Highest capture resolution is 2560x1600 at 60Hz, lossless
The resultant video data containing the coloured bars is then analysed via a set of Perl scripts called FCAT (Frame Capture and Analysis Tool) supplied to PC Perspective by NVIDIA, which generate masses of numerical data. This in turn is used to create specialized graphs of the realtime performance of the system. The coloured bars and the FCAT tool reveal the true rendering performance of the graphics card(s) and any graphics anomalies, such as completely missing frames or runt frames. A runt frame is one in which only a few scan lines are rendered, such as the one below and is quite detrimental to the performance of the system. Only AMD is affected by this, which only happens when using CrossFire with vsync off - a common gaming setting for fast FPS gaming to reduce input lag.
You might be thinking at this point that because AMD's primary competitor, NVIDIA, provided the Perl scripts, that this system would favour NVIDIA over AMD making them look bad. This isn't so however, as only the raw video data directly off the graphics card is captured which is identical to all graphics cards, hence the system is completely brand agnostic.
Runt frame from AMD HD 7950 CrossFire setup is visible along the centre of the screenshot and is clearly highlighted by the thin purple bar
The Frame Rating system can accurately analyze many aspects of rendering performance. First up, is the PLOT diagram below, which shows frame time variances - it's quite serious for the AMD CrossFire setup, hence the mass of orange. It should look more like the thin blue line which represent the NVIDIA SLI setup. Note that the AMD Eyefinity setup below it shows masses of completely dropped frames too, leading to laggy performance that doesn't feel any better than a single card.
AMD Eyefinity CrossFire setup completely drops masses of frames, seriously compromising performance.
The test system also produces the percentile graph which shows you the minimum frame rate that will appear on the screen at any given percent of time during a benchmark run.
Then we have the RUN graph, which shows any runts or completely dropped frames. The example below shows that the AMD CrossFire setup is riddled with runts - every other frame is a runt in fact, seriously reduciing real-world rendering performance, which is why the situation is so damning for AMD. There should just be one line showing, with the Fraps and FPS lines perfectly overlaid, with no orange or red showing at all. There are no dropped frames in this test, hence no red in the graph.
Next up is a Fraps time graph, showing what Fraps captures and is the same as conventional graphics card testing. Note how the AMD CrossFire setup benches only a little way behind the NVIDIA SLI setup in this test.
Next up is the same Fraps time graph called Observed FPS, but with the runts and missing frames removed. Suddenly, CrossFire performs far worse than the SLI setup, offering less than half the performance. This is the actual performance that the gamer sees and is an extremely poor showing for AMD after having said gamer spend a lot of money on two of AMD's HD 7970 cards. In fact, the framerate is actually significantly lower than a single HD 7970! This leads to rough, stuttery gameplay that's highly noticeable even without objective testing like this.
Finally, we have the frame variance graph, which attempts to quantify the difference in rendering times between frames over time. This is done in an attempt to measure the amount of visible stutter and is something which Ryan is still developing, to bring us an ever better objective representation of the experience.
As you can see, this new Frame Rating method of measuring graphics card performance produces masses of data for the reviewer and the reader to chew over, which shows up many more flaws and anomalies in graphics card rendering performance. While it might be a lot to take in, it allows the reader to make a much more informed choice when choosing a graphics card to buy. To me, this makes PC Perspective's graphics card performance tests the new gold standard and the ones to go to until the competition catches up.
Ryan has been in discussion with AMD about these serious performance issues, which AMD have admitted to. Incredibly, it appears that AMD aren't sure of the exact causes of the problems yet and hope to have a fix in 2-3 months' time! To me, this sounds like a good reason to avoid buying their products until they sort this out. We all know how weak AMD is nowadays on the CPU front, but it wasn't known just how bad their CrossFire setups were, which are not cheap. Note that these problems mostly disappear when either vsync or just a single card are used. Certainly runt frames disappear, but there's still too much frame variation and stutter.
This is what Ryan had to say about these performance issues, which won't make pleasant reading for AMD:
Another problem this causes for AMD and its partners is with dual-GPU hardware, the Powercolor Devil 13 or the ASUS ARES II. While both are powerful graphics cards, they depend completely on CrossFire technology to meet the performance expectations set by marketing and by cost and a quick glance at the HD 7970 CrossFire results I’ve showed you today (which basically mirror an ARES II) tells you that neither option is worth the price of admission.
It should come as no surprise that AMD was only recently made aware of these performance issues and was kind of stunned to find out how bad they were. The truth is that AMD could “hack” together a fix to make our frame time graphs look better but it would likely be a solution that introduced more problems than answers without doing the research required to get it right. The driver team has told me several times over the past two weeks that they should have a testable driver to fix the CrossFire problems “in 2 to 3 months.” Until then, buyers that consider a multi-GPU solution a goal or a requirement will want to seriously debate dropping Radeon cards from consideration.
Since there's such a lot of data produced by the new Frame Rating test methodology, PC Perspective are publishing this over several articles to reduce information overload for their readers, according to the following timetable:
- March 27: Frame Rating Dissected: Full Details on Capture-based Graphics Performance Testing
- March 27: Radeon HD 7970 GHz Edition vs GeForce GTX 680 (Single and Dual GPU)
- March 30: AMD Radeon HD 7990 vs GeForce GTX 690 vs GeForce GTX Titan
- April 2: Radeon HD 7950 vs GeForce GTX 660 Ti (Single and Dual GPU)
- April 4: Radeon HD 7870 GHz Edition vs GeForce GTX 660 (Single and Dual GPU)
- April 6: Radeon HD 7850 vs Radeon HD 7790 vs GeForce GTX 650 Ti vs GeForce GTX 650 Ti BOOST
- alexp999 likes this