Tag Archives: benchmarking

Benchmarking tweets

I asked what others did for benchmarking in my last post. Here are the replies on Twitter in a semi-coherent edited form. If I missed any replies, I blame Twitter, whose interface is a magical maze.

First there were some FPS vs. SPF comments:

Richard Mitton: If you’re not measuring in milliseconds then you’re doing it wrong.

Christer Ericson: Yes, ms, not FPS. FPS is not a linear unit for the artists (or anyone).

Marc Olano: FPS isn’t linear. Usual definition of median averages middle 2 for even samples = also wrong. Use ms.

Morgan McGuire notes: FPS *is* a good measure if what you care about is interaction or visual smoothness. SPF is good for computational efficiency.

I replied to Richard & Christer: I’m interested in your reaction to the use of median vs. mean. FPS vs. SPF irrelevant for relative performance.

I also changed the original post to talk about milliseconds instead of frames, to avoid this facet of the discussion.

Christer Ericson: It’s important to catch the spikes, so in the context you’re talking about I would do max. Or mean+variance. Also, don’t think I’ve ever, for profiling reasons, looked at any average. You always look at a specific frame.

Timothy Lottes: I’m personally only interested in worst case ms/frame.

Cass Everitt: Agree with those that concentrate on worst times.

Eric Haines: Right, it depends what you’re looking for, e.g. don’t drop below 60 FPS. I’m mostly warning against using mean.

I added a note to the original post about tracking the max, which makes sense if you’re trying to guarantee a frame rate.

Tobias Berghoff, who benchmarks consoles:

I use min/max/med the most. Averages really only come into play when I need more digits. I spend significant amount of time below the 0.5% mark when wearing my platform tuning hat. I don’t miss trying to get sensible numbers out of PC h/w. But this also comes into play when measuring very short processes. When something only takes a couple of microseconds, you often end up oscillating between states that make the distribution multi-modal. Median won’t catch small shifts.

cupe: Stacked color-coded graph of nested timings (or a subtree of it). Usually unfiltered for analysis, avg for comparisons. Hierarchy is on the left, tooltip displays e.g. “scene/fluid/poisson”, click to restrict. Horizontal lines are milliseconds, orange line is 16.6 ms.

cupe1

E.g. click the big violet bar to see only post (and zoom in to stretch 4ms to screen):

cupe2

Javdev: We use a profiler, Adobe Scout, select multiple frames & see which code is most expensive & iterate it to prevent frame drops.

Björn Blissing: One option is to plot a histogram over the captured data. Reveals if your max/min are outliers or more common occurrences.

Michael Marcin: Try always running circular etwtrace and when frame time dips save and examine the trace.

Mikkel Gjoel: We filter in viewer. Options for all mentioned, and vsync (as that is what we are shipping).

Gjoel

Fabian Giesen: General order statistics (percentiles etc.) are good. Just a plot of frame durations over frame # is helpful, too! And simply recording all frame durations over a few seconds, sorting them and plotting that is quite handy, too. That gives you all the percentiles (and median etc.) and gives you a feel for the shape of the distribution, which matters. (I’m not very happy with single-value summaries; they lose too much information.)

Jaume Sanchez Elias: I like Chrome FPS meter: current, min, max; over time; frequency graph for each framerate

Elias

Krzysztof Narkowicz: Min, max, avg and std dev. Percentiles and med would make a nice addition, but it’s a hassle to compute them.

Anton the Mighty: I always use the standard deviation or standard error and indicated what value n sample size is. Most gfx benchs=bad. It’s usually worth also eyeballing actual data in detail because repeating patterns show either cycles or error in timers. Most recently there was something a friend had with the power manager in windows causing a cycling load on the cpu. I also visually check out timing for cpu+gpu functions across frames with apitrace etc. pretty neat.

All for now – feel free to email or tweet me with anything you want to add.

 

Don’t be mean

[Some on Twitter noted that I should be using milliseconds instead of FPS. This kind of misses the point, but let’s avoid distractions, here’s the article with that change. The sad part is that you then miss my hilarious joke about how I use FPS in the article, because if I used SPF you’d think I was talking about tanning. Which makes me think of another joke about rendering cows and the time it then takes to tan their hides. I’m full of great dad jokes.]

I think I’m reading “The Economist” too much, as I keep trying to come up with punny article titles. Sorry.

So, how do you measure a representative value for milliseconds per frame?

I don’t care about the mechanics, which timer call you use, etc. Just assume you successfully start timer/end timer and get some length of time in milliseconds for the frame. What do you do with these timings?

I usually see things such as an average, or a running average (average of last 20 or 50 or 100 or whatever frame times). I think this is mostly bad. As someone pointed out, almost everyone has more than the average number of legs. I find the same: in a given run there can sometimes be some frames where things noticeably slow down for whatever reason, some load on the computer. What you’re often trying to measure (as a graphics developer) is the performance of the rendering system itself, not the computer’s overall performance.

So, I currently use one of these two, or both: shortest time, or median time, over whatever set of frame times I have. Both have their uses. Shortest time is justifiable (to me, at least) because, assuming you have a very fine-grained timer, your best time is in some sense the “purest” measurement of the time a frame takes. Whatever other processes in your system are slowing down the other frames isn’t your concern. The timer doesn’t lie, you really did go that fast for one frame.

The other measure I’m OK with is the median. If your benchmarking system is going through a series of different frames (an animation or simulation is running, or the camera is orbiting, etc.), then grabbing the median frame is good. Choosing it instead of the average then doesn’t give so much weight to outliers. Better yet, graph the results and see whether the outliers are consistent.

Update: A number of game and VR developers pointed out that their major interest is maximum frame time. Makes sense: for a good experience (especially with VR) you don’t want to drop below your target of 30 FPS, 60 FPS, or 90 FPS.

My point is that the average, the mean, is not so good: often external slowdowns throw off the average enough and at random enough intervals that the average is very noisy and so, pretty useless. Taking the median, the central time of the sorted set, cuts out much of this variance, making each sample have an equal effect on the result.

Anyway, that’s where I’m at with benchmarking. What do you do? Comment here, tweet-reply, or email me at erich@acm.org and I’ll summarize.

p.s. pro tip: walk through your rendering pipeline every once in awhile, watching each step. It’s hard to really know where the time goes without doing so. I did this last week while looking at another bug and found a little logic error was causing a certain path to always do an additional post-process when it usually wasn’t needed. Free performance boost with a two-line fix! But, not something discoverable by benchmarking, because the variance is too much to notice “just” a few frames of difference.

This happens every few years. My favorite lucky find was around 15 years ago, walking through code in an established project and seeing that it was rendering twice for each time it displayed. A one-line change gave us 2x performance.

Seven Things for June 6th

It’s D-Day and it’s been awhile, so let’s get going. This is a LIFO of the 486 backlogged links I’ve collected for this blog:

  • GPUView looks like an interesting profiling tool from some students at Stanford (done as interns at Microsoft, which has a more official page), though I’ve heard it’s a bit of work to set up. If you’ve used it, how did you find it?
  • Open source code for a fast and scalable GLSL GPU implementation of the Perlin noise with functions, not textures.
  • NV Path Rendering is not what you might think, it’s about rendering text and 2D paths with quite a bit of elaboration available (think SVG or other 2D vector descriptions). GTC presentation here.
  • The book “Physically Based Rendering” is now in eBook form, including PDF (so I assume no DRM?). Annoyingly, it costs considerably more than the physical book on Amazon, but that’s the publisher’s doing.
  • Proland looked intriguing, a procedural terrain generator that creates based on view. Appears fairly elaborate, and a quick way to get some plausible-looking terrain data.
  • Geekbench is a cross-platform benchmarking system; from what I’ve heard, mobile platforms kind of set the clock back a fair number of year in terms of performance. Still, 3D is doable (it certainly was in 2002); here’s a starter list of 3D CAD apps for Android (many are on the iPad, too). I need to search out more, I’m interested in what’s out there.
  • Finally, in the category “this looks like a painting but is reality”, a photo taken in Namibia:

Odds and Ends

It’s 5/7/09, a nice odd sequence, so time for a few odds and ends I’ve collected.

OK, this is worth a few minutes of your life: the elevated demo is awe-inspiring. Terrain generation (be patient when you start it), fly-by’s, and music, all in less than 4096 bytes. By way of comparison, an empty MS Word document is 9834 bytes. (thanks to Steve Worley)

Google has put out a browser-based low-level 3D graphics API called O3D. API here. Demos here. Some initial impressions here. It will be interesting to see if they succeed where so many others have failed.

There is a call for participation out for a new book series called “Game Engine Gems“, edited by Eric Lengyel. (thanks to Marwan Ansari)

The main thing I look at on the SIGGRAPH exhibition floor are the book booths. Good books are such a ridiculous bargain: if a book like Geometric Tools saves a programmer 2 hours of time, it’s paid for itself. One new book that I want to see is Real-Time Cameras, by Mark Haigh-Hutchinson, which came out this April. Looking around for more info, I noticed this sad note. I never met Mark, but we corresponded a few times. He came up with a clever idea to avoid performing division when doing a point in polygon test; I folded this into the CrossingsMultiplyTest Graphics Gems code here, crediting him.

I’ve been looking at GPU capabilities and benchmarking information lately. Some nice resources:

  • You probably know about the benchmarking group Futuremark. Me, I hadn’t realized they had useful stats at their site: see the Futuremark ORB links at the bottom of the page and start clicking.
  • Two applications that tell you a ton about your card’s capabilities: GPU-Z, with a ton of information and a statistics page & cute map of downloads at their site, and GPU Caps, which also includes CUDA-related information and some nice little OpenGL benchmarks.
  • Chris Dragan has a web database that provides a fair amount of data on card support for DirectX capabilities and OpenGL extensions.
  • The Notebook Check site had way too much information about many laptop graphics accelerators.
  • nHancer is a utility for NVIDIA cards. It lets you get at all sorts of different capabilities on your GPU, on a per-game basis. There are also interesting antialiasing and anisotropic filtering comparison pages (click on the radio buttons). (thanks to Mauricio Vives)
Some interesting libraries I ran across lately:
  • GTS is an open-source mesh manipulation package.
  • Box2D is a 2D physics engine.
  • Touchlib is a multitouch development kit. (thanks to Morgan McGuire)

Coincidental world: it turns out there’s a different “Eric Haines” out there that made a well-received 3D graphics game for the iPhone, Realmaze 3D. I’m not sure how it compares to his The Magical Flying Pink Pony Game, which looks awesome. (thanks to Nikolai Sander)

I’ve seen similar real-world illusions, but still thought this one was pretty great. (Addendum: Morgan McGuire found this even-better video of the effect.)