I didn’t really intend this post to be a guide to “useful” optimizations though. It’s just intended to highlight some of the quirks and complexities of low-level optimization. If anything, it should be taken as a warning that trying to optimize by fiddling with individual ASM instructions is going to cause a lot of headaches, and you’ll see some unexpected results in many cases.

I agree that remembering what the combination of OoO and superscalar can do is important, and easy to forget. Witness Mike Pall and LuaJIT2 for the payoff…

All things considered, cache layout is probably the most practical performance-related “thing” to keep in mind for most programmers. It’s much easier to reason about memory access patterns than asm dependency chains. But of course I wish I had measurements to back my intuition up.

My favorite example of the effect of a (relatively) obscure hardware structure on code is that inline caches make more sense than C++ style vtbls on a processor without a BTB, but the BTB helps the vtbl more than the inline cache. So modern processors are, in effect, optimized for C++-style virtual dispatch!

True, but there are still areas where such micro-optimizations may be useful. First, if you know what the CPU actually does to your code, you can make some more general assumptions about what is fast and what isn’t. Taking the example in my post above, the exact latency might vary, but I doubt you’ll find a CPU where integer multiplication has less latency than addition. So we can still reason about what is preferable inside a tight loop where the combined latency is what’s holding us back. Or knowing that almost every modern CPU is superscalar, we can determine that a higher instruction count isn’t necessarily worse for performance. We can try to split up our dependencies so that subexpressions can be evaluated in parallel.

But of course you’re right, at a certain point we get down to the really CPU-specific stuff, and that’s probably dangerous to rely on, unless you know the exact hardware configuration on which the program is going to run. (You might be targeting a Playstation 3 specifically, for example, and then you don’t care that your optimizations wouldn’t work on other CPU’s)

There’s a Google Tech Talk called “We have it easy, but do we have it right?” covering the results, for anyone interested. I recommend both the paper and the video!

Other good videos include Elizabeth Bradley’s “Chaos in Computer Dynamics” and Michael Hind’s “The Impact of Multicore Architecture on Software.” Both of those are UWash colloquium videos: [http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/search.cgi]

One interesting result from the Hind video is that, while –O3 overall slightly improves performance at the application level, things look very different at the method level. Most methods are not affected, a few are made much faster, and a few are made much slower! This suggests that JIT and/or tracing compilers will be much better positioned for effective optimization than traditional static compilers.

I’m just finishing up with a computer architecture course, so the complexities of modern CPUs are not (entirely) lost on me. But at a certain point, I can’t help but wonder how much good the extra knowledge brings, precisely because performance is so context-sensitive. An optimization on a Core i7 may be a pessimization on an Atom. Like you said, measure measure measure…