Let’s make it clear though, that the following will have very little practical use. We’re not just into “it doesn’t make a measurable difference” territory, but also deep into “the compiler will do this for you”. So please, don’t try to apply these “optimizations” to your real-world code to save a clock cycle.

This is merely intended as a thought experiment, illustrating some of the factors that makes performance so difficult to predict. And now, with that disclaimer in place, let’s get on with it:

The problem

The “problem” I came up with was the evaluation of x+x+x+x. This was the simplest snippet of code I could think of for which optimization is possible. For the sake of this discussion, let us assume that x is an integer.

A naive compiler will evaluate this code as ((x+x)+x)+x. In other words, it will evaluate one addition, feed that result to the second addition, and then finally feed the result of that to the third addition.

Optimization #1

The optimization I suggested was to evaluate it as (x+x)+(x+x) instead. And why is this faster? A modern CPU is superscalar — that is, it is able to execute multiple instructions every clock cycle. Depending on the CPU model, it can probably execute three or four instructions belonging from the same thread every cycle.

So where the original version would take three times the duration of an add instruction to execute, my optimization can be done in two times the duration: Both the initial subexpressions can be evaluated in parallel. And so, after only the duration of one add instruction, we’ve got the result of two of the additions, and can perform the third and final one. So in this very simple case, we actually reduced the run time by 33%. Not bad, eh?

Optimization #2

My friend then asked if x*4 would be an optimization as well. And now it gets a bit more interesting.

First, of course, x*4 is just a single multiplication. Is that faster than three additions? Is it faster than two additions (which is the time it’d take for my “optimized” version to run)?

That depends on the speed of a multiplication instruction. On common CPU’s, a moment’s research tells us that add has a latency of 1 cycle, and mul has a latency of 3 cycles.¹, so the multiplication takes as long as the original unoptimized version.

Evaluation

So what does this mean? That at a glance, optimization #1 is faster than #2, certainly. #1 yields a result after two clock cycles, where #2 takes a whopping three cycles.

But there are other factors at play. Sometimes the multiplication version may be more efficient. The CPU has a limited number of execution units. It can also only decode a limited number of instructions at a time.

The version using addition requires three instructions to be decoded, and uses two execution units during the first cycle, and one unit in the second. All in all, we’re occupying three “execution-unit cycles”. The version using multiplication does take three cycles, but because modern CPU’s are pipelined, it only occupies the execution unit during the first cycle. In the second cycle, the execution unit is able to begin on a new instruction, while continuing to process the mul instruction. So this version only requires one “execution-unit cycle”. In other words, we’ll free up other execution units so they can execute other instructions. We’re also freeing the front-end from having to decode three instructions.

So we now know that:

• If we need the result as soon as possible, the optimized add version will be more efficient because it finishes sooner.
• But if we need to execute a lot of other instructions as well, the mul version will be more efficient because it uses fewer hardware resources on the CPU

What if we have a lot of instructions we want to execute and we need the result soon? Or if we only have these instructions to execute, and we don’t care about when we’ll get the result (perhaps the next operation is to add the result to that of an ongoing division, which is very slow, so it won’t matter if we take 2, 3 or 15 cycles to get ready)? Hard to say. Either one may be preferable.

Of course on x86 CPUs we also have to take the variable instruction length into account. How many bytes does a mul instruction take? What about three adds? That affects both how much data has to be read from memory and how much space will be taken up in CPU cache, and so that should be taken into account as well.

So what can we learn from this? Mainly that performance is nontrivial. Never assume that you can tell whether some code is “fast” or “slow”. And be especially careful with assumptions about how it can be improved. It is very possible that your “optimization” will actually run slower.

Whenever you optimize code, do as the pros: measure, measure and measure. Measure the speed of the original code. Measure the result of the optimized code. Be careful with the many ways in which your measurement can be invalidated (by the compiler optimizing away the code you wanted to test, or by the CPU cache changing the result in your test case from what you’d expect in the real world by caching — or not caching — the data you’re operating on).

And when performing low-level optimizations, another vital piece of advice is to understand the hardware. Know which instructions are being executed, know the cost of instructions on the relevant hardware, and know what other tricks the hardware uses (Your CPU is probably superscalar and pipelined, and processes instructions out of order. It probably also has a cache of a certain size, with a specific cache line size, and a certain associativity. It has a fixed number of execution units, a known pipeline length and so on. And while we’re at it, the memory subsystem matters too. How long does it take to access RAM? How can the CPU reorder reads and writes? What is its policy for writes? When are they pushed from cache to RAM? If you want to optimize your code on the instruction level, you need to know your CPU. Even the simplest code is affected by dozens such factors, any of which might make a difference.

Well, my test case for this feature now takes ages to run. As I mentioned previously, a simple transaction modifying two integer variables under heavy contention can pull off almost two million transactions per second on my laptop.

My new test, in which each thread takes four variables and alternates between modifying two of them and reading the other two, runs perhaps ten thousand (!) times slower.

Of course I have several leads on how to fix this. The problem is largely all the performance-related “extras” I’ve been leaving out. For example, if a transaction fails to acquire a variable it needs, it simply aborts and immediately retries. In many cases, a better approach would be to block the thread, waiting for that variable to actually become available.

There are several other cases where I have a similar problem: I have to choose between delaying the thread for a moment with sleep() before attempting to continue, blocking it until some condition is true, or aborting the transaction entirely and starting over from scratch. At the moment, I generally just pick the easiest solutions (typically abort, and occasionally call sleep() a few times before we resort to that. Again, implementing some actual meaningful policies here would make a big difference. And tweaking these policies should help still more.

Another problem is that currently, I do not enforce a consistent global order when acquiring objects during a commit. This means I risk livelocks, again causing excessive rollbacks when multiple threads are competing over access to the same variables.

So I’m still optimistic. It should be possible to get performance back on track. But man, it’s depressing watching performance plummet like this.

Edit
And an update. After poking around a bit, it turned out that most of the time was being spent sleeping. When a transaction attempts to commit, if it can not acquire all the all the variables it needs, it retries a few times with a short delay (a couple of milliseconds) in between. If it doesn’t succeed after a few tries, it rolls back the entire transaction and starts over.

It turned out that these few, short sleep() calls brought CPU utilization down to something like 0.01%, and totally destroyed performance. Simply turning the sleep() call into a no-op brought me back to something more or less reasonable. I still need to improve on the above shortcomings, but now at least I can run my tests in less than an hour.