Dogfooding redux

A while ago I wrote a bit about Microsoft’s prac­tice of “dog­food­ing” their soft­ware. That sparked a fair amount of dis­cus­sion on Reddit.

Of course, a few peo­ple assumed I was talk­ing in absolutes, that because the prac­tice of dog­food­ing is not per­fect, it must be evil. That’s a bit of an exaggeration.

I never said that “dog­food­ing is bad”. But dog­food­ing gen­er­ally means using your prod­uct on a daily basis. Not just pok­ing around with it, or check­ing it out, but actu­ally using it as your pri­mary tool for… what­ever the prod­uct is sup­posed to do for you.

This implies that you can’t really use a lot of other prod­ucts as inten­sively while you’re dog­food­ing your own. A devel­oper on the Visual Stu­dio team can’t both use VS10 for all devel­op­ment, and at the same time use Eclipse, Vim and Emacs on a daily basis. If he were to use all those, there would no longer be enough time left to use VS10, and so he wouldn’t be dog­food­ing it.

And that is the prob­lem. Dog­food­ing is a valu­able way to incre­men­tally improve and pol­ish your prod­uct. But it also takes up time that could have been spent using some­one else’s prod­uct. And when you use a prod­uct in your daily life, you become blind to a num­ber of its weaknesses.

One of the Red­dit com­menters men­tioned a lovely exam­ple: Java devel­op­ers, peo­ple who write Java and noth­ing else, don’t really see the need for clo­sures. They just don’t really con­sider it a valu­able fea­ture. But pretty much every­one else does. Does this mean that in Java, alone of all lan­guages, clo­sures are not rel­e­vant? C# pro­gram­mers got clo­sures and are ecsta­tic about them. C++ pro­gram­mers are about to get clo­sures, and everyone’s just itch­ing for it to hap­pen. Func­tional lan­guages always had clo­sures, and pro­gram­mers in those lan­guages just can’t live with­out them. Are we to believe that specif­i­cally in Java, clo­sures just do not mat­ter?

Or does it just mean that the Java devel­oper doesn’t yet real­ize how much eas­ier clo­sures could make his job? He doesn’t real­ize this because he’s never had the chance to use them. He’s stuck in the world of Java as it looks today, and if you were to ask him what he’d like changed about the lan­guage, he’s not going to say “clo­sures” or “higher order func­tions”, or even “tem­plates” or “type infer­ence”. He’s going to come up with some small incre­men­tal improve­ment. Wouldn’t it be nice if class X was added to the class library? Wouldn’t it be nice if Y was named dif­fer­ently? Wouldn’t some short­hand syn­tax for things you can do already be nice?

And the same is true for dog­food­ing. Visual Stu­dio is slow. Many com­mon oper­a­tions cause mul­ti­ple sec­onds of wait time. Adding an empty file to a project in VS2008 is painful some­times. But oddly enough, I only really notice it when I’ve got­ten used to Vim or Emacs or some other alter­na­tive IDE or code edi­tor, which isn’t so slow. Its project struc­ture is fun­da­men­tally bro­ken, but I only really notice this when I’ve just spent a few weeks play­ing around with make­files¹. And the same goes for the Win­dows Mobile phone I had a few years ago. While it worked, I tol­er­ated its quirks. Apart from a few instances when it went really hay­wire, it wasn’t so bad. Sure, it was slug­gish and needed to be rebooted reg­u­larly, and sure, the inter­face wasn’t exactly pretty or con­ve­nient to use. But it worked, most of the time, and I got used to it. I didn’t love it, and I didn’t think it was a par­tic­u­larly good phone, or plat­form or OS, but it was usable. Then the phone broke, and it took me a few weeks with my replace­ment phone to real­ize just how much bet­ter every­thing else was. My Win­dows Mobile phone sucked. I just didn’t real­ize it until I got to use a dif­fer­ent one on a daily basis.

But let me reit­er­ate, dog­food­ing is a good thing. There’s no doubt of that. Your prod­uct can only get bet­ter by hav­ing your devel­op­ers actu­ally use it in the same ways that end users are expected to. But if dog­food­ing is all you do, then the world will pass you by.

So why did I dig this old post up again?

Because Visual Stu­dio 2010 is about to be released, and as we may have expected, this means Microsoft’s devel­op­ers have to beat their drum a bit about how awe­some it is because it’s been dog­fooded.

Soma just seems to for­get that the glar­ing per­for­mance issues in both beta 1 and 2 came as a huge sur­prise to Microsoft until the betas had been released in the wild. As much as they dog­fooded it, it didn’t give them the infor­ma­tion they needed: that VS10 is painfully slow com­pared to every­thing else, includ­ing VS9.

Both betas were released pretty much with the mes­sage “Don’t worry. It might use a lot of man­aged code and use a com­pletely new WPF-based edi­tor. But it runs really well and is just as fast as pre­vi­ous ver­sions of Visual Studio”.

There were two rea­sons for this. One seems to be (accord­ing to another Microsoft developer’s blog post which I can’t seem to find at the moment, unfor­tu­nately) that they sim­ply didn’t col­lect the right met­rics from all their dog­food­ers. A lot of their devel­op­ers did think it was painfully slow, but were never asked how they felt about the cur­rent per­for­mance level.

And the other is obvi­ously that all their dog­food­ers were using VS10 in their day-to-day work. They had noth­ing else to com­pare it with. They weren’t using Eclipse or Vim or Emacs or even VS9 or VS6 in their daily work. So they became used to the down­right painful performance.

But once they released the beta into the wild, it was obvi­ous to every­one else, all those peo­ple who had not been using VS10 on a daily basis for months, that it was a huge step backwards.

So yes Microsoft, you’ve heav­ily dog­fooded VS10. No doubt about that. But let’s not pre­tend that it solved all your prob­lems, or that it gave you the best pos­si­ble prod­uct. In real­ity, it led to you scram­bling the last 6 months to bring the per­for­mance back up to where it should have been all along, and where you’d prob­a­bly have kept it if you’d used other IDE’s occa­sion­ally so that you’d had a basis for com­par­ing performance.

It’s not that dog­food­ing is bad. It’s just not enough. And it is not, in itself, a sell­ing point or a proof of quality.

Programming | RSS 2.0 | Respond | Trackback |