How confident are you with your estimates?

I know you are a busy reader but before going any further I would like you to answer to ten questions. The questions are taken from the book How to Measure Anything: Finding the Value of "Intangibles" in Business by Douglas W. Hubbard. You should be 90% confident with your answers. This means that if you do the test perfectly, you should get 9 correct answers and 1 wrong.

Do the test - answer to the ten questions.

How did it go?  (I appreciate if you write your result to the comments.) When I did the test myself, I got 6 correct answers. When a couple of my colleagues did it, they got 2-5 correct. Two weeks ago I was in ALE 2013 unconference in Bucharest and held an open session there. I asked the questions from a dozen of people. Most of them got 2-4, some of them 5-6 and there was only one person with 7 correct answers.

Too difficult questions?

When people get rather low results from the test, they typically react by saying that the questions were too difficult. This is the key point here. If you don't have enough information, you should not give too narrow range. When you get more information, only then you should make the range narrower.

So how about software development? Have you ever been in a situation where your boss asks you to give a quick "educated guess" about a new feature because there is a management meeting tomorrow where the estimate is needed? And although you don't feel very confident, you say "well, maybe 30 days". And then the boss takes your (educated?) guess with him, goes to the meeting, and let's the guess to become a fact that is used to make an important business decision.

I think this is exactly such a situation where you don't have enough information to give an estimate like that. You are after all giving a range of 30-30, i.e. a single point! Instead, if you are a calibrated person (1), you could say: "with 90% confidence, I think it is 10-300 days". Or if you are not, like me, you should probably just say: "with this amount of information, I don't know".

Unprofessional answers?

When I gave that advice in the ALE13 open session, two people asked me that wouldn't it be unprofessional to give an answer like that? I replied to them that it would be actually quite the opposite.

I think it is unprofessional to pretend that you have information when you don't. I think it is actually unethical. It is much more professional to be honest and say: "I don't know".

How to narrow the range?

If you have a smart boss, he probably wants to know how to make the initial range narrower. The first thing is to spend a bit more time to think about the problem you are about to solve. You can probably identify a couple of parts that are especially uncertain. Maybe you can write a prototype or a technical spike for them?

Another approach is to ask what is the most important thing we need to solve? If your initial estimate for the whole thing was 10-300 days, maybe your calibrated guess for the most important part is 2-20 days. It is still a wide range but perhaps small enough so that you can just do it and see how it goes. And most importantly, to learn by doing.

You will actually learn many things. You may learn technically. You will understand the domain better. And you can start measuring your progress so that you can stop guessing and start forecasting.

Or you may learn that you need to build something else that you were actually supposed to build. If that happens, what is the value of estimating the whole project beforehand then? Another important aspect is that the smaller slices of work you can create, the less need you have for estimating their size.

Cost and value

If we go back to the initial question about the project size, it may actually be the wrong question for another reason as well. When we want to figure out whether to start a new project, we tend to focus on estimating the project cost because that is the "easy" part. We don't try to estimate the value because that would be too difficult.

In his book Douglas Hubbard criticizes such a behaviour. As the book title says, he claims that anything can be measured, so also the value of a project. For doing it he provides you with many tools. I really recommend you to read the book and find out more.

PS. In Finland they are planning to build a health record system that would cost 1.2-1.8 billion Euros. I wonder if that estimate has a 90% confidence interval..?


(1) Calibrated person is such that gets regularly 9/10 right when she is asked to answer with 90% confidence interval. When Douglas Hubbard has a measuring challenge, he trains the key people so that they become calibrated. One tool for that is to answer to other similar questions. After the key people are calibrated, they can give reliable initial estimates for the questions that are created based on the measuring challenge. Based on the initial estimates Hubbard defines with the aid of statistical tools what part of the challenge should be measured more in order to provide most valuable additional information with the least effort.