“Let's say that you have 20 tasks. Each involves rolling a 10-sided die.” (comment)
recommended by 2 users | 2012 | michaelochurch on: Why Software Development Estimations Are Regularly Off | management | original thread |
michaelochurch on Jan 28, 2012 | on: Why Software Development Estimations Are Regularly Off | view on HN

All this is true, and there's another statistically-rooted problem that pertains even to mundane "enterprise" projects where there's little new invention and where reliable time estimates should be possible.

Let's say that you have 20 tasks. Each involves rolling a 10-sided die. If it's a 1 through 8, wait that number of minutes. If it's a 9, wait 15 minutes. If it's a 10, wait an hour.

How long is this string of tasks going to take? Summing the median time expectancy, we get a sum 110 minutes, because the median time for a task is 5.5 minutes. The actual expected time to completion is 222 minutes, with 5+ hours not being unreasonable if one rolls a lot of 9's and 10's.

This is an obvious example where summing the median expected time for the tasks is ridiculous, but it's exactly what people do when they compute time estimates, even though the reality on the field is that the time-cost distribution has a lot more weight on the right. (That is, it's more common for a "6-month" project to take 8 months than 4. In statistics-wonk terms, the distribution is "log-normal".)

Software estimates are generally computed (implicitly) by summing the good-case (25th to 50th percentile) times-to-completion, assuming perfect parallelism with no communication overhead, and with a tendency for unexpected tasks, undocumented responsibilities, and bugs to be overlooked outright.

2 users recommended this comment. What they said:
mwilliamson Apr 23, 2012 | on: Why is estimating so hard? | context

I remember reading a comment on another story which I thought was a wonderful explanation of why estimates tend to be too short:

http://news.ycombinator.com/item?id=3522910

To quote one part:

Let's say that you have 20 tasks. Each involves rolling a 10-sided die. If it's a 1 through 8, wait that number of minutes. If it's a 9, wait 15 minutes. If it's a 10, wait an hour.

How long is this string of tasks going to take? Summing the median time expectancy, we get a sum 110 minutes, because the median time for a task is 5.5 minutes. The actual expected time to completion is 222 minutes, with 5+ hours not being unreasonable if one rolls a lot of 9's and 10's.

x0x0 Sep 15, 2013 | on: Poor developer estimates, a.k.a. guesstimation | context

that "solution" is, basically, ridiculous and not adopted for good reason.

A quote from Michael O'Church I quite like:

      Let's say that you have 20 tasks. Each involves rolling a 10-sided die. 
   If it's a 1 through 8, wait that number of minutes. If it's a 9, wait 15 
   minutes. If it's a 10, wait an hour.
      How long is this string of tasks going to take? Summing the median time 
   expectancy, we get a sum 110 minutes, because the median time for a task is 
   5.5 minutes. The actual expected time to completion is 222 minutes, with 5+ 
   hours not being unreasonable if one rolls a lot of 9's and 10's.
      This is an obvious example where summing the median expected time for the 
   tasks is ridiculous, but it's exactly what people do when they compute time 
   estimates, even though the reality on the field is that the time-cost 
   distribution has a lot more weight on the right. (That is, it's more common 
   for a 6-month project to take 8 months than 4. In statistics-wonk terms, the 
   distribution is log-normal.)
      Software estimates are generally computed (implicitly) by summing the 
   good-case (25th to 50th percentile) times-to-completion, assuming perfect 
   parallelism with no communication overhead, and with a tendency for 
   unexpected tasks, undocumented responsibilities, and bugs to be overlooked 
   outright.
   [1]

The point is, you are unable to know the worst possible outcome. Since it will have huge input into your average, your average is therefore useless. Perhaps the project isn't specced fully enough and, during implementation, a dev discovers an ugly or impossible feature interaction. You now have to either cut features or rearchitect another feature, ie the dice come up 10 in Michael's analogy.

An example from my background: I work on ml code. A learning algorithm was written in parallel with a particular loss function (essentially, this tells you if your estimates are good or bad). The person writing the spec assumed that a different loss function could be dropped in later. Indeed, this is how the math works on paper. But loss function number 2 required a different data layout between the machines. The vision was writing a new loss function should be one or two lines of code dropped into the middle of a hot loop; the reality was it took almost a month of work.

[1] https://news.ycombinator.com/item?id=3522910