Minimum viable discovery and software estimation for engineering work
I recently had to estimate 6 months of work for a new product after just 2 hours of discovery. This is a short deadline for an accurate estimate and I felt uncomfortable providing a number.
I ultimately gave a gut feeling estimate for the work because that's my job and we needed one by the next day. I did specify it was at least 20-30% wrong and highlighted the major risk I saw - a new third party integration that we had never integrated with before.
The whole scenario got me thinking about what works and doesn't work for me when estimating larger pieces of work. Here are some of the thoughts, tips, tricks and learnings from 10 years of providing dodgy software estimates!
You should estimate even though you don't want to
You write software to provide value to your customers. They will eventually pay you for that value. So we need to build things that are most valuable to the business.
Your organisation always has to optimise what work is done to provide the best value for limited resources. These limited resources are a product team's capacity (time) and cost (salaries). You and all the stakeholders need to compare the cost of building and maintaining different pieces of software before deciding what's going to be best value.
Another reason you should always estimate is that schedules force high performing teams to trim back scope and focus on the most useful features. If you don't estimate and hold yourself accountable then you can end up spending time on less valuable features.
Understand the required accuracy from the stakeholders before estimating
Estimating is communication. Make sure you nail the communication with clients and stakeholders from the start. What will the estimate be used for? Clarify if you're being asked for an estimate or a commitment? Do your estimates need 10%, 30%, 50% accuracy?
It's OK to demand a number on required accuracy from the stakeholders. They're asking you for one. Don't accept a subjective response for this - "we just need a rough estimate" isn't good enough. You will be held accountable for your estimate later so make sure you understand the required accuracy and get the stakeholders to agree to this.
Discuss with the stakeholders what will happen if the project takes 2 times as long, 3 times as long? Understand the impact of messing up the estimates. This should feed into how detailed your pre-estimation discovery is, or your accuracy buffer if you can't do discovery.
Don't sweat the small stuff when estimating larger pieces of work
You should roughly estimate an overall piece of work and only focus on detail for the riskiest tasks.
I'm not a fan of estimating every single piece of work in a project. If you need to add some entity CRUD api like you've done 100 times before skip the estimation, just build it and move on.
For that kind of work you can proxy the team's velocity simply by the number of tickets they complete rather than go through estimating, documenting and breaking down every story.
This is particularly true if your team gets 30 minutes to 1 hour to estimate weeks of new work. This is just not enough time for them to understand every task in detail so don't waste time pretending that the estimates are in any way accurate.
Set all those common tasks to "Medium" or "3 days" or whatever a mid level task is in your estimation system. In your 30 minute to 1 hour estimation session just focus on the scary features where you don't really know how long it will take because it's so unknown. These will dominate the time taken to deliver anyway.
Do spend appropriate time investigating unknowns
Your discovery process should be appropriate to the complexity of the solution.
There is some Minimum Viable Discovery that is required for every feature. If it's a simple well known solution then a quick discussion might be enough.
If the solution is a new complex integration then you should include engineering time. Building a POC of a new api integration is incredibly valuable discovery work that can feed into estimations later.
Any vital integration should get a POC during discovery - if you're absolutely dependent on a feature do not take it for granted that it works as the vendor describes in documentation. Build something with it.
These new integrations are the things that will end up taking 10x as long as estimated if you just guess instead of building. 5 I've been burned by this many times in my career in case you can't tell! Learn from my mistakes and ask for time to do a POC before estimating.
Spend time on your processes
If you have a chaotic development process where developers face lots of interruptions from operational issues or support issues then your estimates are never going to be accurate for that team.
If a deployment takes half a day from your best engineers and you have to complete unexpected deployments during a sprint it will blow out your estimates. These kinds of issues add up quickly so keep on top of them.
Story points are not the only way to estimate software problems
Using any tool or process by default always annoys me. Estimating every piece of work with story points is one of those things.
Story points are great for two things
- encouraging shared understanding in a team
- comparing the estimated complexity of different pieces of work in the same team
They're not useful for estimating the length of time a task will take. You should never convert story points to time. Story points are not useful for measuring productivity and they're absolutely never to be used for comparing output of two different teams.
You should deliberately select an estimation technique based on the requirements for the estimation. If you need a time estimate then just use a time estimate (ideally from the engineer(s) that will be building the thing), skip the charade of estimating with points and converting to time later.
If your estimate is required for cost and resourcing then use an estimation technique based on cost and resources.
If you have an existing team and a history of previous estimation then you could consider using Monte Carlo simulation for estimating future work. This provides high levels of accuracy but takes a bit of work to set up.
If you have a new team with no history but accuracy is extremely important (e.g. a fixed price contract) then a detailed bottom-up estimation might be more appropriate for the estimation.
If you're happy with less accuracy then a quick estimate from experts can get you pretty close. But make sure that you add a solid buffer for estimation errors.
This is why it's important to understand the accuracy you're being asked for when providing an estimate.
Tips for improving accuracy of your estimates
If you're an engineer and you don't like giving time estimates you'll have to learn how to get better at it.
1. Identify which tasks have the highest uncertainty
The riskiest tasks are basically going to dominate the mean time to completion for your project. The uncertain tasks will likely have a factor of 10 or more on the overall project completion time.
Identity these tasks early and de-risk them. For example you could complete them first, split them up or build POCs. Whatever it takes to remove this risk is going to improve your overall estimate accuracy.
2. Account for optimism
There are some quick calculations you can do to offset the optimism we usually have as engineers.
First up is calculate the expected, best case and worst case scenarios and plug it in to this formula ((Expected * 4) + best + worst) /6.
An even simpler way to account for optimism is to apply a factor based on experience of the estimator - Expected + (30 - Years of experiences of estimator)%.
3. Use expert estimators
The people estimating should be the people building the software. Individuals and their skill level do matter when it comes to implementing software solutions, so be cautious of having one person estimating for someone else.
Ideally these engineers are also experts in the engineering and business domains being used. If they're not then get them some help from experts.
Make sure to use a technique to prevent the experts from influencing the non-experts or vice versa e.g. use planning poker to prevent bias.
4. Have expert consensus
If you have more than one expert doing the estimating and they all agree on an estimate then it's more likely to be correct.
Avoid having just one expert helping with the estimates. They will likely miss something.
5. It's not just engineering
I do keep mentioning engineers in this post because that's my audience. But estimation should also include input from as many of the roles as possible from your team that makes software better - QA, Ops, PM, Clients.
6. Respect the 6 week rule
We're terrible at estimating but we're truly awful at estimating anything over 6 weeks of work in detail. If you start going over 6 weeks of work either stop and do the 6 weeks and then re-estimate another 6 weeks or take any of the estimates for longer than 6 weeks at a very low confidence.
7. Identify non-functional risks
The team should have a list of non-functional risks before estimating but these should be revised as learnings are made from previously missed estimates. Figuring out why estimates were wrong should be a part of the retrospective cycle.
Here are some examples of things I have been burned by in the past so now I watch out for them in new projects.
- Your internal organisation dependencies don't care about your project as much as you do. They have competing priorities and they won't act on your requests as quickly as you would like.
- Are you in a new team or an existing team? Forming a new team will eat up time. The larger the team, the longer the formation stage.
- What documentation will be needed? No documentation is obviously faster, for a while at least.
- Is this a regulated industry? Who verifies compliance? What is their process?
- Is there regulated data being handled (e.g. PII or payment information)?
- Are there new third party integrations? Do you understand them? Have you built a POC?
- Is it possible that the change needs to be reverted after running for some time? (this adds complexity)
- Is your team also dealing with BAU/bugs? Interruptions with ad-hoc bug work will affect planned work and make estimation less predictable.
Your team and your organisation will have its own non-functional risks. Make sure that you or your team has spent time identifying these if you want to improve accuracy.
8. Compare to previous completed tasks
For critical path tasks you can ask the team to find a similar example they worked on before. If your current estimate is not close to that previous task actual completion time then it would be good to understand why.
9. Be aware of cognitive biases in estimation
Our brains do silly things when we're estimating. We get so much information from discovery that we have to filter it and we end up filtering out important information (Availability Heuristic, Anchoring, Confirmation Bias).
If there are gaps in information we have a tendency to fill it in with constructed stories based on our own experiences (Bandwagon effect, Group attribution error, Anecdotal fallacy).
Estimating at speed with pressure amplifies all of these biases (Conjunction Effect).
It's not possible to overcome all biases but it's important to know that you and the team are being influenced by them. It's worth keeping a look out for biases that are having a significant negative effect on your estimation process.
After estimating
Understand the real deadlines for the piece of work.
Does your riskiest estimate come fairly comfortably within any critical deadline? If not then adjust something and estimate again.
"End of sprint" is not a real deadline. That's a fake deadline.
A marketing campaign or announcement at a conference is a real deadline.
The question to ask is "When is the latest that this piece of work can be delivered and still provide value?".
Document the estimates and any assumptions used
Document any other assumptions you made when estimating.
Share this with the stakeholders in case you missed anything.
Regularly re-estimate
Accept that your first estimate will be out by some %. Continuously review and check on your accuracy as work progresses.
Adjust your estimates wit this new information immediately. This is the cone of uncertainty.
As you move closer to the end of the piece of work the accuracy of your estimates should improve.
What to do when you're going to miss an estimate
1. Communicate the delay right away.
Do not ignore the delay. You will not make up the time later so you have to adjust expectations. Estimation is communicating expectations. Make sure you keep the stakeholders informed.
2. Do not add more people
Adding more engineers to a late project won't make it go any faster. The new team members will take the time of the existing members to bring them up to speed.
If new people are temporarily assigned to a project and they can work on a piece in isolation then they might be faster because they have different goals to the primary team. E.g. quality will likely suffer and the original team will have to pick up the pieces after the heroes leave.
Make sure you try to change the original team's motivations to better align with the organisation's requirements before adding people.
3. Minimise "crunch time"
Crunch time is OK occasionally. Engaged, highly skilled people will generally work hard to meet expectations. But this is unsustainable for any time longer than a couple of weeks.
Ideally you should acknowledge any crunch time with the team and compensate them. But do not glorify this practice or you will create a culture of over-working.
Pushing product teams to meet fake deadlines like end of sprint might work for a few sprints but eventually the team will tire of the constant fake pressure. Aim for sustainable development practices.
Final tips
What you ship at the end will be different from what you estimated.
You will learn more as the project proceeds. The scope usually changes.
Prepare your stakeholders and client for this by focusing on delivering outcomes rather than outputs. Always talk about project outcomes.
The penalty of underestimating is almost always more severe than overestimating.
When you provide an estimate you are setting expectations. Act accordingly. Your client will be more upset if you exceed a deadline than if you finish early. You can greatly reduce the effects of missing the estimate by providing a range instead of a fixed number. "2 months +/- 20%" is better than "2 months".
The negative effects of being behind schedule also has an effect on the team. You will be much happier on a team that is ahead of schedule rather than behind schedule. And happy teams build better products! Don't sabotage yourself before starting by being too optimistic with an estimate.
You can't make your estimate too long either because you might lose business. So there needs to be balance here.
Remember the laws of estimations
Hofstadter’s law: “it always takes longer than you expect, even when you take into account Hofstadter’s Law.”
Parkinson’s law: “work expands so as to fill the time available for its completion”
These laws sound like jokes but they're quite true. Always estimating too conservatively is as much of a problem as estimating too optimistically all the time.
You should be trying to get your estimation accuracy to +/- 10%. This is difficult to do but will make you a more valuable engineer for your organisation.
Summary
Estimating is hard and we all get it wrong.
Estimates are very valuable and you should always do them.
Communicate with the team and the stakeholders any assumptions you're making, document the reasoning behind estimates, definitely document and share the accuracy for your estimates.
Identify the riskiest items. Make special note of these things and try to de-risk them as a priority. Risky unknowns will take up most of the delivery time!