Tom Gash

A decision scientist encounters a knife-wielding attacker. Does she pause to carefully set out and weigh the importance of competing priorities, consult widely on the range of possible approaches to dealing with the situation, calculate probabilities of different outcomes, and create a ‘constrained optimization model’? We hope not. Because the literature from decision science is pretty clear that there are domains where we have to think fast. Equally, there are also plenty of domains where there is no point in thinking things through too much. Unless you opt for that weird one made out of clowns, the font of your internal memos isn’t going to matter too much… 

Studies of decision-making tell us that carefully structured analysis and deliberation supports better decision-making in relation to complex long-term issues. But decision-making in time-critical environments depends much more on experience. Gary Klein is a decision scientist who focuses on how professionals make decision in the real world. His work demonstrates that experienced firefighters, pilots and so forth usually have a much greater ability to spot decision-relevant patterns in their immediate environment than non-experts. They sense when situations are routine (fitting a known pattern) and when something is ‘wrong’ (calling for a different response), for example. And they are adept at finding ‘good enough’ solutions to avert disaster by cycling through their past experiences and possible responses in rapid time. 

To improve decision-making in time-critical contexts, we should be wary of overly complex and formal processes and rules. Instead, we can consider factors such as: 

• Mental state: Fatigue, stress and extremely heightened emotions (fear or anger, for example), have been shown to dramatically undermines human judgement. The focus on employee wellbeing is warranted not just during the current pandemic and not just for the purposes of fairness. It is also vital for decision-making performance. 
• Training: The only substitute for experience is training that as closely resembles ‘on the job’ realities as possible. The military excel at creating more realistic training scenarios and there are exceptional examples the medical sector, where simulation and augmented are an increasingly important part of medical professional development.  
• Rostering/ staffing/ team composition: Workforce planning often pays some heed to ensuring the right experience mix within teams. However, decision-making is fundamentally shaped by the skills, thinking styles and communication styles within a team – and all organisations need to consider how their team structures and skills mix affect decision-making. Experience within a team matters, whenever rapid, complex decision-making is required. 

Of course, saying that experience matters in time-critical situations, is not to say that it’s the only thing to concentrate on. In general, the evidence suggests that experience often only improves decision-making up to a point (20 years is not always better than 10, for example) and it seems that if there is limited feedback on the quality of decisions, then experience carries many fewer benefits. This is why a big focus at Leapwise is supporting people and organisations to understand the consequences of their decisions – and learning the right lessons from success and failure. 

Decision-makers all need information to choose wisely. Who can forget the time that the bomb squad blew up a suspicious vehicle outside Workington Police station? The vehicle had actually been parked there by fellow officers after they helped its owner (who was unwell) but they failed to mention this to their colleagues.

Today, data is everywhere. With the rise of various information-gathering tools – cookies, mobile analytics, sensors and so forth – it accumulates by the second. Yet most organisations still have work to do when it comes to ensuring relevant, accurate information is accessed when needed and presented and processed in ways that support better decisions.  

We need, in my view, much more focus on identifying the decisions that are critical to the organisation and then supporting them with the right information and analysis. At the strategic level, this usually means supporting decision on what you are trying to achieve, who you serve, the capabilities you prioritise, and your culture. But it is equally vital to dig into the decisions made at the sharp end of your organisation – the daily choices made a thousand times that shape the service the public or your customers receive.  

We suggest six steps for organisations seeking to ensure data-driven decision-making: 

1. Shape your data capture to your decision needs. While some data is so cheap and easy to collect, you may as well have it in case it helps with an unforeseen decision later, being intentional about the reason you are collecting information will help shape your information architecture, and allow you to focus your data quality and analytic efforts. Wherever possible, democratise data, so more decision-makers can access it.  

2. Build data literacy across the organisation. Not everyone likes or is comfortable with data, but it simply isn’t feasible for many leaders today to be effective without a good understanding. If leaders don’t ‘get’ the value of information and analysis – or know what good and bad analysis look like – your decision-making will be worse, and your efforts to improve data analysis are doomed to fail.  

3. Present data in ways that overcomes some of our human weaknesses in understanding. Framing of information is powerful. Choice ordering, ways of presenting risk and probability, and even format can ‘nudge’ decision-makers in one direction or another. A picture tells a thousand words, so visualisation can be harnessed to enable better decision making but it can also be misused! 

4. Build stronger feedback on the results of past decisions. At Leapwise, we try to build feedback loops into as much of what we do as we can. We get feedback from clients, we are testing our meetings software on our own meetings, and are trying to build a culture of learning across the organisation.  

5. Automate or partially automate decision-making when it is heavily rules-based. Feedback systems can eventually create self-improving automated decision-making. Our machine learning forecasting partner, Skarp, gets ever-better forecasts precisely by refining how it predicts demands as it collects more information and learns.  

6. Ensure decisions at every level are informed by overarching organisations goals and priorities. Most Agile organisations emphasise the importance of everyone in the firm understanding overall goals, so that they can respond quickly to rapidly changing events. But even more hierarchical organisations understand this. For example, the British Army ensures that when an order is given, it is usually accompanied by the preceding order that it is a response to. Wherever you can, ensure decision-makers have to explain how their decision fits in the broader mission, and there will be less risk of organisational effort becoming fragmented. 

Data collection and storage has become cheaper than ever before – and this trend will likely continue in the coming years. But we need to be supremely vigilant about the fact that data is worthless until it is interpreted and actually influences the decisions we make. 

In a previous article, we set out some techniques to ensure clarity of objectives for any critical decision in your organization. But as well as defining goals, leaders need to understand and respect what decision scientists call ‘constraints’ – the lines that can’t be crossed, and the types of side-effects of any consequence that won’t be tolerated.

In risk critical industries from financial services to healthcare, regulations guide professionals on the constraints that should shape their decisions – particularly for reasons of safety, ethics and legitimacy. For example, banks have capital requirements to protect both themselves and the financial system from collapse. The aviation industry is subject to intense safety protocols and ongoing checks. But regulation can lead to a focusing on specifying ‘means’: steps which have to be taken – or are prohibited. And an equal focus for decision-makers should be specifying the range of outcomes that they do not want, how deeply they wish to avoid them, and why.

Embedding a thorough understanding of such constraints and their underlying rationale is essential or effective decision-making. But it requires engaging with complex areas such as risk appetite and grappling with the practicalities of practical constraints that tie our hands. How many organisational leaders know and factor in the capacity of their staff to absorb new information or protocols in a specific period of time? How good are we at tracking the mental health impacts of different roles and shaping support accordingly? How well do management teams understand their capacity to manage a certain number of programmes or staff? In public services, how clear are we on what decision scientists call ‘equality constraints’ – for example, our tolerances for different geographic areas or populations getting different levels of service?

There are a variety of techniques to ensure decision-making takes account of constraints. For example, for high stakes decisions, it can be worth investing in complex mathematical models (constrained optimization modelling). But progress can be made by following three simpler steps:

1. Ensure you have a standardised decision-making process or ‘checklist’ for high value or regular decisions – and that this includes consideration of decision-making constraints. Processes and checklists may need to vary depending on the type of decision you are making.
2. Where possible, quantify different constraints to support improved decision making. Ideally, ensure information systems provide visibility on how constraints are evolving over time, ideally using intuitive stories and visualisation techniques.
3. For critical constraints affecting high value or repeated organisational decisions, ensure that everyone in the organisation is aware of them

Sometimes understanding what you can’t or aren’t willing to do is as vital for effective decision-making as understanding what you can do.

Key Lessons from Decision Science

Was the 2003 Iraq War a great victory? Saddam Hussain was removed from power in less than a month, which led some to promptly declare the invasion a success.

But this misses the point. The true strategic goals of the invasion should have been (and possibly were at some point) security in the Middle East and the wellbeing of the Iraqi people. And instead of this, the invasion achieved the rise of Al Qaeda and then ISIS, hundreds of thousands of civilian casualties and the destabilization of Syria and the region more generally.

Neglecting to properly define and retain focus on strategic goals lies at the heart of many failures. Setting a single goal gives focus, and can provide a ‘north star’ for any organisation. Less can be more when we need to achieve great things. But, in truth, a sole objective is a rare luxury. A school must care about educational attainment and wellbeing – and consider the appropriate focus on each. A pharmaceutical or consumer goods company must sell lots of their existing products but also create new ones.

Good decision-making is virtually impossible without clarity on the goals being pursued and on which goals are most important. After all, it is only once goals are defined that organisations can harness either intuitive or scientific approaches to assess the optimal path to achieving them.

Unfortunately, goal setting can be harder than it sounds. For organisations, it requires clarifying and being honest about competing goals and then finding ways to thrash out the right balance of priorities (and compromises) given the precise context. It requires tough choices about what is not your focus. It means finding ways to facilitate complex conversations about values, beliefs and evidence.

Sadly, these conversations are often seen as ‘too difficult’ – even though without them, it is very difficult to achieve the genuine collaboration, shared vision and prioritized objectives that can provide the platform for sound strategic and day-to-day decision-making.

As in organisations, individuals can also duck the tricky task of goal-setting. But goal setting does appear to work – even according to the brilliantly sceptical evidence-based HR practitioner Rob Briner. One of the most cited sources of authority on the subject is Locke and Latham – who long ago suggested goals need to have four key attributes to support success:

1. Clarity. It has to be clear what the goal is, a reason many advocate SMART goal-setting processes which specify how and when goals will be achieved, and ensure realism.
2. Challenge. Set a goal that stretches, as these are better than easy or ‘do your best’ goals. Be particularly careful to be realistic about what is required to achieve complex goals – particularly when achieving goals depends on external factors.
3. Commitment. Ensure the goal is genuinely motivating to those who need to deliver it – not always easy, and worth taking time on!
4. Feedback. If you can track progress and evaluate what is working, you can build momentum and motivation, as well as change tactics for achieving goals as you go.

As ever, the right way of approaching goal setting will vary depending on your situation. Some will find goals – particular if imposed on them – deeply demotivating. Others will happily set themselves goals that actually work against their core values. But overall, the benefits of goal clarity will far outweigh the disadvantages. If you want to make good decisions every day this year, you’ll need to be clear on what you’re trying to achieve…

In a previous article, we looked at three different types of bias that can emerge when using algorithms to support (or make) decisions. Here, we put forward some remedies: four approaches that can ensure that making good use of data does not perpetuate or create bias.

1. Involve those who are aware of bias and the risks of algorithmic bias during the design and development of data-driven decision-making projects. This can be helped simply by building more diverse teams, who will be more aware of the risks of bias. But it will also be beneficial to check that as well as technical wizards your team has people with appropriate training or appreciation of social sciences, or even data ethics specialists.

2. Commission external reviews. Several criminal justice sector organisations have reduced bias by opening up their new tools to external scrutiny. An independent reviewer of Durham Police’s Checkpoint programme, for example, identified a risk of bias from including post code as a predictor of future offending behaviour – which could unfairly create bias against people from particular areas. The National Data Analytics Solution (NDAS) project run by West Midlands Police has put considerable work into ensuring transparency and ethical decision-making, inviting a review from the Alan Turing Institute’s Data Ethics Group. The West Midlands PCC has also put in place an independent data ethics panel.

3. Measure algorithmic performance and bias systematically.    

While precise details of how many decision support algorithms are coded is commercially sensitive, all justice organisations will have to be able to explain the tools they are experimenting with and their accuracy and cost-effectiveness in layman’s terms. The US Department of Commerce is leading the field in this area. Its Facial Recognition Vendor Test reports publicly on the accuracy and racial bias of AI systems submitted by a vast number of suppliers. We believe the UK should implement similar systems as a matter of urgency. There is arguably even a case for public sector inspectorates (or a new cross-sector inspectorate) to be given a role in scrutinising algorithms in detail and reporting publicly on any observed bias.

4. Create a more robust legislative framework.

Ensuring inspectorate access to commercially produced algorithms would be likely to require legislation. And further legislative change may be required to ensure proper use of data across the public and private sectors. The US is making some progress here but there is much more to do: not least because global corporations will require (and benefit from) some standardisation across geographies. With the UK exiting the EU, which has led on much work to ensure data protection and other data processing legislation, this is another area where the UK government might need to develop additional expertise post Brexit.