Change: churn or improvement?
Headed to the Institute for Government today for the launch of their report All Change: Why Britain is so prone to policy reinvention and what can be done about it. A good panel and audience, mostly of current and former senior civil servants, offering explanations for why British domestic policy so often sees repeated instigations of similar ideas, often within the space of single governments. The report focuses on three policy areas that have seen particularly high churn — further education, industrial strategy, and regional governance — but views the problems primarily as ones general to all areas.
At the conclusion of the event, IfG director Bronwen Maddox noted that this was going to be a continued focal issue for them, so, because these are issues I think about a lot as part of my research (and because I had wifi too shaky to work on my journey home) are three reflections to throw into that debate, and one alternative tack.
1. Evaluation is the only way to distinguish good change from bad
During the discussion there emerged some disagreement about whether frequent change should necessarily be seen as a bad thing. Both Rachel Wolf and MP Chi Onwurah made the case that change is a necessary part of governments trying to make things better. Nick Macpherson, former permanent secretary to the Treasury, noted the problem that too often policymaking proceeds without much attempt at piloting or evaluation. As a doctoral student in “policy and program evaluation”, I obviously have a special interested in noting its importance, but it simply makes logical sense that if we want to be able to hold governments to account for their changes, we have to know something about what happened after an initiative was launched. Was it an abject failure and therefore its scrapping should be quietly accepted? Or was there something there that should be built upon and therefore we should make more of a fuss if a new Minister, government, or flavour of the month tries to change it?
A key point here is that evaluation can come in many forms. Most government policies are too complex to subject to a randomized trial (although all credit to Goldacre, Halpern & co for keeping up the good argument that many more things could be piloted with random allocation than we think). Because the outcomes of a policy depend on interactions between the intervention, people, and many other system features, and because people ten towards herd behaviors that create nonlinear effects, we cannot always easily isolate the impacts of a policy. But headway in approaches to “evaluating complexity” mean there is no excuse for not trying to get something of a better picture of what a change is doing once introduced to a system. The key people who need educating about evaluation is journalists, who seem to be the main people policymakers listen to. Unless there is more of a media demand for information about quality evaluations, it’s not in the interests of ministers to commission them because it’s time consuming and costly.
Evaluation is particular important if we want to think about change in terms of improvement as opposed to wholescale reform. It would be completely unreasonable to expect ministers and civil servants to get everything right first time, but likewise we seem to currently have no systematic processes for tweaking and improving policies based on feedback. Making improvements systematically is the kind of thing systems thinking gurus have been talking about for years and has taken off under different names in sectors such as healthcare, social innovation, international development, and, in the U.S., education. It should be an established part of the debate about how to get better policymaking.
2. Be cautious about seeing evidence-based policymaking as the answer to the churn problem
The final of the four key recommendations of the report is to “strengthen the policy development process”, including incorporating “evidence checks”. While this is obviously a good idea at an abstract level, one of the challenges of bigging up the role of evidence is that there is often a sparsity of really relevant evidence around. This was an issue raised at the end of the discussion, with a little light bashing of academics. But the problem runs deeper than the lack of incentives for academics to do policy-relevant research. When it comes to major policy initiatives, it is unclear what kind of evidence gathering would really be most helpful. In designing an industrial strategy, for example, the myriad of factors that could potentially determine whether or not a particular policy approach ‘works’ makes it difficult to know what to test. Economists can model possibilities, sure, but there is often insufficient appropriate data to test those models empirically or build models that are suitably encompassing to incorporate the things that can send a policy off in an unexpected direction.
In the absence of sufficient data, evidence on major policy issues often comes in the form of looking to other countries that seem to have done well in this area. As Lord Sainsbury commented in his opening remarks to the event, British policymakers have been looking enviously at Germany’s further education system since the 19th century. It is indeed ridiculous how long we have been harping on this problem, but perhaps a major factor in the repeated failures has been the obsession with benchmarking against — and often even explicitly trying to recreate — a German-style system. As has been carefully documented, Germany’s strong apprenticeship system relies on business associations and labour market regulations that we simply do not have, have not been able to create, and which would be incompatible with our particular variety of capitalism. I know less of the details of what has been attempted in industrial strategy or local governance, but from what I do know it seems that here, too, international models have often led us astray. If we take an institutionalist perspective on what makes policies ‘work’, then borrowing an evidence base from nations with different configurations is rarely likely to be successful.
3. Non-ideological areas are not necessarily apolitical
On several occasions speakers referred to the fact that the level of churn in the three studied domains was particularly striking given that these were not “ideological” domains: in general, the parties have not had much disagreement about what should be done, indeed they have often replicated ideas previously used by the other party. But politics goes beyond partisan ideologies, and there are other reasons why attempts at certain kinds of changes may be particularly hard to make stick. In the case of further education, one of the major challenges to any attempted policy change has been the relative lack of funding for FE in comparison with higher education or pre-16 schooling (some details in here — England has one of the few OECD education systems where institutions receive less per pupil post 16 than they do pre, because typically expert teaching and equipment should make post-16 education more expensive, and where students in vocationally education receive less than their “academic” counterparts). As Marius Busemeyer and colleagues have documented, variation in funding for vocational education across Europe can be explained by political factors: public preferences for funding differ according to the type of education people experience. Thus in the UK further education is weakened by the fact it is not the majority route, and also by a process of policy feedback: the lack of earlier investment creates a weaker constituency of future support. It’s taken Brexit for more funding to open up.
Thinking in terms of underlying political factors which inhibit real improvement highlights that the problem in each of these sectors, it seems, is not too much change but too much surface change and churn in place of real long-term investment. We might think about these areas as having an “immunity to change”, psychologist Robert Kegan’s phrase for the deep commitments that prevent an individual from making changes that they think they want to. In the case of regional governance, for example, properly implementing something like joined up services would require central government to give up some of its powers and also resources, something it seems little prepared to do. In each of the sectors susceptible to churn, therefore, we should look for the deeper small-p political factors that are blocking substantial reform.
One additional answer: Human-centered systems design
All of these problems call for a different way to think about policymaking. While further education, local government, and industrial strategy all involve very different kinds of policies and organisations, they are similar in the extent to which they are all issues where outcomes rely heavily on human interactions and long implementation chains: these are what Lant Pritchett at colleagues at the Harvard Kennedy School would call “transaction intensive” problems. Making policy changes into improvements in these kind of areas is less about coming up with one great or evidence-based idea, and more about working through the myriad of problems that arise when people do not respond to your change in the desired way.
Some (American) colleagues and I have been studying examples of this kind of iterative improvement as something we call “human-centered systems design” — approaches to tackling problem areas with a design mindset that gradually pieces together a new system to produce better outcomes, all in the process of solving particular problems. Design thinking and similar processes have caught on in some sectors because of the way they draw attention to the human factors that mess up attempts to plan solutions through rational linear modelling (essentially, by building in systematic processes to understand user perspectives and so deal with bounded rationality). But design thinking has mostly been applied to creating solutions in terms of products or services. It takes a different kind of approach to plan and implement changes to systems — one that starts with seeing what the relevant system is that is producing the current outcomes.
We’ve hashed out this concept a little in the field of K12 education, but it’s potentially applicable to lots of different sectors. That’s essentially the work of places like Innovation Unit, my sometime-workplace, and Policy Lab, products of sporadic attempts over the years to build these kind of approaches into government. They and others I’m sure have much more to say on the problem of getting from change-as-churn to change-as-improvement, and I look forward to hearing it as IfG continue to pursue this issue.
