How Better Policy Ideas Should Be Tested
Many policy ideas sound convincing in a paragraph.
Some sound convincing in a speech.
Others work well as slogans.
The real test begins after that.
A serious reform proposal should survive several different questions before the public is asked to trust it.
That is the principle behind evidence based policy in New Zealand.
The purpose is not to make reform impossible.
It is to make good ideas stronger and expose weak ones earlier.
Explore supporting research and public evidence in the Data & Truth library.
First Test: Does the Problem Actually Exist?
Policy should begin with a clearly defined problem.
That sounds obvious, but public debate often moves quickly from an alarming example to a proposed solution.
Before designing policy, we should ask:
- What exactly is happening?
- How large is the problem?
- How many people are affected?
- Is the trend getting better or worse?
- Is the evidence national or local?
- Is the problem temporary or structural?
Without this step, governments can end up designing solutions for problems that are poorly understood.
A dramatic case can be real without being representative.
Good policy begins by separating individual examples from wider patterns.
Second Test: Do We Understand the Cause?
Identifying a problem is not the same as understanding why it exists.
Suppose housing costs are rising.
Possible explanations might include:
- land scarcity
- infrastructure constraints
- construction costs
- financing
- population growth
- planning rules
- investor demand
A policy aimed at the wrong cause may produce little improvement.
Worse, it may create new problems elsewhere.
This is why policy design needs a model of how the system works.
A reform should explain not only:
What are we changing?
but also:
Why should this change affect the outcome we care about?
Third Test: What Does It Cost?
Every policy uses resources.
Sometimes the direct government budget cost is obvious.
Sometimes it is not.
A complete cost assessment should also consider:
- administration
- compliance
- enforcement
- infrastructure
- staffing
- technology
- delays
- unintended behaviour
- opportunity cost
A policy that costs $100 million directly may create additional costs elsewhere.
Likewise, an expensive policy may still be worthwhile if its benefits are greater.
The relevant question is not simply:
Is it expensive?
It is:
What do we receive in return for the resources used?
Fourth Test: Who Pays and Who Benefits?
Policy creates winners, losers and trade-offs.
Sometimes this is unavoidable.
A transport project may benefit commuters while requiring taxpayers to fund it.
A housing reform may reduce scarcity value for some existing property owners while improving access for future buyers.
A tax change may improve incentives while reducing revenue available elsewhere.
Fairness does not require every person to receive the same outcome.
But the distribution of costs and benefits should be visible.
Good policy should ask:
- Who benefits?
- Who pays?
- Who carries the risk?
- Are vulnerable groups disproportionately affected?
- Are transitional arrangements needed?
A proposal is easier to debate honestly when these effects are stated clearly.
Fifth Test: How Will People Respond?
People adapt to rules.
Businesses adapt.
Households adapt.
Government agencies adapt.
That means policy cannot be evaluated as though behaviour will remain unchanged.
For example:
A subsidy may encourage more demand.
A tax may change investment decisions.
A regulation may change how businesses structure contracts.
A welfare rule may affect work incentives.
These responses are not always bad.
They are simply part of the policy effect.
A serious reform should therefore ask:
What will people do differently because this rule exists?
Sixth Test: Can Institutions Actually Implement It?
A policy can be theoretically elegant and practically impossible.
Implementation depends on real institutions.
Governments need:
- staff
- data
- systems
- legal authority
- funding
- training
- coordination
If an agency cannot reliably administer a programme, the policy design is incomplete.
Implementation should not be treated as something that happens after the “real” policy work.
It is part of the policy.
A simpler reform that can be administered well may outperform a sophisticated reform that institutions cannot deliver.
Seventh Test: Can It Be Measured?
Policy should define what success looks like.
Otherwise, almost any outcome can later be described as success.
Useful measures should be identified before implementation.
These might include:
- cost
- participation
- waiting times
- household outcomes
- productivity
- housing supply
- employment
- service quality
Not every outcome can be reduced to one number.
But there should be enough measurement to determine whether the intervention is moving in the intended direction.
Eighth Test: What Could Go Wrong?
Good policy analysis includes failure scenarios.
This is not pessimism.
It is risk management.
Questions might include:
- What if costs are higher than expected?
- What if participation is lower?
- What if behaviour changes?
- What if economic conditions deteriorate?
- What if a future government changes part of the system?
- What if the policy creates an unintended incentive?
A proposal that acknowledges risks is stronger than one that pretends they do not exist.
Ninth Test: Can the Policy Be Corrected?
No policy design is perfect.
Governments operate with incomplete information.
That means good reform should include mechanisms for learning.
These may include:
- scheduled reviews
- sunset clauses
- pilot programmes
- independent evaluation
- staged implementation
- clear performance measures
Reform should not become permanent simply because changing it is politically difficult.
The ability to correct policy is part of good policy design.
Tenth Test: Can It Survive Over Time?
A policy can work for two years and fail over twenty.
Long-term durability matters.
The environment may change because of:
- population
- technology
- economic conditions
- demographics
- public finances
- future governments
A reform does not need to remain unchanged forever.
But it should be designed with enough flexibility to survive foreseeable changes.
Short-term political success is not the same as durable reform.
Evidence Does Not Eliminate Values
Evidence can tell us a great deal.
It can show:
- what is happening
- what happened elsewhere
- what something costs
- how people respond
- whether outcomes changed
But evidence cannot decide every political question.
Two people can agree on the evidence and still disagree about priorities.
One may place greater weight on equality.
Another may prioritise individual choice.
Another may value fiscal restraint.
Evidence does not remove those differences.
Its role is to make the trade-offs clearer.
That is why evidence-led policy should not mean pretending that values do not exist.
It means separating factual claims from value judgements wherever possible.
Disagreement Is Part of the Process
A good policy process should allow serious disagreement.
Opposing arguments can reveal assumptions that supporters overlooked.
Criticism can expose:
- hidden costs
- implementation problems
- unintended incentives
- fairness concerns
The objective should not be to eliminate opposition.
It should be to improve the quality of disagreement.
A strong idea should be able to survive scrutiny.
What Should the Solution Lab Test?
FIXNZ uses the Solution Lab to examine reform proposals through a common set of questions.
Evidence
Is the problem supported by credible information?
Mechanism
Does the proposal address the likely cause?
Cost
What resources are required?
Fairness
Who gains, who pays and who carries risk?
Behaviour
How might people respond?
Implementation
Can institutions actually deliver it?
Measurement
How will success or failure be recognised?
Risk
What could go wrong?
Adaptability
Can the proposal be corrected?
Durability
Can it remain workable over time?
Explore and compare policy proposals in the FIXNZ Solution Lab.
The FIXNZ Perspective
Better policy does not begin by asking whether an idea comes from the left, right or centre.
It begins by asking whether the idea survives examination.
Does the evidence support the problem?
Does the proposal address the cause?
What does it cost?
Who benefits?
Who pays?
Can it actually be implemented?
And what happens if it does not work as expected?
The purpose of testing ideas is not to prevent reform.
It is to create reforms that are easier to understand, harder to misrepresent and more likely to work.
Good ideas should welcome scrutiny.
Weak ideas should be improved or rejected.
That is the purpose of the Solution Lab.
Better reform begins with better testing.
