Policy development and implementation in Australia has long been plagued by poor evaluation. As a result, there is very little information about what works well, what doesn’t — and why.
The CIS was fortunate to have Dr Ivy Papps visit this week and talk to a group of our researchers and representatives from the not-for-profit sector about the work she has been doing in the UK, evaluating programs aimed at reducing intergenerational disadvantage. It seems the UK suffers the same problems as Australia — both in the nature of the problem to be addressed and in the government’s means and effectiveness of addressing it.
Policies and programs are frequently implemented without proper trials and pilots. A program that is ‘research-based’ is not enough to guarantee success; what looks good on paper does not always work well in practice. When dealing with human beings, there are always unforeseen factors and unintended consequences.
Even when evaluations do take place, they are often designed after the program commences or ends, severely limiting the reliability of the results. The tests used and data collected are not always appropriate or valid measures of the outcome being sought. Too much use is made of surveys that only provide details of the participants’ feelings and opinions about the impact of the program.
Furthermore, programs are abandoned before they have a reasonable chance to have an effect, partly due to the desire for a quick-fix in short electoral cycles. This instability creates an endless process of program development and implementation that benefits only the providers of the services.
This cannot continue. Some governments and organisations have made more progress on this score than others. The NSW education department’s Centre for Education Statistics and Evaluation (CESE) is making an effort to improve the evidence base for education policy. The Smith Family has also long understood the need for good data to underpin their programs. Hopefully others will follow their lead.
Dr Jennifer Buckingham is a Research Fellow at the Centre for Independent Studies.
Home > Commentary > Opinion > Poor evaluation leads to poor policy
Poor evaluation leads to poor policy
The CIS was fortunate to have Dr Ivy Papps visit this week and talk to a group of our researchers and representatives from the not-for-profit sector about the work she has been doing in the UK, evaluating programs aimed at reducing intergenerational disadvantage. It seems the UK suffers the same problems as Australia — both in the nature of the problem to be addressed and in the government’s means and effectiveness of addressing it.
Policies and programs are frequently implemented without proper trials and pilots. A program that is ‘research-based’ is not enough to guarantee success; what looks good on paper does not always work well in practice. When dealing with human beings, there are always unforeseen factors and unintended consequences.
Even when evaluations do take place, they are often designed after the program commences or ends, severely limiting the reliability of the results. The tests used and data collected are not always appropriate or valid measures of the outcome being sought. Too much use is made of surveys that only provide details of the participants’ feelings and opinions about the impact of the program.
Furthermore, programs are abandoned before they have a reasonable chance to have an effect, partly due to the desire for a quick-fix in short electoral cycles. This instability creates an endless process of program development and implementation that benefits only the providers of the services.
This cannot continue. Some governments and organisations have made more progress on this score than others. The NSW education department’s Centre for Education Statistics and Evaluation (CESE) is making an effort to improve the evidence base for education policy. The Smith Family has also long understood the need for good data to underpin their programs. Hopefully others will follow their lead.
• Subscribe
Subscribe now and stay in the loop with our giving appeals, event alerts, newsletters and research updates.
We are always pleased to hear from you. If you have any questions or feedback, please contact us here: