Building Your Organisation’s Evaluation Capacity and Capability

What obstructs and promotes evaluation in third sector organisations (nonprofits and social enterprises)?

Conducting an evaluation has been a practice used by many third sector organisations to critically examine how a program or intervention was implemented, and if it achieved ‘success’. Evaluations help managers and practitioners learn about how to improve program and service delivery effectiveness, and/or to inform whether an intervention is efficient and effective.

There are different types of evaluations, formative and summative. More recently third sector organisations have been focussing on outcome and impact evaluations which often collect real-time data so that managers can continuously improve a program as it is being delivered. Ongoing outcome and impact measurement allows managers to be more responsive to feedback and insights they gain from outcome data. Rather than waiting till the conclusion of a program or funding agreement, it allows them to make early informed decisions about whether to continue, discontinue, replicate or scale up a program or intervention.

Regardless of the type of evaluation, if organisations do not implement good quality evaluations and outcomes measurement, they may fail to optimise outcomes for beneficiaries, fail to give stakeholders opportunities to give their feedback, and fail to remain accountable to stakeholders and the community in which they operate.

While there are many benefits to conducting evaluations that measure outcomes and impact, the third sector continues to struggle with planning, implementing, and using evaluations effectively. A survey by NPC in the UK found that out of 1,000 UK charities 25 percent of them said they do not evaluate or measure the impact of their work at all.

In Australia, a study of more than 350 small, medium-sized, and large charities found of the 70 percent that do measure their outcomes, around 60 percent spend less than 3 percent of their budget on outcomes measurement which is far lower than recommended levels” (Callis, Seivwright, & Flatau, 2019). A large proportion of the small and medium-sized organisations dedicate less than 1 percent of their budget to outcomes measurement.

Another Australian study by Chilchrist (2020) found that a lack of financial resources is seen as the most significant restriction on organisations’ capacity to identify, measure, analyse and report on outcomes (p. 23). This is closely followed by the challenge of upskilling the workforce to implement and use outcomes data to gain insights about their organisation’s performance.

Why do so many organisations still struggle to conduct evaluations that provide good quality evidence about their outcomes and impact?

To investigate the issue, Bach-Mortensen and Montgomery (2018) conducted a review to explore what practitioners have said are the key barriers (what obstructs) and facilitates (what promotes and enables) third sector organisations to conduct good quality evaluations.

Their review identified a range of factors at play:

The top three cited factors operating as barriers to engagement in evaluation:

the lack of evaluation expertise and internal capacity
mismatch between funder requirements and what organisations perceived to be appropriate evaluation goals
the lack of financial resources to conduct evaluation.

The most-reported factors operating as enablers to evaluation:

funders requiring evaluation
involving stakeholders in identifying relevant outcome indicators and evaluation goals
having appropriately trained staff to undertake evaluation.

How should organisations promote better quality evaluations?

Increasing financial support for organisations to evaluate was found to be a necessary factor to improve evaluation practice, but funding alone is not sufficient in isolation. The review identified that organisations need the appropriate support to undertake evaluation, which includes receiving expert advice and staff training to ensure the organisation has the appropriate capability to run evaluations.

The review also identified that organisational culture is very important, which means organisations need to demonstrate the internal motivation to be accountable in addition to the willingness to mobilise resources towards improving evaluation capacity and capability.

Build your organisation’s evaluation capacity and capability

If you would like to build your organisation’s evaluation capacity and capability, book now for QUTeX’s short course Social Impact Evaluation Principles and Practices which introduces participants to the principles and practice of evaluating the effectiveness of program activities using evaluative and strategic thinking.

References

Bach-Mortensen, A., & Montgomery, P. (2018). What are the barriers and facilitators for third sector organisations (non-profits) to evaluate their services? A systematic review. Systematic Reviews, 7:13 doi:http://dx.doi.org/10.1186/s13643-018-0681-1

Callis, Z., Seivwright, A., and Flatau, P., (2019) Outcomes Measurement in the Australian Community Sector: A National Report Card, Bankwest Foundation Social Impact Series, No. 10, Bankwest Foundation, Western Australia.

Gilchrist, D. J., (2020), “Outcomes: Research into Practice”, A report of the National Outcomes Measurement Research Agenda for Grant Thornton Australia, Melbourne, Australia. https://www.uwa.edu.au/schools/-/media/Not-for-profits-UWA/Managing-NFP-Performance/2020-Grant-Thornton_National-Outcomes-Measurement-Research-Agenda—Working-Paper-No-5.pdf

Michael, L. (11 April 2019) NFPs struggling to measure their impact. Probono Australia. https://probonoaustralia.com.au/news/2019/04/nfps-struggling-to-measure-their-impact/

Ógáin E.N., Lumley T., Pritchard D. (2012). Making an impact: impact measurement among charities and social enterprises. http://www.thinknpc.org/publications/making-an-impact/