Planning for Effective Evaluation

Currently most public engagement is done because morally it’s the “right thing to do.” If you are affecting people’s lives, you should ask them about it. But can we do more? Can we demonstrate the value and impact of public engagement – the difference  that it can make?

This is one of the biggest challenges we face in the field of public engagement. There is an abundance of anecdotal evidence that points to the efficacy of our work, but concrete, measureable data that demonstrates impact is scarce. Planning and building frameworks for evaluation is essential to the long-term success of public engagement in order to collect the comparable evidence we need to demonstrate the value of public engagement to participants, communities and funders.

This was what we set out to do with the Aga Khan Foundation of Canada (AKFC), which does international development work, including public engagement in Canada. Its goal is to increase the engagement of targeted Canadians in activities that support human development. Our role was to help the AKFC build an evaluation framework, logic model and performance-management framework so it could better understand the impact of its work as well as meet the government’s requirements for funding and improve its own programming.

STEP 1: APPLYING THEORY OF CHANGE

Our first step was to engage the AKFC in a Theory of Change (TOC) exercise, where we asked questionslike:

• What does success look like in six months, or a year, if the program works well?
•  What will be different because of the public engagement work we do?
• How do we think change happens? How can our programming best support that?

The TOC helped build a shared understanding of the work and became the roadmap that we used to build the framework.

STEP 2: BUILDING THE EVALUATION FRAMEWORK

More and more we encourage organizations to establish a comprehensive evaluation framework before their program launches and then to commit to carrying it through the life of the program in order to measure the outcomes. Too often in public engagement, people don’t think about evaluation until afterwards. And that’s too late collect most of the data that is needed, so establishing an evaluation framework today is critical to evaluating public engagement activities tomorrow.

In the case of the AKFC, with the TOC completed we were able to start shaping the key evaluation criteria and aligning it with the criteria required by funders. We worked systematically to build an evaluation framework, a process that included:

• Establishing evaluation criteria. What are the key criteria (in sometimes called outputs and outcomes) that we want to keep our eye on and assess? How can we measure progress toward the desired outcomes and outputs? Which criteria should we prioritize? What time and budget limitations might determine how many criteria we can focus on? How do the various criteria relate to the different stakeholder groups?
• Developing key indicators for each criterion. What are the measures that will help assess the extent to which the project’s process and impact were successful? Which indicators are quantitative—related to change in the number of participants, for example? Which are qualitative—e.g. related to participant experience?
• Identify data sources. What are the data sources for these indicators? What data do we have access to? What data collection methods – e.g. surveys, focus groups – will we use?
• Data collection. Where and how will we collect this data, and for what purpose? Who is responsible? How will we store the data?

When possible, we need to establish a baseline within the first year so we can do a comparative evaluation from start to finish.  This is rarely done in public engagement work, but is a key component of measurement and evaluation. When we know what the starting point is, we can develop evaluation and data collection tools that can assess the difference made, e.g. pre- and post-engagement surveys.

Our work with AKFC is one of the biggest and most rigorous  initiatives we’ve undertaken in setting up an evaluation system for a multi-year, multi-faceted public engagement program before it begins. The level of detail and the comprehensive scope of this framework give the AKFC the ability to answer important questions about both program impact and program improvement, such as:

• Is the program having the kind of impact that AKFC hoped it would?
• Are there aspects that are working better than others and achieving the types of change we want to see?
• Do we need to modify the program as we go forward, and if so, how?

Over time, when the evaluation data is shared, this rigorous evaluation framework can help the public engagement field speak more fluently about impact. We know anecdotally that public engagement can yield powerful results; we are working on providing the hard evidence to support that.