When you hear the word “evaluation,” you might think of it as something that is done after the fact. That would be fair, because that’s when many evaluations take place — after an event, program, or course. Evaluations typically tell us how we did, as well as what we could have done better. While this approach can be very useful where you have an established program model that you already understand well, it is not very useful where you are working in completely new territory – where you are trying to figure things out as you go.  Developmental evaluation can be a powerful tool for these situations.

Developmental evaluation is a very useful tool in helping organizations to understand complex social issues. It was used to evaluate and inform the approach for the Health Promoting Schools movement across Canada.

Communities across the country are taking part in an exciting new initiative that is designed to help end youth homelessness. They have been embarking on the process of developing coalitions, researching the issue and starting dialogues in order to develop solutions and to generate community momentum around this issue.

The Mobilizing Local Capacity (MLC) initiative, started by Eva’s Initiatives in Toronto (and funded by the Catherine Donnelly Foundation) was started about two years ago to catalyze action around this issue and to help communities develop broad strategies to deal with youth homelessness. Communities from across Canada applied to participate and, so far, six communities have been selected.

One World worked with Eva’s Initiatives to conduct a developmental evaluation of the MLC and here are some of the learnings that have emerged so far:

1) Thinking upstream: Most communities respond to homelessness by providing shelter beds. While this is a useful – and necessary – short-term response to the issue, the challenge is that the problem does not go away and more and more beds need to be provided. Communities need to start thinking upstream, to start dealing with the causes of the problem rather than just treating the symptoms.

2) Thinking like a system: Thinking upstream automatically means you need to involve a broader group of organizations in developing solutions. In Saint John NB, for example, high school guidance councilors expressed a strong interest in getting involved as they are often the individuals that high school students turn to when they are experiencing problems at home. Intervening at this early stage could help to prevent some young people from becoming homeless. Similarly, police, who often get involved when youth are already on the streets and involved in criminal activity, really want to get involved on the preventive side of the problem.

3) Engaging the ideas and energy of youth: From the very beginning of its MLC process, Kamloops decided that they wanted it to be youth-driven. They wanted to engage mainstream as well as homeless youth and give them a real voice. They found that youth care passionately about the issue of youth homelessness, and eagerly participated. They brought a drive and commitment to the issue that helped to engage the public in a way that likely would not have happened in an agency-driven process. (Check out this video that was produced by the youth). Much of that energy and perspective is reflected in the final community strategy.

4) It takes a coalition: One of the most important parts of the MLC process is that communities need to develop broad coalitions to develop a systemic approach to the issue, and to get the broad support to move the strategies forward. Kingston’s MLC initiative, led by the local United Way was particularly effective in developing a broad coalition of local organizations and local politicians to take action. Youth played a major role in this initiative as well.

5) A catalyst approach works: The MLC approach is a novel way to support communities to take action. Each participating community develops a strategy that is very “local” in flavor, and really resonates with local players. Participating communities do not receive much funding; instead, they are helped with coaching and connected to experts who can provide them with advice, as well as other resources. They are also connected with other communities going through the same process so there are many opportunities for learning and site visits. All of the communities reported they were very happy with this model of support.

We hope that the lessons learned in this initiative can be applied to other groups tackling complex social issues.

How can developmental evaluation help communities take action to address a complex problem such as youth homelessness?

The Problem: Youth Homelessness

Youth homelessness is a significant national problem in Canada. Approximately 235,000 individuals, youth and families experience homelessness per year. Of these, approximately 20% are youth aged 16-24.[1]

Youth homelessness is a largely hidden problem in most communities. Most youth experiencing homelessness do not wind up “sleeping rough” on the street, but spend nights “couch surfing” wherever they can.

We evaluate aspects of our life and work every day: we might start a new fitness routine and check out our progress in six weeks, or switch to a different software program and then evaluate how effective it is, or where we are with it. We evaluate the minor and the major to answer a simple question: Did it work? Then we ask more complex questions: If it did, what made it work? If it did not, what prevented us from achieving the desired outcome? In the field of public engagement, evaluations can be a powerful tool for learning and improvement. 

Evaluations are not one-size-fits-all. The appropriate type of evaluation depends largely on the purpose of the evaluation.

A formative evaluation focuses on ways of improving and enhancing programs. They are often done through a quality-improvement lens, and may be process-oriented or impact/outcome-oriented.

Formative evaluation can be done at any stage of a program. It can take place as the activity unfolds and provide an opportunity to take corrective action in real time to improve outcomes.  It can also include a phase completed at the end of the project to help assess and document lessons learned for next time.

A summative evaluation occurs at the end of a process and focuses on outcomes – e.g., did we achieve what we had intended?  Such an evaluation judges the overall effectiveness of a program. It often used to make decisions about continuing or terminating a program or project.

Public engagement initiatives vary in size, scope, time frame and purpose, from projects with tens of thousands of participants around the world to panels involving 10 citizens from across town. The objectives may be to effect a change, to do things better, to foster involvement, to increase knowledge and/or to build common ground.

Whatever the intent, the ultimate goal is to make a difference. The challenge lies in measuring this difference. What have we accomplished? What difference has the engagement initiative made? Is one kind of engagement better than another to achieve certain goals? In the public engagement field, sometimes a lack of comparable data makes it difficult to answer these questions. For example, a group might be doing a citizens’ panel in Edmonton to advise city council on policy. It has chosen one set of criteria for evaluation. But another group doing a citizens’ panel in Guelph might have a different set and it becomes difficult to compare the success and achievements of the two panels (even allowing for the importance context can play).

As a professional field of endeavour, public engagement is relatively new. In the past, it has not been a priority to collect and disseminate evidence on the impact and efficacy of engagement initiatives, but that is changing. By combining qualitative evidence with quantitative data we can determine to what extent  an initiative was successful, if it had the impact we wanted and how it could be improved upon. There is so much that can be measured that a key challenge is making the evaluation as practical as possible.

Often, when we think about  evaluation, the number of questions we have multiply rapidly as we brainstorm what we want to find out and discover. As we go through the process, the list of items that we want to measure and examine outgrows the time and resources available.

Currently most public engagement is done because morally it’s the “right thing to do.” If you are affecting people’s lives, you should ask them about it. But can we do more? Can we demonstrate the value and impact of public engagement – the difference  that it can make?

This is one of the biggest challenges we face in the field of public engagement. There is an abundance of anecdotal evidence that points to the efficacy of our work, but concrete, measureable data that demonstrates impact is scarce. Planning and building frameworks for evaluation is essential to the long-term success of public engagement in order to collect the comparable evidence we need to demonstrate the value of public engagement to participants, communities and funders.

Developmental evaluation can be especially useful in helping to deal with complex issues where more conventional approaches to evaluation fall short. It is a way to support the development and testing of creative approaches to address complex problems; it is well-suited to those situations where you are “learning as you go”.

Jamie Gamble describes three main features of a developmental evaluation: