Why is it relevant?

Participatory evaluation provides Citizen Observatory participants with an active role in the evaluation of their Citizen Observatory. It helps to foster dialogue and encourages diverse stakeholder groups to consider different perspectives about the progress and impact of the Citizen Observatory.

How can this be done?

Participatory evaluation encourages Citizen Observatory participants to collectively capture and evaluate the impacts of the Citizen Observatory. In turn, they can reflect on the activities and understand what was successful and what needs improvement for future endeavours. It also allows them to further understand the value of collecting data and evidence about the environmental challenge which is pressing to them. The Co-Evaluation Tool, Community-Level Indicators, Insights Workshops and Impact Stories are some of the tools you can use for participatory evaluation of your Citizen Observatory.

Example of completed Co-Evaluation Tool ©GROW Observatory

The Co-Evaluation Tool for citizen science can also be applied to Citizen Observatories. It aims at capturing how participation in a project has changed knowledge, practices and policy. This tool builds on the Place Standard Tool; it focuses on ten indicators of the different types of impact that could result from a project. This tool, which encompasses a method and canvas, is designed for community members and citizen science practitioners wanting to evaluate a project. It provides prompting questions for discussions among participants, allowing communities and project managers to consider these indicators in a methodical way. This tool can be used at the end of a project to assess the impacts and changes achieved. It can also be useful at the beginning of a project as a way to collect an understanding among stakeholders.

Participants of the Making Sense project in Barcelona complete the CLIs tool worksheet to decide which type of indicators they would like to collect ©Making Sense

The Community Level Indicators (CLI) tool is designed for community members and citizen science practitioners wanting to start a new project and is designed to be used throughout a citizen science project. The CLI method involves asking community members to identify extra information that a community in a citizen science project can collect to complement and contextualise sensor data. For example, if a community is concerned about air pollution in their area, they might start a campaign to reduce the number of cars that drive on their street while using a sensor to monitor changes in air quality. The CLI, in this case, could be the measurement of car traffic on the street over the same period of time that the air quality is also measured and comparing the data to see if there is any correlation between the car traffic and the air quality. This toolkit helps participants to track the potential impact of their actions. The CLI tool helps participants to collaboratively choose what information will be collected, and how. This tool can also be used at the end of a data collection period, to see how actions have made a difference.

Insight workshops are a great way to reflect on the collaborative efforts of the participants and organisers, as well as engaging additional stakeholders who are not part of the core membership. The sessions can be formal or informal but should aim at iteratively presenting insights and results from the Observatory as they emerge rather than waiting for traditional publication, as well as conducting rolling evaluations on what went well and what could be improved. There are many different ways of conducting an insight workshop, but having a space where people feel comfortable to express their opinions is key. Other helpful materials can be post-its for noting down pros and cons of the project, and sticker dots, which can be used for voting. Asking participants to fill out feedback questionnaires during these sessions is also a good way to have them reflect on the activities of the project.

A participant fills out a feedback questionnaire about the pilot during a final insight workshop during the Making Sense project in Barcelona. © Making Sense

You can use impact stories to report on changes that result from your Citizen Observatory. An impact story is a narrative about your journey from when you started the Citizen Observatory. In your story, you will explain the issue you were trying to tackle and talk about moments or events when you were aware of a change that had occurred in relation to the problem you were working on. For example, you can use an impact story to report on a moment or event where you felt empowered in a discussion with authorities because you had gathered data and insights through your work as a citizen scientist. The WeObserve Impact Community of Practice developed an approach that focuses on capturing and promoting policy impacts. The Impact Inquiry Instrument is available here.

Example from the Making Sense project

For Making Sense, participatory evaluation was crucial to the development of the project and also the participants’ awareness of what they had learned about the environmental challenge. During a campaign in Barcelona, the participants got together for a workshop on CLI. They first agreed on a sensing strategy, which included where and when they would be collecting data on noise pollution using the Smart Citizen Kit. 

They then used the CLI tool and worksheet to collectively decide on one or two other indicators that could be used for data annotation in combination with the sensor strategy. The workshop helped participants think about the problem differently. For example, some of the participants wanted to monitor the public presence in the area where the data on noise levels was being collected and then compare this information to the noise sensor data. 

This allowed participants to overcome a culture of blame and to instead see the possibilities of collaborative decision-making. Also, the CLI workshop helped them make sense of the sensor data. By discussing the CLIs the participants were able to plan approaches that would build on the sensor’s datasets to reveal deeper insights into the issue of noise pollution. 

In one case, a participant used her phone to photograph the number of people on the street. These images evidenced the source of the problem. Once the community had gathered the sensor data, they used these photographs alongside the data sets to demonstrate the problem to local government officials.

Useful Resources

TOOL: The Co-Evaluation Tool is open source and  has been developed with community-led citizen science projects in mind.

TOOL: The Community Level Indicators Tool is open source  and has been developed with community-led citizen science projects in mind. It is part of Citizen Sensing: A Toolkit, a collection of 25 methods and tools that can be used in citizen science projects.

SCIENTIFIC PAPER: This paper presents a qualitative study on the development of a novel approach to CLI during two projects that focused on noise pollution.

CoP: The WeObserve Impact Community of Practice brings together practitioners of Citizen Observatories and citizen science to share and learn different ways of capturing impacts, including via participatory evaluation.

You may also be interested in:

I want to measure impacts of the Citizen Observatory…

…via a suitable approach

This work by parties of the WeObserve consortium is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.