Finally, projects will want to share and communicate their results, and evaluate their impacts.
Project outputs are resources that the projects produce based on their work, and the insight they generated. They can be many different things, such as data sets, reports, academic papers, online resources like maps or other data visualisations, software, or policy recommendations, to name just a few. These result types have different levels of quality and complexity, and will consequently lead to different types of impacts. For example, an open, well-documented dataset may have a large impact on research, whereas a set of well-formulated policy recommendations will be more useful in discussions with policy makers, and can lead to policy impact. Projects should have set out in the Problem framing phase what kind of outputs they want to produce, and what impact they want to achieve, which should guide how they go about producing and using their results in this phase. Projects may also find that unexpected results can lead to additional outputs.
How projects share their results will differ with the project type, as well as the audiences they want to reach.
- In Action projects, results are commonly shared among participants and with local stakeholders (media, policy makers, community groups) to achieve the policy impact identified at the outset. The evaluation of such projects is often limited, as it may not be seen as required, if the project is not on-going. For example, Citicomplastic created a public report and held a workshop in order to raise awareness of the issues they found with the composting of bioplastic.
- In Conservation projects, results are likely to have direct influence on policy due to direct links between the project host and policy-makers. There is a higher need for evaluation to demonstrate appropriate use of resources to funders, especially if the project is funded through tax income. For example, Noise Maps developed strong links to local communities and policy makers, which enables future discussions and projects about the issues they are interested in.
- In Investigation projects, results are likely to be shared widely, including through formal academic publications. Evaluation and impact assessment are required to justify further funding and enable impact of the results. For example, Sonic Kayaks made all of their outputs – data, reports and presentations – publicly available, to enable future research and support the open source community.
- In Education projects, results will not only include scientific insight that can be communicated, but also learning achievements with individual participants. Therefore, evaluation will likely include an assessment against the projects’ learning objectives. For example, the students in Students, air pollution and DIY sensing presented the results of their research projects at a public conference, and partner NILU published a toolkit to enable others to replicate their work on a national website.
- In Virtual projects, results are likely to be shared widely, including through formal academic publications. Evaluation is likely to be on-going, as platforms need to be maintained and continuously developed to engage participants with state-of-the-art technology.
Following the open science and FAIR principles described in the Data section, all results should be openly accessible, so they can inform future research and innovation. This helps to avoid unnecessary repetition of data collection or analyses for other projects.
To provide contextual details on the intended purpose(s) of gathered data, projects should publish not only their raw data, but also document completed and/or intended analyses alongside datasets. Wherever possible, this should include numerical results, data visualisations and the interpretation and analyses of results, such as a text-based report, which would be stored and disseminated alongside datasets (Roman et al., 2020). All details about the preprocessing of data should also be published alongside datasets and other research outputs. This may include a version history, a methodological description, or pre-processed versions of the dataset.
If stakeholders are to make use of data, then they must first be able to find and access any datasets and outputs. Any public dataset should include a permanent identifier such as a DOI (Digital Object Identifier), as well as a human readable and ideally machine-readable license. Project web-pages should have clear, visible links to external datasets and other resources (Roman et al., 2020).
Projects may also want to communicate the insight they generated to the wider public, or to specific stakeholders they have identified at the beginning of the project or during their work. They should develop a communication strategy based on their stakeholder map, where they prioritise the most relevant stakeholders, and consider for each of them what they want to communicate to them, and what is the best way to reach them. This could include engagement activities for the wider public or local communities, like public presentations, talks, or webinars; or more targeted activities, like discussions with policy makers, giving evidence to public consultations, or discussions with students at a local school.
Communicating insights to different stakeholders requires projects to tailor their messaging depending on the target group. For example, policy makers are rarely interested in data and detailed justifications – rather, they want to know succinctly what they should be doing and why. Therefore, the best way to communicate results to them is in brief reports leading to recommendations for defined actions that can be consumed in the space of a coffee break. On the other hand, engaging the wider public may require to explain the basics of the research process, so that lay people can follow what the project did and what their results mean.
Assessing project impact
Assessing the impact a citizen science project has is crucial for any project, though the focus of this assessment might be different between short- and long-term projects. While for longer-term projects an impact assessment can be used iteratively to improve the project, for short-term projects it is a way to map the results in an accurate way and enable the project to communicate their results to different stakeholders. If a project with a single goal has achieved that goal, such as a change in local policy, this may serve as a single measure of success – but relying on this alone will miss out on all the other impacts the project has had. The overall goal of impact assessment is to bring about a more ecologically, socio-culturally and economically sustainable and equitable environment. ACTION has developed an impact assessment framework to assess the scientific, social, economic, political and environmental impact of citizen science projects, which is accompanied by supporting materials, which are all listed under Activities below.
This guide includes many useful resources, but most importantly, walks citizen science projects through six steps to set up their communication plan.
Data visualisation tools
Grafana or Tableau are free tools for data visualisation. Projects can use them to create a large array of different visualisations from their data, such as heat maps, histograms, or even complete dashboards. Both come with extensive documentation and tutorials.
Web portal where you can publish open datasets and their metadata, to make them available for the community to reuse. Keep track of the data produced/ consumed/ processed by our citizen science initiatives.
GUIDELINES & RECOMMENDATIONS
Concerning the results of their research, project managers should consider the following questions:
- Which specific outputs will the project produce? This could be a report, an open dataset, an academic publication, a flyer to inform citizens, a business model, or many other things.
- What are the short term and long terms social, economic, political and environmental impacts of your project?
- Who will be interested in these results? Who should be interested?
- How will you communicate your results to these groups?
The ACTION team has hosted several webinars on data, how it can be used and published:
- Webinar on the open data portal, which explains how projects can use Zenodo to publish their data and other outputs.
- Webinar on Grafana, which demonstrates a free tool that projects can use for data visualisation.
- Slides for a talk on communication and outreach, providing guidance on how to build a communication strategy and tools to use to reach specific target groups.
- The ACTION team also developed a webinar on impact assessment that explains in a synthetic way the overall approach and how to use the impact assessment canvas.
Socio-economic, political and environmental impact self-assessment methodology and tools
ACTION has developed an impact assessment framework and methodology to help citizen science projects understand what scientific, social, economic and political impact they have. The framework also helps CS project managers in analysing the transformative potential of their project, i.e. the degree to which the project can help to change, alter, or replace current systems, the business-as-usual in one or more fields such as knowledge production or environmental protection.
Following a mixed methods approach, the methodology is designed to be modular and flexible, in order to be adaptable to the specific characteristics of different CS projects. Indeed, not all the impact dimensions considered are (equally) relevant for all projects, depending on their nature, their specific focus and the level of citizen engagement.
The process for assessing the impact of a CS project works as follows:
- Define the project outputs, stakeholders and relevance of various impact dimensions. This can be done by using the ACTION impact assessment canvas: a four pages graphic form, accompanied by guidelines supporting projects in filling it in.
- Define an impact assessment process, and when and how to collect the required data. Projects can use ACTIONs impact assessment matrix, which lists the different variables for each of the impact dimensions, and advises who needs to supply the data (project managers and/or citizens), and when (only at the end of the project (ex-post), or also at the beginning (ex-ante)).
- Gather data. This can be done by using questionnaires developed and successfully tested in ACTION.
- Analyse the data and draft a report. An example is available for the Students, air pollution and DIY sensing pilot in ACTION.
The project has published all of their data – survey results and measurements – and analysis on Zenodo, making them free for anyone to use. They also used their results (from a pre-ACTION stage of the project) to write a research paper together with their participants, which explains the concept of sonic kayaks and how their manifold usage opportunities: to monitor water quality, allow people with visual impairments to kayak, or as an art installation.
Ars ACTION Lab
Several of the ACTION pilots presented their findings at Ars Electronica in September 2020 – a festival that celebrates Art, Technology and Society:
- Noise Maps presented their findings on the sound heritage of Barcelona, and hosted a discussion on the nature of noise and its role in our culture.
- Students, air pollution and DIY sensing hosted a workshop that demonstrated how to build air pollution sensors and measure air quality.
- Citicomplastic ran a workshop on the (issues with the) disposal of biodegradable plastic, and what citizens can do about them.
- Street Spectra hosted a workshop on the problem of light pollution, demonstrating how participants can use a spectrograph in combination with their smartphone to help tackle the issue.
- Loss of the Night held a global observation campaign, followed by a workshop explaining light pollution and how it can be measured.
Summaries and videos of the presentations are available on the linked websites. All of them are fantastic examples for how citizen science projects and their results can be used to engage new, sometimes unexpected audiences.