Toolkit

Guiding questions

Concerning the results of their research, project managers should consider the following questions:

  • Which specific outputs will the project produce? This could be a report, an open dataset, an academic publication, a flyer to inform citizens, a business model, or many other things.
  • What are the short term and long term social, economic, political and environmental impacts of your project?
  • Who will be interested in these results? Who should be interested?
  • How will you communicate your results to these groups?

Results

At the end of their implementation work, projects will want to share and communicate their results, and evaluate their impacts. 

Project outputs are resources that the projects produce based on their work, and the insight they generated. They can be many different things, such as data sets, reports, academic papers, online resources like maps or other data visualisations, software, or policy recommendations, to name just a few. These result types have different levels of quality and complexity, and will consequently lead to different types of impacts. For example, an open, well-documented dataset may have a large impact on research, whereas a set of well-formulated policy recommendations will be more useful in discussions with policy makers, and can lead to policy impact. Projects should have set out in the Problem framing phase what kind of outputs they want to produce, and what impact they want to achieve, which should guide how they go about producing and using their results in this phase. Projects may also find that unexpected results can lead to additional outputs. 

How projects share their results will differ with the project type, as well as the audiences they want to reach. A grass-roots citizen science initiative focused on local issues is unlikely to have — or to need — the same reach as a national initiative funded by policy makers or government. It is important to consider the best way to communicate and disseminate relevant findings on a project-by-project basis. Citicomplastic created a public report and held a workshop in order to raise awareness of the issues they found with the composting of bioplastic. Noise Maps developed strong links to local communities and policy makers, which enables future discussions and projects about the issues they are interested in. To achieve wider reach, Sonic Kayaks made all of their outputs – data, reports and presentations – publicly available, to enable future research and support the open source community. Similarly, the students in Students, air pollution and DIY sensing presented the results of their research projects at a public conference, and partner NILU published a toolkit to enable others to replicate their work on a national website. Projects should carefully consider which outputs they have to share and how best to disseminate those.

Publishing data

Following the open science and FAIR principles described in the Data section, all results should be openly accessible, so they can inform future research and innovation. This helps to avoid unnecessary repetition of data collection or analyses for other projects. 

To provide contextual details on the intended purpose(s) of gathered data, projects should publish not only their raw data, but also document completed and/or intended analyses alongside datasets. Wherever possible, this should include numerical results, data visualisations and the interpretation and analyses of results, such as a text-based report, which would be stored and disseminated alongside datasets (Roman et al., 2020). All details about the preprocessing of data should also be published alongside datasets and other research outputs. This may include a version history, a methodological description, or pre-processed versions of the dataset. 

If stakeholders are to make use of data, then they must first be able to find and access any datasets and outputs. Any public dataset should include a permanent identifier such as a DOI (Digital Object Identifier), as well as a human readable and ideally machine-readable licence. Project web-pages should have clear, visible links to external datasets and other resources (Roman et al., 2020).

TOOLS

Data documentation tools

As well as the ACTION Data Management tool, there are two checklist tools which support documentation of datasets. Datasheets for Datasets and Model Cards are both templates consisting of a set of questions which should be considered and answered when preparing a dataset for publication. The answers are designed to prevent yes or no answers and cover issues that other stakeholders will need to be aware of if they are to use your data. Datasets for Datasheets is more general, while Model Cards is tailored towards machine learning and AI models.

Data publication tools

Zenodo is an Open Science platform, where data and any outputs can be stored, and receive a DOI. The ACTION Open Data Portal is dedicated to citizen science, and can be used to publish open datasets and their metadata, to make them available for the community to reuse, or keep track of the data produced/ consumed/ processed by our citizen science initiatives.

In the future, the ACTION Data Portal will integrate ACTION’s ASSET Research Objects, which allows it to visually display the resources generated in each pilot, and the relations between them. A research object is an aggregation of research resources to exchange scholarly information on the Web. In a research object we can include papers, data, software, images, slides and any other research artifacts that were used in your research. This can be useful for other researchers (or CS projects) to reproduce and replicate your experiments in other conditions. One of the obstacles is that generating a resource object is a tedious task, which sometimes hampers its use . Thus, ACTION has developed a visual tool based on the tool ASSET to facilitate this task. Research objects generated with this tool may be published on other platforms like RoHub or visualised in our future Knowledge Portal (an evolution of our Open Data Portal).

Data visualisation tools

Grafana or Tableau are free tools for data visualisation. Projects can use them to create a large array of different visualisations from their data, such as heat maps, histograms, or even complete dashboards. Both come with extensive documentation and tutorials. Another easy-to-use online tool specifically for the development of dashboards is Infogram.

ACTION Webinars

The ACTION team has hosted several webinars on data, how it can be used and published:

  • Webinar on the open data portal, which explains how projects can use Zenodo to publish their data and other outputs.
  • Webinar on Grafana, which demonstrates a free tool projects can use for data visualisation, explains the main charts used to visualise data and the use of Grafana to display dashboards.
  • Slides for a talk on communication and outreach, providing guidance on how to build a communication strategy and tools to use to reach specific target groups.
  • The ACTION team also developed a webinar on impact assessment that explains in a synthetic way the overall approach and how to use the impact assessment canvas.
CASE STUDIES

Sonic Kayaks

The project has published all of their data – survey results and measurements – and analysis on Zenodo, making them free for anyone to use. They also used their results (from a pre-ACTION stage of the project) to write a research paper together with their participants, which explains the concept of sonic kayaks and how their manifold usage opportunities: to monitor water quality, allow people with visual impairments to kayak, or as an art installation. 

Dragonflies and Pesticides

The Dutch Butterfly Conservation publishes the data on pesticides collected by their citizen scientists on Zenodo. The method developed with University for Applied Sciences Leiden will also be made available through Zenodo in order to allow others to study these compounds in the environment themselves. Furthermore the activity itself raised awareness for an important threat to biodiversity that is invisible and often ignored.

Publishing insights

Projects may also want to communicate the insight they generated to the wider public, or to specific stakeholders they have identified at the beginning of the project or during their work. They should develop a communication strategy based on their stakeholder map, where they prioritise the most relevant stakeholders, and consider for each of them what they want to communicate to them, and what is the best way to reach them. This could include engagement activities for the wider public or local communities, like public presentations, talks, or webinars; or more targeted activities, like discussions with policy makers, giving evidence to public consultations, or discussions with students at a local school. 

Communicating insights to different stakeholders requires projects to tailor their messaging depending on the target group. For example, policy makers are rarely interested in data and detailed justifications – rather, they want to know succinctly what they should be doing and why. Therefore, the best way to communicate results to them is in brief reports leading to recommendations for defined actions that can be consumed in the space of a coffee break. On the other hand, engaging the wider public may require explaining the basics of the research process, so that lay people can follow what the project did and what their results mean.

GUIDELINES & RECOMMENDATIONS

Guidelines for engagement

The Scivil Communication Guide includes many useful resources, but most importantly, walks citizen science projects through six steps to set up their communication plan.

The Data Refuge Toolkit is a downloadable resource collection, which enables projects to create public engagement activities. It was created by the Data Refuge project.

CASE STUDIES

Ars ACTION Lab

Several of the ACTION pilots presented their findings at Ars Electronica in September 2020 – a festival that celebrates Art, Technology and Society:

Summaries and videos of the presentations are available on the linked websites. All of them are fantastic examples for how citizen science projects and their results can be used to engage new, sometimes unexpected audiences.

WOWNature

The WOWNature project worked with the air quality sensor provider WiseAir, who supplied the sensors for the duration of the project, and also conducted the analysis of the readings, which was too technical and specific to be done by the project. The project team did support the analysis and report creation by providing specific local knowledge about the forests, such as where trees were cut here, or where barbecues take place. The team also provided information about the forest structure and stakeholders, based on its certification for forest management.

Wiseair analysed the data and provided a report about the analysis, that was written in a way that the citizen scientists could understand. A preliminary report was shared with the project and their participants, to which citizen scientists gave feedback, such as where details or conclusions were not sufficiently explained. The final report was then improved by Wiseair based on this feedback, and published by the project. In addition to the report about the analysis, the team has written and published another report about the whole WOWNature project that is used for dissemination, and may also support similar initiatives elsewhere, as well as the raw data collected through the sensors.

Assessing project impact

Assessing the impact a citizen science project has is crucial for any project, though the focus of this assessment might be different between short- and long-term projects. While for longer-term projects an impact assessment can be used iteratively to improve the project, for short-term projects it is a way to map the results in an accurate way and enable the project to communicate their results to different stakeholders. If a project with a single goal has achieved that goal, such as a change in local policy, this may serve as a single measure of success – but relying on this alone will miss out on all the other impacts the project has had. The overall goal of impact assessment is to bring about a more ecologically, socio-culturally and economically sustainable and equitable environment. ACTION has developed an impact assessment framework to assess the scientific, social, economic, political and environmental impact of citizen science projects, accompanied by supporting materials, all listed under Activities below.

Data collected through such an impact assessment methodology, along with a summary of the most relevant information on the project, can then be visualised in an infographic, such as the ACTION Pilot’s Dashboard. An infographic is an effective solution to graphically narrate a story about the project’s achievements, with the end-goal of helping the audience understand the context, the purpose of the various charts, and the progress made by the project over time. In the case of the ACTION pilots, it provides a set of intuitive, easy-to-use online widgets for different types of indicators, including socio-economic information, number of records gathered and validated, publication, events, etc. Besides infographics, impact assessment results can be described in ad hoc reports such as those included in the ACTION final impact assessment (Passani et al. 2022) or scientific papers (Grossberndt et al., 2021).

TOOLS

Socio-economic, political and environmental impact self-assessment methodology and tools

ACTION has developed an impact assessment framework and methodology to help citizen science projects understand what scientific, social, economic and political impact they have. The framework also helps CS project managers in analysing the transformative potential of their project, i.e. the degree to which the project can help to change, alter, or replace current systems, the business-as-usual in one or more fields such as knowledge production or environmental protection.

Following a mixed methods approach, the methodology is designed to be modular and flexible, in order to be adaptable to the specific characteristics of different CS projects. Indeed, not all the impact dimensions considered are (equally) relevant for all projects, depending on their nature, their specific focus and the level of citizen engagement.

The process for assessing the impact of a CS project works as follows:

  1. Define the project outputs, stakeholders and relevance of various impact dimensions. This can be done by using the ACTION impact assessment canvas: a four pages graphic form, accompanied by guidelines supporting projects in filling it in.
  2. Define an impact assessment process, and when and how to collect the required data. Projects can use ACTION’s impact assessment matrix, which lists different variables for each of the impact dimensions, who needs to supply the data (project managers and/or citizens), and when (only at the end of the project (ex-post), or also at the beginning (ex-ante)).
  3. Gather data. This can be done by using questionnaires developed and tested in ACTION.
  4. Analyse the data and draft a report. An in depth analysis is available for the Students, air pollution and DIY sensing pilot in ACTION, while shorter reports are available for each of the ACTION’ pilots in the ACTION final impact assessment deliverable, together with an aggregated analysis (available on ACTION website and on Zenodo).
CASE STUDIES

In My Backyard

CS projects can use the ACTION impact assessment canvas as a self-reflecting tool, as a guideline for discussing within the pilot team about the expected impacts and ways of maximising them and presenting them to their stakeholders. In My Backyard included the main concepts of the impact assessment framework in their final report. In a visual and communicative way they presented their main achievements in terms of social, economic, political and environmental impact.