Research & task design
During the research design phase, projects define their research questions and agree on the methodology or protocol they will follow to carry out the data collection. They develop suitable data collection instruments to answer their research questions, and define tasks that participants will work on.
The research design itself begins with the project goals, and involves creating a plan for the research implementation as a whole, including data collection and analysis, and the role of participants with potentially different skill-sets at different points in the project. The research design is the grand picture to the focused scenarios of the research tasks.
The tasks for participants are also designed at this stage, and include details for the different kinds of contributions participants can make, and how. Not every participant will contribute at all stages and in all the possible ways to a project. Where and how they engage will depend on their skills, abilities, technology available, and motivation. Therefore, projects need to consider and design their tasks, so that participants with different backgrounds and in different situations can complete them.
Research design depends on project goals and framing, and can therefore vary across project types. Especially if the project wants their research results to be robust enough to serve as evidence for policy-makers or professional researchers, it will require a high level of expertise, which projects may have to seek externally.
- In Action projects, the research design follows naturally from the project goals. For example, if the issue citizens want to address is air pollution, the question to ask is how bad that pollution is in their area, and measuring this pollution with some kind of sensor to answer that question will be a natural fit. For example, Citicomplastic wanted to know whether bioplastic could be composted at home, so they placed composters in participants’ gardens and let them try it out.
- Conservation projects, who often have a longer project duration, are more likely to require a more complex design. Research designs in such projects may have multiple stages or parallel processes of data collection and analysis, or repeat and adapt them multiple times. For example, De Vlinderstichting monitors butterfly and dragonfly populations across the Netherlands over time. To do this, they created a structure of routes (transects) at which they want to measure populations, with different schedules based on life cycles of butterflies and dragonflies. Citizen scientists can sign up to walk these routes and conduct counts, and De Vlinderstichting provides ways for them to record their results – starting with paper or web forms, and now using an app.
- In Investigation projects, the research design is often developed centrally and at large scale, for a high number of participants who then conduct small tasks in their own locality. For example, Loss of the Night wants to understand the issue of light pollution globally, so they require measurements from all over the world. To enable this, they rely on the eyes of their participants, who report which stars they can see from their location.
- In Education projects, the focus is on engagement around the intended learning outcomes. Therefore, more than in other cases, these projects have to consider the age of the target participants, especially with regards to data gathering instruments. For example, Students, air pollution and DIY sensing wants to educate students about air pollution. They work with high schools and teachers to engage pupils, who then carry out small research projects independently: They define their own research questions, and build sensors to collect data to answer them.
- In Virtual projects, the research design has a similar scale as in investigation projects, relying on a high number of participants. It includes the development of a platform through which data can be submitted electronically. This can be done via a website or app, through social media, or text messages. Virtual projects benefit from many small contributions from a large number of volunteers, and their task design needs to maintain the interest of and suit the types of technology their intended participants can use. Therefore, they often use gamification elements.
Depending on the complexity of the project, designing the research and tasks, and defining research questions and methodologies, may or may not be iterative. Sometimes a defined research question is not necessary at all, while other times projects have multiple questions they pursue at the same time. When they are relevant, research questions should be clearly stated, and refer to a single issue individually – there can be multiple questions to cover a range of related issues. They can be rephrased during the course of the project to match the data that can be collected. Each question should be accompanied by at least one hypothesis that can be confirmed or rejected based on findings from the research.
The research design of a citizen science project requires particular care and planning. Many projects will need to consider face to face activities or remote engagement with their participants, and tutorials or dedicated guidance to enable their participants to conduct research activities in their environment. For Online projects specifically, there is a large variety of tools available to support project organisers in the development of their tasks and workflows. General design tools can be used to build the layout and appearance of virtual and computer or smartphone-based tasks, and dedicated citizen science platforms offer ad-hoc task setup tools that support the design of new tasks, or specifings aspects like the task workflow. Similar tools are available on crowdsourcing platforms, which are often used to recruit participants in online projects. These tools allow project organisers to reach out to specific citizen scientists, choose reward for their engagement, filter the contributors, and assign specific qualifications following their engagement.
While defining their research design and methodology, projects may want to identify and reach out to stakeholders who would benefit from the data or outputs from the project, to ensure their results will be relevant and reusable for them (Roman et al., 2020).
At the end of the research design phase, projects should know what they want to find out, what data they will need in order to do so, how they will collect this data, and how their participants will be engaged in achieving the project goals.
The research design process itself is also iterative. While the research is conducted, it is important to continuously assess the ongoing work. Is the project going in the expected direction? Are the planned activities contributing to the set goals? Are volunteers properly involved, and creating the expected output? Did something emerge during the activities that could influence/improve the research design? How is it possible to manage it?
Sometimes the reality of citizen science – or really any research – projects is different from the expectation. Therefore it is important to be flexible while the activities are ongoing, to ensure the main project goal is achieved. However, it is necessary to specify that sometimes the results are more exciting than expected, and they could push the team to plan further activities. It is important to understand the difference between the scope of the current project goals and resources, and possible future initiatives.
At the end of the project, the analysis should allow citizen science projects to respond to the research questions set out in the beginning, and provide the data to support these answers. Furthermore, the research design should be reviewed, allowing for it to be refined and potentially replicated in other places and / or in other contexts.
A free and easy to use tool that allows anyone to quickly and easily design, implement and launch their citizen science project. The tool supports four task types and assets including images, videos, text and sound files. If desired, upon completion of the design and beta testing process, projects can be launched to the main Zooniverse website to recruit from potentially millions of volunteers.
Free and open-source python based citizen science and crowdsourcing task design tool, featuring templates for a number of popular tasks such as image classification and allowing a great deal of customisation. Requires additional software and running on a web-server and likely to be less suitable for those without more technical knowledge.
GUIDELINES & RECOMMENDATIONS
In designing their citizen science research design, projects should consider the following questions:
- What resources do you need to implement and run this project, and how will you access them?
- You could look into support or funding programs for citizen science, or look into free tools and resources that you can use.
- What expertise do you have, and what are you missing? How will you fill those gaps?
- This could be by learning about aspects of the project yourself, by finding volunteers or paid services, or partnering with individuals or organisations who can provide them.
- Are there any individuals or organisations you could partner with, and for what purposes?
- You could reach out to researchers at local universities, NGOs with goals similar to yours, or councillors with a political interest in the issue you are investigating.
- Where and how will citizen scientists be involved throughout the project? What contribution can they make? How will you engage with them?
- Citizens could be involved only for data collection, for example by using an app you provide them with; or they could be involved in the entire process, advising the project on key questions and issues.
- What data do you need to collect to answer your research question? How much data will you need? What will you do with it?
- What is the best way to collect the data required to answer the research questions?
- What tools will you use to collect the data? How will you ensure data quality?
Guidelines for task design
These guidelines are high-level recommendations for designing and implementing citizen science initiatives developed by the ACTION team. They are based on research findings from within the ACTION project – to find out more, you can read the report of our findings.
- Choose a suitable approach
Citizen science is most suitable for gathering or analysing research data where participants can be motivated to engage without requiring payment or rewards — i.e., where a task is inherently engaging, supports research for the public good or can be designed in such a manner that they are inherently fun and enjoyable (e.g., Games With A Purpose). Prior to selecting citizen science as an approach, it is important to consider whether the task and research aims align with these goals. If not, consider alternative methods such as paid microtask crowdsourcing or more traditional, lab-based or field studies.
- Formulate a suitable problem
Consider the task that is to be presented to volunteers. While overarching research questions made be broad, citizen science works best when the questions presented to volunteers are specific and discrete, with limited ambiguity. Think about how the task mapsto the activities that volunteers will complete and what resources volunteers will need as well as any specificities and restrictions that will define the task — for example, will the task require being present in a specific location? If so, the resources, input devices and task steps will be different to a task that can be carried out at home.
- Account for trade-offs
The use of citizen science entails inevitable trade-offs between the quantity of data to be gathered, the speed at which data is to be gathered and the accuracy of the gathered data.Prior to commencing the research process, it is essential to consider and identify which of these factors is to be prioritised and take appropriate steps to safeguard this factor, while taking steps to mitigate threats to the additional trade-off factors. For example, if a project is to emphasise accuracy and quality of submissions, the task completion time is likely to increase and this can limit engagement. It is important to then streamline and simplify the task completion process to support faster data gathering or otherwise take steps to encourage engagement to account for these trade-offs.
- Account for technology
As outlined in the initial guidelines reiterated above, it is important to consider the technology and software that volunteers are likely to use to complete your task. Does the task need to support both mobile and desktop devices or is the task designed to be completed outside of the home? Does the task support multiple browsers? Whereverpossible, support diverse technologies to lower any barriers to entry. If participants cannot access your task, then they are unlikely to put in the effort to overcome these barriers and continue contributing. If these barriers are technological, it is also possible that volunteers will not be able to overcome these barriers or will not know how. While it may require a significant commitment of time and resources, it is essential that these issues are resolved upfront and prior to task publication, as participants who encounter these barriers may be unwilling or unable to return to tasks and/or may otherwise remain unaware that these issues have been resolved.
- Provide Context
Citizen science tasks can often be designed and implemented in such a way that they are trivial and simple for volunteers to complete. This is essential for encouraging accessibility and gathering high quality data, but can obfuscate or trivialise their research value, with the potential to harm volunteer engagement. Tasks, project resources and educational resources should provide additional context on the value that volunteer contributions pose for the research process.
- Provide Feedback
While citizen science tasks are generally designed to be easily understood and completed by all participants, not all projects are able to achieve this. Moreover, even where tasks are otherwise easily understood, participants want and need feedback on the accuracy of their responses and the value of their contributions to scientific research. Providing feedback to participants — either within tasks or through communication features such as forums — can encourage participant engagement with citizen science.
- Solicit Feedback
Tasks should not necessarily remain static. The design process involves a number of assumptions and trade-offs which may not align with the expectations of participants.Soliciting feedback from participants is key to ensure that the needs of all stakeholders are met, with the potential for increased task quality and engagement, as well as volunteer engagement.
- Avoid Ambiguity
While the requirements and processes involved within a task may be clear to task designers, these do not necessarily align with the understanding and motivations of volunteers. To avoid misunderstanding, miscommunication and other issues, avoid ambiguity wherever possible. Support participants through the task process by using discrete, clear questions and limit the need for autonomy and personal judgement.Consider offering multiple choice answers rather than free text responses, for example.
- Consider Time-scales
While citizen science is an effective way to gather large volumes of data for scientific purposes, volunteer engagement is sporadic, asymmetrical and often brief. It can therefore take a significant amount of time to gather larger datasets. This can be offset by focusing on restrictive, limited-time activities such as BioBlitzes, where volunteers are asked togather or analyse data over a short period of time. While this approach can be very effective, it is less effective for tasks with more longer-term aims such as public engagement and education. It is essential to consider the implications and long-term aims of the approach to be used and which factors are most important — is it essential to gather data quickly or in large quantities? Do the research aims warrant longer term engagement and community building or is one off engagement desirable?
- Do not pre-prepare input data
Scientific research often entails concepts or data with which volunteers may be unfamiliar.While these data must be selected carefully, our findings suggest that there is no significant value to be gained from the pre-formulation of input data, for example grouping data according to perceived difficulty. Instead, presenting tasks at random requires minimal time and resources, with no negative impacts on participant engagement.
Street Spectra has created this tutorial for their participants, which explains how to use the spectrograph they provide together with their mobile phone to take pictures of street lamps, and then use the images to categorise the type of lamp they found.
Dragonflies and pesticides developed this tutorial to guide their participants – who would already be familiar with counting butterflies or dragonflies on their transects – on how to collect water samples for the project.
Students, air pollution and DIY sensing developed this tutorial to help other projects who want to set up air quality measurement projects with high school students. It includes an overview of the process they used, and materials that were developed for workshops and events.
The project was developed by citizens for citizens, and allowed for several routed to engagement. In workshops with participants, they co-created a protocol for data collection, which also helped to select points of interest for data collection, and designed their whole data collection process. Participants could further host audio sensors at their own home, or engage through guided walks, where they would collect data at specified stops. These different routes were meant to allow citizens with different engagement preferences and available time to contribute to the project in the best way that was possible for them.
The project had planned to engage participants in the data collection and results phases. However, due to the pandemic-related lockdown in 2020, they had to adapt their approach. Instead of hosting composters on a farm accessible to a large group of participants, especially disadvantaged youths, they found participants who could host a composter and conduct the experiment in their own backyard. Participants were asked to set up the composter, containing manure and bioplastic, with help from the project team. They then proceeded to take weekly temperature measurements and photos of the decay of the bioplastic.