Researchers are increasingly shifting their experiments online. One of the many good reasons for this is that online data collection allows researchers to access a worldwide participant pool.
For those researchers wanting to increase their knowledge about online studies, I’ll be walking you through the different options and considerations for setting up your own online study.
This is not a tutorial on how to setup or code an online experiment. Instead I want to give you more of a general overview of the topic of online studies so you’ll know exactly what’s involved, and can make an informed decision on what tools to explore.
What will we cover?
First, I’ll talk briefly about how participants access online studies through URL links, and I’ll give you some handy tips based on my previous experiences. Then I’ll cover the current best platforms and services to use for creating online surveys and experiments.
As an alternative option, you could custom-code the entire experiment using JavaScript. This can be necessary if you have a complex experiment that can’t be built with the existing tools, but you’ll be surprised at what you can do even without coding.
A quick caveat, the platforms I discuss here are not free. However, academic institutions often provide licenses to use these platforms so that researchers don’t have to pay themselves. I will also discuss some free alternatives so you can still get started even if you can’t access a license.
Part 1
Creating a URL entry point to your online study
You need a way for participants to access your online study. This is typically done via a URL link embedded in your study advertisement (e.g. via an email sent to your participants).
The URL is generated when you build your online study using one of the online platforms I’ll discuss below. So when participants click your link, they’ll be taken directly to your study.
Sometimes you might want to include extra data in the URL to identify who the participant is or where they came from. This can often be stored or used by the study platform to change what the participant sees (e.g. which experiment condition), or tag their data for later analysis (e.g. whether the participant is a control). This extra data is often sent in the form of URL parameters (https://en.wikipedia.org/wiki/Query_string).
Quick Tip
Click here to learn how to set up mail merge
Part 2
Collecting online survey data with Qualtrics
Qualtrics is a dedicated platform for building online surveys. It’s a great option if your study doesn’t require any complex experimental data collection or reaction times.
It’s easy to set up a basic survey and distribute it via an automatically generated URL link. Qualtrics also stores your data so it’s simple to download in .csv or .spss format.
If you dig a little deeper you’ll soon find that Qualtrics is rich in features. Questions can be displayed as Likert scales, drop down lists, sliders, text entry fields and more. You can also apply your own validation, randomisation, and custom logic branches.
A feature I particularly like is the ‘survey flow’ which gives you maximum control over your survey. It also allows you to set all kinds of custom variables such as unique participant id codes. You can pass these id codes and other variables as embedded data via URL query strings (described above) which allows you to combine Qualtrics with other platforms while keeping track of your participants. This is great if you want to combine the powerful survey features of Qualtrics with a more dedicated experiment building platform like those discussed in the next section.
There are lots of options for modifying your survey using the Qualtrics interface
Alternatives to Qualtrics?
Part 3
Collecting online experimental research data: Exploring the best platforms
Creating an online experimental study goes beyond the capabilities of survey builders due to the added complexity of the data (e.g. reaction times, graphics, videos, location of mouse clicks etc.). To collect experimental data you’ll need a more powerful experiment building platform.
There are many experiment builders out there, so in this section I’ll focus on the three platforms I think are most worth exploring. Each platform provides enough features to set up a fully functional online experiment. In addition, they all conveniently host and store your data so it’s easily downloadable as a .csv file.
The choice of which platform to use often comes down to personal preference, as they each take a slightly different approach to building your experiment (e.g. coding vs. graphical interfaces). However it’s good to explore them all so that you can choose which platform best suits your needs.
Let’s start by looking at the first option.
Testable
Testable promotes itself as simple to learn and easy to use. They say you can learn the platform in an hour and create a basic experiment from scratch in about 10 minutes. It’s quite popular with a community of over 4000 researchers and students from over 100 leading Universities.
There’s no need to learn any coding and experiments are created using their natural language form (see below image). The form will generate a basic experiment template and you can make further refinements by editing a downloadable file of trials with additional options available on the dashboard.
Testable covers a variety of experiment designs and includes features such as counterbalancing, advanced logic/branching, staircase procedures, and support for different response types (including voice). Testable also has an integrated survey builder which is handy if you want to avoid mixing platforms (e.g. with Qualtrics).
No software installation is required to use Testable, and it can run in most web browsers.
Testable’s natural language form sets you up with a basic experiment template
Inquisit web
Inquisit web is a good choice for experiments that require timing precision which is important for reaction time studies.
You build Inquisit experiments using its own scripting language (based on JavaScript). Having to do the coding yourself can be a steep learning curve for some. But the good news is that the syntax isn’t too complex and there’s plenty of documentation and tutorials to follow online. Inquisit also hosts a library of pre-existing tests which can be a useful learning reference when making your own experiment.
Unlike Testable Inquisit does not run in the browser, instead participants must download a small piece of software to run the experiment. This could be considered a limitation and some participants may feel uncomfortable opening additional software on their computer. On the other hand, the software does provide superior input timing accuracy and it also allows your experiment to run completely full screen (great for keeping the participant focused!).
I’ve had a lot of success in the past with Inquisit. It’s also fairly easy to pass variables between Qualtrics and Inquisit via query strings (mentioned above) for more complex design setups.
Click here to learn how to build your first Inquisit experimentHere is an example Inquisit script for a survey
And here is the result of the output
Gorilla
Gorilla is another popular choice for people who don’t want to code. You use a graphical interface to create questionnaires and tasks, and then combine it all together in their experiment builder. The experiment builder works by dragging and dropping elements on a grid to build a series of branching nodes that represent your experiment workflow. Additional options and features can be applied (e.g. randomisation) using the dashboard menus.
Since Gorilla offers an integrated Questionnaire Builder and Task Builder all your bases are covered for setting up a fully featured study.
There are enough features here to handle a wide variety of study designs. Although some coding may be required as the complexity of your experiment increases. However, this could also be seen as a positive as it gives you fine-grained control over your design.
Like Testable, no software installation is required, and it can run in most web browsers.
Gorilla have a range of tutorials for learning their platform
In Testable you build questionnaires and tasks using a combination of built-in tools and optional custom coding, then you put it all together in the experiment builder grid
Honourable mention: PsyToolkit
PsyToolkit is a free-to-use, non-commercial platform for programming and running cognitive experiments and surveys. This platform has a bit more of a coding learning curve, but they do host a library of pre-existing surveys and experiments in addition to some very handy tutorials.
Concluding Remarks
You should now have a solid grounding on the different options that are available for building online studies. I’ve covered the main platforms that will get you set up but there are undoubtedly even more platforms that are just as good. Feel free to leave a comment if you think I’ve missed out an important one.
I deliberately avoided making this into a more technical tutorial since there are already excellent resources on the web for building experiments. Instead, I hope this has given you more of a general overview of the topic.
What’s next? In a future post I’d like to make a tutorial on how to build and host your own online experiment from scratch using JavaScript. I’d also like to explore options for collecting data via smartphone devices.
Good luck getting started with your first online experiment!