A brief introduction to: Test order

Science series materials are brought to you by TestGorilla’s Assessment Team: A group of IO psychology, data, and psychometric experts with a deep understanding of the science behind skills-based hiring.

TestGorilla takes a deeply scientific and multi-layered approach to creating tests for skills-based hiring. This series of brief introductions aims to conjure transparency when it comes to how we build our tests and how we advise they are used.

In this post, we’ll introduce you to test order and offer some test order best practices that will help you increase the reliability of your assessments.

What is test order?

When we refer to test order, we’re talking about the order in which tests are placed within an assessment – where an assessment is made up of two to five different tests and/or custom questions. 

Note that this is specifically how we understand tests and assessments at TestGorilla. (It’s also common to use the terms interchangeably, or to understand a test as a type of assessment where there are correct and incorrect answers.) 

Why is it important?

When you’re choosing the order of tests in an assessment that you’re creating, it’s important to consider what the candidate is going through. Taking tests is hard – it’s very cognitively demanding and requires a lot of both deep and time-bound thinking, especially when the test is timed. 

Because of this, the order in which tests are taken by candidates can have an impact on their assessment-taking experience and overall performance. But it’s difficult for people who don’t have expertise in testing to know how to properly order tests within an assessment.

Without this guidance hiring managers can make suboptimal decisions about test order, resulting in a worse experience for both the candidates and themselves. 

Despite test order’s importance, we don’t expect employers, who aren’t experts in testing or assessment psychology, to know how to do it properly. The purpose of this science series article is to support you with test ordering expertise, so that you can improve your test reliability and candidate experience.

2 best practices for test order

Assessment scientists use the term “fatigue effect” to refer to the cognitive demand that makes taking tests hard – the longer an assessment goes on, the more fatigued the candidate will get. Not only is enhanced fatigue a poor experience for the candidate; it can also worsen the psychometric properties of a test. In other words, when the fatigue effect is in play, the data you get about the candidate is noisier and less reliable.

Because of this, there is a broad consensus among assessment scientists in support of two (very simple!) test order best practices:

1. More cognitively demanding tests should be placed first

More cognitively demanding tests should be placed at the start of an assessment, when fatigue effect is low. The impact of this is that candidates can spend their mental energy on the more difficult tests when they are fresh and energized.

2. Less cognitively demanding tests should be placed last

Intuitively, less cognitively demanding tests should be placed at the end of an assessment, when fatigue effect is highest. The impact of this is that when candidates begin to get tired as a result of taking tests, they are working on easier tests that require less concentration.

A test order case study: TestGorilla’s Problem-solving test

Our Problem-solving test has been used hundreds of thousands of times in assessments created by a multitude of employers. Due to the widely-applicable nature of this test, assessments that include it have been used to hire for a wide range of job roles. Customers can, and do, place this test anywhere from first to fifth in the assessments they create.

The Problem-solving test is a hard test for candidates. It’s challenging and cognitively demanding, and requires that candidates demonstrate substantial problem-solving skills in a short amount of time (10 minutes). In short, it’s exactly the kind of test that, as best practices suggest, should be placed at the start of an assessment.

We gathered some data in order to understand the impact that test order has on the performance of our Problem-solving test – where ‘performance’ is determined by the reliability of the test (where reliability is a statistical measure of how consistent candidate scores are). Below is a summary of our findings.

The problem-solving test is more reliable when placed first in an assessment  

Note that when considering the reliability of a test for use in hiring, a reliability score of 0.70 is the minimum acceptable standard, with higher values being more desirable.

As you can see, when the Problem-solving test is placed first in an assessment, it has a reliability estimate of 0.772. This score sits nicely above the minimum acceptable score of 0.7. 

As the test is placed further and further along in the test order of an assessment, its reliability score steadily decreases. By the fifth slot, it barely meets the minimum accepted standard for test reliability. Test order really does make a difference. 

Our data also shows that the majority of people using this test place it within the first three tests of an assessment. This shows there is already some intuitive understanding among TestGorilla users that our Problem-solving test should appear early on in an assessment.

The best way to understand test order? Try it out for yourself!

Being intentional with test order can help you make the most of your TestGorilla assessments. It’s low-hanging fruit: by switching your test order around slightly, you can improve the candidate experience and the reliability of your test, enabling you to make better hiring decisions. 

The best way to develop your understanding of test order is by simply trying out the tests for themselves and seeing what feels hardest. 

Place the ones that you find toughest at the start of your assessment, and move the easier ones to the end. You’ll end up with a more reliable assessment, and your candidates will be happy you took that additional step. Why not experiment with some of our tests for free by signing up for a free plan?

You can check out additional blogs about the science behind TestGorilla here:

• A brief introduction to: Candidate satisfaction

• A brief introduction to: Test validation

• A brief introduction to: Behavioral competencies