A brief introduction to: Test order

ShareShare on FacebookShare on TwitterShare on LinkedIn

Science series materials are brought to you by TestGorilla’s Assessment Team: A group of IO psychology, data, and psychometric experts with a deep understanding of the science behind skills-based hiring.

TestGorilla takes a deeply scientific and multi-layered approach to creating tests for skills-based hiring. This series of brief introductions aims to conjure transparency when it comes to how we build our tests and how we advise they are used.

In this post, we’ll introduce you to test order and offer some test order best practices that will help you increase the reliability of your assessments.

What is test order?

When we refer to test order, we’re talking about the order in which tests are placed within an assessment – where an assessment is made up of two to five different tests and/or custom questions. 

Test order definition

Note that this is specifically how we understand tests and assessments at TestGorilla. (It’s also common to use the terms interchangeably, or to understand a test as a type of assessment where there are correct and incorrect answers.) 

Why is it important?

When you’re choosing the order of tests in an assessment that you’re creating, it’s important to consider what the candidate is going through. Taking tests is hard – it’s very cognitively demanding and requires a lot of both deep and time-bound thinking, especially when the test is timed. 

Because of this, the order in which tests are taken by candidates can have an impact on their assessment-taking experience and overall performance. But it’s difficult for people who don’t have expertise in testing to know how to properly order tests within an assessment.

Without this guidance hiring managers can make suboptimal decisions about test order, resulting in a worse experience for both the candidates and themselves. 

Despite test order’s importance, we don’t expect employers, who aren’t experts in testing or assessment psychology, to know how to do it properly. The purpose of this science series article is to support you with test ordering expertise, so that you can improve your test reliability and candidate experience.

2 best practices for test order

Assessment scientists use the term “fatigue effect” to refer to the cognitive demand that makes taking tests hard – the longer an assessment goes on, the more fatigued the candidate will get. Not only is enhanced fatigue a poor experience for the candidate; it can also worsen the psychometric properties of a test. In other words, when the fatigue effect is in play, the data you get about the candidate is noisier and less reliable.

Because of this, there is a broad consensus among assessment scientists in support of two (very simple!) test order best practices:

1. More cognitively demanding tests should be placed first

More cognitively demanding tests should be placed at the start of an assessment, when fatigue effect is low. The impact of this is that candidates can spend their mental energy on the more difficult tests when they are fresh and energized.

2. Less cognitively demanding tests should be placed last

Intuitively, less cognitively demanding tests should be placed at the end of an assessment, when fatigue effect is highest. The impact of this is that when candidates begin to get tired as a result of taking tests, they are working on easier tests that require less concentration.

A test order case study: TestGorilla’s Problem-solving test

Our Problem-solving test has been used hundreds of thousands of times in assessments created by a multitude of employers. Due to the widely-applicable nature of this test, assessments that include it have been used to hire for a wide range of job roles. Customers can, and do, place this test anywhere from first to fifth in the assessments they create.

The Problem-solving test is a hard test for candidates. It’s challenging and cognitively demanding, and requires that candidates demonstrate substantial problem-solving skills in a short amount of time (10 minutes). In short, it’s exactly the kind of test that, as best practices suggest, should be placed at the start of an assessment.

We gathered some data in order to understand the impact that test order has on the performance of our Problem-solving test – where ‘performance’ is determined by the reliability of the test (where reliability is a statistical measure of how consistent candidate scores are). Below is a summary of our findings.

The problem-solving test is more reliable when placed first in an assessment  

Note that when considering the reliability of a test for use in hiring, a reliability score of 0.70 is the minimum acceptable standard, with higher values being more desirable.

Where was the Problem-solving test placed within the assessment?

What was the reliability score of the test under this placement?

What percentage of Problem-solving tests appear in this order?

1st test in assessment

0.772

23.22%

2nd test in assessment

0.735

24.043%

3rd test in assessment

0.726

29.458%

4th test in assessment

0.718

13.204%

5th test in assessment

0.703

10.074%

As you can see, when the Problem-solving test is placed first in an assessment, it has a reliability estimate of 0.772. This score sits nicely above the minimum acceptable score of 0.7. 

As the test is placed further and further along in the test order of an assessment, its reliability score steadily decreases. By the fifth slot, it barely meets the minimum accepted standard for test reliability. Test order really does make a difference. 

Our data also shows that the majority of people using this test place it within the first three tests of an assessment. This shows there is already some intuitive understanding among TestGorilla users that our Problem-solving test should appear early on in an assessment.

A table showing how the reliability of the Problem-solving test increases when you place it earlier in an assessment

The best way to understand test order? Try it out for yourself!

Being intentional with test order can help you make the most of your TestGorilla assessments. It’s low-hanging fruit: by switching your test order around slightly, you can improve the candidate experience and the reliability of your test, enabling you to make better hiring decisions. 

The best way to develop your understanding of test order is by simply trying out the tests for themselves and seeing what feels hardest. 

Place the ones that you find toughest at the start of your assessment, and move the easier ones to the end. You’ll end up with a more reliable assessment, and your candidates will be happy you took that additional step. Why not experiment with some of our tests for free by signing up for a free plan?

You can check out additional blogs about the science behind TestGorilla here:

Share on FacebookShare on TwitterShare on LinkedInShare

Hire the best candidates with TestGorilla.

Create pre-employment assessments in minutes to screen candidates, save time, and hire the best talent.

Free resources

Anti-cheating checklist

This checklist covers key features you should look for when choosing a skills testing platform

Checklist

Onboarding checklist

This resource will help you develop an onboarding checklist for new hires.

Checklist

How to find candidates with strong attention to detail

How to assess your candidates' attention to detail.

Ebook

How to get HR certified

Learn how to get human resources certified through HRCI or SHRM.

Ebook

Improve quality of hire

Learn how you can improve the level of talent at your company.

Ebook

Case study: How CapitalT reduces hiring bias

Learn how CapitalT reduced hiring bias with online skills assessments.

Case study

Resume screening guide

Learn how to make the resume process more efficient and more effective.

Ebook

Important recruitment metrics

Improve your hiring strategy with these 7 critical recruitment metrics.

Ebook

Case study: How Sukhi reduces shortlisting time

Learn how Sukhi decreased time spent reviewing resumes by 83%!

Case study

12 pre-employment testing hacks

Hire more efficiently with these hacks that 99% of recruiters aren't using.

Ebook

The benefits of diversity

Make a business case for diversity and inclusion initiatives with this data.

Ebook