Inside A nugget challenge

The creation of each challenge and its components involves very rigorous generation and review processes. Let’s go over how we create each of these pieces (for more of the science behind each of these pieces, click here!).

Rank-ordered questions

Each rank-ordered question can be broken into two key parts: a scenario and its corresponding response options. The scenario outlines what is happening at a (imaginary) workplace, placing the user into a specific work context. The response options then provide the user with four options of what they could do in the scenario to allow them to outline what they would be most likely to do. When creating the scenario, we use the following considerations:  

• Skill Relevancy: The scenarios must involve a work situation that is relevant to the skill being assessed (e.g., Critical Thinking) and more specifically, the behavioural indicators of this skill (e.g., using data to guide problem solving).  

• Key Information: The scenario must be written so that there is enough information for the person completing the challenge to make an informed decision about what they would do. We want to make sure there is as much context in the scenario as possible so participants can make a decision without confusion, while avoiding so much information that the person feels as though only one of the response options is desirable.

• Simplicity: We avoid using jargon in the wording of the scenario so that everyone and anyone completing the rank-order question can understand the scenario.

• Widely Applicable: We must ensure the scenarios are not too specific to situations or tasks that are only relevant to one or a few jobs. We often keep the scenarios broad/vague to ensure those from various jobs could make informed decisions about what they would do.  

Taking into account all of these considerations, an example scenario from our challenge library is:  

“You are a team lead running a meeting with Avery, the junior lead recently assigned to your team. The meeting is with your company’s most important client, Hayden. The stakes are high with this meeting, so you have been dreading the thought of it for weeks. It is the halfway point now and you call for a 15-minute break.

You exit the meeting room, with Avery following behind you.

Avery says, “Is it just me, or are we doing just absolutely horribly right now?” You reply, “Yeah, the client hates pretty much everything. It seems to me that Hayden’s just not interested in changing any part of this process and that the company just wants to stick to their old ways.”

Avery nods in agreement and says, “Yeah, I definitely agree with you. It doesn’t seem like Hayden is really listening anymore. They are just taking every opportunity to shoot down what we say. So, what do we do?”  

You can see that the scenario is work-related, broad while providing key pieces of information required to make a decision and the language is not specific to any one type of job.

The next step in the process is to create the four response options for the rank-order question. Each response option reflects a different level of the target skill. Some additional considerations we take into account when creating the response options for the rank-order questions are:

• Equal Appeal: We make sure that each option is appealing and appears to be a reasonable and realistic option. If only one or two of the response options appear like good options, everyone will select one of those options, regardless of what their level of each soft skill is. This is the result of them wanting to select the options they think will make them look the best to the company they work for or are trying to be hired by. It is particularly important to make the items appealing that are indicative of low levels of each skill as these should not evidently be the ‘worst’ options (or no one will choose them).

• Length Similarity: We make the response options as similar in length as possible. This helps to ensure that people completing the challenge don't just assume that the longest item is the best item and therefore the one they should select.  

With these considerations, response options for the scenario above are:  

1. "Plan to go back into the room and ask the client, Hayden, for feedback to see if you should change directions.

2. Suggest that you change directions to try to appease Hayden based on what you believe would be more interesting to them.  

3. Suggest you stick to your current ideas because you believe in your approach and have spent a lot of time on your meeting prep.

4. Plan to go into the meeting and ask Hayden to pay attention if they seem distracted as they might just not be fully understanding your ideas."

So with that we have our rank-ordered question laid out! Are we ready to move ahead? Not quite.  

We always make sure we do a thorough review of the content before moving onto the next step. When reviewing the content, we revisit all the things we were meant to consider during the creation of the rank-order question to ensure each consideration was properly adhered to. Specifically, we use the following list:

1. Is the rank-order question scenario rich enough for people to read, understand and respond to?

2. Is the scenario broad enough that it will be applicable across a variety of different jobs?

3. Do the response options tap into various levels of the target skill?  

4. Are the response options balanced in desirability and length?

Once the rank-ordered question has been thoroughly reviewed and we verify that it adheres to each of these criteria, we are able to move it into production.  

Open format question  

For the open format questions, we must create a scenario for the person to respond to naturally and flexibly. We may ask the person completing the challenge to either write a message on a specific platform (e.g., Teams, Slack), write an email, write a script of what they would say to someone in person, write a brief action plan, or another work activity that is relevant in that scenario. When creating the open format questions, we make the following considerations:  

• Skill Relevancy: The open format questions, much like the rank-ordered questions, must elicit the target skill.  

• Information Richness: We aim to ensure the scenario is rich enough in context that the person completing it can respond appropriately and demonstrate their skills.  

• Opportunity for Expression: We try to make sure there is enough information provided in the scenario that someone could write at least a couple sentences in their response. This helps us to better assess the soft skills of the person completing the challenge as there is more information about the person that can be evaluated.

• Logical Flow: Because most of our open format questions in our current library naturally flow from the rank-ordered questions, we aim to ensure that the plot of the scenario in the open format questions realistically and reasonably build on the storyline of the preceding scenario.  

An example open format question from our library is:  

“After communicating your plan to Avery, you just finish filling up your water bottle and look down at your watch. You wince at the realization that the break is coming to an end.  

You nudge Avery to start walking towards the meeting room. You both enter the room and see that Hayden is just getting off what sounded like an important call.

Hayden lets out a sigh and says, “What’s next?”

You take a deep breath and say….

Write a script of what you would say to Hayden.”

We will then review each work sample. When doing so, we ask ourselves the following questions:  

• Does the open format question tap into the target skill?    

• Is the scenario rich enough in context for the user to fully understand it as they complete it?  

• Are we asking the user to do something in the open format question that everyone will understand or will only those in a specific job be able to adequately complete the work sample?

• Does the open format question naturally flow like a narrative from the preceding question in the challenge flow?  

• Is the open format question written so that it could capture some unique information from what was already provided in the preceding section?  

Taking it all in, it seems like a lot, right? Although it is a time and resource intensive process, we believe our challenge creation process is totally worth it. This process helps to give us confidence in how we are measuring each of our soft skills and helps us to provide clients and users with accurate soft skill scores. We should note, however, that this is just the first step in ensuring we are doing the best things possible for our clients.  

How do we make sure that we are accurately assessing our soft skills beyond the thorough challenge creation process? Check out this post to find out more!