Test Case Variables

Overview

When you run a test cell during an exam, sets of input data are run through your solution and the output is checked for accuracy. If any test case fails, the test case variables are provided so you can see the exact inputs and outputs where your solution didn't match the expected result.

Passing the test cell means you can submit your solution to the autograder, which tests your code against a larger set of inputs. These additional tests are randomly constructed, just like the tests run in the test cell, meaning they still conform to the requirements given for that exercise.

Passing the test cell doesn't guarantee your code will pass when submitted

The autograder uses a randomized set of inputs that may reveal bugs in your solution not found by cases run in the test cell.Follow the workflow below to learn how to read the autograder output and use test case variables to debug.

What exactly are test case variables?

All test cells across every exam and most homework notebooks use the same four variables. Each is a dict where the keys match the parameter or output names from the exercise instructions.

Variable	Purpose
input_vars	The inputs passed to your function. Keys are the parameter names.
original_input_vars	A copy of inputs before your function runs. Useful for checking if your solution accidentally modifies the input.
returned_output_vars	What your function returned. If there are multiple outputs, keys match the names in the exercise instructions.
true_output_vars	The expected (correct) output. Same key structure as returned_output_vars.

Typical Workflow

This workflow focuses on testing and debugging your code. For guidance on breaking down problems and writing pseudocode, see Exam Prep Advice.

Debugging Flowchart

This chart shows you the expected flow for solving exercises. The orange sections roughly represent the debugging steps, including test case variable usage.

Written out:

1. Write/edit code - Read the exercise context, write and edit code to solve it
2. Run solution/demo cell -> Confirm your output and the printed demo look the same; this is a simple check for you to iterate quickly
3. Run test cell -> Confirm it passes; if it fails, the test case variables are provided for you to inspect
4. Iterate -> View test case variables to see failure, jump back to step 1, and keep iterating until you pass
5. Submit and ensure points are awarded -> If no points are awarded, you can click details > view grading report to see the failure

Your code can fail due to other issues as well

The test case variables compare the data going into your function against the expected output. However there are other errors that could cause you to not receive points, such as timing out the autograder, key errors, or other more fundamental errors. These errors should all be visible when running the test cell, or in the grade report. Simply put, you should rely on the traceback shown after the test cell, then review test case variables if needed.

Quick walkthrough video

Accessing Variables

Each dict has one key per input or output (e.g., a function with 3 outputs produces 3 keys). Print the entire variable to see everything, or access a specific key to narrow it down.

Quick access:

print(returned_output_vars)
print(true_output_vars)

Access specific key

1) Print out all keys for the var (may only be one)

print(returned_output_vars.keys())

2) Use the key name you want to see

print(returned_output_vars[{key_name}])

Comparing Variables

Copy/paste is disabled during exams, so use short variable names and keep comparison code brief. The approach depends on the data type.

my_out = returned_output_vars[{key_name}]
true_out = true_output_vars[{key_name}]

Lists/Sets

len(my_out), len(true_out)

my_out == true_out

set(my_out) - set(true_out)  # items in yours but not expected

Dictionaries

my_out.keys() == true_out.keys()

[k for k in my_out if my_out[k] != true_out[k]]  # differing keys

DataFrames

my_out.shape, true_out.shape

my_out.equals(true_out)

my_out.compare(true_out)  # shows differences

Numbers/Floats

abs(my_out - true_out) < 1e-6

Tips

• Use code, not visual inspection, to compare variables — differences like data types are not always obvious.
• Consider defining a helper function once at the start of your exam and reusing it throughout:

def qc():
    r = returned_output_vars
    t = true_output_vars
    print("Match:", r==t)
    print("Mine:", r)
    print("True:", t)
    return r, t

More information

This page should have covered the relevant information for the test case variables. There is more content located within your LMS, which contains two long-form videos involving debugging with test case variables. There is also a mandatory quiz there that you should have completed already.

That can be found here:

• GT: Canvas > Modules > Supplemental Topic: How Student Code is Tested on Exams and Homework Notebooks
• EdX: Module 0: Fundamentals > Supplemental Topic: How Student Code is Tested on Exams and Homework Notebooks

Relevant bootcamp notebook links:

• Bootcamp notebook - Colab data debugging and test case variable
• bootcamp notebook - Data troubleshooting example

FAQ

The test case variables look like they're showing me data from a different exercise?

The test case variables persist until they're overwritten. Since the test case variables have the same name for each exercise (but may have different keys), the variables will hold values until you run the relevant test cell. That means if you are skipping around and forget to run the test cell that generates the test case variables you're looking at, you may be looking at ones previously generated. This has happened to people before.