Exam Test Case Variables

Overview

When you run a test cell during an exam, the autograder creates variables that you can use to debug your solution. These variables let you inspect the actual inputs and expected outputs for test cases.

This page is a quick reference

For detailed videos and additional grading information, see the supplemental course content:

GT: Canvas > Modules > Supplemental Topic: How Student Code is Tested on Exams and Homework Notebooks
EdX: Module 0: Fundamentals > Supplemental Topic: How Student Code is Tested on Exams and Homework Notebooks

Quick walkthrough video

How to Access Test Case Variables

Variable	Purpose
`input_vars`	The inputs passed to your function
`original_input_vars`	A copy of inputs before your function ran
`returned_output_vars`	What your function returned
`true_output_vars`	The expected (correct) output

Typical Workflow

Run solution/demo cell -> Confirm your output and the printed demo look the same
Run test cell
If it doesn't pass, access test case variables and identify differences, then use them to debug

Variables persist until they're overwritten

Since the test case variables have the same name for each exercise (but may have different keys), the variables will hold values until you run the relevant test cell. That means if you are skipping around and forget to run the test cell that generates the test case variables you're looking at, you may be looking at ones previously generated. This has happened to people before.

Accessing Variables

Each variable is a dict containing the inputs or outputs. Each of these dict variables will have as many keys as there are inputs, or outputs. For example if the exercises requires writing a function to generate 3 outputs, then the returned output vars will contain 3 keys. You can simply print the variables to see all of the information, but if you need to look at a specific part, or run out of screen real estate, you may need to print or display just that key.

Quick access:

print(returned_output_vars)
print(true_output_vars)

Access specific key

1) Print out all keys for the var (may only be one)

print(returned_output_vars.keys())

2) Use key name you want to see

print(returned_output_vars[{key_name}])

Comparing Variables

Comparison approach depends on data type. Keep code short since copy/paste is disabled.

Shorten variable names to save time and typing effort, since you won't be able to copy and paste.

my_out = returned_output_vars[{key_name}]
true_out = true_output_vars[{key_name}]

Or even shorter, there's nothing wrong with using even shorter variable names, as long as you can keep track of them (and don't overlap with other names).

Lists/Sets

len(my_out), len(true_out)

my_out == true_out

set(my_out) - set(true_out)  # items in yours but not expected

Dictionaries

my_out.keys() == true_out.keys()

[k for k in my_out if my_out[k] != true_out[k]]  # differing keys

DataFrames

my_out.shape, true_out.shape

my_out.equals(true_out)

my_out.compare(true_out)  # shows differences

Numbers/Floats

abs(my_out - true_out) < 1e-6

Tips

Test case variables let you programmatically compare data. While a visual inspection may work if the error is obvious, you should plan on using code to debug your solution. Things like data types, for example, may not be obvious.
Since copy/paste is disabled, you will want to be efficient with your test code, using shortened, quick versions of things. You can also create functions to help you compare.

For example, a quick compare function may save you a few seconds here and there, at the cost of a few seconds at the start. You can save this code and type it from your notes at the start of the exam if you wish. By defining this in your notebook and running it once, you can then run it again at your leisure

def qc():
    r = returned_output_vars
    t = true_output_vars
    print("Match:", r==t)
    print("Mine:", r)
    print("True:", t)
    return m, t

Remember, the same way the function will persist in memory once it's run, so will the test case vars. So if you're jumping around, ensure you just ran the the most recent test cell you want to debug. ===> If things look of, quickly run that function code and your qc or equivalent function again, you may be looking at data from a different exercise.