This lab works on phones, but writing code is much easier on a laptop.
LLM · Evaluation, Inference and Reliability · LM9-L1 · Intermediate
Checkpoint B scores three points higher than checkpoint A. Is it better, or did it get lucky on this test set? The answer depends on how many items you used, whether you compared them in pairs, and whether you decided those things before looking.
You will watch the p-value wander as items are added, write a paired bootstrap, and compare perplexities with an uncertainty.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 compares the paired and unpaired tests as the number of items grows. Stage 2 writes a paired bootstrap interval. Stage 3 compares two perplexities with a standard error. The per-item results and log-probabilities are generated numbers; no language model is evaluated. Your code is graded on hidden results.
Educational simplification: binary correct/incorrect outcomes from a random process, 50 pre-drawn bootstrap resamples (real analyses use thousands), and a normal approximation for the interval on perplexity differences. All data in this lab are invented.
Get notified when new robotics labs are released. One email per new lab, nothing else.