This lab works on phones, but writing code is much easier on a laptop.
LLM · Evaluation, Inference and Reliability · LM9-L2 · Intermediate
A model that scores 92% on one phrasing of a question and 55% on another does not have an accuracy of 92%. A good evaluation says which prompts were used, applies criteria that are written down, lists what failed, and shows how much the result depends on the wording.
You will see how reporting choices change the headline, write the harness itself, and find the prompt-sensitive items.
You have passed every stage. Now connect it back to the theory and move on:
Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().
Stage 1 shows how wordings and aggregates change a reported accuracy. Stage 2 writes a small deterministic harness with three kinds of criteria. Stage 3 finds prompt-sensitive items. The model outputs and outcomes are invented; no language model is run. Your code is graded on hidden data.
Educational simplification: five or a few dozen items, three criterion types, a handful of wordings, outcomes as 0/1. A real harness also handles many metrics, model calls, retries and logging, but the structure is the same. All data in this lab are invented.
Get notified when new robotics labs are released. One email per new lab, nothing else.