FixTheRobot Explore Robotics / LLM / LM6 / LM6-L1

This lab works on phones, but writing code is much easier on a laptop.

LLM · Fine-Tuning for Classification · LM6-L1 · Beginner

Classification Metrics and Failure Cases

A classifier that says “no hazard” every time is right 92% of the time on a dataset where 8% of the messages are hazards. Accuracy alone would rank it as excellent.

You will pick a threshold with that trap in mind, write the metrics that look at the rare class, and then hunt for the other common reason that a good score is not trustworthy: a feature that works only by accident.

Your task

    Simulation
    solution.pyLoading Python…
    Ctrl+Enter runs
    What Python can I use?

    Plain Python with math (and slamkit for small matrix helpers: matmul, transpose, solve, inverse, identity). Images, point clouds and detections arrive as lists inside the dictionary D; an image is a list of rows, each row a list of pixels. There is no NumPy or OpenCV: you write the algorithm yourself, which is the point of the lab. Print with print().

    Hints

    What this lab teaches

    Stage 1 explores the threshold of a score-based classifier on imbalanced data. Stage 2 computes the confusion matrix and the four main scores. Stage 3 finds a shortcut feature by comparing training and validation behaviour. The scores and examples are generated numbers; no language model is used. Your code is graded on hidden data.

    Theory for this lab

    Fine-Tuning for Classification labs

    1. LM6-L1: Classification Metrics and Failure Cases: this lab.
    2. LM6-L2: Fine-Tune a Language Model for Classification

    Educational simplification: binary labels, one score per message, a handful of binary features, small tables of examples. Fine-tuned language models are evaluated in the same way, with the same metrics, on text examples. All data in this lab are invented.

    Get the next lab

    Get notified when new robotics labs are released. One email per new lab, nothing else.