Robot Learning · Theory · M7

M7-T2 — System Identification

If the shape of the physics is known, learning a model is two numbers and a linear solve. What matters is whether the data can tell the two numbers apart.

Lesson 2 of 6
Module 7: Model-Based Robot Learning
Theory
About 15 minutes to read

System identification is the old, careful branch of model learning: choose a model structure from physics, then estimate its parameters from input-output data (Ljung 1999). For Scout with momentum, the structure is a velocity update with two unknowns, a drive gain and a drag. This lesson derives the least-squares estimate, solves it by hand for four transitions, then measures with real experiments how the estimate improves with data, how it suffers from noise, why a log of constant commands cannot separate the two parameters, and why a simulator’s nominal numbers may not describe the floor.

Learning objectives

After this lesson, you should be able to:

Write the identification problem for v′ = v + dt (gain · a − drag · v) as a linear regression and derive the normal equations.
Compute the estimate by hand from four transitions, and say how far such a small data set can be trusted.
Predict, and measure over seeds, how the error of the estimates scales with the number of transitions and the noise level.
Judge whether data are informative: read a condition number or confidence interval, and name the pitfalls of regression (poor excitation, noisy regressors).

The intuition

You can find out how a shopping trolley behaves with two experiments: push it a bit and see how fast it speeds up (the gain), and let it roll and see how fast it slows down (the drag). One experiment alone does not do it. If you only ever push at the same steady strength, the trolley soon reaches a speed where push and friction cancel, and all you learn is the ratio of the two. A pair of unknowns needs data in which the push and the speed vary independently.

Least squares is the arithmetic that does both experiments at once on logged data: it finds the two numbers that make the predicted change of velocity closest, in the sense of squared error, to the change that was logged. And since every transition is noisy, it is also an estimate with an error, one that shrinks as data accumulate. The figure shows that shrinkage for the drag of Scout’s floor model: at 0.05 noise, 10 transitions give a spread of 0.157 in the drag, 200 give 0.029, and 800 give 0.015. The curves fall roughly like the inverse square root of the data, so each halving of the error costs four times the data.

Spread (standard deviation over 100 data sets) of the least-squares drag estimate against the number of transitions, for three noise levels, on the Scout floor dynamics (true drag 3.4). Real runs.

The mathematics

From the dynamics to a regression

The velocity update on each axis is v_t+1 = v_t + dt (gain · a_t − drag · v_t) + noise · w_t, and the position follows exactly as x_t+1 = x_t + dt · v_t+1 (positions need no fitting if velocity is measured). Move v_t to the left and the model is linear in the unknowns:

regression form

y_{t}=v_{t+1}-v_{t}=\alpha \cdot v_{t}+\beta \cdot a_{t}+e_{t}

\alpha =-\mathrm{dt}\cdot \text{drag}

\beta =\mathrm{dt}\cdot \text{gain}

e_{t}=\text{noise}\cdot w_{t}

\phi _{t}=[v_{t},\,a_{t}]

\theta =[\alpha ,\,\beta ]

y_{t}=\phi _{t}\cdot \theta +e_{t}

Each transition on each axis gives one row; with N transitions of a two-axis robot there are 2N rows (both axes share the same two parameters, so they are stacked). Least squares chooses θ to minimise the sum of squared residuals Σ (y_t − φ_tθ)².

The normal equations

Setting the derivative of the sum of squares to zero gives the normal equations. With Φ the matrix whose rows are φ_t and y the vector of targets:

normal equations

\sum _{t}(y_{t}-\phi _{t}\,\theta )^{2}\;\mathrm{is\ minimised\ where}

\frac{\partial }{\partial \theta }=-2\,\sum _{t}\,\phi _{t}^{\top}(y_{t}-\phi _{t}\,\theta )=0

(\Phi ^{\top}\,\Phi )\,\theta =\Phi ^{\top}\,y

[\sum \,v^{2}\,\Sigma \,v\,a][\alpha ]

[\sum \,v\,y]

[\sum \,v\,a\,\Sigma \,a^{2}][\beta ]=[\sum \,a\,y]

\text{drag}=-\frac{\alpha }{\mathrm{dt}}

\text{gain}=\frac{\beta }{\mathrm{dt}}

RLCore.ridgeFit(X, Y, 0, false) solves exactly this

For two unknowns the solution has a closed form: with D = Σv² · Σa² − (Σva)², α = (Σvy · Σa² − Σva · Σay) / D and β = (Σv² · Σay − Σva · Σvy) / D. The determinant D is the thing to watch: it is zero when the v column is proportional to the a column, and then the problem has no unique answer.

How good is the estimate? Noise, bias and confidence

If the model form is right and the errors have zero mean and variance σ², the estimate is consistent (it converges to the truth as data grow) and its covariance is approximately the expression below. The regressor v_t depends on earlier noise, so a very small data set can be slightly biased (3.375 at 10 transitions against a truth of 3.4 in the experiment below), but the bias is not visible in the printed digits from 100 transitions up.

covariance of the estimate

\operatorname{Cov} (\hat{\theta})=\sigma ^{2}(\Phi ^{\top}\,\Phi )^{-1}

\hat{\sigma}^{2}=\frac{\mathrm{RSS}}{n-2}

so the standard errors fall as 1/√n, because ΦᵀΦ grows like n, and rise with the noise σ: the pattern of the figure. The same matrix shows whether the data are informative: an ill-conditioned ΦᵀΦ (a large ratio of its largest to smallest eigenvalue, the condition number) has a large inverse, hence a wide confidence interval. This is the idea of persistent excitation in the identification literature (Ljung 1999): the inputs must keep exciting every mode of the system that you want to estimate. For this two-parameter problem, what is needed is that v and a are not nearly proportional to each other.

Pitfalls of regression

Regression pitfalls
Pitfall	What goes wrong	Check
Poor excitation	v and a nearly proportional: individual parameters are poorly determined, though a combination may be fine	condition number of ΦᵀΦ, confidence intervals, correlation of the two columns
Noise in the regressors (errors in variables)	the measured v_t has sensor noise, which is both in the regressor and, with the opposite sign, in the target v_t+1 − v_t; ordinary least squares then pushes the drag estimate up	compare with an experiment with a better sensor, or repeat measurements
Wrong structure	the residuals are not noise: they depend on v, on a, or on time	plot residuals against the regressors; compare the residual spread with the noise you expect
Data from a narrow region	the fit says nothing about commands or speeds never used	validate on held-out data from where the plan will go (lesson M7-T3)

A worked example

All numbers are from Node runs against RLCore (dynStep with RLCore.DYN.floor: gain 2.2, drag 3.4, dt 0.25). The 100-data-set figures use the seeds 1 to 100; the data are random commands uniform in [−1, 1] on each axis with a reset to a random start every 20 steps, unless stated.

1. Four transitions, by hand. One axis, Scout starts at rest, noise 0.05, commands 1, 1, −1, 0.5. The logged (v_t, a_t, v_t+1), rounded to three places, are (0, 1, 0.598), (0.598, 1, 0.637), (0.637, −1, −0.389) and (−0.389, 0.5, 0.135), so the targets y = v_t+1 − v_t are 0.598, 0.039, −1.026 and 0.524. The sums are

sums for the four transitions

\sum \,v^{2}=0+0.3576+0.4058+0.1513=0.914694

\sum \,v\,a=0+0.598-0.637-0.1945=-0.2335

\sum \,a^{2}=1+1+1+0.25=3.25

\sum \,v\,y=0+0.0233-0.6536-0.2038=-0.834076

\sum \,a\,y=0.598+0.039+1.026+0.262=1.925

D=0.914694\cdot 3.25-(-0.2335)^{2}=2.918233

solution

\alpha =\frac{-0.834076\cdot 3.25-(-0.2335)(1.925)}{D}=-\frac{2.261259}{2.918233}=-0.77487

\beta =\frac{0.914694\cdot 1.925-(-0.2335)(-0.834076)}{D}=\frac{1.566072}{2.918233}=0.53664

\text{drag}=\frac{0.77487}{0.25}=3.099

\text{gain}=\frac{0.53664}{0.25}=2.147

The truth is drag 3.4 and gain 2.2: the drag is 9% low, the gain 2.4% low. That is not a mistake in the arithmetic (I checked it in Node); it is what four noisy transitions give. Repeating the same four-command experiment for 200 different noise draws gave drag 3.42 ± 0.21 (mean ± standard deviation) and gain 2.20 ± 0.11, so errors like this one are normal.

2. More data, less error. Table of the mean and standard deviation of the estimates over 100 data sets (true drag 3.4, gain 2.2):

drag / gain: mean ± standard deviation over 100 data sets
Noise	N = 10	N = 50	N = 200	N = 800
0.03	3.385 ± 0.096 / 2.196 ± 0.048	3.396 ± 0.036 / 2.200 ± 0.024	3.400 ± 0.017 / 2.200 ± 0.010	3.401 ± 0.009 / 2.200 ± 0.005
0.05	3.375 ± 0.157 / 2.193 ± 0.080	3.393 ± 0.060 / 2.200 ± 0.041	3.400 ± 0.029 / 2.200 ± 0.017	3.401 ± 0.015 / 2.200 ± 0.009
0.08	3.360 ± 0.245 / 2.190 ± 0.128	3.388 ± 0.093 / 2.200 ± 0.065	3.400 ± 0.045 / 2.200 ± 0.028	3.402 ± 0.023 / 2.200 ± 0.014

Two readings. The means sit on the truth from 50 transitions up (the bias is negligible from 100 transitions up and slightly low below that), so the error is almost all spread. And the spread scales as predicted: from 200 to 800 transitions (four times the data) the standard deviation of the drag at noise 0.05 goes from 0.029 to 0.015, about half; and at N = 200, going from noise 0.03 to 0.08 raises it from 0.017 to 0.045, about the ratio of the noises (2.7). This is the evidence behind the 1 to 2% recovery from 200 transitions in lesson M7-T1.

3. Data that cannot separate the parameters. Same noise (0.05), 200 transitions, but different command logs. The 95% interval is 1.96 standard errors from σ̂² (ΦᵀΦ)⁻¹ divided by dt; "condition" is the ratio of the eigenvalues of ΦᵀΦ; "corr" is the correlation between the v and a columns.

Same number of transitions, different excitation
Command log (200 transitions, noise 0.05)	drag	gain	condition	corr(v, a)
random, seed 1	3.41 ± 0.07	2.20 ± 0.04	3.3	0.06
steps of 10 samples between the levels 0.8, −0.8, 0.3, −0.3	3.38 ± 0.09	2.16 ± 0.05	12.7	0.82
constant command (1, 0.5), reset every 20 steps, seed 1	3.51 ± 0.16	2.28 ± 0.10	79.8	0.97
the same, seeds 2 and 3	3.31 ± 0.17 and 3.42 ± 0.16	2.15 ± 0.11 and 2.20 ± 0.10	77.3 and 81.1	0.97
constant 0.8, one run of 200 steps, no reset	3.36 ± 0.31	2.18 ± 0.20	347.6	0.99

The estimates are still centred near the truth, but the interval widens by 2 to 4.5 times as the log loses variety. Over 100 data sets, the standard deviation of the drag estimate was 0.029 for random commands and 0.078 for the constant-command log with resets, and for the gain 0.017 and 0.049. One combination, however, stayed well determined: gain / drag, the steady-state speed per unit command (true 0.6471), had standard deviation 0.0072 with random commands and 0.0040 with the constant ones. A constant push teaches Scout’s top speed very well and says very little about how that speed splits into drive and drag. A planner that only ever drives at constant speed might not notice; one that must brake, or accelerate from rest, will. I did not run a perfectly steady-state log, where the columns are exactly proportional and the problem is singular.

4. Noise in the measured velocity. I added Gaussian sensor noise to the logged velocity (400 transitions, process noise 0.03, 100 data sets; each time step is measured once and the same measurement serves as the end of one transition and the start of the next). Estimated drag: 3.400 at sensor noise 0, 3.403 at 0.02, 3.416 at 0.05 and 3.457 at 0.1 m/s; gain 2.200, 2.200, 2.200, 2.201. The drag is biased upward, and the bias does not shrink with more data, unlike the spread. The direction agrees with the algebra (sensor noise enters the regressor and, negatively, the target), but the size is only an experiment on this setup: 1.7% at 0.1 m/s here. I did not test other noise structures such as correlated or delayed measurements.

5. Nominal against floor. The simulator’s nominal parameters (gain 3.0, drag 3.0) imply a top speed of 1 m/s; the floor’s (2.2, 3.4) give 0.647 m/s. Scoring the logged velocity changes with each parameter set, the root-mean-square one-step residual per axis on three random-command logs was 0.129, 0.131 and 0.124 for the nominal parameters and 0.052, 0.053 and 0.047 for the parameters fitted to that same log. The fitted residual is about the process noise of 0.05, which is as small as it can get; the nominal one is two and a half times that, and so not noise. This is the simulator-to-floor mismatch in numbers, and the reason to identify the floor rather than trust the simulator (lesson M10-T3 treats calibrating against randomising). What it costs a 12-step prediction is lesson M7-T3.

What this simulation simplifies. The model form is exactly right here, the noise is Gaussian and white, and the velocity is observed (with at most independent noise). On a real robot there are delays, saturation, friction that depends on speed in more than one way, and a velocity that is itself estimated; the logs would then show structure in the residuals, and the numbers above would not carry over.

Engineering implications and limitations

A procedure for identifying a robot model

Choose the structure from physics. Mass and drag give a first-order velocity response; a motor with lag gives a second-order one. Start with the smallest structure that could explain the logs.
Design the experiment. Excite what you will use: steps and ramps in the commands, speeds from rest to top speed, both directions. Random commands are the baseline; the constant log of example 3 is the thing to avoid. Reset to rest often enough that the transients are in the data.
Fit by least squares and read the diagnostics. Report the parameters with intervals, the condition number of ΦᵀΦ, and the residual spread.
Validate on a log you did not fit, with different commands, by predicting the velocity (one step) and the position (twelve steps) (lesson M7-T3). Plot residuals against v and a.
Re-identify when the robot changes: a payload, worn wheels, a different floor. Lab M7-L1 (fit the dynamics) practises the first three steps with a log from the floor.

Practical notes

Solve the normal equations with a stable method for larger problems; for two unknowns the closed form is fine, but a QR or an SVD is safer when the columns are nearly proportional. I used the Gaussian elimination of RLCore.solve, which gave sensible estimates at the conditions seen here (up to about 350); I did not study its numerical limits.
Scale the regressors before comparing condition numbers. The condition number depends on the units of v and a; it is a relative warning, and confidence intervals are the unit-free check.
Fit the discrete update that matches the simulator’s integration order. Position follows the new velocity here; an explicit-Euler ordering that uses the old one gives a different model (a classic mistake in lab M7-L1).
Measure what you will compare: velocity noise, the process noise (the residual spread) and the latency between command and response. A value of 0.05 as the residual spread is a statement about the model; a larger one on held-out data is a warning.

Where this meets learning

System identification is supervised learning with a strong prior: the model form. When the form is right it needs very few data and extrapolates; when it is wrong, the residuals will say so, and a flexible model (lesson M7-T3) can take over. It is also the first step of calibrating a simulator to the floor before training a policy in it (module 10). As in all supervised learning, the data distribution is the thing to guard (lesson M2-T5 on generalisation, and lesson M3-T3 on distribution shift).

Common misconceptions

“The fitted parameters are the true parameters.”

They are an estimate with a spread: four transitions gave drag 3.10 for a truth of 3.4, and even 200 noisy ones give ±0.03 (one standard deviation). Report the interval.

“More data always fixes it.”

More data shrinks the spread, not a bias. The sensor-noise bias in the drag (3.457 at 0.1 m/s) did not shrink with data, and a log without excitation widens the interval at any size.

“Constant commands are enough to calibrate a robot.”

They fix the top speed (gain / drag, standard deviation 0.0040) but split it into gain and drag poorly (the drag spread was 2.7 times larger than with random commands at the same size).

“The simulator’s numbers are close enough.”

The nominal parameters left a one-step residual of about 0.13, 2.5 times the noise floor of 0.05, because the floor is slower and heavier than the simulator thinks.

Engineering takeaways

A model with known structure is a linear regression: y = v′ − v = α v + β a, with α = −dt · drag and β = dt · gain. The normal equations (ΦᵀΦ) θ = Φᵀ y give the estimate.
The estimate is unbiased with covariance σ² (ΦᵀΦ)⁻¹: the spread falls like 1/√n and rises with the noise (0.029 at 200 transitions and noise 0.05, 0.015 at 800).
Data must be informative. Constant commands give a condition number of 80 to 350 and 2 to 4.5 times wider intervals; only the ratio gain / drag is well determined.
Noise in the measured velocity biases the drag upward (+0.057 at 0.1 m/s in this setup), and that bias does not go away with more data.
The simulator’s nominal parameters leave a residual 2.5 times the noise floor: identify the floor, and check the residuals.

References and further reading

Ljung’s book is the standard reference for system identification, including persistent excitation and the covariance of the least-squares estimate; Sutton and Barto place model learning inside reinforcement learning.

L. Ljung, System Identification: Theory for the User, 2nd edition, Prentice Hall, 1999.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, MIT Press, 2018, chapter 8.
M. P. Deisenroth and C. E. Rasmussen, “PILCO: A Model-Based and Data-Efficient Approach to Policy Search”, Proceedings of the 28th International Conference on Machine Learning (ICML), 2011.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

Write the two normal equations for α and β, in terms of the sums Σv², Σva, Σa², Σvy and Σay.

Explanation

Σv² · α + Σva · β = Σvy and Σva · α + Σa² · β = Σay. Then drag = −α / dt and gain = β / dt.

Question 2

A log has Scout driven at a constant command of 0.8 for 200 steps. Why are its individual estimates of gain and drag uncertain although the data are plentiful, and what would you do?

Explanation

v rises to a steady value proportional to the command, so the v and a columns are nearly proportional (correlation 0.99, condition number about 350): the matrix ΦᵀΦ is nearly singular and its inverse is large. Drive varied commands (steps or random), starting from rest, and re-fit.

Question 3

You double the data from 200 to 400 transitions. By about how much do you expect the standard deviation of the drag estimate to fall, and what will not change?

Explanation

It falls by about 1/√2, i.e. to 0.7 of its value (0.029 to 0.021 in the experiment). A bias from sensor noise in the velocity, or from a wrong model form, does not change.

What’s next

Next: lesson M7-T3 asks how wrong a model is after twelve steps rather than one, why the answer depends on the dynamics, what a black-box network does outside its data, and how that connects to the distribution shift of lesson M3-T3. Lab M7-L1 fits the floor’s dynamics from a log.

Practice this lesson

Fit the Dynamics

Scout's simulator and the real floor disagree: fit friction and drive gain from a log, and check the prediction on new commands.

Next lesson

M7-T3 — Learned Dynamics and Model Error

Train a one-step model and show how its error compounds over a prediction horizon.

Nothing below is required to finish this lesson.

Before
M7-T1: Why Learn a Model?
Compare model-free and model-based learning by sample efficiency, accuracy and failure modes.
Lab
M7-L1: Fit the Dynamics
Scout's simulator and the real floor disagree: fit friction and drive gain from a log, and check the prediction on new commands.

Learning objectives

The intuition

The mathematics

From the dynamics to a regression

The normal equations

How good is the estimate? Noise, bias and confidence

Pitfalls of regression

A worked example

Engineering implications and limitations

A procedure for identifying a robot model

Practical notes

Where this meets learning

Common misconceptions

“The fitted parameters are the true parameters.”

“More data always fixes it.”

“Constant commands are enough to calibrate a robot.”

“The simulator’s numbers are close enough.”

Engineering takeaways

References and further reading

Knowledge check

Question 1

Question 2

Question 3

What’s next

Fit the Dynamics

M7-T3 — Learned Dynamics and Model Error

Related content

Get notified when the next lesson and new labs are released.