Sensor selection: robotic manipulators

1. Engineering problem

Robotic manipulator: the sensor set determines what the robot can perceive, how it localises and how it fails. Choosing it from the failure cases backwards is cheaper than discovering them in the field.

2. When is this needed?

• Starting a new robotic manipulator design.
• Replacing or adding a sensor after a field failure.
• Reviewing a sensor architecture before purchase.

3. Selection and design decisions

Typical combinations

• Joint encoders + wrist force/torque: Compliant interaction and contact detection.
• Add RGB-D camera (and tactile fingertips): Object pose for grasping plus contact feedback.

Sensor types that do not fit this application

• Event camera: Not typical for this application.
• Thermal (long-wave infrared) camera: Not typical for this application.
• 2D LiDAR (scanning): Not typical for this application.
• 3D LiDAR (mechanical, spinning): Not typical for this application.
• Non-spinning LiDAR (flash, MEMS, optical phased array, hybrid): Not typical for this application.
• FMCW LiDAR: Not typical for this application.
• GNSS (multi-constellation): GNSS is not available indoors.
• RTK GNSS: GNSS is not available indoors.
• GNSS/INS (integrated unit): GNSS is not available indoors.
• Barometer / pressure sensor: Not typical for this application.
• Magnetometer: Not typical for this application.
• Radar (mmWave and automotive-class): Not typical for this application.
• Ultrasonic range sensor: Not typical for this application.
• UWB (ultra-wideband) ranging: Not typical for this application.
• Sonar (underwater acoustic): Underwater acoustic sensor; not for operation in air.

Try your own requirements in the Sensor Selection Calculator; read the per-sensor detail in the family guides for cameras, LiDAR, inertial and positioning and other sensors.

4. Advantages and disadvantages

• RGB-D (active depth) camera: Dense depth on texture-less indoor surfaces; but short range, mainly indoors.
• Wheel / joint encoder: Precise, high-rate proprioception; basis of odometry and joint control; but odometry drifts from wheel slip and uncertain wheel radius/track.
• Force/torque sensor: Direct contact-force measurement for compliant and safe interaction; but expensive and fragile against overload.

5. Requirements and constraints

• Range, accuracy and rate requirements written as numbers before shortlisting.
• Mass, power and compute budgets (see the power and compute guides).
• Environment: lighting, weather, surfaces (glass, reflective, dark).
• Interfaces and ROS 2 driver availability for your distribution.
• Cost and supply for the quantities you need.

6. Architecture

RGB-D (active depth) camera  ·  Wheel / joint encoder  ·  Force/torque sensor
   │ time-synchronised, calibrated extrinsics
   ▼
state estimation / SLAM  ──▶  planning and control

7. Implementation

Integration complexity: Medium: hand-eye calibration and force sensing need care, and the sensors are close to people and objects.

1. Select the core sensors and write down what each cannot see.
2. Add the common sensors that cover those gaps.
3. Define time synchronisation and calibration for every sensor pair that is fused (Modules 6 and 7).
4. Define the redundancy and failure behaviour (below), then build and validate sensor by sensor.

Redundancy: Encoders on motor and output side, software force limits plus a safety-rated force or speed limiting function where people share the workspace, and recalibration checks on a fixed target at every start.

8. Real examples

A pick-and-place arm: joint encoders, a wrist force/torque sensor for contact detection, an RGB-D camera for object pose and tactile fingertips for grasp verification.

9. Failure scenarios

• Hand-eye calibration drifting after a knock
• Force/torque drift or overload damage
• Depth cameras failing on shiny, dark or transparent objects
• Encoder zero offsets after maintenance
• Cable strain at the wrist

10. Troubleshooting

1. Reproduce the failure and record all sensor streams (ROS bag).
2. Check which sensor lost data or went wrong first (rates, stamps, covariance).
3. Compare against the failure list above and the per-sensor failure scenarios in the family guides.
4. Add the missing modality or monitor, then repeat the test.

11. Validation

• Each listed failure condition is provoked deliberately (glass, sun, GNSS loss, slip) and the robot behaves as specified.
• Each sensor’s data rate, stamps and extrinsics are verified on the integrated robot.
• Redundancy is tested by unplugging each sensor in turn.
• Results are saved with the configuration.

12. Engineering checklist

• Failure cases listed before sensors chosen
• Core sensors cover the main task; common sensors cover their blind spots
• Datasheet specifications checked at operating conditions
• Interfaces, power and ROS 2 drivers checked
• Time sync and calibration planned for every fused pair
• Redundancy and failure behaviour defined and tested
• Safety-rated functions identified separately from navigation sensing

13. Further references

• Class-level guidance written from general engineering knowledge; it contains no product specifications. Product-specific numbers must come from the manufacturer’s datasheet for the exact model and operating mode.
• ROS 2 message types: sensor_msgs (jazzy branch).

Related on FixTheRobot

• Complementary Strengths and Weaknesses of Cameras and LiDAR Perception · Lesson