STATE
Robots usually control things they cannot measure directly.
A mobile robot may care about position, velocity, heading, terrain slope, wheel slip, contact, map alignment, obstacle motion, battery state, and actuator health. Only some of those quantities are directly sensed, and even direct measurements are imperfect.
State estimation turns measurements and models into a best current estimate of what the system is doing. Sensor fusion is one way to improve that estimate by combining information with different strengths and failure modes.
An inertial sensor can update rapidly but drifts. GNSS can provide an absolute position reference but may be noisy, blocked, reflected, jammed, or delayed. Wheel odometry can be smooth until the wheels slip. Vision can provide rich geometry until lighting, texture, motion blur, or occlusion become hostile. Lidar provides range structure but has its own reflective, geometric, weather, and timing limitations.
The estimator’s job is not to vote. It is to reason about what each observation says, how uncertain it is, how the system should have evolved, and whether the pieces remain mutually consistent.
UNCERTAINTY
A number without uncertainty is half a measurement.
If one sensor reports position as 10.0 m and another reports 10.4 m, the arithmetic mean is 10.2 m. That answer is only sensible if the sensors deserve equal weight, have comparable error distributions, are properly calibrated, refer to the same time and coordinate frame, and are not sharing a hidden common bias.
Estimation systems therefore carry uncertainty explicitly. In Gaussian-style estimators that often appears as a covariance matrix. Covariance does more than assign a vague confidence level. It describes uncertainty in each state component and how errors in different components are related.
If uncertainty is understated, the estimator trusts itself too much and may reject useful measurements. If uncertainty is overstated, the system may wander or respond sluggishly because it cannot commit to the state it already knows fairly well.
Calibration is therefore not a one-time act of making readings “look right.” It is part of the statistical contract between sensors and estimator.
CORRELATION
Two sensors are not independent just because they have two part numbers.
Fusion algorithms often benefit most when sensors fail differently. If two measurements share a common error source, combining them can produce false confidence.
Two cameras mounted on the same flexible mast may both inherit the same vibration. Two navigation receivers may share the same multipath environment. Several software “sensors” may all depend on the same underlying camera frame. Two temperature probes can share the same bad calibration procedure. Multiple localization algorithms may all consume the same incorrect map.
If the estimator treats correlated errors as independent, repeated evidence can be counted multiple times. The covariance shrinks as though several independent witnesses agreed, even though the witnesses all copied the same memo.
System diagrams should therefore include information ancestry, not just physical components. Ask where each estimate ultimately comes from.
TIME
Measurements must describe the same world before they can describe it together.
A lidar scan, camera frame, wheel encoder sample, and inertial update may arrive at different rates and different delays. If they are fused according to arrival time instead of measurement time, a moving robot can combine observations of different physical states as though they were simultaneous.
This is especially damaging during rapid rotation or acceleration. A few tens of milliseconds can convert a valid geometric relationship into apparent disagreement.
Good systems preserve acquisition timestamps and transform measurements into a consistent temporal frame. Some estimators maintain a history and apply delayed measurements at the correct historical state before propagating forward again. Others predict measurements to a common reference time.
Whatever the architecture, “latest message” is not the same thing as “measurement of now.”
FRAMES
Coordinate mistakes can impersonate sensor disagreement.
A robot may use body, world, map, odometry, sensor, tool, camera, inertial, and joint coordinate frames simultaneously. Every transformation has direction, units, orientation conventions, and timestamps.
A 5 cm translation error in a camera-to-body transform can create apparent motion as the robot rotates. A sign convention error can flip yaw. Mixing degrees and radians can ruin an otherwise excellent estimator with almost comedic efficiency. A stale transform can make a perfect sensor look noisy.
Before tuning statistical parameters to “fix” disagreement, verify frame definitions and transforms. Many estimation problems are geometry problems wearing probability makeup.
OBSERVABILITY
Some states cannot be inferred from the available measurements.
An estimator cannot estimate information the system never reveals.
Observability asks whether internal state can be reconstructed from available measurements over time. The exact mathematics depend on the system, but the engineering intuition is useful everywhere.
A stationary inertial system may have poor information about some biases. A monocular camera may not determine absolute scale without additional constraints. Wheel encoders cannot distinguish perfect rolling from certain forms of slip without outside evidence. A robot moving through a featureless corridor can lose geometric constraints that were abundant in a textured room.
When a state becomes weakly observable, estimator uncertainty should grow. If the software continues reporting a tight confident estimate anyway, the estimator is not demonstrating intelligence. It is hiding ignorance.
BIAS
Noise averages down. Bias does not politely disappear.
Random zero-mean noise can often be reduced by averaging or repeated observations. Bias is different. A sensor that is consistently offset can make a long sequence of measurements converge very precisely on the wrong value.
Bias can come from calibration error, temperature, mounting stress, scale-factor error, magnetic disturbance, optical distortion, clock offset, environmental effects, or model mismatch. Some estimators include bias terms as states so the system can infer and track them. Others require separate calibration or compensation models.
Whenever a fused estimate becomes strangely precise while remaining wrong in a repeatable direction, suspect an unmodeled bias or common-mode error before adding more sensors.
OUTLIERS
A measurement can be validly formatted and still deserve rejection.
Real sensors occasionally return observations that do not match the assumed noise model. GNSS jumps. vision matches the wrong feature. lidar catches a reflective anomaly. encoders miss counts. magnetic sensors encounter a new field source. cameras saturate.
Robust estimators use consistency checks, gating, residual analysis, or alternative loss functions to reduce the damage from outliers. But aggressive rejection has its own danger: a filter can become so committed to its current model that it rejects the first evidence that reality has changed.
Innovation or residual signals are therefore valuable diagnostics. They expose the difference between what the model predicted and what the sensor observed. Persistent structure in residuals often indicates model error, calibration problems, timing errors, or a sensor that has entered a new regime.
FAILURE DETECTION
Fusion should make disagreement visible, not bury it.
A polished fused output can hide the most operationally useful information: the sensors are fighting.
Systems should expose individual measurement health, residuals, rejected observations, uncertainty growth, time since last accepted update, calibration state, and which sensors are currently contributing to each state estimate.
If GNSS disappears, the operator should know the estimator is coasting on inertial and odometry. If visual localization loses features, that state should be explicit. If uncertainty is growing, the planner or controller may need to reduce speed, increase clearance, seek a better observation geometry, or enter a degraded mode.
A fused position without fusion health is a dashboard number pretending to be an instrument.
AI PERCEPTION
Machine-learned perception does not abolish measurement science.
A neural detector can act as a sensor in the larger system: it consumes observations and produces inferred quantities such as classes, locations, masks, depth, or landmarks. Its outputs still have failure distributions, domain dependence, latency, correlation, and uncertainty.
Adding a learned detector to a probabilistic estimator does not magically make the detector calibrated. A softmax score is not automatically a physical probability. Model errors may be strongly correlated across frames or environments. Training data can create systematic blind spots rather than random noise.
The same discipline still applies: characterize performance, preserve timestamps, understand inputs, measure failure conditions, expose uncertainty where possible, and keep independent checks for claims that matter.
FIELD METHOD
How to debug a fused estimate.
1. Plot raw sensors separately. Do not begin with the fused output.
2. Preserve timestamps and latency. Verify measurement time rather than message arrival time.
3. Check coordinate transforms. Confirm units, axes, handedness, mounting geometry, and transform age.
4. Plot residuals. Look for bias, trends, state-dependent error, and changes after environmental transitions.
5. Disable sensors one at a time. A useful ablation reveals which sensor actually constrains each state.
6. Compare claimed uncertainty with observed error. An estimator that says “±2 cm” while missing by 40 cm is miscalibrated even if the trajectory looks smooth.
7. Test correlated failures. Remove the shared reference, disturb the common mount, change the lighting, alter the map, or otherwise attack the assumptions sensors share.
8. Make degradation operational. Let uncertainty affect speed, planning, control authority, or fail-safe state instead of merely coloring a dashboard icon.
BOTTOM LINE
Fusion is disciplined disagreement.
The value of multiple sensors is not that they agree. It is that their different information and different failure modes can constrain one another when the estimator understands those differences.
More sensors do not guarantee more truth. Better models of uncertainty can produce better estimates.
A robot should know not only where it thinks it is, but how much evidence supports that belief and what would make the belief fail.
SOURCE TRAIL
Measurement and estimation references.
NIST — Quantifying Uncertainty Towards Information-Centric Unmanned Navigation
NIST — Tolerances and Uncertainty in Robotic Systems
NIST — Perception Performance of Robotic Systems
MIT OpenCourseWare — State Estimation