FAULT MODEL
Start with a specific claim about what can fail.
“Test robustness” is too vague. Define the component, failure mode, duration, timing, and expected system response. A sensor can fail silent, freeze at its last value, report a plausible bias, become noisy, lag badly, or produce impossible values. Those failures exercise different detection logic.
Map each injected fault to a requirement or architectural assumption. The experiment should answer whether the system behaves as intended under that defined condition, not merely whether engineers can make it unhappy.
SAFETY BOUNDARY
Controlled failure requires controlled energy.
Fault injection can create unexpected motion, heat, current, data loss, or unsafe states. Use current-limited supplies, physical restraints, disabled high-energy actuators, simulation, hardware-in-the-loop, sacrificial fixtures, protective barriers, and emergency stops as appropriate.
Do not begin by shorting random rails on a fully energized prototype because the phrase “fault injection” sounded scientific. Inject the smallest condition that represents the failure mechanism while preserving a safe test environment.
POWER
Reset timing deserves more than one clean power cycle.
Cut input power at different phases of flash writes, firmware updates, boot, radio transmit, motor starts, and storage commits. Vary ramp rate and source impedance. Introduce brief sags that cross peripheral thresholds but may or may not trigger the MCU supervisor.
The objective is to expose partial-state failures: one rail dead, another alive; flash interrupted; peripheral wedged; configuration half-committed; charger source switching at the wrong moment.
BUSES
Make the wire misbehave in the way the architecture claims it can survive.
Hold I²C SDA or SCL low through a controlled fixture. Disconnect CAN termination. Introduce a node that stops acknowledging. Disable an SPI peripheral mid-transaction. Delay responses beyond the normal timeout.
Observe whether the driver identifies the actual failure class, whether retries are bounded, whether unrelated devices remain usable, and whether recovery returns the bus to a known state without a full system reboot.
SENSORS
Plausible wrong values are more dangerous than obvious nonsense.
Open-circuit and out-of-range faults are easy to detect. A sensor frozen at yesterday's plausible value, drifting slowly, lagging, or correlated with another bad sensor can defeat naive range checks.
Inject bias, delay, scale error, noise, freezes, dropouts, and physically inconsistent combinations. Verify that fusion, control, and health monitoring distinguish uncertainty from certainty rather than simply averaging the bad input into the answer.
STORAGE
Corrupt metadata, not just data.
Flip CRC bits, damage generation counters, erase part of a journal, duplicate valid records, truncate a firmware manifest, and interrupt writes at different boundaries. Recovery logic should select the newest coherent state or deliberately fall back rather than trusting whichever structure is easiest to parse.
Preserve the corrupted artifact when possible. A reproducible bad storage image is more useful than a note saying “flash got weird.”
THERMAL + COOLING
Kill the fan before the thermal limit does it for you.
Disconnect fan tach, stop airflow, block an inlet, alter ambient temperature, or force a high-power duty cycle within safe lab limits. Verify the system detects reduced cooling, derates before emergency shutdown, records the event, and recovers with sensible hysteresis.
Do not require literal component overheating to test logic that should respond before damage becomes relevant.
NETWORKS
Loss, delay, duplication, and reordering are separate faults.
Network test tools can add latency, jitter, packet loss, bandwidth limits, duplication, and reordering. Wireless tests can attenuate signal or create controlled interference in appropriate shielded or lawful test conditions.
Watch what the application does before the connection fully disappears. Queues may grow, retries may amplify congestion, stale commands may arrive late, and operator displays may continue saying “connected” long after control quality is unacceptable.
FIRMWARE UPDATE
Test the candidate that should never be accepted.
Use corrupted images, wrong hardware targets, invalid signatures, older blocked versions, candidates that crash before confirmation, and configuration migrations that fail intentionally.
The update mechanism should reject unauthorized or incoherent images, survive interruption, preserve a known-good path, record why the candidate failed, and avoid infinite rollback loops.
OBSERVABILITY
The system should explain its degraded state.
A successful fault-injection test produces evidence: fault detected, affected component isolated, capability reduced, operator informed, recovery attempted, and final state recorded.
If engineers can see the injected fault on laboratory instruments but the product telemetry remains cheerful and blank, the resilience architecture is incomplete even if the machine stayed running.
AUTOMATION
Repeat failures until timing luck stops protecting you.
Manual testing finds obvious issues. Automated fault campaigns expose timing windows. A programmable supply can interrupt power at different delays. A relay or switch fixture can disconnect a bus or sensor. Software can corrupt files or add network impairment. Hardware-in-the-loop systems can inject impossible and borderline sensor states reproducibly.
Record the fault seed, timing, firmware version, configuration, expected behavior, and observed outcome so a failure can be replayed after the fix.
PASS CRITERIA
“Did not crash” is not enough.
Define pass conditions before the experiment: detection latency, maximum unsafe command duration, preserved capability, retry ceiling, alarm visibility, log completeness, recovery time, data integrity, and whether human intervention is required.
A system can remain powered while doing the wrong thing. Resilience is mission behavior under fault, not merely continued CPU execution.
BOTTOM LINE
Failure is a test input.
Fault injection converts resilience from design language into observed behavior. It also exposes the gaps between component protection, firmware recovery, operator awareness, and actual mission survival.
If a recovery path matters, make it perform on command.
SOURCE TRAIL
Technical starting points.
NASA — Fault Management Handbook
NASA Technical Reports Server — fault management and fault injection literature