THE WRONG TEST
“Does it stay on?” is not the same as “does it remain useful?”
A system can remain powered while losing the capability that made it valuable. A robot may stay online after localization collapses. An aircraft computer may continue executing while an actuator is unavailable. A server may answer health checks while its dependency graph is partially dead. A vehicle may retain propulsion but lose reliable sensing.
Graceful degradation asks a stronger question: after a fault, can the system move into a known reduced-capability state that preserves the highest-priority functions without creating a new hazard?
That requires more than redundancy. It requires failure detection, isolation, capability modeling, fallback behavior, operator communication, and a recovery path.
PRIORITIES
You cannot preserve everything, so decide what matters before the fault.
Graceful degradation begins with a hierarchy of function.
For a mobile robot, preventing uncontrolled motion may outrank mission progress. For a spacecraft, attitude control and thermal survival may outrank science collection. For a web platform, preserving read-only access may outrank accepting new writes. For a manufacturing cell, safe stop and state preservation may outrank production throughput.
These priorities should not be invented during the failure. They belong in requirements and hazard analysis. A degraded mode is coherent only if the system knows which capabilities may be sacrificed to preserve more important ones.
DETECTION
The system has to know something changed.
Fault tolerance begins with evidence. Sensors disagree. heartbeat messages stop. checksums fail. actuator response diverges from command. temperatures rise. timing budgets are missed. control residuals grow. storage errors appear. a dependency returns invalid data.
Detection should be designed around observable failure symptoms, not merely the failures engineers remembered to name.
There is always a tradeoff between sensitivity and false alarms. A detector that trips on every transient creates nuisance degradation and trains operators to ignore it. A detector that waits for absolute certainty may identify the problem after the useful recovery window has closed.
Good systems preserve the evidence that triggered the state change so later analysis can distinguish a real fault from an overactive detector.
ISOLATION
Knowing that “something is wrong” is not enough.
If the system cannot determine what has failed, it may disable healthy resources or continue trusting the damaged one.
Fault isolation narrows the problem to a component, channel, function, or confidence class. That might mean identifying one failed wheel encoder, one bad network path, one stuck control surface, one corrupted data source, one overheating compute module, or one unavailable backend service.
Isolation does not always need perfect diagnosis. A robot may only need to know that global localization is unreliable, even if it cannot identify whether the root cause is GNSS multipath, visual failure, or map corruption. The degraded behavior can be tied to the lost capability rather than the exact failed part.
CAPABILITY MODEL
Components fail. Missions lose capabilities.
Hardware-centric thinking says “motor three failed.” Operational thinking asks what that means for the system.
Can the vehicle still translate? Can it rotate? At what acceleration? Can it stop safely? Can it hold position? Can it return home? Can it still sense obstacles? Can it still communicate? Can it still write state safely?
A capability model maps component health into what actions remain feasible. This is especially important in systems with redundancy because loss of one element may reduce the reachable performance envelope without eliminating a function entirely.
Degraded mode should be based on the remaining envelope, not a binary alive/dead flag.
RECONFIGURATION
Use what remains differently.
Some resilient systems can reconfigure around a fault. A flight-control system may redistribute command to remaining control surfaces. A robot may switch localization sources. A server may route around a failed region. A power system may shed noncritical loads. A storage system may become read-only to avoid further corruption.
Reconfiguration needs explicit assumptions. A backup channel is useful only if it is sufficiently independent from the failed path. Two redundant components powered by the same failed regulator are not two independent resources. Two services in different containers on the same dead host are not availability architecture.
Fault containment boundaries should therefore be visible in system design: power, compute, software process, network, clock, sensor, mechanical, thermal, and operator dependencies.
DEGRADED MODES
Name them like real operating states.
“Fallback” is too vague for serious operations.
A system might define:
NOMINAL — full sensing, full control authority, normal mission envelope.
DEGRADED-NAV — global position unavailable; speed and route complexity reduced; local obstacle avoidance retained.
DEGRADED-ACTUATION — reduced control authority; acceleration and maneuver envelope restricted.
READ-ONLY — state can be served but not safely modified.
SAFE-HOLD — stop mission progress, maintain stabilization and essential environmental control.
RETURN / ABORT — remaining capability is sufficient for a controlled exit but not continued mission work.
SAFE-SHUTDOWN — no useful controlled operating envelope remains.
Explicit states make testing possible. They also make operator interfaces much less ambiguous than a yellow triangle labeled “warning.”
HUMAN FACTORS
The operator needs to know what the machine has stopped promising.
A degraded system that silently changes behavior can be more dangerous than a system that stops.
Operators should see which capability is lost, which functions remain, what limits changed, what automatic actions occurred, what evidence triggered the transition, and what recovery actions are allowed.
If control gains changed, maximum speed fell, autonomous planning was disabled, or localization became relative-only, those are operational facts. The interface should not continue presenting the nominal experience while the machine is living under a different contract.
NASA research on graceful degradation in complex air-traffic systems also highlights that degradation can arise from interactions among technology, environment, and human operators. Resilience is not merely a component property.
FAIL-SAFE VS FAIL-OPERATIONAL
Sometimes stopping is success. Sometimes stopping is the hazard.
Different functions require different failure philosophies.
A machine tool may be safest if power is removed and motion stops. An aircraft cannot simply turn off flight control. A life-support system may need continued operation under multiple failures. A database may be safer refusing writes while continuing reads. An autonomous vehicle may need enough control to reach a safe stopping location rather than stopping instantly in the current lane.
Fail-safe means the failure drives the system toward a safe state. Fail-operational means the system must continue providing some function after failure. Many real systems need both depending on the subsystem and phase of operation.
The design should state the intended behavior explicitly instead of using “fault tolerant” as a comforting adjective.
COMMON-MODE FAILURE
Redundancy that shares the same failure is inventory, not resilience.
Duplicate sensors can share a power rail. redundant computers can share software defects. backup networks can share a switch. replicated services can share credentials, configuration, or a bad deployment. multiple navigation algorithms can share the same corrupted map.
Graceful degradation depends on independence where independence matters.
Architecture reviews should ask what faults can remove several supposedly redundant channels at once. Power, timing, software, environmental exposure, physical location, data ancestry, and maintenance procedures all create common-mode coupling.
RECOVERY
Returning to nominal is its own transition.
A fault clearing does not automatically mean the system should resume full capability.
Data may be stale. controller state may have drifted. caches may need rebuilding. actuators may need recalibration. localization may need reinitialization. queued work may no longer be valid. the original fault may have been intermittent.
Recovery should therefore have criteria. Some systems require manual acknowledgement. Others demand a period of healthy observation. Some need a complete restart or re-homing sequence.
A robust state machine treats recovery as deliberately as failure entry.
TESTING
Fault modes belong in the lab, not only in the incident report.
Pull the sensor. delay the network. cap the actuator. kill the process. corrupt the message. drop the power rail in simulation. force the estimator uncertainty high. remove a storage dependency. overheat the compute module in a controlled test environment. disconnect the redundant link.
Then observe:
Was the fault detected? Was the correct capability marked unavailable? Did the system transition into the expected degraded state? Did it preserve the higher-priority function? Was the operator informed? Were logs sufficient to reconstruct why? Could the system recover safely?
Fault injection turns resilience from a claim into an experiment.
FIELD METHOD
A practical degradation design review.
1. List mission functions by priority. Distinguish essential, useful, and expendable capabilities.
2. Map dependencies. Power, compute, network, sensors, actuators, clocks, software services, people, and external data.
3. Define detectable symptoms. Avoid relying only on one component to report its own failure.
4. Define degraded states. Give each state an explicit capability envelope and exit criteria.
5. Expose state to operators and higher-level software. Planning must know when the machine has fewer options.
6. Test common-mode faults. Remove shared infrastructure, not only leaf components.
7. Test transitions in both directions. Failure entry and recovery can each create hazards.
8. Preserve evidence. Fault state, trigger, health telemetry, timestamps, commands, and recovery actions belong in the incident record.
BOTTOM LINE
Resilience is controlled loss.
No engineered system has infinite redundancy, infinite power, infinite control authority, or infinite bandwidth. Failure eventually removes something.
Graceful degradation means losing capability without losing the system’s understanding of what it can still safely do.
The architecture should know what remains, operate inside that smaller envelope, and make the change visible.
SOURCE TRAIL
Fault tolerance and graceful degradation references.
NASA NTRS — Fault Propagation and Fault Containment in Aerospace Systems
NASA NTRS — Restructurable Aircraft Control with Graceful Performance Degradation
NASA NTRS — Fault Tolerant Control Laws
NASA NTRS — Designing Graceful Degradation into Complex Systems