The basic idea behind the circuit breaker is very simple. You wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all. Usually you'll also want some kind of monitor alert if the circuit breaker trips.
Martin Fowler
Use Your Reset Button To Change Your Vibration
What if a network resource is not available? Or a call to a remote service fails for whatever reason? Should you keep on firing requests to the service or resource and maybe run out of critical resources yourself, with a potential of cascading failures across multiple systems?
Surprisingly a Gang of Fortunate Thinkers are constantly refining and inventing new patterns to overcome real-world software catastrophes like these. The Circuit Breaker pattern is described in Michael T. Nygard's book, Release It! Design and Deploy Production-Ready Software.
The Circuit Breaker
A circuit breaker acts as a proxy for operations that might fail. The proxy can be implemented as a state machine with the states closed, open, half-open. In the diagram the Client receives two timeout exceptions. After x number of timeouts, the Circuit Breaker trips to the open state. Any subsequent requests from the Client will immediately result in a circuit-open-exception.
Any request for the resource will result in some sort of circuit-open-exception as long as the timeout is running. Once the timeout is reached, the Circuit Breaker enters the half-open state and one request is processed and sent to the Resource. If this request succeeds the Circuit Breaker enters the open state and all operations are processed as normal.
Circuit Breaker States
Closed
This is the normal state on a healthy system, operations are executed without issues. If an operation throws an exception, the failure count is incremented (+ logged, monitored, ...). If the failure count exceeds the threshold, the circuit breaker trips into the open state. If a call succeeds before the threshold is reached, the failure count is reset.
Open
All calls to the operation will fail immediately and throw an exception. A timeout is started when the circuit breaker trips. Once the timeout is reached, the circuit breaker enters a half-open state.
Half-Open
The circuit breaker allows one operation to execute. If this operation fails, the circuit breaker re-enters the open state and the timeout is reset. If the operation succeeds, the circuit breaker enters the closed state and resumes the normal operation.
Considerations
Multi-threading
It is likely that a pattern like this would be used in a multi-threaded environment, therefore it should be able to function properly when multiple threads are executing operations. Updating the failure count variable should lock the variable in an atomic operation, so other threads wanting to update the failure count must wait until it is released.
More Info
You can find more about implementation considerations on Tim Ross's blog here.
Inspiration
Code Examples
NuGet Package
Related Design Patterns
Retry Pattern
Health Endpoint Monitoring Pattern
State Machine PatternMicroservices ChassisAPI GatewayServer-side Discovery
Health Endpoint Monitoring Pattern
State Machine PatternMicroservices ChassisAPI GatewayServer-side Discovery
