Skip to content

Circuit Breaker Pattern: Build More Resilient Apps

Technology

May 10, 2021 - 3 minute read

1654 Blog Circuit Breaker Pattern 416X300
Bartlomiej Paletta Software Developer

An experienced back-end developer specialising in .NET, MS SQL and the implementation of Microsoft-based solutions. He’s passionate about developing cloud-based solutions and discovering interesting design patterns. Privately, he enjoys hiking and motorcycling.

See all Bartlomiej's posts

2988 HC Digital Transformation 476X381

Share

The Problem

Services often collaborate with each other when handling requests. This is the case especially for microservices which are distributed in nature. When one service synchronously invokes another (an external service such as payment APIs, warehouse automation, etc.) there’s always the possibility that the other service will be unavailable or under heavy load. It’s common to use the retry pattern for unsuccessful calls, but it can be dangerous for the system. It will try to call the third-party service again and again using resources such as threads while waiting for the response. As a result, this slows down your system and might lead to resource exhaustion, making the calling service unable to handle other requests. The failure of one service can potentially cause a cascade to other services throughout the application.

The Solution

The circuit breaker pattern prevents an application from performing operations that are likely to fail, doesn’t allow failures to cascade to other services and helps you create resilient systems. It acts as a proxy and monitors the number of recent failures, then uses this information to decide whether to allow the operation to proceed or return an exception immediately. In a nutshell, it works like an electrical circuit in a house: when it’s closed, the device works, when it’s broken (e.g., by a switch), the device stops working.

The Circuit Breaker Pattern States

The circuit breaker can be implemented as a state machine with the following states:

  • Closed — The operating state. The request to flow through the specified route works and doesn’t cause any system failures. When the number of recent failures exceeds a specified threshold, it causes a circuit break and the closed state changes into the open one. At this point the proxy starts a timeout timer, and when it expires, the proxy is placed into the Half-Open state.

  • Open — When an application enters this state, the incoming requests fail immediately for a specified timeout.

  • Half-Open — The breaker passes a specified number of requests to test the status of the resource. The half-open state determines whether the circuit returns to the closed or the open state.

The most common thresholds when implementing the circuit breaker pattern are connection timeouts, error spikes, incorrect status codes, and unexpected response types.

Implementation

It’s quite complicated to implement a circuit breaker from scratch and it would take quite a while to do it correctly. There are many libraries that allow developers to use circuit breaker policies, for example Polly for .NET or Resilience4j for Java. Here is a simple example of using a circuit breaker pattern with the Polly library. It defines a circuit that opens after 3 exceptions and closes again after 1 minute.

circuit breaker pattern example

Application runs the method 4 times, the fourth time you get an open circuit exception:

circuit breaker pattern exception

Simply wait one minute before the fourth call:

circuit breaker pattern exception

As expected, the circuit will be closed again, and the method will be executed for the fourth time:

circuit breaker pattern exception

Conclusion

The circuit breaker pattern is often used together with the retry pattern. Both patterns, properly implemented and configured, are powerful tools for application resilience policies. The circuit breaker pattern helps the system prevent sending unnecessary loads to a failed service. It’s potentially easy to implement using the existing libraries, however, it may be challenging to select the appropriate thresholds without creating false alarms or introducing excessive latency. There are ways to achieve this, for example by implementing a detailed logging mechanism where the circuit breaker pattern is used. By analysing the information in the logs, the thresholds can be adjusted to the correct level. Sometimes this operation requires long observation and multiple adjustments, but once you find the right thresholds, you can benefit from the implemented pattern and protect your application from an overload.

2988 HC Digital Transformation 476X381
Bartlomiej Paletta Software Developer

An experienced back-end developer specialising in .NET, MS SQL and the implementation of Microsoft-based solutions. He’s passionate about developing cloud-based solutions and discovering interesting design patterns. Privately, he enjoys hiking and motorcycling.

See all Bartlomiej's posts

Related posts

You might be also interested in

Contact

Start your project with Objectivity

CTA Pattern - Contact - Middle