Category: WebApp

In today’s cloud-native world, building resilient systems is more important than ever. As teams adopt microservices and rely on various downstream dependencies, it becomes crucial to handle failures gracefully. One common challenge when working with Azure Functions is how to temporarily pause message processing. In this post, we’ll explore how to implement a circuit breaker pattern that allows you to programmatically disable and re-enable function triggers, helping your system recover smoothly without losing data or overwhelming dependent services.

If you’re using Azure Functions at scale, you’ve likely encountered scenarios where you need to temporarily pause processing:

• A critical downstream service is experiencing an outage
• You’ve hit API rate limits with an external service
• You’re performing maintenance on dependent systems
• You need to recover from an incident without processing new events

Manually disabling function triggers through the Azure Portal is cumbersome and doesn’t scale well. Automating this process provides more reliability and helps prevent cascading failures across your architecture.

The Solution

The solution is based on an Azure Function App called called FuncTriggerManager which provides a circuit breaker pattern implementation for other Azure Function Apps. It allows you to programmatically enable or disable function triggers.

Let’s walk through the flow at a high level:

1. Function Triggered by Message
   An Azure Function is triggered by a message from a Service Bus queue or topic.
2. Processing and API Call
   The function processes the message and attempts to forward it to a downstream API.
3. Failure Detected
   If the API responds with an HTTP 503 (Service Unavailable), the function logs the error and sends a control message to an Azure Storage Queue.
4. Trigger Manager Disables Function
   A separate Function App (FuncTriggerManager) listens to this queue, disables the affected Function App, and places another message on the queue with a scheduled delay.
5. Function Re-enabled
   After the delay, the manager function re-enables the original Function App, allowing message processing to resume once the downstream service is likely to have recovered.

Key Features

• Queue-based control: Send simple messages to enable/disable specific functions
• Automatic re-enablement: Set a time period after which functions automatically resume
• Minimal permissions: Uses managed identity with least privilege access
• Simple integration: Minimal code changes needed in your existing functions

How it works

The solution consists of an Azure Function with a queue trigger that:

1. Monitors a designated storage queue for control messages
2. Parses instructions about which function to enable/disable
3. Uses Azure Resource Manager APIs to modify the function’s settings
4. If configured, automatically re-enables functions after a specified period

Here’s what the control message looks like:

{
  "FunctionAppName": "YourFunctionAppName",
  "FunctionName": "SpecificFunctionName",
  "RessourceGroupName": "YourResourceGroupName", 
  "DisableFunction": true,
  "DisablePeriodMinutes": 30
}

When this message is processed, the specified function will be disabled for 30 minutes, then automatically re-enabled. If you set DisablePeriodMinutes to 0 or omit it, the function will remain disabled until explicitly enabled.

Full source code of the solution can be found here https://github.com/connectedcircuits/funcappshortcct

Implementation Details

Under the hood, FuncTriggerManager uses the Azure Management APIs to modify a special application setting called AzureWebJobs.<FunctionName>.Disabled that controls whether a specific function trigger is enabled. This is the same mechanism used by the Azure Portal when you manually disable a function.

The FuncTriggerManager service authenticates using its managed identity with the following permissions applied to the Function App you wish to control.

• Reader access to the resource group of function apps to control
• Website Contributor permissions on target function apps to control

Testing with a Sample Function App

I have included a sample Function App so you can follow how to send a control message onto the storage queue. This sample Function App is triggered when a message is placed onto a Service Bus queue. It then sends the message to a HTTP endpoint which represents an API. If it returns a status code of 5xx a control message is placed onto the Storage Account queue to disable its trigger.

The test function simulates a real-world scenario where you might need to temporarily disable message processing triggered messages from a Service Bus queue.

The source code for sample simulator can be found here – https://github.com/connectedcircuits/FuncSbProxy-funcappshortcct-

Getting Started

To implement this pattern in your environment:

1. Deploy the FuncTriggerManager function app with managed identity
2. Set up the required app settings:
   • StorageConnection: Connection string to your Azure Storage
   • QueueName: Name of the queue to monitor
   • AzureSubscriptionId: Your Azure subscription ID
3. Develop your Function App service and copy the DisableFuncMessenger.cs (as in the sample FuncSbProxy)
4. Setup the required app setting your Function App service
   • StorageConnection: Connection string to your Azure Storage
   • QueueName: Name of the queue to monitor
   • ResourceGroupName: Name of the Azure Resource where this function app will be deployed to.
   • DisableFuncPeriodMin: Time in minutes you want disable the trigger for.

Real-World Use Cases

Here are some scenarios where I’ve found this pattern particularly useful:

Scenario 1: Downstream Service Outage

When an external API reports a 503 Service Unavailable error, your error handling code can place a message in the circuit breaker queue to disable the corresponding function until the service recovers.

Scenario 2: Rate Limit Protection

If your function processes data that interacts with rate-limited APIs, you can temporarily disable processing when approaching limits and schedule re-enablement when your quota resets.

Conclusion

The circuit breaker pattern is essential for building resilient systems, and having an automated way to implement it for Azure Functions has been invaluable in our production environments. FuncTriggerManager provides a simple yet powerful approach to managing function trigger states programmatically.

By incorporating this pattern into your Azure architecture, you can improve resilience, reduce cascading failures, and gain more control over your serverless workflows during incidents and maintenance windows.

---

Note: Remember to secure your circuit breaker queue appropriately as it provides control over your function apps’ operation.

Enjoy…