I typically come across solutions that incorporate a service bus where monitoring of the dead-letter queues are simply ignored. Upon questioning the reason, the normal excuse for not monitoring the DLQ (dead-letter queue) is quote “I don’t require it because I have added exception handling therefore no messages will end up on the DLQ”. From experience there will be scenarios where the exception logic did not capture an edge case scenario and the message will end up onto the DLQ without anyone knowing about.
A simple solution is to have a single process to monitor all the DQL messages and raise an alert when one occurs. Below is one of my building blocks which a typically incorporate when there is a service bus involved in a solution.
To use the DLQ building block to centrally capture any DLQ message, simply set the “ForwardDeadLetteredMessagesTo” property on each of the messaging queues to a common DLQ handler queue as shown below.
Now when a message gets dead lettered, it will end up in this common queue which is monitored by an Azure Function. Note the current NuGet version v3.0.4 of the ServiceBus DLL has the DeadLetterSource property and is not currently available in the Logic App Service Bus connector. The function writes the DLQ meta data, any custom properties and the message payload to a blobstore file. By using an Azure Storage Account V2 for the blobstore, a new blob creation event will be fired to any interested subscribers which in this case is a Logic App.
Azure Function Code
Below is the code for the Azure Function. Here I am using the IBinder interface to allow me to set the folder path and file name imperatively. The connection strings (ServiceBusConn, StorageAccountConn) are defined in the Application settings of the Azure App Service.
- using Microsoft.Azure.ServiceBus;
- using Microsoft.Azure.WebJobs;
- using Microsoft.Extensions.Logging;
- using Newtonsoft.Json;
- using System;
- using System.Collections.Generic;
- using System.IO;
- using System.Threading.Tasks;
- namespace ServiceBusDLQMonitoring
- {
- public static class Function1
- {
- [FunctionName("Function1")]
- public static async Task RunAsync([ServiceBusTrigger("dlq-processor", Connection = "ServiceBusConn")] Message dlqQueue,
- Binder blobBinder,
- ILogger log)
- {
- log.LogInformation($"C# ServiceBus queue trigger function processed message: {dlqQueue.MessageId}");
- //set the filename and blob path
- var blobFile = Guid.NewGuid().ToString() + ".json";
- var path = string.Concat("dlq/",DateTime.UtcNow.ToString("yyyy/MM/dd"),"/" ,dlqQueue.SystemProperties.DeadLetterSource,"/", blobFile);
- var dlqMsq = new DLQMessage
- {
- DeadletterSource = dlqQueue.SystemProperties.DeadLetterSource,
- MessageId = dlqQueue.MessageId,
- SequenceNumber = dlqQueue.SystemProperties.SequenceNumber,
- SessionId = dlqQueue.SessionId,
- UserProperties = dlqQueue.UserProperties,
- DeadLetterReason = dlqQueue.UserProperties["DeadLetterReason"].ToString(),
- EnqueuedDttmUTC = dlqQueue.SystemProperties.EnqueuedTimeUtc,
- ContentType = dlqQueue.ContentType,
- DeliveryCount = dlqQueue.SystemProperties.DeliveryCount,
- Label = dlqQueue.Label,
- MsgBase64Encoded = Convert.ToBase64String(dlqQueue.Body, Base64FormattingOptions.None)
- };
- var blobAttributes = new Attribute[]
- {
- new BlobAttribute(path),
- new StorageAccountAttribute("StorageAccountConn")
- };
- using (var writer = await blobBinder.BindAsync<TextWriter>(blobAttributes))
- {
- writer.Write(JsonConvert.SerializeObject(dlqMsq,Formatting.Indented));
- }
- }
- }
- public class DLQMessage
- {
- public string DeadletterSource { get; set; }
- public long SequenceNumber { get; set; }
- public string MessageId { get; set; }
- public string SessionId { get; set; }
- public string Label { get; set; }
- public string DeadLetterReason { get; set; }
- public DateTime EnqueuedDttmUTC { get; set; }
- public int DeliveryCount { get; set; }
- public string ContentType { get; set; }
- public IDictionary<string,object> UserProperties { get; set; }
- public string MsgBase64Encoded { get; set; }
- }
- }
Logic App Implementation
A Logic App is used to the retrieve the message from the blob store when it is triggered by the EventGrid HTTP webhook. The basic workflow is shown below and can be expanded to suit your own requirements.
The expression for the ‘Get blob content using path’ action is @{replace(triggerBody()[0][‘data’][‘url’],’https://dqlmessages.blob.core.windows.net/’,”)}. Here I am just replacing the Domain name with an empty string as I only want the resource location.
The Parse JSON action has the following schema. This makes it easier to reference the properties downstream.
{
"properties": {
"ContentType": {
"type": [
"string",
"null"
]
},
"DeadLetterReason": {
"type": "string"
},
"DeadletterSource": {
"type": "string"
},
"DeliveryCount": {
"type": "integer"
},
"EnqueuedDttmUTC": {
"type": "string"
},
"Label": {
"type": [
"string",
"null"
]
},
"MessageId": {
"type": "string"
},
"MsgBase64Encoded": {
"type": [
"string",
"null"
]
},
"SequenceNumber": {
"type": "integer"
},
"SessionId": {
"type": [
"string",
"null"
]
},
"UserProperties": {
"type": "any"
}
},
"type": "object"
}
The last action ‘Set variable MsgBody’ has the value set to: “@{base64ToString(body(‘Parse_JSON’)?[‘MsgBase64Encoded’])}”
Blob creation event configuration
Next is to setup a subscription to the Blob creation event. Click on Events under the Storage account for the DLQ messages as shown below.
Then click on the +Event Subscription to add a new subscription.
Setup the subscription Basic details with a suitable subscription name and the blob storage account resource name. Uncheck the ‘Subscribe to all event types’ and select Blob Created event. Set the endpoint details to Web Hook and the URL of the Logic App Http trigger endpoint address.
Under Filters, enable subject filtering. Add the following prefix ‘/blobServices/default/containers/’ to the name of the DLQ container (in this example its called ‘dlq’) and add it to the ‘Subject Begins With’ textbox. Then in the ‘Subject Ends With’, set it to the filename extension ‘.json’. Now click the Create button at the bottom of the page to create the subscription.
Sample Logic App output
Once everything is wired up and if there are any messages that have been placed onto the DLQ, you should see some logic app runs. Below is an example of the outputs from the last two actions.
All you need to do now is extend the Logic App to deliver the DLQ alert to somewhere.
Enjoy…
Really good pattern. We are looking into a similar scenario and will be looking to implementation a similar setup to create tickets upon dead letter messages.
Thanks for the kind comments. Reach out if you need any help.
Thanks a ton!