Providing DR By Using An Azure Service Bus In Two Regions

This is a Disaster Recovery option I used on one of my projects. A client required a DR solution which guaranteed not to lose any messages during the DR failover process. In essence, no messages were allowed to be lost in mid flight whilst the DR process was taking place.

Solution

The approach I took was to use an Azure Service Bus  in a primary Azure region and another service bus in a secondary Azure  DR region. The publisher would then send the same message to both service bus endpoints. For my solution I used Azure APIM to post the same message to both primary and secondary regions, thereby simplifying the code for the publisher. 

image

The primary region would simply process the messages as per normal using a Logic App to poll for new messages on the service bus. However in the DR region,  the equivalent Logic App would be set in a disabled state. The messages in the secondary region would remain on the bus until the Time-To-Live (TTL) threshold is reached before either being moved onto the Deadletter queue, or dropped from the queue altogether by setting the EnableDeadLetteringOnMessageExpiration property to false. Using this technique provides automatic pruning of the older messages that would have definitely been processed by the primary region before failing over.

The value chosen for the TTL property is determined by how long it would take to failover to the DR region and the time to detect an issue being realised in the primary region.

Failing over would simply involve enable the Logic App the DR region and failing back would involve disabling this Logic App again. 

Pros

  • No messages are lost during the failover process.
  • Low monetary transaction cost by the DR environment as no Logic Apps are being triggered during the normal process flow through the primary region.
  • Simplistic DR design which involves just another another queue.
  • Simple failover process.

Cons

  • There is a delay before any new messages would be processed while  the older messages on the service bus are reprocessed first.
  • The backend system processing the messages must be idempotent, meaning the same message maybe processed by the backend system multiple times and produce the same outcome.
  • Requires the publisher to send the same message to 2 different endpoints. This may be mitigated by using APIM to manage sending the messages to both service bus endpoints.

Enjoy…

APIM backup & restore using Azure Automation Services

In this post I will describe the steps of using PowerShell scripts to backup APIM and using the Automation service to schedule the backup every month. The restore function also allows you to restore  APIM into another resource group or APIM service. For the project I am working on now, this is what I am doing to move the configuration settings between each environment.

First you need to create a blob store which ideally should be Read-Access geo-redundant storage (RA-GRS). This is where the APIM backups will be stored. After the blob store has been provisioned, create a container for the backup file as shown below.

image

Once the container is created, take note of the Storage account name and Access key for the blob store. These values will be used in the PowerShell script later.

image

Next provision an Azure Automation service and ensure the Create Azure Run As account is set to “yes”.

image

Once it has been provisioned, ensure the modules have been updated by clicking on the “Modules” link on the left hand navigation panel and then “Update Azure Modules”. Note this does take a while to complete.

image

After the update has been completed, click the “Browse gallery” link and in search textbox type “apim”. Once found,  double click on the row to open the import blade.

image

Now click the Import icon to import the cmdlet. This can take several minutes to import.

image

After the PowerShell module has been imported, create a new Runbook and ensure the type has been set to “PowerShell”. Then click the Create button at the bottom of the page.

image

This will open up a new blade where we can add and test the PowerShell script to backup the APIM settings.

image

Now add the following script below into the text editor and remember to update the variables with your environment settings. Once you have added the script, click the “Save” button and then the “Test pane” button to ensure the script runs successfully.

   1: Disable-AzureDataCollection

   2: Write-Output "Starting backup of APIM..."

   3:

   4: # sign in non-interactively using the service principal

   5: $connectionName = "AzureRunAsConnection";

   6: $storageAccountName = "apimstorebackup";

   7: $storageAccountKey = "<storage account key>";

   8: $resourceGroupName = "APIMService";

   9: $apimName = "apimmanager";

  10: $targetContainerName = "backup";

  11: $targetBlobName "AzureAPIM.apimbackup"

  12: try

  13: {

  14:     # Get the connection "AzureRunAsConnection "

  15:     $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName

  16:

  17:     Write-Output "Logging in to Azure..."

  18:     Add-AzureRmAccount `

  19:         -ServicePrincipal `

  20:         -TenantId $servicePrincipalConnection.TenantId `

  21:         -ApplicationId $servicePrincipalConnection.ApplicationId `

  22:         -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint

  23: }

  24: catch {

  25:     if (!$servicePrincipalConnection)

  26:     {

  27:         $ErrorMessage = "Connection $connectionName not found."

  28:         throw $ErrorMessage

  29:     } else{

  30:         Write-Error -Message $_.Exception

  31:         throw $_.Exception

  32:     }

  33: }

  34:

  35: $sourceContext = (New-AzureStorageContext -StorageAccountName $storageAccountName  -StorageAccountKey $storageAccountKey);

  36:

  37: Write-Output "Starting backup of APIM instance";

  38: Backup-AzureRmApiManagement `

  39:             -ResourceGroupName $resourceGroupName `

  40:             -Name $apimName `

  41:             -StorageContext $sourceContext `

  42:             -TargetContainerName $targetContainerName `

  43:             -TargetBlobName $targetBlobName;

  44:

  45: Write-Output "Backup of APIM completed.";

Here are the description of the variables:

  • $connectionName = “AzureRunAsConnection” – this is the default connection account that was created when the Automation service was provisioned.
  • $storageAccountName = “apimstorebackup” – name of the blob storage account that was created in the first step.
  • $storageAccountKey = “<storage account key>” – the blob store access key obtained from the portal.
  • $resourceGroupName = “APIMService” –  name of the Azure resource group.
  • $apimName = “apimmanager” – the name of the APIM service.
  • $targetContainerName = “backup”  – name of the backup container in blob store.
  • $tartgetBlobName = “AzureAPIM.apimbackup” – file name of the backup file.  This can be omitted and will create a default filename {apimName}-{yyyy-MM-dd-HH-mm}.apimbackup

Once you have confirmed the script executes without any errors, you can now set up a recurring schedule by creating a new schedule in the Automation service blade under Shared Resources.

imageimage

Next you need to link your Runbook to this schedule by double clicking on your runbook name and then the schedule button on the top menu. This will open another blade where you can view all your schedules that you can select from.

image

That is the automated back process completed now. Below is the PowerShell script required to restored the backup file.

#get the storgae context
$sourceContext = (New-AzureStorageContext `
-StorageAccountName “<blob storage name>” `
-StorageAccountKey “<blob storage account key from Azure portal>”)

#restore the backup
Restore-AzureRmApiManagement -ResourceGroupName “<name of resource group>” `
-Name “<name of the APIM service>” `
-StorageContext $sourceContext  `
-SourceContainerName “<blob storage container name>” `
-SourceBlobName “<backup file name>”

More details on these scripts can be found here: https://docs.microsoft.com/en-us/powershell/module/azurerm.apimanagement/restore-azurermapimanagement?view=azurermps-4.3.1

Enjoy.