Azure APIM Content Based Routing Policy

Content based routing is where the message routing endpoint is determined by the contents of the message at runtime. Instead of using an Azure Logic App workflow to determine where route the message, I decided to use Azure APIM and a custom policy. The main objective of using an APIM policy was to allow me to add or update the routing information through configuration rather than code and to add a proxy API for each of the providers API endpoints.

Below is the high level design of the solution based on Azure APIM and custom policies. Here the backend ERP system sends a generic PO message to the PO Routing API which will then forward the message onto the suppliers proxy API based on the SupplierId contained in the message. The proxy API will then transform the generic PO to the correct format before being sent to the supplier’s API.

The main component of this solution is the internal routing API that accepts the PO message and uses a custom APIM Policy to read the SupplierId attribute in the message. The outbound URL of this API is then set to the internal supplier proxy API by using a lookup list to find the matching reroute URL for the SupplierId. This lookup list is a JSON array stored as a Name-Value pair in APIM. This allows us to add or update the routing information through configuration rather than code.

Here is the structure of the URL lookup list which is an array of JSON objects stored as a Name-value pair in APIM.

[
   {
     "Name": "Supplier A",
     "SupplierId": "AB123",
     "Url": https://myapim.azure-api.net/dev/purchaseorder/suppliera
    },
    {
     "Name": "Supplier B",
     "SupplierId": "XYZ111",
     "Url": https://myapim.azure-api.net/dev/purchaseorder/supplierb
    },
    {
     "Name": "SupplierC",
     "SupplierId": "WAK345",
     "Url": https://myapim.azure-api.net/dev/purchaseorder/supplierc
    }
 ]

PO Routing API

The code below is the custom policy for the PO Routing API. This loads the above JSON URL Lookup list into a variable called “POList”.  The PO request body is then loaded into a JObject  type and the SupplierId attribute value found. Here you can use more complex queries if the required routing values are scattered within the message.

Next the URL Lookup list is converted into a JArray type and then searched for the matching SupplierId which was found in the request message. The  variable “ContentBasedUrl” is then set to the URL attribute of the redirection object. If no errors are found then the backend url is set to the  internal supplier’s proxy API and the PO message forwarded on.


  // Get the URL Lookup list from the Name-value pair in APIM 
    <set-variable name="POList" value="{{dev-posuppliers-lookup}}" />


    <set-variable name="ContentBasedUrl" value="@{ 

        JObject body = context.Request.Body.As<JObject>(true);

        // Create an JSON object collection
        try
        {

          // Get the AccountId from the inbound PO message
            var supplierId = body["Purchase"]?["SupplierId"];

            var jsonList = JArray.Parse((string)context.Variables.GetValueOrDefault<string>("POList"));

          // Find the AccountId in the json array 
            var arr = jsonList.Where(m => m["SupplierId"].Value<string>() == (string)supplierId);

            if(arr.Count() == 0 )
            {
                return "ERROR - No matching account found for :- " + supplierId;
            }

            var url = ((JObject)arr.First())["Url"];
            if( url == null )
            {
                return "ERROR - Cannot find key 'Url'";
            }          

            return url.ToString();
        }
        catch( Exception ex)
        {
            return "ERROR - Invalid message received.";
        }

    }" />
    <choose>
        <when condition="@(((string)context.Variables["ContentBasedUrl"]).StartsWith("ERROR"))">
            <return-response>
                <set-status code="400" />
                <set-header name="X-Error-Description" exists-action="override">
                    <value>@((string)context.Variables["ContentBasedUrl"])</value>
                </set-header>
            </return-response>
        </when>
    </choose>
    <set-backend-service base-url="@((string)context.Variables["ContentBasedUrl"])" />
</inbound>
<backend>
    <base />
</backend>
<outbound>
    <base />
</outbound>
<on-error>
    <base />
</on-error>

Supplier Proxy API

A proxy API is created for each of the suppliers in APIM where the backend is set to the suppliers API endpoint.  The proxy will typically have the following policies.

The primary purpose of this proxy API is to map the generic PO message to the required suppliers format and to manage any authentication requirements. See my other blog about using a mapping function within a policy https://connectedcircuits.blog/2019/08/29/using-an-azure-apim-scatter-gather-policy-with-a-mapping-function/ There is no reason why you cannot use any other mapping process in this section.

The map in the Outbound section of the policy takes the response message from the Suppliers API and maps it to the generic PO response message which is then passed back to the PO Routing API and then finally to the ERP system as the response message.

Also by adding a proxy for each of the supplier’s API provides decoupling from the main PO Routing API,  allowing us to test the suppliers API directly using the proxy.

The last step is to add all these API’s to a new Product in APIM so the subscription key can be shared and passed down from the Routing API to the supplier’s proxy APIs.

Enjoy…

Enterprise API for Advanced Azure Developers

For the last few months I have busy developing  the above course for EC-Council. This is my first endeavour doing something like this and wanted to experience the process of  creating an online course.

You can view my course here at https://codered.eccouncil.org/course/enterprise-api-for-advanced-azure-developers and would be keen to get your feedback on any improvements I can make if I do create another course in the future.

The course will take you through the steps of designing and developing an enterprise-ready API from the ground up, covering both the functional and non-functional design aspects of an API. There is a total of 7 chapters equating to just over 4 hours of video content covering the following topics:

  • Learn the basics of designing an enterprise level API.

  • Learn about the importance of an API schema definition.

  • Learn about the common non-functional requirements when designing an API.

  • Develop a synchronous and asynchronous API’s using Azure API App and Logic Apps

  • Understand how to secure your API using Azure Active Directory and custom roles.

  • Set up Azure API Management and customizing the Developer Portal.

  • Create custom APIM polices.

  • Configure monitoring and alerting for your API.

Enjoy…

Always Linter your API Definitions

An API definition may be thought of as an electronic equivalent of a receptionist for an organisation. A receptionist is normally the first person they see when visiting an organisation and their duties may include answering enquires about products and services.

This is similar to an API definition, when someone wants to do business electronically, they would use the API definition to obtain information about the available services and operations.

My thoughts are, API’s should be built customer first and developer second. Meaning an API needs to be treated as a product and to make it appealing to a consumer, it needs to be designed as intuitive and usable with very little effort.

Having standardised API definition documents across all your API’s, allows a consumer to easily navigate around your API’s as they will all have a consistent look and feel about them. This specially holds true when there are multiple teams developing Microservices for your organisation. You don’t want your consumers think they are dealing with several different organisations because they all look and behave differently. Also, when generating the client object models from several different API definitions from the same organisation, you want the property names to all follow the same naming conventions.

To ensure uniformity across all API schema definitions, a linter tool should be implemented. One such open-source tool is Spectral https://github.com/stoplightio/spectral from Stoplight, however there are many other similar products available.

Ideally the validation should be added as a task to your CI/CD pipelines when committing the changes to your API definition repository.

Using Spectral in MS DevOps API Projects

Spectral provides the capability to develop custom rules as a yaml or json file. I typically create a separate DevOps project to store these rules inside a repository which may be shared across other API projects.

Below is how I have setup the repositories section in the pipeline to include the linter rules and the API definition.

image

After downloading the Spectral npm package in the pipeline, I then run the CLI command to start the linter on the schema specification document which is normally stored in a folder called ‘specification’ in the corresponding API repository.

clip_image001

An example of the pipeline job can be seen below. Here the job failed due to some rule violations.

clip_image002

The build pipeline and sample rules can be found here https://github.com/connectedcircuits/devops-api-linter

Hope this article now persuades you to start using Linter across all your API definitions.

Enjoy…

Using an Azure APIM Scatter-Gather policy with a Mapping Function

This is a follow up from a previous blog “Azure APIM Scatter-Gather Pattern Policy” where I wrote about using the Wait policy to create a scatter-gather pattern.  A few colleges were asking about being able to map the inbound request to the different schemas required by each of the Microservices.

A high level design is shown below using two “Wait” polices and an Azure Function for the mapping. The first policy ‘Translation’ sends the request to the mapping function and when completed the second policy ‘Scatter’ is executed to send the mapped request to each of the Microservices.

image

The internals of the Azure Function that maps the incoming request is shown below as an example. Here I am using a route on the supplier name and a simple If-Then statement to select which static translator method to call.

public static class MappingService
{
[FunctionName("DataMapping")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = "Pricing/{suppliername}")]
HttpRequest req,
string suppliername,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic response = null;
if (suppliername.Equals("Supplier1", StringComparison.OrdinalIgnoreCase))
response = Translator.TranslateSupplier1(requestBody);
else if (suppliername.Equals("Supplier2", StringComparison.OrdinalIgnoreCase))
response = Translator.TranslateSupplier2(requestBody);

return response != null
? (ActionResult)new OkObjectResult(response)
: new BadRequestObjectResult("Invalid message.");
}
}

Below is the code for the Translation Policy which calls the two Azure function resources in parallel that accepts the inbound price request message. The function access code is stored as a named-value pair in APIM called “functionapkey”.

<!-- Call the mapping service -->
<wait for="all">
<send-request mode="copy" response-variable-name="res_SupplierMap1" timeout="120" ignore-error="false">
<set-url>https://mapsvc.azurewebsites.net/api/Pricing/supplier1?code={{funcmapkey}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
<send-request mode="copy" response-variable-name="res_SupplierMap2" timeout="120" ignore-error="false">
<set-url>https://mapsvc.azurewebsites.net/api/Pricing/supplier2?code={{funcmapkey}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
</wait>
 

The code for the Scatter Policy is shown below which is similar to the original blog post. However it uses the mapped outputs from the Translation policy that are stored in the res_SupplerMap1 and res_SupplerMap2 context variables instead.

<!-- Call the pricing services -->
<wait for="all">
<send-request mode="copy" response-variable-name="response_1" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_1}}</set-url>
<set-method>POST</set-method>
<set-body>@(((IResponse)context.Variables["res_SupplierMap1"]).Body.As<string>())</set-body>
</send-request>
<send-request mode="copy" response-variable-name="response_2" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_2}}</set-url>
<set-method>POST</set-method>
<set-body>@(((IResponse)context.Variables["res_SupplierMap2"]).Body.As<string>())</set-body>
</send-request>
</wait>

The last policy checks the status of each of the pricing services and returns the results as a composite message if there were no errors encountered. This is similar to the original blog post but instead of returning a JObject I am now returning a JArray collection.

<choose>
<when condition="@((int)((IResponse)context.Variables["response_1"]).StatusCode != 200)">
<return-response response-variable-name="response_1" />
</when>
<when condition="@((int)((IResponse)context.Variables["response_2"]).StatusCode != 200)">
<return-response response-variable-name="response_2" />
</when>
<otherwise>
<return-response>
<set-status code="200" reason="OK" />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body>@{
JArray suppliers = new JArray();
suppliers.Add(((IResponse)context.Variables["response_1"]).Body.As<JObject>());
suppliers.Add(((IResponse)context.Variables["response_2"]).Body.As<JObject>());
return suppliers.ToString();
}</set-body>
</return-response>
</otherwise>
</choose>

The completed policy for the method looks like this below. Take note the request payload is stored in the variable named “requestPayload” initially to avoid locking body context in the <Wait> policies.

<policies>
<inbound>
<set-variable name="requestPayload" value="@(context.Request.Body.As&lt;string>(true))" />
<!-- Call the mapping service -->
<wait for="all">
<send-request mode="copy" response-variable-name="res_SupplierMap1" timeout="120" ignore-error="false">
<set-url>https://mapsvc.azurewebsites.net/api/Pricing/supplier1?code={{funcmapkey}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
<send-request mode="copy" response-variable-name="res_SupplierMap2" timeout="120" ignore-error="false">
<set-url>https://mapsvc.azurewebsites.net/api/Pricing/supplier2?code={{funcmapkey}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
</wait>
<!-- Call the pricing services -->
<wait for="all">
<send-request mode="copy" response-variable-name="response_1" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_1}}</set-url>
<set-method>POST</set-method>
<set-body>@(((IResponse)context.Variables["res_SupplierMap1"]).Body.As<string>())</set-body>
</send-request>
<send-request mode="copy" response-variable-name="response_2" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_2}}</set-url>
<set-method>POST</set-method>
<set-body>@(((IResponse)context.Variables["res_SupplierMap2"]).Body.As<string>())</set-body>
</send-request>
</wait>
<choose>
<when condition="@((int)((IResponse)context.Variables["response_1"]).StatusCode != 200)">
<return-response response-variable-name="response_1" />
</when>
<when condition="@((int)((IResponse)context.Variables["response_2"]).StatusCode != 200)">
<return-response response-variable-name="response_2" />
</when>
<otherwise>
<return-response>
<set-status code="200" reason="OK" />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body>@{
JArray suppliers = new JArray();
suppliers.Add(((IResponse)context.Variables["response_1"]).Body.As<JObject>());
suppliers.Add(((IResponse)context.Variables["response_2"]).Body.As<JObject>());
return suppliers.ToString();
}</set-body>
</return-response>
</otherwise>
</choose>
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>

Using the tracing feature in APIM, you can see the initial price request message below. This will be sent to the mapping function Pricing method.

 image

Below is the trace output from APIM showing the two different messages returned from mapping function and assigned to the res_Supplier variables.

image

Below is the composite message returned from APIM as an JSON array containing each supplier.

image

In conclusion, using the two <Wait> polices to send multiple requests in parallel yields a request/response latency of around 200ms on average in this scenario. Also instead of using an Azure Function to map the incoming request, you could replace it with a couple of “Transformation” policies.

Enjoy…

Azure APIM Scatter-Gather Pattern Policy

Recently I have been reading about some of the advanced policies available in APIM and the one that caught my eye was the “Wait” policy found here https://docs.microsoft.com/en-us/azure/api-management/api-management-advanced-policies. This policy will execute child policies in parallel and will wait until either one or all of the child polices have completed before continuing. 

I was pondering on a good use case for this policy and the first one came to mind was a scatter-gather pattern shown below. I could easily create this pattern without having to provision any other required Azure services.

 

image

The Scatter-Gather pattern is created by exposing a REST endpoint through APIM which will accept a JSON payload. When a request comes into APIM, a custom policy will forward this request onto the service endpoints defined in the policy and wait until they have all responded before continuing on with the rest of the statements in the parent policy. Once all the destination endpoints have responded, the return status code on each service is then checked for success or an error. If an error does occur, then the error is returned immediately otherwise the message from each service is combined into a composite JSON message which is returned back to the caller as the response message.

For the scenario, I will be sending a price request for a product to 3 suppliers simultaneously. In Azure APIM, I created a method which will accept a JSON price request message as shown below.

image

 

Then I added an Inbound processing policy to that method which is broken down into two parts. Below is the first part of the policy where it sends the inbound request to multiple endpoints in parallel using the <send-request> tag. Named values are used for the service endpoint address defined in the <set-url>{{serviceendpoint_xxx}}</set-url> tags. The response from each service call is inserted into the variable response-variable-name=”response_xxx” which is used in the second part of this policy.

<set-variable name="requestPayload" value="@(context.Request.Body.As&lt;string>(true))" />
<wait for="all">
<send-request mode="copy" response-variable-name="response_1" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_1}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
<send-request mode="copy" response-variable-name="response_2" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_2}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
<send-request mode="copy" response-variable-name="response_3" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_3}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
</wait>

After the closing </wait> tag,  the response code from each of the services is checked inside a <choose> block. If its not equal to 200 then the response is immediately returned, otherwise each of the responses is wrap inside a JSON object and returned as the composite response message.
 
<choose>
<when condition="@((int)((IResponse)context.Variables["response_1"]).StatusCode != 200)">
<return-response response-variable-name="response_1" />
</when>
<when condition="@((int)((IResponse)context.Variables["response_2"]).StatusCode != 200)">
<return-response response-variable-name="response_2" />
</when>
<when condition="@((int)((IResponse)context.Variables["response_3"]).StatusCode != 200)">
<return-response response-variable-name="response_3" />
</when>
<otherwise>
<return-response>
<set-status code="200" reason="OK" />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body>@(new JObject(
new JProperty("service_1",((IResponse)context.Variables["response_1"]).Body.As<JObject>()),
new JProperty("service_2",((IResponse)context.Variables["response_2"]).Body.As<JObject>()),
new JProperty("service_3",((IResponse)context.Variables["response_3"]).Body.As<JObject>())
).ToString())</set-body>
</return-response>
</otherwise>
</choose>
 
Below is the whole policy for the method. More service endpoints may be added or removed from this policy as desired.
<policies>
<inbound>
<set-variable name="requestPayload" value="@(context.Request.Body.As&lt;string>(true))" />
<wait for="all">
<send-request mode="copy" response-variable-name="response_1" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_1}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
<send-request mode="copy" response-variable-name="response_2" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_2}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
<send-request mode="copy" response-variable-name="response_3" timeout="120" ignore-error="false">
<set-url>{{serviceendpoint_3}}</set-url>
<set-method>POST</set-method>
<set-body>@(context.Variables.GetValueOrDefault<string>("requestPayload"))</set-body>
</send-request>
</wait>
<choose>
<when condition="@((int)((IResponse)context.Variables["response_1"]).StatusCode != 200)">
<return-response response-variable-name="response_1" />
</when>
<when condition="@((int)((IResponse)context.Variables["response_2"]).StatusCode != 200)">
<return-response response-variable-name="response_2" />
</when>
<when condition="@((int)((IResponse)context.Variables["response_3"]).StatusCode != 200)">
<return-response response-variable-name="response_3" />
</when>
<otherwise>
<return-response>
<set-status code="200" reason="OK" />
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body>@(new JObject(
new JProperty("service_1",((IResponse)context.Variables["response_1"]).Body.As<JObject>()),
new JProperty("service_2",((IResponse)context.Variables["response_2"]).Body.As<JObject>()),
new JProperty("service_3",((IResponse)context.Variables["response_3"]).Body.As<JObject>())
).ToString())</set-body>
</return-response>
</otherwise>
</choose>
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>

 
For testing purposes, I just created 3 simple logic apps triggered by a HTTP request which returned some static data. Using Postman, I sent the following message.
image
 
The response message from APIM is shown below.
 
image
 
Enjoy…

Exposing Azure Service Bus through APIM, generating a SAS Token and setting the Session Id

On one of my recent projects, a client application was required to place a message onto an Azure Service Bus by using a HTTP endpoint rather than using the Service Bus SDK and with the following constraints.

  • The client is unable to generate the Service Bus SAS token.
  • Service Bus Session Id needs to be set to the customer number found in the message to ensure ordered delivery by the consumer.
  • All  messages are to be sent to a Service Bus Topic and require to add a custom Service Bus property called ‘MgsType’.
  • Custom HTTP Request Headers may be used.

I decided upon a solution that uses Azure APIM to expose the Service Bus endpoint.  A custom policy will be used to generate the Service Bus SAS token and to parse the message for the customer Id. The customer Id will then be used to set the SessionId on the Service Bus. The client then only has to register for a subscription key in the Azure Developer portal and then pass this key with each HTTP request in the header.

image

The first step of this solution  is to create an Azure Service Bus with a Topic called ‘transactions’ and to create a Shared access policy called ‘Sender’ which has only Send claims.

image

Next add a subscription to the topic with ‘Require Session’ enabled and the following  rule “MsgType = ‘Deposits’”

image

Before we start developing the custom policies, we need to setup 3 Name Values in the APIM blade as shown below. These values are used for generating the Service Bus SAS token and are obtained from the Service Bus properties.

image

Where,

  • SB_Key – primary key for the ‘Sender’ shared access policies
  • SB_KeyName  – name of the Shared access policy
  • SB_Uri – is the topic URL which can be found by clicking on the topic name under the ‘Topics’ blade shown below.

image

Next is to create an API using the ‘Blank API’ template similar to what I have done below. Note the ‘Web service URL’ value is the base address of the Service Bus topic URL. (ie without the topic name resource location)

image

Then add an operation to the service as below. Note the URL should be the name of your topic with the resource ‘messages’ appended to the end.

image

Once the operation has been created, we can now add the custom policy to the ‘Inbound processing’ stage.

image

The APIM policy consists of several code blocks inside the inbound processing stage which are describe in detail below. To improve performance I will be caching the generated SAS token.

The  first code block looks up cache for a value. If nothing is found then the variable “cachedSasToken” is assigned to null as there is no default value specified.

<cache-lookup-value key="crmsbsas" variable-name="cachedSasToken" />

Next a control flow is used to check the variable “cachedSasToken” for null and if true then a SAS token is generated using the values stored in APIM Name-Value pairs. Once the token is  calculated then it is stored in cache and set to expire in 120 seconds. The cache is then read again to assign the variable “cachedSasToken” with the generated SAS token.

Both the signature and resource URL is required to be UrlEncoded. My first choice was to use the System.Web.UrlEncode function to encode the values. Unfortunately this function is not available in the APIM policy expressions as it only has a subset of the .Net Framework types. To work around this issue, I ended up using the System.Uri.EscapeDataString method instead.

<choose>

            <when condition="@(context.Variables.GetValueOrDefault&lt;string>("cachedSasToken") == null)">

                <cache-store-value key="crmsbsas" value="@{

                        string resourceUri = "{{SB_Uri}}";

                        string keyName = "{{SB_KeyName}}";

                        string key = "{{SB_Key}}";

                        TimeSpan sinceEpoch = DateTime.UtcNow - new DateTime(1970, 1, 1);

                        var expiry = Convert.ToString((int)sinceEpoch.TotalSeconds + 120);

                        string stringToSign = System.Uri.EscapeDataString(resourceUri) + "\n" + expiry;

                        HMACSHA256 hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key));

                        var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

                        var sasToken = String.Format("SharedAccessSignature sr={0}&amp;sig={1}&amp;se={2}&amp;skn={3}", 

                                        System.Uri.EscapeDataString(resourceUri),

                                        System.Uri.EscapeDataString(signature), expiry, keyName);

                        return sasToken;

                    }" duration="100" />

                <cache-lookup-value key="crmsbsas" variable-name="cachedSasToken" />

            </when>

        </choose>

The SAS token is then added to the header using the value from the variable ‘cachedSasToken’

<set-header name="Authorization" exists-action="override">

<value>@(context.Variables.GetValueOrDefault<string>("cachedSasToken"))</value>

</set-header>

I then set the message content type to application/json and remove the APIM subscription key header from being sent to the Service Bus endpoint.

<set-header name="Content-type" exists-action="override">

          <value>application/json</value>

      </set-header>

      <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />

The last part is to extract the customer number from the message and assign it to the SessionId property of the Service Bus. The standard set of Service Bus properties are required to be added to  a custom header called ‘BrokerProperties’

<set-header name="BrokerProperties" exists-action="override">

            <value>@{

               string reqData = context.Request.Body?.As<string>(true);

               dynamic data =  JsonConvert.DeserializeObject(reqData);

               string order =  data?.CustomerNumber;

               return string.Format("{{\"SessionId\":\"{0}\"}}", order);

            }</value>

        </set-header>

The full code for the custom policy is here:

<!--

    IMPORTANT:

    - Policy elements can appear only within the <inbound>, <outbound>, <backend> section elements.

    - Only the <forward-request> policy element can appear within the <backend> section element.

    - To apply a policy to the incoming request (before it is forwarded to the backend service), place a corresponding policy element within the <inbound> section element.

    - To apply a policy to the outgoing response (before it is sent back to the caller), place a corresponding policy element within the <outbound> section element.

    - To add a policy position the cursor at the desired insertion point and click on the round button associated with the policy.

    - To remove a policy, delete the corresponding policy statement from the policy document.

    - Position the <base> element within a section element to inherit all policies from the corresponding section element in the enclosing scope.

    - Remove the <base> element to prevent inheriting policies from the corresponding section element in the enclosing scope.

    - Policies are applied in the order of their appearance, from the top down.

-->

<policies>

    <inbound>

        <base />

        <cache-lookup-value key="crmsbsas" variable-name="cachedSasToken" />

        <choose>

            <when condition="@(context.Variables.GetValueOrDefault&lt;string>("cachedSasToken") == null)">

                <cache-store-value key="crmsbsas" value="@{

                        string resourceUri = "{{SB_Uri}}";

                        string keyName = "{{SB_KeyName}}";

                        string key = "{{SB_Key}}";

                        TimeSpan sinceEpoch = DateTime.UtcNow - new DateTime(1970, 1, 1);

                        var expiry = Convert.ToString((int)sinceEpoch.TotalSeconds + 120);

                        string stringToSign = System.Uri.EscapeDataString(resourceUri) + "\n" + expiry;

                        HMACSHA256 hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key));

                        var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

                        var sasToken = String.Format("SharedAccessSignature sr={0}&amp;sig={1}&amp;se={2}&amp;skn={3}", 

                                        System.Uri.EscapeDataString(resourceUri),

                                        System.Uri.EscapeDataString(signature), expiry, keyName);

                        return sasToken;

                    }" duration="10" />

                <cache-lookup-value key="crmsbsas" variable-name="cachedSasToken" />

            </when>

        </choose>

        <set-header name="Authorization" exists-action="override">

            <value>@(context.Variables.GetValueOrDefault<string>("cachedSasToken"))</value>

        </set-header>

        <set-header name="Content-type" exists-action="override">

            <value>application/json</value>

        </set-header>

        <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />

        <set-header name="BrokerProperties" exists-action="override">

            <value>@{

               string reqData = context.Request.Body?.As<string>(true);

               dynamic data =  JsonConvert.DeserializeObject(reqData);

               string order =  data?.CustomerNumber;

               return string.Format("{{\"SessionId\":\"{0}\"}}", order);

            }</value>

        </set-header>

    </inbound>

    <backend>

        <base />

    </backend>

    <outbound>

        <base />

    </outbound>

    <on-error>

        <base />

    </on-error>

</policies>

 

Now lets use Postman to send a message to the URL endpoint exposed by APIM to test the policy. The headers contain the APIM subscription key and a custom header for the ‘MsgType’ which will be used for the Service Bus subscription filter.

image

The message body simply contains the Customer number and the amount to deposit.

image

After posting the message to the APIM endpoint URL, we can see the message was successfully forwarded to the Service Bus by using Service Bus Explorer to view the message properties and content.

Notice the message custom properties has the MsgType and the ‘SessionId’ is populated with the customer number.

image

Enjoy…

Using Azure APIM Policies to Route on HTTP Verbs & Resources

The policies available in APIM are indeed very powerful. It provides the ability to modify the backend flow of a request using rules based on the payload contents or the request context properties. I have yet to find a scenario where I could not resolve a requirement using the inbuilt statements.

One example is I had to send a request to different endpoints depending on the request verb (Post,Put,Delete) and the resource location.

image

A policy inspects the incoming request Url and the operation method to determine which Azure Logic App or Function to forward the request onto.

The inbound policy section for this scenario is shown below. It uses the Xslt Choose function to match on the Url path and the operation. If a match is found, it sets the variable “logicappEndpoint” to the name-value variable defined in the Azure APIM blade.

<inbound>

    <choose>

        <when condition="@(context.Request.Url.Path.Equals("inventory")  && context.Operation.Method.Equals("PUT"))">

            <set-variable name="logicappEndpoint" value="{{UAT-AXProductUpdateUrl}}" />

        </when>

        <when condition="@(context.Request.Url.Path.Equals("inventory")  && context.Operation.Method.Equals("POST"))">

            <set-variable name="logicappEndpoint" value="{{UAT-AXProductCreateUrl}}" />

        </when>

        <when condition="@(context.Request.Url.Path.Equals("inventory") && context.Operation.Method.Equals("DELETE"))">

            <set-variable name="logicappEndpoint" value="{{UAT-AXProductDeleteUrl}}" />

        </when>

        <when condition="@(context.Request.Url.Path.Equals("stockmovement") && context.Operation.Method.Equals("PUT"))">

            <set-variable name="logicappEndpoint" value="{{UAT-AXStockMovementUpdateUrl}}" />

        </when>

        <when condition="@(context.Request.Url.Path.Equals("stockmovement") && context.Operation.Method.Equals("POST"))">

            <set-variable name="logicappEndpoint" value="{{UAT-AXStockMovementCreateUrl}}" />

        </when>        

    </choose>

    <base />

</inbound>

To reference the Named-Value properties defined in the portal from within a policy, simply wrap the property name with double braces as shown here as an example:-  {{UAT-AXStockMovementCreateUrl}}

These Name-Values pairs are then created in APIM blade of the Azure Portal shown below.

image

Once the Inbound policy section has been created, we then need to create the rule for the Backend section shown below. Here I am also setting a retry condition to 3 times before sending back a status error code of 500.  Also I am always forwarding the request to the Logic App as a POST.

<backend>

    <retry condition="@(context.Response.StatusCode == 500)" count="3" interval="5" max-interval="10" delta="2" first-fast-retry="false">

        <send-request mode="copy" response-variable-name="tokenstate" timeout="120" ignore-error="true">

            <set-url>@(context.Variables.GetValueOrDefault<string>("logicappEndpoint"))</set-url>

            <set-method>POST</set-method>

            <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />

        </send-request>

    </retry>

</backend>

 

I hope this short blog gives you a basic understanding of how to redirect a request based on the HTTP verb and a resource location. Enjoy…

Using an Azure APIM Policy to call an OAuth endpoint and cache the token

Recently I have been involved as the Integration Architect and tech lead for a project to integrate between MS Dynamics 356  AX/CRM and 3rd party systems.

One of the challenges was to provide single endpoints for internal systems calling web services outside of the organisation that utilised OAuth2.0.  This is where a consumer normally has to call an OAuth token endpoint first and then append the token to the request before calling the actual web service.

By using APIM as a proxy and policies in APIM, I managed to achieve the goal of providing a single URL endpoint for the consumer. The policy initially gets the token from the authorisation endpoint, caches the token and then passes the token to the web service being called. This process is known as fragment caching, where parts of the responses are cached for subsequent requests. Also by caching the bearer part of the token improved the performance significantly for subsequent calls.

Below were the steps I used to add a web API to create transfers orders in Dynamics AX and a policy using the Azure APIM management portal.

1. First create the properties for the oAuth clientId and client secret. A good tip is to prefix the property name and set the Tags with the name of the API you are calling. This helps latter on when you have multiple properties to manage.

 

image

The Properties page should look like something like below after adding your custom properties:

 

image

2. Next, go to the API’s page to either add or import the API’s definition you are calling. Here I am just going to add the API manually by first entering the name, API endpoint address and the public facing URL suffix. Remember to add it to an existing product.

image

3. After the API has been created we will then add the operations. For this demo I will just add one operation to create a transfer order.

image

4. Now that the API we wish to call has been added to the APIM service we can finally pay our attention to creating a policy which does the background task of obtaining the OAuth token. Here I will setup a policy on the CreateTransferOrder operation of the Web API. Once selected, click the ADD POLICY button to begin creating the policy from the template.

image

5. The “<inbound>” policy section is applied to the incoming request before forwarding the request to the backend service. This is where we will check the cache for the authorisation token and if no hit, then we call the OAuth token endpoint to obtain a token.

   1: <inbound>

   2:     <cache-lookup-value key="token-{{Dev-Web1-ClientId}}" variable-name="bearerToken" />

   3:     <choose>

   4:         <when condition="@(!context.Variables.ContainsKey("bearerToken"))">

   5:             <send-request mode="new" response-variable-name="oauthResponse" timeout="20" ignore-error="false">

   6:                 <set-url>https://login.microsoftonline.com/xxxxxx.onmicrosoft.com/oauth2/token</set-url>

   7:                 <set-method>POST</set-method>

   8:                 <set-header name="Content-Type" exists-action="override">

   9:                     <value>application/x-www-form-urlencoded</value>

  10:                     <!-- for multiple headers with the same name add additional value elements -->

  11:                 </set-header>

  12:                 <set-body>@("grant_type=client_credentials&client_id={{Dev-Web1-ClientId}}&client_secret={{Dev-Web1-ClientSecret}}&resource=https%3A%2F%2Fxxxx.xxxxx.ax.dynamics.com")</set-body>

  13:             </send-request>

  14:             <set-variable name="accessToken" value="@((string)((IResponse)context.Variables["oauthResponse"]).Body.As<JObject>()["access_token"])" />

  15:             <!-- Store result in cache -->

  16:             <cache-store-value key="token-{{Dev-Web1-ClientId}}" value="@((string)context.Variables["AccessToken"])" duration="3600" />

  17:         </when>

  18:     </choose>

  19: </inbound>

Lets go through the key points of this definition below.

  • Line 2: – Assigns the value in cache to the context variable called “bearerToken”. On first entry, the cache value will be null and the variable will not be created. Note I am adding the ClientId as part of the cache key name to keep it unique. Property values are accessed by surrounding the key name with double braces. eg  {{myPropertyName}}
  • Line 4: – Checks if the context variable collection contains a key called “bearerToken” and if not found executes the code between the opening and closing “<when>” XML elements.
  • Line 5: – Initiates the request to the OAuth endpoint with a response timeout of 20 seconds. This will put the response message into the variable called “oauthResponse”
  • Line 6: – Is where you set the URL to send the request to. In this scenario I am using the Azure AD OAuth token endpoint below as our STS service: https://login.microsoftonline.com/xxxxxx.onmicrosoft.com/oauth2/token
  • Line 12: – This is where you define the body payload for the request and this is defined  as a typical client credentials grant type payload. Here I am getting the values for the client Id and secret from the user definable properties set in the Properties page. The resource parameter is just hardcoded to the Urlencoded resource URL of the API but can also parameterised.

<set-body>@(“grant_type=client_credentials&client_id={{Dev-Web1-ClientId}}&client_secret={{Dev-Web1-ClientSecret}}&resource=https%3A%2F%2Fxxxx.xxxxx.ax.dynamics.com”)</set-body>

  • Line 14: – Casts the response as a JSON object to allow the retrieval of the “access_token” value using an indexer and assigns it to the context variable “accessToken”.
  • Line 16: – Is where we add the contents of the variable “accessToken” into cache for a period of 3600 seconds.

6. Now that the “<inbound>” section has been completed, we can look at the “<backend>” section of the policy. This is where the policy forwards your request to the backend web service as defined in the API configuration page.

   1: <backend>

   2:     <send-request mode="copy" response-variable-name="transferWSResponse" timeout="20" ignore-error="false">

   3:         <set-method>POST</set-method>

   4:         <set-header name="Authorization" exists-action="override">

   5:             <value>@("Bearer " + (string)context.Variables["bearerToken"])</value>

   6:         </set-header>

   7:         <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />

   8:         <set-header name="Content-Type" exists-action="override">

   9:             <value>application/json</value>

  10:         </set-header>

  11:     </send-request>

  12: </backend>

Lets go through this section as before.

  • Line 2: – Creates the request to the backend web service. Here we are placing the response from the web service into the variable called “transferWSResponse”.
  • Line 4: – Is the creating the “Authorization” header to be sent with the request.
  • Line 5: – Adds the bearer token value from the context variable “bearerToken” the authorisation header.
  • Line 7: – Removes the APIM subscription from being forwarded to the backend web service.

7. Now we need to return the response message from the backend web service to the caller. This is done in the “<outbound>” policy section. Here we just simply return the value of the variable “transferWSResponse” back to the caller.

   1: <outbound>

   2:     <return-response response-variable-name="transferWSResponse">

   3:     </return-response>

   4:     <base />

   5: </outbound>

That’s the whole policy defined which will call the OAuth endpoint the get the token and cache it for subsequent calls.

Using the tracing feature in APIM, when the first request is made, the cache will be null and the variable will not be set as shown below.

image

The next trace shows any subsequent requests will hit the cache and set the context variable to the bearer token until it expires.

image

One important note about retrieving data from cache is its an out-of-process call and can add tens of milliseconds onto a request.

Working with the APIM polices can make a huge impact on your API development efforts as logging and access management can be off-loaded using the polices available in APIM. A full list of expressions used in polices can be found here: https://docs.microsoft.com/en-us/azure/api-management/api-management-policy-expressions

Enjoy…

APIM backup & restore using Azure Automation Services

In this post I will describe the steps of using PowerShell scripts to backup APIM and using the Automation service to schedule the backup every month. The restore function also allows you to restore  APIM into another resource group or APIM service. For the project I am working on now, this is what I am doing to move the configuration settings between each environment.

First you need to create a blob store which ideally should be Read-Access geo-redundant storage (RA-GRS). This is where the APIM backups will be stored. After the blob store has been provisioned, create a container for the backup file as shown below.

image

Once the container is created, take note of the Storage account name and Access key for the blob store. These values will be used in the PowerShell script later.

image

Next provision an Azure Automation service and ensure the Create Azure Run As account is set to “yes”.

image

Once it has been provisioned, ensure the modules have been updated by clicking on the “Modules” link on the left hand navigation panel and then “Update Azure Modules”. Note this does take a while to complete.

image

After the update has been completed, click the “Browse gallery” link and in search textbox type “apim”. Once found,  double click on the row to open the import blade.

image

Now click the Import icon to import the cmdlet. This can take several minutes to import.

image

After the PowerShell module has been imported, create a new Runbook and ensure the type has been set to “PowerShell”. Then click the Create button at the bottom of the page.

image

This will open up a new blade where we can add and test the PowerShell script to backup the APIM settings.

image

Now add the following script below into the text editor and remember to update the variables with your environment settings. Once you have added the script, click the “Save” button and then the “Test pane” button to ensure the script runs successfully.

   1: Disable-AzureDataCollection

   2: Write-Output "Starting backup of APIM..."

   3:

   4: # sign in non-interactively using the service principal

   5: $connectionName = "AzureRunAsConnection";

   6: $storageAccountName = "apimstorebackup";

   7: $storageAccountKey = "<storage account key>";

   8: $resourceGroupName = "APIMService";

   9: $apimName = "apimmanager";

  10: $targetContainerName = "backup";

  11: $targetBlobName "AzureAPIM.apimbackup"

  12: try

  13: {

  14:     # Get the connection "AzureRunAsConnection "

  15:     $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName

  16:

  17:     Write-Output "Logging in to Azure..."

  18:     Add-AzureRmAccount `

  19:         -ServicePrincipal `

  20:         -TenantId $servicePrincipalConnection.TenantId `

  21:         -ApplicationId $servicePrincipalConnection.ApplicationId `

  22:         -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint

  23: }

  24: catch {

  25:     if (!$servicePrincipalConnection)

  26:     {

  27:         $ErrorMessage = "Connection $connectionName not found."

  28:         throw $ErrorMessage

  29:     } else{

  30:         Write-Error -Message $_.Exception

  31:         throw $_.Exception

  32:     }

  33: }

  34:

  35: $sourceContext = (New-AzureStorageContext -StorageAccountName $storageAccountName  -StorageAccountKey $storageAccountKey);

  36:

  37: Write-Output "Starting backup of APIM instance";

  38: Backup-AzureRmApiManagement `

  39:             -ResourceGroupName $resourceGroupName `

  40:             -Name $apimName `

  41:             -StorageContext $sourceContext `

  42:             -TargetContainerName $targetContainerName `

  43:             -TargetBlobName $targetBlobName;

  44:

  45: Write-Output "Backup of APIM completed.";

Here are the description of the variables:

  • $connectionName = “AzureRunAsConnection” – this is the default connection account that was created when the Automation service was provisioned.
  • $storageAccountName = “apimstorebackup” – name of the blob storage account that was created in the first step.
  • $storageAccountKey = “<storage account key>” – the blob store access key obtained from the portal.
  • $resourceGroupName = “APIMService” –  name of the Azure resource group.
  • $apimName = “apimmanager” – the name of the APIM service.
  • $targetContainerName = “backup”  – name of the backup container in blob store.
  • $tartgetBlobName = “AzureAPIM.apimbackup” – file name of the backup file.  This can be omitted and will create a default filename {apimName}-{yyyy-MM-dd-HH-mm}.apimbackup

Once you have confirmed the script executes without any errors, you can now set up a recurring schedule by creating a new schedule in the Automation service blade under Shared Resources.

imageimage

Next you need to link your Runbook to this schedule by double clicking on your runbook name and then the schedule button on the top menu. This will open another blade where you can view all your schedules that you can select from.

image

That is the automated back process completed now. Below is the PowerShell script required to restored the backup file.

#get the storgae context
$sourceContext = (New-AzureStorageContext `
-StorageAccountName “<blob storage name>” `
-StorageAccountKey “<blob storage account key from Azure portal>”)

#restore the backup
Restore-AzureRmApiManagement -ResourceGroupName “<name of resource group>” `
-Name “<name of the APIM service>” `
-StorageContext $sourceContext  `
-SourceContainerName “<blob storage container name>” `
-SourceBlobName “<backup file name>”

More details on these scripts can be found here: https://docs.microsoft.com/en-us/powershell/module/azurerm.apimanagement/restore-azurermapimanagement?view=azurermps-4.3.1

Enjoy.