One of my most
valuable take from Integrate 2019 is the tip from Derek Li regarding fault code
429, Rate Limit Exceeded, in Logic Apps.
Why is this valuable? Well we have some integrations where we pick up files from on-prem via the File Connector and Azure Data Gateway. Sometimes files arrives in bursts, maybe 200-300 files at the same time. The problem we get then is 429. Especially when the retry is exceeded and the Logic App fails.
So back home from London and Integrate 2019 I tried the tip from Derek Li and it worked like a charm.
One backdraft is that the designer cannot decide which connector being used (as you can see in the picture).
The trick is to define a number of api connections (5 in this case) and when using the connection in “Get_file_content_using_path” name it dynamically like this: “@parameters(‘$connections’)[concat(‘filesystem_’,rand(1,6))][‘connectionId’]”. One of the connection “filesystem_1” to “filesystem_5” will then be used. Randomly in each loop. Below is a snippet of the ARM template.
In my current project we had an integration which should pick up files based on a name pattern. When we tested the integration in our test subscription it looked just fine, some files matched and some did not. After a few weeks in production we noticed an interesting thing. A lot of actions were run and cost was much higher than normal integrations and in test. When we looked into the reason why, we found that there were a lot more files and folders (with files in them) that did not match. This led to a lot of loops (For-Each action) with the condition to see if the file matched. Our integration did the job but at a high cost.
After some research we found a way to optimize the workflow. Logic Apps Filter Array, or Query Action which is the name in the documentation, helped us to reduce the number of actions. Just put the Filter Array after the List files in folder action and do the For each on the output from Filter array. When doing this you can also skip the Condition action within the For each.
Logic Apps Filter array before For each is a nice little pattern which reduces the number of actions, especially if you have a lot of items that do not match the filter.
One thing we miss with the Query action is that you cannot build advanced filters in the same way as you now can do with the Conditon action. Of course you can do it in codeview or even advanced mode. Maybe it is just a matter of time before we get the GUI part as in Condition.
More information about the Query Action is found here.
My customer asked me to create a list of all servers that our Logic Apps are either receiving from or sending to. In BizTalk this is easy, just go to All Artifacts and choose Send Ports or Receive Locations. In the Azure Portal there is no such a thing (as far as I know). Instead you must check the Azure API Connections and choose “Edit”. On the edit page you will see the properties, “Host Server Address” in this case.
Azure API Connection
Solution with Powershell
So to be able to complete my task I wrote this script in Powershell. The script fetches all the resources within an subscription and for each API Connection it gets the information needed.
The script is based on Get-AzureRmOperationalInsightsSearchResults cmdlet so Log Analytics and OMS must be enabled, You can read more about how to get started with Log Analytics and OMS in the Microsoft Azure Documentation pages.
Some input are required by the script:
ResourceGroup – the name of the resource group where the OMS workspace is located
Workspace – the name of the OMS workspace
Workflow – the name of the Logic App workflow
Action – the name of the actual action within the Logic App
The output from the script includes the datetime, status and the actual runId, which can be used when searching for more details in the Logic Apps runs history or with another script (maybe in a future blog post).
This is one way of doing it, of course you can go in to the OMS portal and do the search but I am too lazy for that, PowerShell scripts to the rescue…
Logic Apps polling for files or messages can be rather costly if the requirements are such that the file or message must be taken care of within shortly.
Polling every 10th second (one Logic App) for one month will generate 6 * 60 * 24 * 30 => 259200 actions.
The monthly cost for this will be 250000 * 0,000675 + 9200 * 0,000338 = 171,8596 Euro Actual prices 2017-08-11, See Logic App Pricing
Suppose we have at least 10 scenarios like this and the yearly cost will be approximately 172 * 10 * 12 = 20640 Euro
One way to eliminate (almost) the polling cost is to let Azure Function triggers do the job. The cost for that is almost nothing (less than one Euro), only cost for storage account and execution of function when the function is triggered.
So how do we do this?
In this example we want to take action when a new file is stored in a Dropbox folder.
Create a new Azure function and choose External File as input trigger. Select Dropbox as the API Connection.
Create a message to the Logic App containing data about the actual file. The function will then call the Logic App. Function:
The Logic App is triggered by the call from the Azure Function and the file can be processed.