Azure Durable Functions

I will be exploring  Azure Functions and recently with more formal documentation, looking at durable functions , which are starting to make sense to me as valid pieces of a tech stack.

So what are Azure functions?

They are functions that might look similar  to a micro-service, they work based on some type of trigger whether its a timer  a queue, or an http request. They can be bound to many data types.

They are server less, so you don’t need to provision a setup for these functions. Azure will determine the function’s needs based on how many events run, and its activity,  so it’s consumption based.

What are durable functions?

imagine a chain of events that needs to run, or an event that triggers multiple other events,(Fan out , fan in) and  it needs to have something manage\track\ensure  reliable completion of that process. This  concept of having something orchestrate those multiple activities is what durable functions revolve around. Durable functions will store a state to track its current state, values returned , and completed events.

The key as I mentioned is the orchestrator  or durable function because it defines the workflow, triggers all the activity functions , and put itself to sleep  while it waits for events to complete. In the end it uses Azure storage to store  the current queues , tables to store state and finally uses as concept known as event sourcing  to track what events have been worked on; its an append only  list of events that have already been successfully run.

So durable functions store all this in a storage account, which when you set it up in a production environment you would need to provide a connection string  . Durable Functions refer to this persistence layer as a Task  Hub.

Setting up Durable Functions

Local setup requirements are:

1. Visual Studio  (15.3 or greater )
2. Include  “Azure  Developer ” in setup of  Visual Studio 2017
3. Ensure you have Microsoft Azure  Storage Emulator

Setting up your project for Durable functions

1. Create a new  Azure Functions Project
2. in the second screen I selected Azure v1 functions, which has a little more flexibility but you are free to use azure .net core functionality.
3. Wait for project startup to completed.
4. Install Azure.WebJobs.Extensions.DurableTask (or if .net core there is a Core package).

5. 

Orchestrator Function Rules & Constraints

The recommended\prescribed solution for durable functions is to prep the data you need and pass it through to the function.

Durable Orchestrator Functions must be non-blocking.
So don’t block  the thread, or perform operations which you would normally have to await for, you should do those in your activity functions and then wait on them.

Don’t create infinite loops.
The easiest way to avoid this is to use the orchestrator function as a run through process  (which can be run infinite # of times independent of each other) but if you need something to continue to run  a workflow forever  and infinitely looped that could be done using ContinueAsNew.

Logging  in a Function Orchestrator

Recommended that you should use the built-in TraceWriter.  Consider that every Log can be rewritten because of how an orchestrator is re-run each time so you can use the attribute DurableOrchestrationContext.IsReplaying to see if its replaying  this specific line.

Application Creation

I created  a Orchestrator Starter, an Orchestrator, and then I created a class with mock Activity Functions I  wanted to run. It is a good idea to have an orchestrator starter as this can do all the preliminary work to prepare the data for the orchestrator.

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using System.Threading.Tasks;

namespace DurableFunctions
{
    public static class ProcessVideoOrchestrator
    {
        [FunctionName("O_ProcessVideo")]
        public static async Task<object> ProcessVideo(
            [OrchestrationTrigger] DurableOrchestrationContext context, /* tells azure runtime that this is an orchestrator function */
            TraceWriter log 
            )
        {
            var videoLocation = context.GetInput<string>();
            if (!context.IsReplaying) { 
                log.Info("Start A_TranscodeVideo");
            }
            var transcodeLocation   = await context.CallActivityAsync<string>("A_TranscodeVideo", videoLocation); // will put process to sleep while it waits
            if (!context.IsReplaying)
            {
                log.Info("Start A_getThumbnail");
            }

            var thumbnailLocation =  await context.CallActivityAsync<string>("A_getThumbnail", transcodeLocation);
            if (!context.IsReplaying)
            {
                log.Info("Start A_prependIntro");
            }
            var introLocation = await context.CallActivityAsync<string>("A_prependIntro", transcodeLocation);
            return new {transcodeLocation, thumbnailLocation,introLocation};
        }
    }
}

using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;

namespace DurableFunctions
{
    public static class ProcessorActivities
    {
        [FunctionName("A_TranscodeVideo")]
        public static async Task<string> TranscodeVideo( [ActivityTrigger] string inputVideo, TraceWriter log)
        {
            log.Info($" Transcoding {inputVideo}" );
            await Task.Delay(5000);
            return "transcoded.mp4";
        }
        [FunctionName("A_getThumbnail")]
        public static async Task<string> GetThumbnail([ActivityTrigger] string transcoded, TraceWriter log)
        {
            log.Info($" generating Thumbnail for {transcoded}");
            await Task.Delay(5000);
            return "thumbnail.jpg";
        }
        [FunctionName("A_prependIntro")]
        public static async Task<string> PrependIntro([ActivityTrigger] string transcoded, TraceWriter log)
        {
            log.Info($" generating prepended intro for {transcoded}");
            await Task.Delay(5000);
            return "prependedvideo.mp4";
        }

    }
}

Output

{
id: "d96145604927468ba7fb747582d3e49a",
statusQueryGetUri: "http://localhost:7071/admin/extensions/DurableTaskExtension/instances/d96145604927468ba7fb747582d3e49a?taskHub=DurableFunctionsHub&connection=Storage&code=dsYv5U1Rjw6IaQUBMEtEYiGGfhEWjMW8iPcre7QS0aub3GjL6LFnbw==",
sendEventPostUri: "http://localhost:7071/admin/extensions/DurableTaskExtension/instances/d96145604927468ba7fb747582d3e49a/raiseEvent/{eventName}?taskHub=DurableFunctionsHub&connection=Storage&code=dsYv5U1Rjw6IaQUBMEtEYiGGfhEWjMW8iPcre7QS0aub3GjL6LFnbw==",
terminatePostUri: "http://localhost:7071/admin/extensions/DurableTaskExtension/instances/d96145604927468ba7fb747582d3e49a/terminate?reason={text}&taskHub=DurableFunctionsHub&connection=Storage&code=dsYv5U1Rjw6IaQUBMEtEYiGGfhEWjMW8iPcre7QS0aub3GjL6LFnbw=="
}

{
instanceId: "d96145604927468ba7fb747582d3e49a",
runtimeStatus: "Completed",
input: "this.mp4",
customStatus: null,
output: {
transcodeLocation: "transcoded.mp4",
thumbnailLocation: "thumbnail.jpg",
introLocation: "prependedvideo.mp4"
},
createdTime: "2018-07-13T18:31:42Z",
lastUpdatedTime: "2018-07-13T18:32:00Z"
}

So its important to understand that the function will return a traceable response, but you still need to get the data from the statusQueryGetUri.

In my next post I am going to explore into Durable Function rest API function.

Please note the information provided here was heavily based on  Mark Heath‘s excellent   Durable Functions Fundamentals  in Pluralsight.

Valuable Links

Durable Functions Repository
Durable Functions Documentation
Durable Functions  Project