StockFunctions 1.0

September 25, 2022

I have been investing in several ETF’s (Exchange Traded Fund), and one of them is The Rize Sustainable Future Of Food UCITS ETF. The full fund holdings are downloadable as a .xlsx-file from the company’s site. I thought it would be useful to get a notification or summary of the changes in the fund’s holdings. This was the kickstarter for the first iteration of the StockFunctions-project.

The idea was to get an e-mail notification every day on a scheduled time about the current situation of the fund’s holdings. The e-mail message would have information about any sold holdings, or if there were any recently bought stocks.

Durable Functions

Azure Functions seemed like an ideal choice for this. I could have just created one function with all logic implemented in it, but I wanted to explore the Durable Functions capabilities. I wanted to create separate functions which all had their own responsibility and were independent from each other. When the Orchestration Function was triggered, it would invoke all the functions in order, and catch any exceptions occurred in the process. This is also called function chaining pattern.

The full list of supported Durable Functions application patterns are as follows:

Function chaining
Fan-out/fan-in
Async HTTP APIs
Monitoring
Human interaction
Aggregator (stateful entities)

For more info about Durable Functions, see https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview.

Solution architecture

The solution architecture was pretty simple: Just a plain function app which uses App Service plan and Azure Storage under the hood.

Why did I use time-triggered pipeline in Azure DevOps when I could just used time-triggered Azure Function? Actually, at first, I used it and it worked for some time. However, I encountered some weird issues with it not triggering on schedule, so after banging my head against the wall for a while I decided to use standard HTTP-triggered function and invoke it from Azure DevOps.

I used SendGrid API to send the summary e-mails. SendGrid offers one hundred emails a day for free, so it was a perfect choice for this project.

Function app architecture

Here follows the high-level explanation of the orchestrated process:

DownloadFileFunction fetches new Excel-file from fund company url.
CreateStockHoldingsFromExcelFunction creates StockHolding-entities from the Excel file.
GetExistingHoldingsFunction fetches existing holdings, if any, to Azure Table Storage.
UpdateHoldingsFunction updates new holdings to Azure Table Storage.
CompareStockHoldingsFunction returns the result of comparison between existing and new stock holdings.
GenerateHtmlReportFunction generates HTML-based report from comparison results.
CreateSendGridMessageFunction creates SendGridMessage-entity which is consumed by SendGrid API.
SendSendGridMessageFunction sends the message via SendGrid API.

OrchestratorFunction

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Extensions.Logging;
using StockFunctions.Function.Dto;
using StockFunctions.Function.Factories;
using StockFunctions.Function.Models;
using System;
using System.Threading.Tasks;

namespace StockFunctions.Function.Functions
{
    public class OrchestratorFunction
    {
        [FunctionName("OrchestratorFunction")]
        public async Task Run(
            [OrchestrationTrigger] IDurableOrchestrationContext context,
            ILogger log)
        {
            try
            {
                var orchestratorFunctionArgs = context.GetInput<OrchestratorFunctionArgs>();
                var downloadFileFunctionResult = await context.CallActivityAsync<DownloadFileFunctionResult>(nameof(DownloadFileFunction), orchestratorFunctionArgs.ExcelUrl);
                var createStockHoldingsFromExcelFunctionResult = await context.CallActivityAsync<CreateStockHoldingsFromExcelFunctionResult>(nameof(CreateStockHoldingsFromExcelFunction), downloadFileFunctionResult);
                var getExistingHoldingsFunctionResult = await context.CallActivityAsync<GetExistingHoldingsFunctionResult>(nameof(GetExistingHoldingsFunction), createStockHoldingsFromExcelFunctionResult);
                var updateHoldingsFunctionResult = await context.CallActivityAsync<UpdateHoldingsFunctionResult>(nameof(UpdateHoldingsFunction), getExistingHoldingsFunctionResult);
                var compareStockHoldingsFunctionResult = await context.CallActivityAsync<CompareStockHoldingsFunctionResult>(nameof(CompareStockHoldingsFunction), updateHoldingsFunctionResult);
                var generateHtmlReportFunctionResult = await context.CallActivityAsync<GenerateHtmlReportFunctionResult>(nameof(GenerateHtmlReportFunction), compareStockHoldingsFunctionResult);

                var createSendGridMessageFunctionArgs = CreateSendGridMessageFunctionArgsFactory.Create(generateHtmlReportFunctionResult, orchestratorFunctionArgs);

                var createSendGridMessageFunctionResult = await context.CallActivityAsync<CreateSendGridMessageFunctionResult>(nameof(CreateSendGridMessageFunction), createSendGridMessageFunctionArgs);
                await context.CallActivityAsync(nameof(SendSendGridMessageFunction), createSendGridMessageFunctionResult);
            }
            catch (Exception e)
            {
                log.LogError($"Error encountered while running the orchestrator function: {e.Message}, {e.StackTrace}");
            }
        }
    }
}

DownloadFileFunction

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using StockFunctions.Function.Dto;
using StockFunctions.Function.Services;
using System;
using System.Threading.Tasks;

namespace StockFunctions.Function.Functions
{
    public class DownloadFileFunction
    {
        private readonly IFileDownloaderService _fileDownloaderService;

        public DownloadFileFunction(IFileDownloaderService fileDownloaderService)
        {
            _fileDownloaderService = fileDownloaderService ?? throw new ArgumentNullException(nameof(fileDownloaderService));
        }

        [FunctionName("DownloadFileFunction")]
        public async Task<DownloadFileFunctionResult> Run(
           [ActivityTrigger] IDurableActivityContext context)
        {
            var url = context.GetInput<string>();

            return await _fileDownloaderService.DownloadExcel(url);
        }
    }
}

CreateStockHoldingsFromExcelFunction

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using StockFunctions.Function.Dto;
using StockFunctions.Function.Models;
using StockFunctions.Function.Services;
using System.Threading.Tasks;

namespace StockFunctions.Function.Functions
{
    public class CreateStockHoldingsFromExcelFunction
    {
        private readonly IStockHoldingParserService _stockHoldingParserService;

        public CreateStockHoldingsFromExcelFunction(IStockHoldingParserService stockHoldingParserService)
        {
            _stockHoldingParserService = stockHoldingParserService;
        }

        [FunctionName("CreateStockHoldingsFromExcelFunction")]
        public async Task<CreateStockHoldingsFromExcelFunctionResult> Run(
       [ActivityTrigger] IDurableActivityContext context)
        {
            var file = context.GetInput<DownloadFileFunctionResult>();

            return await Task.FromResult(_stockHoldingParserService.ParseStockHoldingsFromExcel(file.Content));
        }
    }
}

CompareStockHoldingsFunction

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using StockFunctions.Function.Models;
using StockFunctions.Function.Services;
using System;
using System.Threading.Tasks;

namespace StockFunctions.Function.Functions
{
    public class CompareStockHoldingsFunction
    {
        private readonly IStockHoldingComparerService _stockHoldingComparerService;

        public CompareStockHoldingsFunction(IStockHoldingComparerService stockHoldingComparerService)
        {
            _stockHoldingComparerService = stockHoldingComparerService ?? throw new ArgumentNullException(nameof(stockHoldingComparerService));
        }

        [FunctionName("CompareStockHoldingsFunction")]
        public async Task<CompareStockHoldingsFunctionResult> Run(
        [ActivityTrigger] IDurableActivityContext context)
        {
            var result = context.GetInput<UpdateHoldingsFunctionResult>();
            return await Task.FromResult(_stockHoldingComparerService.CompareHoldings(result.ExistingStockHoldings, result.NewStockHoldings));
        }
    }
}

GetExistingHoldingsFunction

using Microsoft.Azure.Cosmos.Table;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using StockFunctions.Function.Models;
using StockFunctions.Function.Services;
using System.Threading.Tasks;

namespace StockFunctions.Function.Functions
{
    public class GetExistingHoldingsFunction
    {
        private readonly IStockHoldingService _stockHoldingService;

        public GetExistingHoldingsFunction(IStockHoldingService stockHoldingService)
        {
            _stockHoldingService = stockHoldingService;
        }

        [FunctionName("GetExistingHoldingsFunction")]
        public async Task<GetExistingHoldingsFunctionResult> Run(
      [ActivityTrigger] IDurableActivityContext context,
      [Table("stockholdings")] CloudTable cloudTable)
        {
            return await _stockHoldingService.GetExistingHoldings(cloudTable);
        }
    }
}

UpdateHoldingsFunction

using Microsoft.Azure.Cosmos.Table;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using StockFunctions.Function.Models;
using StockFunctions.Function.Services;
using System.Threading.Tasks;

namespace StockFunctions.Function.Functions
{
    public class UpdateHoldingsFunction
    {
        private readonly IStockHoldingService _stockHoldingService;

        public UpdateHoldingsFunction(IStockHoldingService stockHoldingService)
        {
            _stockHoldingService = stockHoldingService;
        }

        [FunctionName("UpdateHoldingsFunction")]
        public async Task<UpdateHoldingsFunctionResult> Run(
        [ActivityTrigger] IDurableActivityContext context,
        [Table("stockholdings")] CloudTable cloudTable)
        {
            var input = context.GetInput<UpdateHoldingsFunctionArgs>();

            return await _stockHoldingService.UpdateWithNewHoldings(cloudTable, input.NewHoldings, input.ExistingHoldings);
        }
    }
}

CompareStockHoldingsFunction

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using StockFunctions.Function.Models;
using StockFunctions.Function.Services;
using System;
using System.Threading.Tasks;

namespace StockFunctions.Function.Functions
{
    public class CompareStockHoldingsFunction
    {
        private readonly IStockHoldingComparerService _stockHoldingComparerService;

        public CompareStockHoldingsFunction(IStockHoldingComparerService stockHoldingComparerService)
        {
            _stockHoldingComparerService = stockHoldingComparerService ?? throw new ArgumentNullException(nameof(stockHoldingComparerService));
        }

        [FunctionName("CompareStockHoldingsFunction")]
        public async Task<CompareStockHoldingsFunctionResult> Run(
        [ActivityTrigger] IDurableActivityContext context)
        {
            var result = context.GetInput<UpdateHoldingsFunctionResult>();
            return await Task.FromResult(_stockHoldingComparerService.CompareHoldings(result.ExistingStockHoldings, result.NewStockHoldings));
        }
    }
}

GenerateHtmlReportFunction

using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs;
using StockFunctions.Function.Models;
using System;
using System.Threading.Tasks;
using StockFunctions.Function.Services;

namespace StockFunctions.Function.Functions
{
    public class GenerateHtmlReportFunction
    {
        private readonly IHTMLReportGeneratorService _htmlReportGeneratorService;

        public GenerateHtmlReportFunction(IHTMLReportGeneratorService htmlReportGeneratorService)
        {
            _htmlReportGeneratorService = htmlReportGeneratorService ?? throw new ArgumentNullException(nameof(htmlReportGeneratorService));
        }

        [FunctionName("GenerateHtmlReportFunction")]
        public async Task<GenerateHtmlReportFunctionResult> Run(
      [ActivityTrigger] IDurableActivityContext context)
        {
            var input = context.GetInput<CompareStockHoldingsFunctionResult>();
            return await Task.FromResult(_htmlReportGeneratorService.GenerateHTMLReport(input));
        }
    }
}

CreateSendGridMessageFunction

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using StockFunctions.Function.Models;
using StockFunctions.Function.Services;
using System;

namespace StockFunctions.Function.Functions
{
    public class CreateSendGridMessageFunction
    {
        private readonly ISendGridMessageService _sendGridMessageService;

        public CreateSendGridMessageFunction(ISendGridMessageService sendGridMessageService)
        {
            _sendGridMessageService = sendGridMessageService ?? throw new ArgumentNullException(nameof(sendGridMessageService));
        }

        [FunctionName("CreateSendGridMessageFunction")]
        public CreateSendGridMessageFunctionResult Run(
          [ActivityTrigger] IDurableActivityContext context)
        {
            var input = context.GetInput<CreateSendGridMessageFunctionArgs>();

            return _sendGridMessageService.Create(input.Message, input.ReportSubject, input.SendReportFrom, input.SendReportTo);
        }
    }
}

SendSendGridMessageFunction

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using SendGrid.Helpers.Mail;
using StockFunctions.Function.Models;

namespace StockFunctions.Function.Functions
{
    public class SendSendGridMessageFunction
    {
        [FunctionName("SendSendGridMessageFunction")]
        public void Run(
          [ActivityTrigger] IDurableActivityContext context,
          [SendGrid(ApiKey = "SendGridAPIKey")] out SendGridMessage message)
        {
            var input = context.GetInput<CreateSendGridMessageFunctionResult>();

            message = input.Message;
        }
    }
}

As you might notice, the functions themselves do not do much else than invoke corresponding service. This might be a little bit overkill here, because the service’s functionality is not really reused across functions, but at least the functions look super clean.

The services are registered using DI (dependency injection) using FunctionsStartup:

using AutoMapper;
using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection;
using StockFunctions.Function;
using StockFunctions.Function.Factories;
using StockFunctions.Function.Services;

[assembly: FunctionsStartup(typeof(Startup))]
namespace StockFunctions.Function
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            builder.Services.AddHttpClient();

            builder.Services.AddLogging();
            builder.Services.AddScoped<IStockHoldingParserService, StockHoldingParserService>();
            builder.Services.AddScoped<IFileDownloaderService, ExcelDownloaderService>();
            builder.Services.AddScoped<IStockHoldingComparerService, StockHoldingComparerService>();
            builder.Services.AddScoped<ICloudTableServiceFactory, CloudTableServiceFactory>();
            builder.Services.AddScoped<IHTMLReportGeneratorService, HTMLReportGeneratorService>();
            builder.Services.AddScoped<IStockHoldingService, StockHoldingService>();
            builder.Services.AddScoped<ISendGridMessageService, SendGridMessageService>();

            builder.Services.AddAutoMapper(typeof(Startup));
        }
    }
}

In unit tests I wanted to explore AutoFixture’s functionalities.

using AutoFixture.Xunit2;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Moq;
using StockFunctions.Function.Dto;
using StockFunctions.Function.Functions;
using StockFunctions.Function.Services;
using StockFunctions.Function.Tests.Utils;
using System.Threading.Tasks;
using Xunit;

namespace StockFunctions.Function.Tests.Functions
{
    public class DownloadFileFunctionTests
    {
        [Theory, AutoMoqData]
        public async Task Run_ValidParameters_ShouldCallCorrectServices(
            [Frozen] Mock<IFileDownloaderService> fileDownloaderService,
            Mock<IDurableActivityContext> durableActivityContext,
            DownloadFileFunction downloadFileFunction,
            DownloadFileFunctionResult downloadFileFunctionResult,
            string request)
        {
            durableActivityContext.Setup(x => x.GetInput<string>())
                .Returns(request);

            fileDownloaderService.Setup(x => x.DownloadExcel(It.IsAny<string>()))
                .ReturnsAsync(downloadFileFunctionResult);

            var result = await downloadFileFunction.Run(durableActivityContext.Object);
            Assert.Equal(downloadFileFunctionResult, result);

            fileDownloaderService.Verify(x => x.DownloadExcel(request), Times.Once);
        }
    }
}

I am using custom AutoMoqData attribute, which allows me to use AutoFixture and Moq together.

using AutoFixture;
using AutoFixture.AutoMoq;
using AutoFixture.Xunit2;

namespace StockFunctions.Function.Tests.Utils
{
    public class AutoMoqDataAttribute : AutoDataAttribute
    {
        public AutoMoqDataAttribute()
          : base(() => new Fixture().Customize(new AutoMoqCustomization()))
        {
        }
    }
}

The Frozen attribute is used to “freeze” corresponding type and indicate that the same instance should be returned every time the IFixture creates an instance of that type. More info on AutoFixture’s quick start page https://autofixture.github.io/docs/quick-start.

Example of sent e-mail message

CI/CD pipeline

The Build stage of the pipeline has pretty standard steps: Build, test and publish artifacts.

The Deploy stage consists of Bicep definitions from which the Azure resources are created. I am a big fan of IaC (Infrastructure as Code) and especially Bicep. Bicep makes it really easy to separate your resource definitions into digestable modules.

The last step in the Deploy stage is the Function app deployment.

name: NetCore-CI-azure-pipeline.yml
trigger:
  branches:
    include:
      - master

pool:
  vmImage: "ubuntu-latest"

variables:
  - group: Development

stages:
  - stage: Build
    displayName: Build solution
    jobs:
      - job: Build
        displayName: Build and publish solution
        steps:
          - task: UseDotNet@2
            inputs:
              packageType: "sdk"
              version: "3.x"
            displayName: "Use .NET Core SDK 3.x"

          - task: DotNetCoreCLI@2
            inputs:
              command: "restore"
              projects: "$(SolutionPath)"
            displayName: "Restore NuGet packages"

          - task: DotNetCoreCLI@2
            inputs:
              command: "build"
              projects: "$(SolutionPath)"
            displayName: "Build solution"

          - task: DotNetCoreCLI@2
            inputs:
              command: "test"
              projects: "$(SolutionPath)"
            displayName: "Run tests"

          - task: DotNetCoreCLI@2
            inputs:
              command: "publish"
              publishWebProjects: false
              projects: "$(SolutionPath)"
              arguments: "--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)/$(buildConfiguration)"
            displayName: "Publish solution"

          - task: PublishBuildArtifacts@1
            inputs:
              PathtoPublish: "$(Build.ArtifactStagingDirectory)"
              ArtifactName: "drop"
              publishLocation: "Container"
            displayName: "Publish build artifacts"

          - task: PublishBuildArtifacts@1
            inputs:
              PathtoPublish: "$(Build.SourcesDirectory)/deployment"
              ArtifactName: "deployment"
              publishLocation: "Container"
            displayName: "Publish deployment files"

  - stage: Deploy
    displayName: Deploy solution
    jobs:
      - job: Deploy
        steps:
          - download: current
            artifact: drop
          - task: AzureCLI@2
            inputs:
              azureSubscription: "Pay-As-You-Go (68ba0741-2984-46af-872e-8597f8a6ec37)"
              scriptType: "pscore"
              scriptLocation: "inlineScript"
              inlineScript: |
                az deployment group create `
                --resource-group $(resourceGroup) `
                --template-file .\main.bicep `
                --parameters 'sendGridApiKey=$(sendGridApiKey)' `
                'sendReportTo=$(sendReportTo)' `
                'sendReportFrom=$(sendReportFrom)' `
                'reportSubject=$(reportSubject)'
              workingDirectory: "deployment"
            displayName: "Bicep deployment"

          - task: AzureFunctionApp@1
            inputs:
              azureSubscription: "Pay-As-You-Go (68ba0741-2984-46af-872e-8597f8a6ec37)"
              appType: "functionApp"
              appName: "$(funcAppName)"
              package: "$(Pipeline.Workspace)/drop/**/*.zip"
              deploymentMethod: "auto"
            displayName: "Deploy function app"

What I learned from this project

Always use Application Insights with Azure Functions. Debugging of previous runs was impossible without Application Insights as the logs were not saved anywhere. I could not find reliable information about this, but I guess previously it was possible to view these logs in the linked Storage account. For development purposes, you can always use Log Stream functionality in Azure.
Use the latest possible version of Azure Functions host and .NET. I encountered many issues, such as ‘Could not load file or assembly xxx in Azure Functions’-error with certain packages (AutoMapper) on Azure when using Azure Functions host version 3 and .NET Core 3.1. With Azure Functions host v4 and .NET 6 everything worked like a charm.
Do not use disallowed characters in Azure Table Storage’s PartitionKey and RowKey properties (see https://learn.microsoft.com/fi-fi/rest/api/storageservices/Understanding-the-Table-Service-Data-Model#characters-disallowed-in-key-fields). I learned this the hard way, when the holding data contained a company with a forward slash in its name, which caused the whole function to crash.

View project’s source code in https://dev.azure.com/jacker92/BlogAssets/_git/StockFunctions-v1