Browsed by
Category: Technology

The world of technology is one of innovation and excitement. Technology for me is the one force in the world that can truly transform lives and improve the world. For me, its the one thing where we build something that leaves the world a better place.

Leveraging Private Dev Containers

Leveraging Private Dev Containers

So this should be a pretty quick post, but I thought I would share this tip and trick I found while playing around with the implementation.

Dev Containers really are an amazing advancement in the development tools that are out there. Gone are the days of being handed a massive document and spending a day or two configuring your local development machine.

Dev containers make it easy to leverage docker to spin up and do development inside a container, and then putting that container reference into your repo to be rendered.

Now, the problem becomes, what if you have private python packages or specific internal tools that you want to include in your dev container? What can you do to make it easier to developer to leverage?

The answer is, you can host a container image on a registry that is private and exposed to your developers via their azure subscription. The benefit to this is it makes it easy to standardize the dev environment with internal tools, and make it easy to spin up new environments without issue.

So the question becomes “How?” And the answer is a pretty basic one. If you follow the spec defined here, then you will see in the devcontainer spec for the json file, there is an initializeCommand option, which allows you to specify a bash script to run during the initialization of the container.

But inside that script, you can add the following commands to make sure your dev container works:

az login --use-device-code
az acr login --name {your registry name}
docker pull {repository/imagename}:latest

And then when you build the DockerFile, you just point to your private registry. This means that whenever your team is able to start up their dev container, they will get a login prompt to enter the code, and log into the private docker registry. And that’s it!

Building a magic mirror – Part 1 – The project

Building a magic mirror – Part 1 – The project

For this post, I thought I would share a project I’ve been working on for my family. For our family, we have a marker board in the kitchen that helps keep track of everything going on for myself, my wife and our kids. And while this is great in practice and does help. The fact that this is analog has been driving me nuts for YEARS. So I wanted to see if we could up this with a magic mirror.

Now I have a magic mirror in my office that I use to help stay focused, and I have here how I manage it via Azure Dev Ops. But I’ve never really done a post detailing how I built this mirror for those interested.

First Hardware Requirements:

In my case I’m using an old Raspberry Pi 3 that I happen to have just sitting around the office, and I’ve installed the Raspberry Pi linux OS on that device.

Outside of that, I’ve got the basics:

  • Power supply cable
  • HDMI Capable
  • Monitor
  • 64 GB Micro SD card
  • SD Card Reader

Now I have plans to hook this to a larger TV when I set it up in the kitchen, but for right now I’ve just got a standard monitor.

Goal of the Project

For me, I found this video on YouTube, and thought it was pretty great, so this is my starting point for this project.

Setting up the Raspberry PI – Out-of-the-Box

To download and install the OS, I used the Raspberry Pi image manager found here. I used the SD card reader I had in the office to format the SD card, and then install the OS.

Once that was completed, I booted up my raspberry pi, and finished the setup which involved the following (there is a wizard to help with this part):

  • Configure localization
  • Configure Wifi
  • Reset password
  • Reset Hostname
  • Download / Install Updates

Finally, one step I did to make my life easier is to enable SSH to the raspberry pi, which allows me to work on it from my laptop rather than setting up a keyboard / monitor / mouse permanently.

You do this by going to the “Settings” on the Raspberry Pi, and going to the “Interface” tab, and selecting “Enable” for “SSH”.

Now that my Raspberry Pi is running, we come to the meat of this, and that’s getting the magic mirror running:

Step 1 – Install Node.js

You need Node.JS to run everything about the magic mirror, so you can start by running these commands against your raspberry pi:

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt install -y nodejs

From there, I cloned the magic mirror repo to Raspberry Pi.

git clone https://github.com/MichMich/MagicMirror

Then enter the repo from the command prompt:

cd MagicMirror/

Then you need to install NPM to be able to work with the MagicMirror installation. This takes the longest, and the first time I ran this I actually had to use sudo to make sure it completed the install.

sudo npm install

A good recommendation from the magic mirror site is to copy the default config so you have a backup. You can do that with this command:

cp config/config.js.sample config/config.js

Finally you can start your MagicMirror with:

npm run start

Now, the next part was tricky, if you reboot your RaspberryPi, the magic mirror will not start automatically, and you need to do some more configuration to make that happen. Most documentation will tell you to use pm2, and I would agree with that, but if you try to run the commands on the most recent Raspberry Pi, you’ll find that pm2 is not installed. You can resolve that with this command:

npm install pm2 -g

Then run the following commands to configure your MagicMirror to run on startup.

pm2 startup

After running this command you will be given a command to run to enable pm2 on startup, run this command.

Then run the following:

cd ~
nano magicmirror.sh

Put the following in the magicmirror.sh file, and then hit Ctrl-X then Y

cd ~/MagicMirror
DISPLAY=:0 npm start

Finally run these commands to finish configuration

chmod +x magicmirror.sh
pm2 start magicmirror.sh

pm2 save
sudo reboot

After that you’ll see your monitor displaying the default configuration for the MagicMirror.

Next post I’ll walk you through the steps I took for configuring some of the common modules to get it working.

Reconciling the Temporal Nature of Software Development

Reconciling the Temporal Nature of Software Development

Stop me if you heard this one before, you are working on a code project, and you really want to make this one perfect. You spend a ton of time brainstorming ideas to make this code “future proof”, this code is going to be elegant and perfect, and the code that will last forever.

Or here’s another one, you open up a project you worked on a long time ago, and when you look at the code it is just awful. I once saw a quote in a software comment that says, “When I wrote this, only God and I knew how it worked, now only God knows.”

Now both of these statements are extremely common among a lot of engineers I know and have worked with, and I’ve fallen into these traps myself (a lot), way more than I care to admit.

But over the past year, I’ve come to the conclusion, that these types of behaviors are fool’s errands.

I can’t tell you the number of times that I’ve seen this confirmed for, and that these type scenarios never lead to a positive outcome. And if we’re being honest, the past decade with the adoption of agile processes, in many ways addressed these same problems. Waterfall as a development methodology is built on this falsehood, so why do we continue to chase this “Holy Grail” that always turns out so poorly.

Realistically, I have found that these situations lead to either:

  • Setting unrealistic expectations and forcing additional stress on yourself to deliver.
  • Making it really easy to fall into imposter syndrome.

Now I’m not writing this and claiming to be some kind of psychology expert, all I wanted to do here is share my own thoughts and feelings on this, and honestly your mileage may vary, but this is based on my experience.

The simple fact I’ve come to realize is that Software Development is a temporal activity, like any act of creation. At the end of the day, all you are capable of doing is creating the best code that you can at the present moment. Period.

Any act of creation, whether it’s writing, art, architecture, etc, all have one thing in common, once you go through the process, they are frozen in time. Let’s face it, the Mona Lisa isn’t going to change, and any work being done on it is strictly to maintain it.

When you boil it down, at the project level, Agile focuses on this through the concept of “Definition of Done”, or a “just ship it” mentality.

But I would argue that this mindset needs to extend much further to help prevent ourselves from inflicting burnout upon ourselves. Carol Dweck talks about this in her growth mindset to a certain extent, specifically questioning the idea that talent is fixed, and pointing out that we as humans have the ability to grow in our ability to do the things we care about.

Let me give you an example, there are whole college courses that talk about the differences between Van Gough’s work over the course of his career. The simple fact is that every day we get better at our craft. So ultimately, it’s important to embrace that coding is no different.

My point in this post is this…it’s not worth putting the stress on yourself to build something that’s going to “stand the test of time.” Remember that at the end of the day, the best you can do is the intersection of these constraints:

  • Resources – The tools you have at your disposal.
  • Skill – Your ability to do the thing you are trying to do.
  • Knowledge – Your knowledge of the problem being addressed, or the environment your work will live in.
  • Time – How much time you have to create the thing.
  • Focus – The number of distractions getting in your way.
  • Desire – How much your heart is in the work.

These 6 pillars are ultimately the governing constraints that will determine the level of work your going to do.

I have been writing and re-writing this post for a while, but as we approach the end of 2021, I’m feeling reflective. My point is this, when you do your work, or build your solution you need to embrace the idea that you are not going to build the digital equivalent of the Great Wall of China, and your work is not going to stand the test of time. There will be new technologies, techniques, and even learnings you will be bringing back. So don’t put yourself through that pain, rather do the best job you can, within those 6 pillars, and move onto the next thing.

When you start to consider this, if you’re like me, you will realize that you are free to do the best you can, and not put that additional pressure on yourself.

The joke I tell people is this:

  • Past Kevin is an idiot who writes terrible code and makes bad choices. And he likes to procrastinate.
  • Future Kevin is a genius who can solve any problem in an elegant and amazing manner.
  • Present Kevin is stuck in the middle, doing the best he can, and trying to figure out how to get it done.

I wish you all the best in the coming year, and hope you have a great holidays.

Cool Nerdy Gift Idea – Word Cloud

Cool Nerdy Gift Idea – Word Cloud

The holidays are fast approaching, and this year I had a really cool idea for a gift that turned out well, and I thought I would share it. For the past year and a half, I’ve had this thing going with my wife where every day I’ve sent her a “Reason X, that I love you…” and it’s been a thing of ours that’s been going for a long time (up to 462 at the time of this post).

But what was really cool was this year for our anniversary I decided to take a nerdy approach to making something very sentimental but easy to make. Needless to say, it was very well-received, and I thought I would share.

What I did was used Microsoft Cognitive Services and Power BI to build a Word Cloud based on the key words extracted from the text messages I’ve sent her. Microsoft provides a cognitive service that does text analytics, and if you’re like me you’ve seen sentiment analysis and other bots before. But one of the capabilities, is Key Word Extraction, which is discussed here.

So given this, I wrote a simple python script to pull in all the text messages that I exported to csv, and run them through cognitive services.

from collections import Counter
import json 

key = "..."
endpoint = "..."

run_text_analytics = True
run_summarize = True 

from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

class KeywordResult():
    def __init__(self, keyword, count):
        self.keyword = keyword
        self.count = count 

# Authenticate the client using your key and endpoint 
def authenticate_client():
    ta_credential = AzureKeyCredential(key)
    text_analytics_client = TextAnalyticsClient(
            endpoint=endpoint, 
            credential=ta_credential)
    return text_analytics_client

client = authenticate_client()

def key_phrase_extraction(client):

    try:
        if (run_text_analytics == True):
            print("Running Text Analytics")
            with open("./data/reasons.txt") as f:
                lines = f.readlines()

                responses = []
                for i in range(0, len(lines),10):
                    documents = lines[i:i+10]
                    response = client.extract_key_phrases(documents = documents)[0]

                    if not response.is_error:
                        for phrase in response.key_phrases:
                            #print("\t\t", phrase)
                            responses += [phrase]
                    else:
                        print(response.id, response.error)
                # for line in lines:
                #     documents = [line]
                    

                with open("./data/output.txt", 'w') as o:
                    for respone_line in responses:
                        o.write(f"{respone_line}\n")
            print("Running Text Analytics - Complete")
        
        if (run_summarize == True):
            print("Running Summary Statistics")
            print("Getting output values")
            with open("./data/output.txt") as reason_keywords:
                keywords = reason_keywords.readlines()
                keyword_counts = Counter(keywords)
                print("Counts retrieved")

                print("Building Keyword objects")
                keyword_list = []
                for key, value in keyword_counts.items():
                    result = KeywordResult(key,value)
                    keyword_list.append(result)
                print("Keyword objects built")

                print("Writing output files")
                with open("./data/keyword_counts.csv","w") as keyword_count_output:
                    for k in keyword_list:
                        print(f"Key = {k.keyword} Value = {k.count}")
                        print()
                        key_value = k.keyword.replace("\n","")
                        result_line = f"{key_value},{k.count}\n"
                        keyword_count_output.write(result_line)
                print("Finished writing output files")
         
    except Exception as err:
        print("Encountered exception. {}".format(err))
        
key_phrase_extraction(client)

Now with the above code, you will need to create a text analytics cognitive service, and then populate the endpoint and the key provided. But the code will take each row of the document and run it through cognitive services (in batches of 10) and then output the results.

From there, you can open up Power BI and point it at the text document provided, and connect the Word Cloud visual, and you’re done. There are great instructions found here if it helps.

It’s a pretty easy gift that can be really amazing. And Happy Holidays!

How to leverage templates in YAML Pipelines

How to leverage templates in YAML Pipelines

So now secret that I really am a big fan of leveraging DevOps to extend your productivity. I’ve had the privilege of working on smaller teams that have accomplished far more than anyone could have predicted. And honestly the key principle that is always at the center of those efforts is treat everything as a repeatable activity.

Now, if you look at the idea of a micro service application, at it’s core its several different services that are independently deployable, and at it’s core that statement can cause a lot of excessive technical debt from a DevOps perspective.

For example, if I encapsulate all logic into separate python modules, I need a pipeline for each module, and those pipelines look almost identical.

Or if I’m deploying docker containers, my pipelines for each service likely look almost identical. See the pattern here?

Now imagine, you do this and build a robust application with 20-30 services running in containers. In the above, that means if I have to change their deployment pipeline, by adding say a new environment, I have to update between 20 – 30 pipelines, with the same changes.

Thankfully, ADO has an answer to this, in the use of templates. The idea here is we create a repo within ADO for our deployment templates, which contain the majority of the logic to deploy our services and then call those templates in each service.

For this example, I’ve built a template that I use to deploy a docker container and push it to a container registry, which is a pretty common practice.

The logic to implement it is fairly simple and looks like the following:

resources:
  repositories:
    - repository: templates
      type: git
      name: "TestProject/templates"

Using the above code will enable your pipeline to pull from a separate git repo, and then you can use the following to code to create a sample template:

parameters:
  - name: imageName
    type: string
  
  - name: containerRegistryName
    type: string

  - name: repositoryName
    type: string

  - name: containerRegistryConnection
    type: string

  - name: tag
    type: string

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: 'docker build -t="${{ parameters.containerRegistryName }}/${{ parameters.imageName }}:${{ parameters.tag }}" -t="${{ parameters.containerRegistryName }}/${{ parameters.imageName }}:latest" -f="./Dockerfile" .'
    workingDirectory: '$(Agent.BuildDirectory)/container'
  displayName: "Building docker container"

- task: Docker@2
  inputs:
    containerRegistry: '${{ parameters.containerRegistryConnection }}'
    repository: '${{ parameters.imageName }}'
    command: 'push'
    tags: |
      $(tag)
      latest
  displayName: "Pushing container to registry"

Finally, you can go to any yaml pipeline in your project and use the following to reference the template:

steps:
- template: /containers/container.yml@templates
  parameters:
    imageName: $(imageName)
    containerRegistryName: $(containerRegistry)
    repositoryName: $(repositoryName)
    containerRegistryConnection: 'AG-ASCII-GSMP-boxaimarketopsacr'
    tag: $(tag)
Poly-Repo vs Mono-Repo

Poly-Repo vs Mono-Repo

So I’ve been doing a lot of DevOps work recently, and one of the bigger topic of discussions I’ve been a part of recently is this idea of Mono-Repo vs Poly-Repo. And I thought I would way in with some of my thoughts on this.

So first and foremost, let’s talk about what the difference is. Mono-Repo vs Poly-Repo, actually refers to how you store your source control. Now I don’t care if you are using Azure Dev Ops, GitHub, BitBucket, or any other solution. The idea here is whether you put the entirety of your source code in a single repo, or if you split it up into multiple repositories.

Now this doesn’t sound like a big deal, or might not make sense depending on the type of development code, but this also ties into the idea of Microservices. If you think about a micro-services, and the nature of them, then the debate about repos becomes apparent.

This can be a hot-debated statement, but most modern application development involves distributed solutions and architectures with Micro-services, whether you are deploying to a server-less environment, or even to Kubernetes, most modern applications involve a lot of completely separate micro-services that provide the total functionality.

And this is where the debate comes into play, the idea that let’s say your application is actually made of a series of smaller micro service containers that are being used to completely overall functionality. Then how do you store them in source control. Does each service get it’s own repository or do you have one repository with all your services in folders.

When we look at Mono-Repo, it’s not without benefits:

  • Easier to interact with
  • Easier to handle changes that cut across multiple services
  • Pull Requests are all localized

But it isn’t without it’s downsides:

  • Harder to control from a security perspective
  • Makes it easier to inject bad practices
  • Can make versioning much more difficult

And really in a lot of ways Poly-Repo can really read like the opposite of what’s above.

For me, I prefer poly-repo, and I’ll tell you why. Ultimately it can create some more overhead, but I find it leads to better isolation and enforcement of good development practices. But making each repo responsible for containing a single service and all of it’s components it makes for a much cleaner development experience and makes it much easier to maintain that isolation and avoid letting bad practices slip in.

Now I do believe is making repos for single purposes, and that includes things like a templates repo for deployment components and GitOps pieces. But I like the idea that to make a change to a service the workflow is:

  • Clone the Services Repo
  • Make changes
  • Test changes
  • PR changes
  • PR kicks off automated deployment

It helps to keep each of these services as independently deplorable in this model which is ultimately where you want to be as opposed to building multiple services at once.

Building modules in Python

Building modules in Python

So recently I’ve been involved in a lot more development work as a result of changing direction in my career. And honestly it’s been really interesting. Most recently I’ve been doing a lot of work with Python, which up until now was a language that I was familiar with but hadn’t done much with.

And I have to tell you, I’ve really been enjoying it. Python really is a powerful language in it’s flexibility, and I’ve been doing a lot of work with building out scripts to do a variety of tasks.

As mentioned in previous blog posts, I’m a big fan of DevOps and one of the things I try to embrace quickly is the idea of packaging code to maximize re-use. And to that end, I recently took a step back and went through how to build python modules, and how to package them up for using a pip install.

How to structure your Python Modules?

The first thing I’ve learned about Python is that it very much focused on the idea of convention. And what I mean by that is that Python focuses on the idea of using convention to define how things are done over have a rigid structure that requires configuration. And putting together these kinds of modules is no different.

So when you go through the setting up of the configuration, you would use the following structure:

  • {Project Folder}
    • {Module Folder}
    • __init__.py
    • {Module Code}
  • setup.py
  • requirements.txt

From there the next important part of the setup is to create a setup.py. As you start to flush this out, you are going to be identifying all the details of your package here along with dependencies that you want pip to resolve. A great post on project structure is here.

import setuptools 
  
with open("README.md", "r") as fh: 
    long_description = fh.read() 
  
setuptools.setup( 
    name="kevin_module", 
    version="1.0.0", 
    python_requires = '>=3.7',
    packages = setuptools.find_packages(),
    author="...", 
    author_email="...", 
    description="...", 
    long_description=long_description, 
    long_description_content_type="text/markdown", 
    license="MIT", 
) 

And next is the requirement for a readme.md, which will outline the specifics of your package. This where you are going to put the documentation.

Next the important part is how to implement the __init__.py, this is important and is basically a manifest for the namespace of the library. Below is the sample.

from .{python code file} import {class}

From there you can actually use a utility called twine to bundle the package, and information on twine can be found here. Below is the command to create the bundle. There is a great post on that found here.

Threading in Python

Threading in Python

Just another quick tip and trick for Python, and that is how you implement threading. This is surprisingly simple in Python, but basically it involves installing the threading library using pip

pip install threading

From there the process is the following:

numThreads = 10
threadList = []

def RunTest(self, name):
	print(f"Hello {name}")

for x in range(numThreads):
	t = threading.Thread(target=RunTest, args=(x))
	t.start()
	threadList.append(t)

This will then kick off each thread and let them run in the background, now the next logical question being how do I wait for a thread to complete, and that would be the following:

for t in threadList:
    t.join()

That’s really it, overall pretty easy.

Updating version numbers for Python Packages in Azure DevOps

Updating version numbers for Python Packages in Azure DevOps

So I did a previous post on how to create package libraries in Python, and I wanted to put in a post here on how to solve a problem I immediately identified.

If you look at the setup.py, you will see that the version number, and other details are very much hard coded into the file. This is concerning as it requires a manual step to go and update this before you can do a build / publish activity. And honestly, nowadays CI/CD is the way of the word.

So to resolve this, I built a script to have the automated build agent inject the version number created by the CI/CD tool. And that code is the following:

import fileinput
import sys

filename = sys.argv[1]
text_to_search = sys.argv[2]
replacement_text = sys.argv[3]
with fileinput.FileInput(filename, inplace=True, backup='.bak') as file:
    for line in file:
        print(line.replace(text_to_search, replacement_text), end='')

I then updated my setup.py with the following:

name="packageName", 
    version="{{__BuildNumber__}}", 
    python_requires = '>=3.7',
    description="{{__BuildReason__}}", 

And that’s it, from their you just trigger this tile and inject the new build number into the file.

Generating Dummy Data for Event Hubs or Blob Storage

Generating Dummy Data for Event Hubs or Blob Storage

So I was working on this as part of another project, and I thought I would share. Basically, one of the most annoying aspects of building data pipelines is getting test data to verify the results of that data.

So nothing overly ground breaking, but I thought this might be useful for anyone trying to pipe data into a data pipeline, whether that be blob storage or event hub.

So what I did was build a small generic utility to build text files full of JSON objects and then parse those files putting them onto event hub.

Now for the sake of this instance, I decoupled the code for the event hub, so that I could get more utility, and implemented this as part of a dotnet core console application. Below is the method for generating the files:

static void Main(string[] args)
        {
            var builder = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);
            var configuration = builder.Build();

            var appSettings = new AppSettings();

            ConfigurationBinder.Bind(configuration.GetSection("AppSettings"), appSettings);

            for (var f = 0; f < appSettings.NumberOfFiles; f++)
            {
                var fileName = $"{appSettings.FilePrefix}-{f}-{ DateTime.Now.ToString("MM-dd-yyyy-hh-mm-ss")}.txt";

                Console.WriteLine("-----------------------------------------------------------------------");
                Console.WriteLine($"Creating file - {fileName}");
                Console.WriteLine("-----------------------------------------------------------------------");
                Console.WriteLine("");

                //Create records for entry
                var list = new List<LogEntryModel>();
                for (var x = 0; x < appSettings.MaxNumberOfRecords; x++)
                {
                    var logEntry = new LogEntryModel();

                    logEntry.LogDateTime = DateTime.Now;
                    logEntry.LogMessage = $"Test { x } - { DateTime.Now.ToString("MM-dd-yyyy-hh-mm-ss")}";
                    logEntry.SequenceNumber = x;

                    list.Add(logEntry);
                    Console.WriteLine($"Creating line entry - { logEntry.LogMessage}");

                    var randomTime = RandomNumber(1, appSettings.MaxWaitBetweenEntries);

                    Console.WriteLine($"Thread sleep for { randomTime }");
                    Thread.Sleep(randomTime);
                    Console.WriteLine($"Sleep over - Processing file");
                }

                var filePath = $@"C:\temp\{fileName}";
                //Create text file"
                using (StreamWriter file = File.CreateText(filePath))
                {
                    JsonSerializer serializer = new JsonSerializer();
                    serializer.Serialize(file, list);
                    Console.WriteLine("Pushing Json to file");
                    Console.WriteLine("");
                }

                //Push to blob storage
                BlobServiceClient blobServiceClient = new BlobServiceClient(appSettings.BlobConnectionString);

                //Create a unique name for the container
                string containerName = "logs";

                // Create the container and return a container client object
                var containerClient = blobServiceClient.GetBlobContainerClient(containerName);

                BlobClient blobClient = containerClient.GetBlobClient(fileName);

                Console.WriteLine("Pushing File to Blob Storage");
                Console.WriteLine("");
                using FileStream uploadFile = File.OpenRead(filePath);
                var uploadTask = blobClient.UploadAsync(uploadFile, true);

                uploadTask.Wait();

                uploadFile.Close();

                Console.WriteLine("File Uploaded to Blob storage");
                Console.WriteLine("");

                var randomFileTime = RandomNumber(1, appSettings.MaxWaitBetweenFiles);
                Console.WriteLine($"Thread going to sleep for - { randomFileTime}");
                Thread.Sleep(randomFileTime);
                Console.WriteLine("Thread sleep down, moving onto next file");
                Console.WriteLine("");

                Console.WriteLine($"Started Deleting file {filePath}");
                File.Delete(filePath);
                Console.WriteLine($"Finished Deleting file {filePath}");
            }

            Console.WriteLine("All Files Processed and uploaded.");

            Console.ReadLine();
        }

In addition to creating staggered entries, it additionally outputs in an easy readable format to the console screen. Below is the method I use to generate the random numbers:

static int RandomNumber(int min, int max)
        {
            return _random.Next(min, max);
        }

Overall nothing to special, but it at least creates an easy method of generating the json objects required for pumping through a data pipeline.

Below is all I leverage for a data model for this but this could easily be swapped for any data model you like with some random elements:

public class LogEntryModel
    {
        public DateTime LogDateTime { get; set; }
        public string LogMessage { get; set; }
        public int SequenceNumber { get; set; }
    }

Now on the back end, I needed to take these blob files and parse them. And did so by doing the following:

using (var sr = new StreamReader(logFile, Encoding.UTF8))
            {
                var logs = new List<LogEntryModel>();

                var str = sr.ReadToEnd();

                logs = JsonConvert.DeserializeObject<List<LogEntryModel>>(str);

                await using (var producerClient = new EventHubProducerClient(connectionString, hubName))

                {
                    using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();

                    foreach (var logEntry in logs)
                    {
                        var txt = JsonConvert.SerializeObject(logEntry);
                        eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(txt)));
                    }

                    await producerClient.SendAsync(eventBatch);
                    log.LogInformation($"Log of {name} with {logs.Count} rows processed.");
                }
            }

Anyway, I hope you find this helpful to get data pushed into your pipeline.