Lessons Learned in Software Development | Henrik Warne’s blog

Henrik Warne’s Lessons Learned in Software Development

I wanted to share this post from 2015 from Henrik Warne’s Blog for a while, as I agree with most of it and have been trying to follow it from the first reading.

If you are a software engineer, or is involved with software development in any level, this post will certainly be worth of your time.

Azure Web Jobs – Creating your first Azure Web Job from Scratch

So, you have just created your first Web App in Azure, and it is working like a charm. You have all the scalability you need and you know exactly how to store your data in the many options that Azure provide. What else do you need? Well, what will you do if you need to run some asynchronous background processing, like processing images or batch data? That is when Azure Web Jobs comes in handy.

What are Azure Web Jobs?

Straight from Azure documentation web site:

WebJobs is a feature of Azure App Service that enables you to run a program or script in the same context as a web app, API app, or mobile app. The purpose of the WebJobs SDK is to simplify the code you write for common tasks that a WebJob can perform, such as image processing, queue processing, RSS aggregation, file maintenance, and sending emails.

So, as you can see, it can perfectly fit the role of asynchronous background processing and with one interesting advantage, Web Jobs have very powerful integration with other Azure Services like the Service Bus and the Storage Services, making it incredibly easy not only to read/write data to this services but also to use them as TRIGGERS, avoiding the use of scheduled processing.

Let’s begin.

Creating your first ‘Azure Web Job’ from Scratch

The idea here is to create a simple Web Job that will be triggered after an item is added to an Azure Queue, and after very little parsing, save it to the Azure Tables Service.

The Setup

The first thing we are going to need is an Azure Account. If you don’t already have one, you can create one here, for free. Then we will install the Azure CLI 2.0, the installation instructions for all platforms can be found here. You may also want to install the Azure Storage Explorer, a tool that we will use to manage our Storage Account for testing.

Now we need to setup the resources in Azure to which the Web Job will connect. I’ll give you a list of commands from the Azure CLI that must be execute in a shell prompt.

First things first, we need to login to Azure through the console.

Then we need to create the Resource Group that will group all our resources.

Storage Account

Next step, create the storage account.

Then we create the Queue, that will be the source for our data, and the Table, which will be the destination of the processed data.

That’s all we need in the Storage service. Now we need to create the App that will host our Web Jobs.

App Service

And last, but not least, let’s configure our Connection Strings for the App:

And that is it for the Setup. Now, we can begin the coding.

Finally! Web Jobs

The Coding

For the coding part we can use whatever code editor we prefer. I will use Visual Studio Code, as it offers great support for coding C#.

As for the programming language, you may be surprised to know that Azure supports a lot of programming/scripting languages in the Web Jobs SDK, so you can choose the one you like the most. For this exercise, I will use C# because I feel like doing so 🙂

Start the Project

As I want this article to be as multi platform as possible, I will create a new project using the dotnet CLI. Open a shell prompt, navigate to the directory of your Web Job and type the following command:

Unfortunately the Web Jobs SDK has no support yet for the .NET Core Framework, so we have to make a small modification to the .csproj file, so we can use .NET Framework v4.5x.

The final file should look like this:

It is important to understand that a .NET Core Application can be executed by a Web Job, it only cannot make use of the Web Jobs SDK, which makes it much easier to integrate with other Azure Services

Azure Web Jobs SDK

Back to the shell prompt, we need to do three things. Add the Web Jobs SDK Package, restore the packages and build the project.

Code Host

It is worth mentioning now that one Web Job project has, at least, two main components:

  • The Code Host
    • This is the Entrypoint for the application, and this is where we should do all the configuration for our Web Job
  • A Trigger
    • Our project must have at least one Trigger, but it can have many others. In this case we will use only one trigger, fired by a new entry in our Queue.

Let’s start with the Code Host, which will be placed in the Main method of our code and will be the starting point of the application

That’s it 😀

Seriously Web Jobs?

No! I am not. The Host is, as the name implies, a piece of code that will configure your web job and control the lifetime of the code. It can be a lot bigger than this, including all sorts of configurations, but that is all we need for a simple Web Job.

The Trigger

The trigger is where the real business logic of our Web Job live or, at least, where it begin. In our case we will have a single method that takes a QueueTrigger and records the data to a Table. The table will also be represented through a parameter, to which we will save the data.

This is the method code:

Simple enough, isn’t it?

The method gets three parameters, the first one is the trigger itself, the message in the queue, notice that the attribute of the parameter specifies which queue the message comes from. The second parameter has the “out” keyword, specifying that is the output of the code and the parameter attribute also specifies the Table to which the data must go. The last parameter is a TextWriter object that the Azure SDK injects so you can log information to the Azure Log Console.

The model for the Queue is a simple POCO class, nothing special about it. The model for the Table inherits from “TableEntity” making it suitable to be saved to an Azure Table.

We are almost there! Only one step missing!

Deploy Web Jobs!

The Deploy

Now we have to deploy our Web Job to Azure, which is actually pretty simple. First thing we want to do is to build the project, to make sure that everything is in place, back to the shell prompt!

We should get a result like this:

Deploy Web Jobs

Navigate to the directory showed in the build command, in my case C:\Code\AzureCoderWebJob\bin\Debug\net45\, then we will zip the content of the folder.

Zipped Web Job

Now you have to open the Azure Portal, and login with your account.

In the Dashboard page, look for the App Service you created during the setup and open it. In the App Service Blade, under “Settings”, look for the option “WebJobs” and click on it, the WebJobs Blade will be opened, and at the top you should see a button “Add”, click on it. The Blade to add a new WebJob will be opened, and we will have to fill some information:

Add Web Job

The name is simply an identifier for the WebJob, the File must be the Zip file we have just created.

The possible “Type”s are “Continuous” and “Triggered”, it may cause a little confusion but we want to keep the Continuous Type.

The scale is what define if the WebJob will run in only one instance of the App Service or in all the instances, scaling with the Web App, it is not so relevant for our WebJob, so we will keep the default “Multi-instance”.

After filling all the fields, click in “OK”, wait for the deployment and process and… that’s it 🙂 You have just deployed your first WebJob created from scratch! But, how can we test it?

The Test

Let’s start by opening the “Logs” for our deployed WebJob. After we deployed the WebJob, we will be sent back to the WebJobs list Blade. In this list, select the WebJob and click in “Logs”. The Logs will be opened in another Tab of the Web Browser. We should see something like this:

Web Jobs Logs

Click on the title of the Web Job and you should see a small console, with the execution details for the Web Job:

Web Jobs Log

As you can see, our “ProcessQueue” method has been detected!

Now we must add a new item to the azure-coder-queue, using the Azure Storage Explorer…

…and then look at the log again:

Log Web Jobs

Our message has been successfully processed! Now, if we take a look at the contents of the AzureCoderTable, also using Azure Storage Explorer, we should see our item added to the table:

Web Jobs Table

Happy Web Job

And, finally, that is all you need to know to create a very simple WebJob from scratch! Most of what I showned here should work fine in Linux and Mac OS (apart from the usage of .NET Framework) but, unfortunatly, I am not able to test every step in these platforms, so let me know if you try and have any problem!

I am really sorry for the long post 🙂

Long Web Job Post

Sources:

 

[Small Talk] New Country & New Job – Part 2

Hello again 😀

(I promise my next post will be out in a shorter interval and it will be about a technical subject :P)

So, as I was saying in the last post, the interviews were done and the offer was formally made by the company, that was when things really got real. Damn! I was going to work abroad! How does that even work?

Of course that, since I decided that I wanted this position, my wife and I started researching the process of moving to The Netherlands for work, but when it went from a possibility to a question of time, I got admit that I felt a little lost.

First thing was to start dealing with the bureaucracy of the documentation. It is not a VERY complicated part, but everything  takes a lot of time. The good news? The company put us in contact with some people that would take care of most of the process. Thank. God. I can’t imagine what kind of trouble we would have been into without does guys.

The bad news? Me and my wife realized that our passports were expired for a couple of months…

idiot

Unforgivable, I know… I scheduled the visit in the beginning of November and the first available date was the second week of January. Yes, seriously. So I did what I had to do. I panicked. A little. After that we went on with the next step, telling the people that should know about it.

We started with our parents, of course, which was quite easy as both families supported and encouraged the idea. The next, and more complicated part, was to tell my partners at my, then, company that I would  leave the team in a few months. In a general way everyone was understanding and then we started the transition process. I worked for Class Solutions for more than 8 years and I am glad I could leave the company in a good way.

Parallel to these, we started preparing the smaller things. Selling some things, archiving some things, buying some things. I won’t go into details but until our last week in Brazil I was still delivering things that I sold and archiving things at a storage we rented. Hell of a process.

Back to the bureaucracy, despite the fact that I had accepted the job in the beginning of November, it wasn’t until the end of December that we really started moving with the documents.

Then emerged the next big challenge, the relocation process. What does this process comprehends exactly? That is exactly what I asked to the consultant who helped me with the process. She will help me with things like finding a place to stay, setting up some mandatory meetings at the municipality, setting up utilities and even the bank account. Sounds somewhat simple, right? It isn’t.

At this point I got stop and tell you that it would have been impossible, and I mean it, for us to move here if it wasn’t for the help of the consultant and the HR people of my current company. They really took a heavy weight out of my back in this process. If you are reading this, Thank you!

Actually, when I stop to think about it, our greatest contribution in this matter was to be really worried that everything was going to be doomed, and then realize that someone had already solved the problem for us.

At the first week of January we got married (Red heart) and the week after we FINALLY would have our visit to the Federal Police Bureau to do the processing for our passports. After that we would still have to wait 2 weeks (!) to have our passports issued. We finally got our passports by the end of January and thus began the process of obtaining our visas! Between sending the documents to the IND (the organ responsible for immigration processes) and finally getting our visas, it took about 4 weeks for the whole process and when I finally got everything in place we had only 1 week and a half before my starting date at the new company!!

In the last week we sold the last few things, got our flight tickets, finished the documents for the cats (yes, we took our 3 cats with us, it was a completely different story) and said goodbye for a few people.

On the 25th of February, we boarded the plane headed to the Old World, and here we are now, one month (and a few days) later.

It was probably one of the more stressing processes I have ever been through, but it could have been much worse if it wasn’t for the help of some key people both in Brazil and in The Netherlands.

Uff… This, of course, doesn’t cover all the details of the process, but it gives a general picture.

So far I can only say that it has been worth. I am loving every step of the new life. From the work/workplace to the new City (I am in love with Breda). From the weather (which is awesome Open-mouthed smile) to the People.

So, that is most of what I wanted to get out of my chest about this process. If you have read it this far, thanks for spending the time. Smile

My next post will be about Web Jobs! Though Microsoft seem to be cleaning the way for the Azure Functions to take this Role, Web Jobs still have some features that are not provided by Azure Functions.

See you soon! (Soon, I promise, hahaha)

[Small Talk] New Country & New Job – Part 1

Hi everyone, it’s been a while 🙂 I haven’t written anything for the last few weeks because of the big life change I am going through right now.

By the end of the last year I found a great opportunity to work abroad, more specifically to work in The Netherlands.

I had been thinking about working and living abroad for quite some time and that seemed like a perfect opportunity, why not give it a try? What did I have to lose? So, I contacted the head-hunter and started my research on the company, the products, the country and everything I could think about back then. It would also be my first time working for a really big company (awesome). I had tried other companies before but this time it was different, for the first time I really felt able to live in a different country with a completely different language and culture.

So, after a couple of days the head-hunter came back to me to schedule a call for an interview. Ok, no problem, I did that before, right? Right, but never in English. That was a major change for me, first job interview entirely in a foreign language with a person that didn’t even understood my mother-tongue. I did the interview and it went great, the interviewer was the head-hunter himself (great gut, by the way) and it was just a primary assessment of my objectives/capacities and also an alignment of expectations. For me it was awesome, then I knew that I was capable of speaking in English with another professional without major problems. Ha! Felt great. Therefore, the next step would be to have a interview with two leaders of the technical department. We scheduled the date and I’ve started to prepare myself for the interview, quite tricky as I didn’t know what exactly I would be working with (apart from a big picture of the system environment). What did I knew?

  • It was a major system for a specific business segment
  • It was on the Azure Cloud (Yeah!)
  • It was a multi-tenant system

That was pretty much all I knew. So I decided to focus on understanding the type of system that I’ll be dealing with.

The day of the interview came and it was once again a great experience. I spoke with two professionals that were involved with the same product that I would be working with. One worked in the development team and the other one in the QA team. So, another first, now I had to show for that two guys that I would make a good new team member and that I could aggregate to the development of their product. We talked about some general development techniques, about some patterns for cloud development and basic concepts, no problems so far. That interview showed me two weak spots in the way I talk about development. The first one came up when we started talking about scalability of the system. As a multi-tenant application intended to be used by thousands of users, maybe hundreds of thousands, everyday it had to be able to escalate up and down with a certain ease, right? How to prepare for that? I spoke for quite some time about the techniques that I would use to help me with that objective but the point was that  they were expecting me to talk about in a more conceptual way, they need to check if I understood the concept of scalability, so… a negative point for me there (I think). The second point, that one a bit more complicated to handle, was that I did knew a LOT of the technologies that they were using in their environment BUT, I haven’t really used most of them at real projects so far, just fooled around with some of them.

I ended the interview with a mixed feeling. For one side I had been able to speak with them about almost everything, and that should prove that I wasn’t a fraud, I had been able to shown my knowledge in most of the core technologies that they used and I talked about new technologies that were interesting to them (Thank you .NET Core!), for the other side It went clear, even to me, that I didn’t had the seniority that they were expecting for the position. So I would have to wait for their feedback.

A few days later the head-hunter contacted me saying that the technical interview went fine, but I won’t fit the seniority level, damn… BUT, they would be glad to consider me for a Software Engineer position. YEAH! So what? Now I would have to go for a third and final interview with the head of the Technology department, a little pressuring but what the hell? Things were going great, let’s keep it moving!

By that time, I knew that I could have a conversation in English and I didn’t really know what I could do to prepare for an interview with a Senior VP of a big company, so all I could do was to get mentally prepared for the talk. It went as a nice surprise that the VP was actually a nice guy and the pressure faded within a few minutes! This interview seemed to me like the most informal of the three that I went through and it sounded more like a conversation with a friend.

The last interview took place on October, 31st and in this same day I’ve been informed that they were preparing a formal offer for me. God!  What know? I barely knew that the challenges were just beginning :s

[Azure Functions] Function wide Log Service

I confess, I am log-addicted. If I have to guess I would say that at least 1/3 of my code are log calls. It despairs me to look at a console and don’t know exactly what is going on.

That said, Azure Functions provides a very simple-to-use system to log to the execution console. You just have to add a TraceWriter object as a parameter for the Run method.

Couldn’t be easier, I think. But when your function is a little more complex than that you will probably have some separated code files for your business logic (I hope, for the sake of you sanity) and that is when it becomes a little complicated to log your processing.

The obvious choice is to pass the TraceWriter object around through all your methods and simply use it.

It really can be a viable solution if we are talking about two or three methods, but it will be a real mess if you have more than this.

So, I came up with a solution that worked really well for me 😀
The solutions consists of initializing a global object at the beginning of the Function that will be used throughout the code, just importing the Service code file.

The Service code file can be found here. It is really a just simple piece of code. To use it, you just have to instanciate the object at the beginning of the function Execution:

The Log object in line 6 is global and static and is defined in the LogService.csx file. The LogService class is also defined in the same file.

And then you can use it anywhere in the code, just referencing the Service code file

How does it look? 🙂

Connecting to SharePoint in Azure Functions using App Only Permissions

So, recently I needed to create a service that would read and write some data to SharePoint Online, easy enough, isn’t it? Problem was that the scenario demanded that the processing were done in Azure, specifically in Azure Functions. Hmmm… How difficult can it be? (Well, it has to do with SharePoint, so…)

The first thing I did was to import the usual packages that I use to connect to SharePoint:

Both packages had worked perfectly in all previous scenarios, then I had no reason to not try it.

Problem was that the version that the runtime downloaded didn’t have an option for obtaining an App Only Authorization Token, which was necessary in my case.

So, cutting to the chase, after some struggle I found a version of TokenHelper.cs that had the methods that I needed and then I’ve been able to successfully convert it to .csx. The file is available in the following link:

TokenHelper.csx

This file will allow you to connect to SharePoint Online using App-Only Authorization like this:

This is as simple as it gets 😀

I hope this helps anyone who is starting with Azure Functions and SharePoint!