Saturday, 7 January 2017

XUnit Test Lifecycles

You are unit testing right? I hope so. If you are you may have run into some scenarios where things are not working quite right. One issue I’ve personally run into is things which attempt to maintain state between tests (eww, I know). Unfortunately, while it is good practice to make sure all of the tests you are writing are completely independent from each other, sometimes shared state will creep in due to other factors (like using an in-memory database because Microsoft kills kittens* and didn’t make Entity Framework easy to mock out).

XUnit Test Context

How many times does a constructor get called for a class in C#? If you answered one, you’d be completely wrong when it comes to XUnit (it kind of surprised me too when I first learned about it). Turns out that as part of the XUnit lifecycle the constructor is called before each test is run, likewise you can implement IDisposable and have Dispose() called after each test is run. While it runs kind of counter to expectations, no state set in any instance variables will be shared between any tests. If you need the ability to share state XUnit provides not one, but two separate options: 1) Class Fixtures and 2) Collection Fixtures. The choice of which to use is entirely dependent upon the scope of what needs to share state. As you’ll see below by default a class is by default a collection although you can also build collections composed of the tests of multiple classes.

Class Fixtures

In order to create a class fixture a small amount of setup needs to be done: a new class needs to be created which will maintain the shared state across all runs of all tests that are part of the class. The test in question needs to implement IClassFixture<YourFixtureName> and your fixture will be passed in as a constructor argument.

An example fixture might be something like:

public class MyFixture{
   private MySuperObject _myInstanceVar;
   public MyFixture(){
       _myInstanceVar = new MySuperObject();
   }

while consuming the fixture would look like this:

public class FixtureTests : IClassFixture<MyFixture>
{
    private MyFixture _fixture;
    public FixtureTests(MyFixture <span class="hiddenGrammarError" pre=""><span class="hiddenGrammarError" pre=""><span class="hiddenGrammarError" pre="">fixture)
    {
         _fixture</span></span></span> = fixture;
    }
}

From a test writers perspective the Fixture mechanism almost looks like a dependency injection engine (except with absolutely zero magic in it).

Collection Fixtures

Collection fixtures operate pretty similarly to class fixtures, although they do require a slightly larger amount of setup. In order to setup a collection fixture, much like a class fixture, you must first define the actual fixture. The wiring logic is a bit different:
First define a collection:

[CollectionDefinition("Awesome Collection")]
public class MyCollection : ICollectionFixture<MyFixture>
{
    // Intentionally left empty. 
}

The actual test class would look like the below:

[Collection("Awesome Collection")]
public class MemberOfMyCollectionTests
{
    private MyFixture _fixture;
    public FixtureTests(MyFixture fixture)
    {
         _fixture = fixture;
    }
}

Note on best practice: it is probably almost always best to pull the collection name out to a constants file to prevent simple typos from dramatically changing the behavior of your tests. But I’m in favor of just using constants everywhere unless there is a compelling reason not to…
Second note: the fixtures must all be in the same assembly. XUnit doesn’t let us get fancy with collection definitions spanning multiple assemblies (not that there would be any big benefit from that)

Fixture Lifetimes

For either of the possible fixture types the lifetime logic is basically the same: the fixture is created immediately prior to the first invocation of the first test in the collection (be it a class-collection or a collection-collection). The fixture is destroyed immediately after the last test in the collection.

XUnit Test Collections

The lifecycle and parallelization of tests is largely driven by the concept of Test Collections. The official documentation does a good job of giving a quick overview. To summarize: by default all tests in a given class are part of the same collection thus will run in a serial fashion. Classes in the same assembly will run their collections in parallel. This behavior is able to customized by specifying a couple of assembly level attributes:

  1. Forces all tests in all classes to be in a single collection – i.e. force serial execution of all tests in the assembly.
    [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]
  2. Determines the maximum parallelization of tests. Setting this to 1 only allows one test to execute at a time, although multiple tests may be executed in an interleaved fashion. By default this is equal to the number of virtual CPUs on the PC (which seems to be a good default unless you have some very specific use-cases).
    [assembly: CollectionBehavior(MaxParallelThreads = n)]
  3. Disables test paralleization assembly-wide. Note that this is different than MaxParallelThreads since it actually turns off the parallelization infrastructure in xunit (for the assembly this attribute decorates)
    [assembly: CollectionBehavior(DisableTestParallelization = true)]


*Just kidding. No kittens were harmed in the writing of this post.

Sunday, 1 January 2017

Goals for 2017

It’s a new year, so in the tradition of setting a few resolutions, here are a few of my professional goals for 2017:

  1. More Networking
  2. More Side Projects
  3. More Reading
  4. More Writing
  5. Take Better Care of Myself

All of those things should be pretty easy, but there is the all important question of “Why?”

Networking

I work as a senior full-stack engineer at my full-time job, and also work as (the only) infrastructure engineer for my wife’s businesses (1, 2) both of which keep me fairly busy, but as a professional I’m always curious what other people are working on, and what other stacks are out there. In order to start getting more exposure to other ways people are doing things I’m going to start trying to attend at least one meetup a month just to chat with other developers and engineers. A stretch goal to this would be to find one or more conference (ideally within the Greater Los Angeles area) to attend.

More Side Projects

This one is kind of easy resolution as I’ve already started working on another side project. Historically my issue with side projects is that I’m great at working on something for one or two days, but staying focused on it long-term has been an issue. The goal with this resolution isn’t so much to just start a bunch of projects, it’s to work on actually shipping things. I did ship a couple of (very small) side projects last year (both are .NET centric):

  1. BlockListChecker – A tool to check if an email address is on any of many ESPs suppression lists.
  2. NuGetVersionChecker – A tool to generate a (very simplistic) report of packages in a package.config file.

I would like to ship at least 6 projects this year (assuming some mix of very simple to somewhat complex).

More Reading

I’ve always been a bit of a bookworm, although I’ve gravitated towards fiction. I recently got a Kindle Paper-white and have found it to be very easy to read on (even for technical books). I’ve already started reading “The Pragmatic Programmer”, and while the book itself is a bit dated, I’m finding there to be a lot of great content. At the tail end of last year I also signed up as a volunteer reviewer for Manning Publication Review, and have already reviewed one book (which was a fantastic read, I’m excited for the remainder of the book to be released so I can finish it). I’ve been adding programming books to an Amazon wish list for a few years now, and 2017 will the year I’ll try to get through at least a few of them. To steal an idea from “The Pragmatic Programmer” I’m going to try to get through at least one book per quarter (ideally even quicker than that though).

More Writing

I’m going to assume that as of the writing of this post I have exactly zero regular readers on this blog. There are probably two major reasons for that: a) I don’t write on a regular basis; and b) I need to become a better writer. The nice thing is that the resolution to both of those issues is the same: write more. Writing is a skill like any other, where the more it is done, the better it will become (getting feedback is helpful in that as well).

Take Better Care of Myself

I think every person has this resolution every year. I spend 60+ hours a week in front a computer between my job, my wife’s businesses, and working on side projects/playing games/etc. I do a few things which help: I ride my bicycle to work most days, and try to either go for a long hike or a long bike ride every weekend. There are a few low hanging fruit and a few habit-changes I can make which will make me healthier and also prevent RSI in my hands/wrists (something I periodically struggle with).

  1. Stop slouching – proper posture would go light-years to keeping me in good shape.
  2. Drink more water – as my coworkers can attest I drink way to much coffee (3-5 cups most days), and in the afternoon I like carbonated and heavily caffeinated drinks. To start I am going to start trying to drink one glass of water for every cup of coffee I have in the mornings. For my afternoon drinks I’m going to stop with the energy drinks and start sticking to sugar-free soda (which obviously has less sugar, and also much less caffeine).
  3. Eat healthier – While I get a fair amount of cardio in every day (bike rides and afternoon walks), most things I’ve read lately show that weight control is really a function of your diet (i.e. working out is great for your heart, but if you want to lose or maintain your current weight, calorie restriction is the only real sure-fire way to do it).

Let’s Make 2017 a Great Year

I’ve never tried publicly blogging my resolution before, let’s see if it makes me a bit more accountable. If nothing else, I’ll be able to point my browser to this page anytime I need a bit of motivation to keep on track with my goals for the year. These are all fairly conservative goals, so I’m optimistic of being to both meet and exceed them, which should put me in a great spot for 2018.

Tuesday, 18 October 2016

A Gentle Introduction to Onion Architecture in ASP.NET MVC – Part 2

In part 1 of this series we discussed what an onion architecture application would look like and discussed the technologies that we can leverage in .Net 4 in order to make that work. In this section we’ll go over how the project is structured, including spending a bit of time looking at how the IoC container is configured. This being a simple application the configuration is significantly easier to understand than it can be in more complex applications.

Project Structure

The application consists of 4 projects: Core, Infrastructure, Infrastructure.Tests, and Web. Each one of these projects has a unique purpose and it behooves all developers to ensure that they don’t mix concerns between projects.

Core Project

The core project is responsible for defining all interfaces for all services which will be implemented in the infrastructure layer, and it also is responsible for having all domain models. In Entity Framework Code First projects, the EF Entity Models can exist in Core. The models that exist in the sample application are not “true” domain models, instead they are just POCO representations

Infrastructure and Test Project

The infrastructure project is responsible for the implementation of all of the services defined in the Core Project. One of the critical distinctions between onion architecture and traditional layered applications is that the data access code (if there is any) will live in an infrastructure style project instead of living in the base/core layer. In the sample application the Infrastructure project only calls out to third-party services. The test project only tests the behavior of the composite service as I have not written this application in a sufficiently decoupled fashion to pass in the RestClient and ideally an abstraction would also be built around the ConfigurationManager

Web Project

As the name probably implies, this is where the “web” parts of the application go, including controllers, views, front-end assets, etc. For this application front end assets are just managed by downloading and saving them into /Scripts, and then everything is manually wired up using the BundleConfig.cs in App_Start. The interactivity within the application is achieved by using a little bit of jQuery.

Dependency Resolution

This application exposes 4 services in total, but only has 2 interfaces. This is due to the fact that the CompositeBounceCheckerService is composed of both MailgunService and SendGridService, hence all three of them share the same interface. The final service, the SuppressionListCheckService just consumes the CompositeBounceCheckerService. This final layer of indirection isn’t, strictly speaking, necessary, however it does afford the ability easily pass one of the service-specific IThirdPartyBounceService services as its dependency if we only wanted to check for suppression in a single ESP. The DefaultRegistry below shows how to get that all setup.

            For<IThirdPartyBounceService>().Use<CompositeBounceCheckerService>()
                .EnumerableOf<IThirdPartyBounceService>().Contains(x =>
                {
                    x.Type<SendGridService>();
                    x.Type<MailgunService>();
                });

This code basically tells StructureMap to scan all registered assemblies (all assemblies listed in the Scan call above this) and register the SendGridService and MailgunService as services within the composite service.

StructureMap is capable of doing a lot more, such as having custom life-cycles for certain services or handling weird object hierarchies (you can do a lot more than just have interfaces and services).

Testing

Testing is fairly direct as we just mock out the services that feed into the service we want to test (generally known as the “SUT” or the System Under Test). One easy example of a test is this:

        [Fact]
        public void Get_Bounces_Returns_A_List_Of_View_Models()
        {
            // Arrange
            var returnList = new List<SuppressedEmailViewModel>
            {
                new SuppressedEmailViewModel
                {
                    AddedOn = DateTime.Now,
                    EmailAddress = "[email protected]",
                    EmailServiceProvider = EspEnum.UNKNOWN,
                    ErrorCode = string.Empty,
                    ErrorText = string.Empty
                }
            };

            var mockService1 = new Mock<IThirdPartyBounceService>();
            mockService1.Setup(x => x.GetBounces()).Returns(returnList);
            // Initialize the composite service with an array of one third party service.
            var compositeService = new CompositeBounceCheckerService(new[] { mockService1.Object });

            //Act and Assert
            var result = compositeService.GetBounces();

            Assert.Equal(returnList, result);
            mockService1.Verify(x => x.GetBounces(), Times.Once);
        }

The code above will first: create data for the Mock to return, then: create and setup the mock, then: inject it into the service, then: call the service, and finally: make sure that the behavior of the service was correct. Having a robust suite of tests allows us to change the implementation of any of the services and still verify that the output is correct.

When writing tests, I generally follow the AAA pattern (Arrange, Act, Assert) and leave comments in the code where those things are happening as an easy way to make sure that my tests are structured in a consistent fashion.

That’s all for now folks. I hope that this two part series on the onion architecture made the benefits of using it a bit more clear.

Please, check out the code on Github, and drop me a line if you have any questions!

Sunday, 18 September 2016

A Gentle Introduction to Onion Architecture in ASP.NET MVC – Part 1

Welcome to part one of a multi part series on Enterprise Application in .Net Core! In this series we’ll go over everything that is necessary to build a best in breed enterprise application with the Onion Architecture at its core. The word enterprise is integral in describing the software being described as it is going to be software which is capable of being extended year after year in a clean fashion while allowing things to stay DRY and testable. We’ll cover the following:

  1. System to Design
  2. Initial Architecture and Architectural Considerations
  3. Proper Abstractions Around Each Layer of the “Onion”
  4. Unit Testing

This series of posts isn’t going to exhaustively walk you through every decision that may need to be considered (i.e. we’ll only very briefly discuss localization). My intention with these posts is to put forth my ideas of what a solid architecture looks like. With that being said, the application we are going to design is called “Block List Checker” which is an application which will allow one or more Email Service Providers (ESPs) such as Mailgun, SendGrid, etc. to be queried simultaneously to return a list of all email addresses which are on a suppression list. This is a very simple application which is intentionally somewhat over-engineered for the purposes of illustrating the various components of the system.

Inversion of Control and Dependency Injection

In order to design this software appropriately there are several tools that will need to be incorporated into the solution. The first (and arguably the most important) of these is StructureMap a .Net-friendly IoC/DI Container. For those not familiar: IoC (or Inversion of Control) and DI (or Dependency Injection) are design patterns which help make code much more testable. They do this by “inverting” the dependency chain. The way this plays out in MVC applications is that controllers will be injected with all of the dependencies that all of the services which are used by those controllers require. The IoC container will basically maintain a mapping of interfaces to services and, at runtime, will provide the controller with an instance of the service requested. Due to the fact that this effectively requires loose-coupling between components, writing unit tests becomes much easier (note that the description of IoC here is just barely skimming the surface, entire books have been written on this subject).

Unit Testing

Because this application is so simple, I have opted not to write comprehensive tests around it, however I have written a few tests just to demonstrate using xunit with this application and how the loose coupling allows the testing to be achieved.

Known Issues

This application is very basic and the UI could use a bit of love. It currently targets Asp.Net MVC5 instead of .Net Core MVC6 due to the fact that I prefer writing API access code using RestSharp, and that currently doesn’t seem compatible with .Net Core. I may release another sample project in the not distant future which targets .Net Core. It also currently doesn’t make use of either a proper front end framework, or of most of the functionality that MVC5 exposes on the front end. I haven’t written this application to be overly robust, and due to the nature of how it consumes API keys without any authentication I wouldn’t expose it on a publicly facing web server.

Get the code

With all that being said, the code is available here: https://github.com/lbearl/BlockListChecker.

Part Two of the series is available here

Monday, 25 April 2016

PucciThe.Dog – Python/Flask on an RPI

Overview

Flask is an amazing little Python web micro-framework for those who aren’t familiar with it. It allowed me to build out this entire application in about 8 hours of total dev time (including a whole bunch of time just not quite grokking flask-login). The minimal goals I wanted to achieve was to expose a very basic web presence for Pucci (the dog), along with building a very basic puppy cam. Since I had an old Raspberry Pi lying around, I figured that this might be the ideal project to use it for.

Please feel free to take a look at the code

Bill of Materials

  1. Raspberry Pi
  2. Microsoft LifeCam NX-3000
  3. Sweetbox Case (optional)

The Plumbing

In the interest of making this work as quickly as possible, the actual picture-taking logic is just a shell script running in cron (specifically once every 5 minutes). It likewise will examine all of the files in the directory and delete any that are more than a day old. The real magic is actually done by fswebcam as documented here.

fswebcam -r 320x240 --jpeg 80 -D 3 -S 13 \ 
/home/pi/poochpics/$DATE.jpg

Technically the webcam should support higher resolution pictures, but I suspect that it isn’t quite as compatible as I was led to believe it was. The -D 3 -S 13 are very important for me as the camera was corrupting 70+% of the images that it was capturing. These arguments will first delay the capture for 3 seconds and then skip the first 13 frames captured, finally generating a photo based on the 14th frame captured. This numbers were very scientifically found by simply playing with the webcam until it was returning reliable results 100% of the time (it is possible that others will be able to operate the webcam without any delay or skipping any frames).

Now that all of the photo generation and automatic purging of old images is handled the actual responsibilities for the web application are pretty slim: basically just a regular old mostly static web page with the ability to login to a secure area which will have the actual photos on display.

I made the jump to almost 100% Windows at home (since I have been working on Windows machine for pretty much my entire career), and decided to use this project as an excuse to try out the Python Tools now available in Visual Studio (spoiler alert: they are awesome). I can’t say that I have ever really used an IDE for Python development before (traditionally I’ve done it in a mix of Vim and Sublime Text), so I can’t compare it to some of the other Python IDEs out there, but as someone who gets paid to do C# in Visual Studio 2015, the experience was very nice.

Implementing the Web Application

In order to actually get things working I just started with the default template that comes with VS for Flask + Jinja2 templating (I contemplated doing this project as an Angular app with a Flask RESTful backend, but decided against it). The code is pretty basic especially considering the entire application only consists of 4 routes and doesn’t really do any magic (there isn’t even a CRUD component to it). The one thing that isn’t exactly standard was my choice for authentication. As I mentioned earlier, this application does use flask-login, and while my original thought was to just hardcode the credentials, I ended up not going down that path as I couldn’t come up with a non-hacky way to persist the credentials that wasn’t less complex than just adding a SQLite database with some credentials tossed into it. To that end if you look at __init__.py you’ll notice the need to have a SQLite db called test.db. This database just has a single table called “user” which will store ids, usernames, emails, and bcrypt hashed passwords:

Deployment

It really is an exciting time to be alive when I can build something on a Core i7 desktop with 32 GB of RAM, and then relatively effortlessly deploy it onto a computer the size of a microcontroller with a whopping 512MB of RAM and a sub-1Ghz ARM processor. The deployment is actually pretty standard: it’s just Apache2 with mod_wsgi and the appropriate wiring as described here. The most interesting part of all this is probably the way that DNS is being handled: Namecheap now offers Dynamic DNS, so I have ddclient running on the Pi, automatically updating my home IP address to Namecheap’s DNS servers. If you want to try it out for yourself and give me a little boost in the wallet as well sign up here.

The only thing I’m a little nervous about with this setup is whether or not the relatively underpowered RPi really is going to do that well connected to the public internet. That being said, it is pretty heavily firewalled, with only port 80 exposed so its a fairly limited attack surface for any of those internet hooligans.

I hope you enjoyed reading about my very small flask application on a RPi, please reach out to me on Twitter @lukebearl or head over to Github.

Saturday, 9 April 2016

One Month Retrospective

One Month In

The company I work for recently started a new initiative: a brand new piece of software completely divorced from the software we traditionally worked on. I along with three of my colleagues were selected to lead the entire architecture and build out all of the core features. All four of us are somewhat experienced developers, but none of us has ever worked on an architecture team at the very beginning of a software project which is very much enterprise-class software.

Just to not confuse anyone, when I say enterprise-class or “big” software I’m specifically referring to software that is expected to grow to encompass multiple modules and have a code base in excess of 500 KLOC. For smaller applications the concerns I bring up here probably aren’t relevant. In this article I am specifically referring to software which is (hopefully) going to have a lifetime measured in many years if not decades with hopefully only minimal refactoring of the core structure.

What We Did

As happens so often in the software world, we were given a whole bunch of pretty “loose” requirements, and it was up to the four us with minimal direction to prioritize and implement everything that we needed. The project basically ended up being implemented in the following order:

  1. Project Layout (we used the Onion Architecture, which I have become a very big fan of)
  2. Initial Core Application Services
  3. Some development niceties that are ASP.NET specific (like things to ensure that we always have an Anti-Forgery Token, etc.)
  4. The Authentication and Authorization systems (using ASP.NET Identity)
  5. The logging framework
  6. Site Navigation and initial site pages for core things

What We Probably Should Have Done

While everything works with the order we did the work in, it is also causing a number of issues because as we finish certain tasks we then have to go back and update other code. The most painful item on this list is the logging framework. I am now thoroughly convinced that one of the first things that should be setup in any new large project is a logging framework. It doesn’t help much for development at the very beginning, but as soon as your first release happens and you push the application live to a web server or anywhere that isn’t a developer’s desktop that logging framework becomes your best friend when something breaks. We happen to be a 100% Microsoft shop, so we could always do something fancy with remote debugging, but that has issues of its own (including ensuring that you can reproduce the issue).

Lessons Learned

Large enterprise-scale software is challenging to design, there will always be a number of moving parts. As developers and architects it makes our lives easier if we take a step back from the initial problem and try to see if there are dependencies which don’t necessarily block development, but which will require a fair amount of rework in order to incorporate after the fact. Logging is probably an example that will be true for just about every software project out there, but another great example is a localization framework (I developed one that doesn’t use resx files because I don’t like that approach). At minimum before any views are created, the localization should be fleshed out, otherwise there will be a massive amount of rework (depending on what the view contains it may be almost a full rewrite).

I hope that anyone who reads this can not repeat the mistakes made, although these mistakes are hardly the most costly out there.