cucumber_logo So as I’m sure you have guessed by my last bunch of posts, I’ve been hooked on Ruby and Rails development.  The more I use it, the more I love it.  I felt I was loving it a bit too much, so decided to divert my attention back to ASP.NET MVC for a while.  While developing with Rails, I was practicing BDD with Cucumber.   I felt that Cucumber scenarios (written Gherkin) were far more useful for the majority of my testing on a web application.  Sure unit tests are worthwhile for bits of code, but I’ve found that they don’t always make sense when testing a web application (like Steve Sanderson has mentioned).  Some may argue that these tests are brittle, but honestly, I feel it defines my application better and it definitely tests the actual interactions better.  I’m fully on board for unit tests for complex processes, but the majority of web work just isn’t that complex.

I’m not going to go fully into what BDD is here, but my goal was to use a similar process when developing ASP.NET MVC applications as I did in Rails.  With Rails, it’s simple, Rails, Cucumber, Pickle, ActiveRecord, Factory Girl, Webrat, etc. work perfectly together.   In the .NET world, there are options, but nothing that really fit together as nicely.  I started with just wanting to test my basic application (menus, breadcrumbs, etc) and didn’t venture into model work yet, so I figured it would be as simple and efficient as it is in Rails.  Not so much.  I tweeted about this exact thing a while ago.

SpecFlow and WatiN

SpecFlow Logo Did I mention how much I love Cucumber?  Well, Matt B had recommended SpecFlow to me on the .NET side, so I went and started using it.  Steve Sanderson actually has a great, detailed post on this exact topic, so I won’t go into all the things with SpecFlow.  In short, while SpecFlow is pretty cool and very much like Cucumber, using it for web testing isn’t exactly great.  There are a lot of things wrong in that space, but really the thing that turned me off was the speed of the tests.  My tests are interacting with a website, and sadly there’s not much in way of web testing under .NET.  For my tests I used WatiN.  WatiN is pretty cool, but the fundamental problem is that it uses the actual browser to run the test (I was using IE), and man, were my tests slow.  I had a suite of just 20 scenarios with 46 simple steps, for example, verifying the title of the page and such.  Nothing complex at all.  Running the scenarios with SpecFlow/WatiN, it takes around a minute and a half to run.  Yikes.  In my Rails project I’ve been working on, I have 167 scenarios with 1912 steps, and that takes about 2 minutes to run (and that’s running on a machine with only 2GB of RAM vs 8GB on the machine running the SpecFlow tests).  Quite a difference, not to mention the Rails tests are actually inserting things into a database as well.  The times to run these tests in .NET was just horrible, not very conducive to using BDD for these types of tests.  My hunch is that the instance of the browser was causing all of the delays in my test, and why my hunt was on for how to do my testing without using a full browser like Webrat can do.

** Update ** **I started writing this article after just using Steve’s example as a guide, however I didn’t finish it.  I’ve been playing with other stuff, but got around to getting back to the article.  Anyway, He is using a new browser instance for each scenario (which seems like what you want so that you get a fresh instance for each scenario).  By changing the browser instance creation to be for each Feature instead, it drops the time to 30 seconds.  A bit better, but still rather slow.  Continue on for what I feel is a better option.**

SpecFlow vs. Cucumber

SpecFlow has the advantage of writing your steps in C#, but with my new found love of Ruby, I don’t have a problem whipping up steps in Ruby.  I’m not one of those people who can only use one language, and really, the steps flow much nicer in Ruby than in C#.  For example:

Here’s a step defined in C#:

[When("I follow "([^"]*)"")]
public void WhenIFollow(string linkText)
{
    var link = WebBrowser.Current.Link(Find.ByText(linkText));
    Assert.IsNotNull(link, string.Format("Could not find a link with the text {0}", linkText));
    link.Click();
}

And the Ruby version (with Webrat):

When /^I follow "([^"]*)"$/ do |link|
  click_link(link)
end

Where the click_link method is provided by Webrat, but they both do the same thing: follow a link on a page.  The point is the syntax, not so much what the method body looks like.  Doesn’t the Ruby look cleaner?  For example, there is no extraneous escaping needed within the regular expression, and no need for the extra method signature markup.

One thing that SpecFlow has is integration with Visual Studio and integration with the Unit Test runner in ReSharper and such.  That’s pretty cool, but again, I’m looking for speed in these things.

Webrat and Mechanize

Webrat is really designed for testing Rack-based web applications, like those running on Rails or Sinatra.  I previously assumed you could only use it for testing Rack-based applications, but thanks to a gem called Mechanize, we can use Webrat with ANY website, oh yea!  Using Mechanize in Webrat is easy, it’s just a simple change to the env.rb file (typically found in the path /features/support/env.rb).  With that change, you can pretty much work with your ASP.NET MVC site just like you can with tests against Rails.  In fact, I just snagged a copy of web_steps.rb from my Rails application and the steps work just great.  The only difference is that you need to fully qualify your URIs.  Within a /features/support/paths.rb file (like you would have in Rails), I simply prefixed the relative paths with the absolute path to my local web server.  There is a good starting project from @cayblood on github.

In order to run the Cucumber tests, simply fire up a command prompt.  Then browse to the root folder where your features directory is located and it’s just a simple “cucumber” command to run all the tests.  Of course, you need to make sure your MVC web application is running or else your results are going to be very red.

Big Speed Improvements

Time So I duplicated the features in my Ruby/Cucumber/Webrat/Mechanize project (and tweak them a bit, totaling 21 scenarios and 61 steps).  Since the SpecFlow features are written in Gherkin, it’s easy to move the files over (just exclude the .feature.cs files from SpecFlow).  The tests took around one second to run.  That’s right… A minute and half with SpecFlow/WatiN down to a second.  Sweetness.  By the way, this is running on Windows as well where Ruby isn’t the speediest (note the tests in the post were using Ruby 1.8.7 and not 1.9.1 like my tweet).

Conclusion

For web testing, there is just such a difference with the Ruby stack vs. the C# stack.  I would expect much better performance if there were a headless browser option available in .NET.  I don’t have any issue with using Ruby code to test C# stuff, and the more I use Ruby, the more I dislike static-typing.  I chalk it up to just “using the right tool for the job.”  Aside from using something else that isn’t C#, the second disadvantage is VS integration.  Notice I said run your tests from a command prompt.  I know you Windows guys (well most) shudder when you hear the “command line,” but some of us aren’t scared :).  Personally, for the speed improvements, I can live without C# and VS integration.  Of course, here I go again, falling further in love with Ruby.