Okay, so, have you heard about NUnit attributes, and how they’re awesome?
Sure you’ve heard of Setup, Test, TestFixture, and TearDown? But how about Values? How about Range?

The Values attribute allows you to run the same test several times with different values. It makes it where you’re not doing something crazy like copy and pasting the same test, over and over again! I was close to that situation, but it hurt my soul too much to do that. So, in order to save my soul, I had to seek a path for redemption. Here’s what I found:

Okay, to try this, you may need to set up a sample test. If you need to start from scratch, check out my End-to-End Example: http://thetestingblog.com/2009/09/10/selenium-rc-in-c-using-nunit-an-end-to-end-example/

My example test verifies that the text “Daniel Brown” is present on the main testing blog page–a simple pass or fail. But imagine we wanted to run the test 3 times, each time verifying the presence of a different Testing Blog author. Here’s how we would do it using the Values attribute:

        [Test]
        public void ValuesTest([Values("Daniel Brown", "Marisa Seal", "Josh Carroll")] string textPresent)
        {
            selenium.Open("http://www.google.com");
            selenium.Type("q", "thetestingblog.com");
            selenium.Click("btnI");
            selenium.WaitForPageToLoad("60000");
            Assert.IsTrue(selenium.IsTextPresent(textPresent));
        }

As you can see, we’re basically passing the argument “textPresent” into the method. The values are what you want the argument to be. For each value, run the test once with “textPresent” = to that value:
[Values("Daniel Brown", "Marisa Seal", "Josh Carroll")] string stringPresent

So, in the case of this test; here’s how it would run:
SetupTest();
ValuesTest(“Daniel Brown”);
TeardownTest();
SetupTest();
ValuesTest(“Marisa Seal”);
TeardownTest();
SetupTest();
ValuesTest(“Josh Carroll”);
TeardownTest();

This is how it looks in the test runner:

Okay, that wasn’t too bad.  Now, let’s go a crazy.  Let’s get nuts.  Potentially throwing best practices to the wind.  We’re going make this example where it pulls the values from a text file instead of hard coding them.  To do that, let’s add a text file to our project:

Right click on your project (“TestingBlogSeleniumRc”) in the Solution Explorer -> Add -> New Item…

Select Text File and call it ValueList.txt

Open the file in your editor and type the three values into the text file.  One per line:

Next we want to make sure that the text file is copied over into the bin folder on build.

Right click on your project (“TestingBlogSeleniumRc”) in the Solution Explorer and select Properties.  Then select Build Events.  Under the Pre-build event command line: (or post-build; it doesn’t matter) paste in:

copy "$(ProjectDir)ValueList.txt" "$(TargetDir)"

Our test is going to be looking for ValueList.txt so that copy command assures that it’s in the same folder as the test dll.

(FULL CODE EXAMPLE BELOW)

• We need to add a reference to System.IO
• Next we’re going to add a global string array to our class called textPresentList
• We are going to read the ValueList.txt file into an array in our setup.  As coded, it should only be read in the first time the SetupTest() method is run.  If you’re a Unit Test purist, you could easily change it to read the file in every time, but I thought it seemed like a waste.
• Next we need to add the Range attribute to a copied and renamed version of our first test.  The range should be hard coded from 0 to 2.  (I tried to do something clever like put it from 0 to textPresentList.Length-1, but the compile wouldn’t have it.  These NUnit attribute values, it appears have to be somewhat static.  This kludge that I’m showing is way to get around the static nature if only a little bit.  This way the values can be dynamic even if the number of values can’t be.  By the way, if anyone has a better idea to make it more dynamic, please post it.)
• Change the selenium.IsTextPresent(textPresent) to selenium.IsTextPresent(textPresentList[i]).
• Then Compile and Run.

You won’t be able to see the values from the file in the Test Runner, but you could write the value translations to the console so that you could see which index corresponds to which value:
Console.WriteLine("Index " + i +": " + textPresentList[i]);

In the Test Runner, select the Text Output tab to view the output:

The whole point of this post is not to show you what to do, but instead to open up your mind to possibilities.  NUnit has some cool functionality, and most of us never scrape the surface of it’s potential.  I challenge you to go and explore the various attributes available for yourself.  Here’s the link: http://nunit.org/index.php?p=attributes&r=2.5.7

Remember upgrade your NUnit to the newest version.  Chances are that you’re not current.

FULL CODE:

using System;
using System.Text;
using System.IO;
using System.Text.RegularExpressions;
using System.Threading;
using NUnit.Framework;
using Selenium;

namespace TestingBlogSeleniumRc
{
	[TestFixture]
	public class FirstTest
	{
		private ISelenium selenium;
        private string[] textPresentList;

		[SetUp]
		public void SetupTest()
		{
            if (textPresentList == null)
                textPresentList = File.ReadAllLines("ValueList.txt");
			selenium = new DefaultSelenium("localhost", 4444, "*iexplore", "http://www.google.com/");
			selenium.Start();
		}

		[TearDown]
		public void TeardownTest()
		{
		    selenium.Stop();
		}

        [Test]
        public void DynamicValuesTest([Range(0,2)] int i)
		{
			selenium.Open("http://www.google.com");
			selenium.Type("q", "thetestingblog.com");
			selenium.Click("btnI");
			selenium.WaitForPageToLoad("60000");
            Console.WriteLine("Index " + i +": " + textPresentList[i]);
            Assert.IsTrue(selenium.IsTextPresent(textPresentList[i]));
		}
        [Test]
        public void ValuesTest([Values("Daniel Brown", "Marisa Seal", "Josh Carroll")] string textPresent)
        {
            selenium.Open("http://www.google.com");
            selenium.Type("q", "thetestingblog.com");
            selenium.Click("btnI");
            selenium.WaitForPageToLoad("60000");
            Assert.IsTrue(selenium.IsTextPresent(textPresent));
        }
	}
}

Déjà vu all over again! Another basic Selenium example. What the heck!? Can’t this guy do anything else?

Well, I agree this is somewhat repetitive, but I hope that this sequel will be in the league of Empire Strikes Back or Star Trek: The Wrath of Khan. Both movies, though they were sequels, were arguably greater than the originals. These sequels introduce us to the likes of Ricardo Montalban playing “Khan” and Billy Dee Williams playing “Lando Calrissian”.  And of course,  a muppet playing “Yoda”.

Of course, many sequels start off with the desire to be Empire, and instead they end up being:

• Caddyshack II
• Beastmaster 2: Through the Portal of Time
• Speed 2: Cruise Control
• Batman and Robin (No, we don’t need nipples on the Batman suit, Joel Schumacher.  But thanks for asking.)
• Mannequin: On the Move.
• Or the biggest sacrileges of all time Godfather III
• And Episode I (Empire, it is not!).

My hope is that this post doesn’t suck like those movies.  At the very least it’s nipple and Clooney free.

• Selenium RC
• Firefox
• Selenium IDE
• Java Runtime Environment (You should already have this)
• Visual C# 2008 Express Edition (Only get this if you don’t have a full version of Visual Studio)

Selenium RC
-No installation necessary. Just find a permanent location and unzip it there. I unzipped mine at C:\ just to make things simple

Firefox, Visual Studio Express, and Java
-Just run the respective installs with all defaults selected.

Selenium IDE
-To install, right click on the file and select “Open With…”
-Select Firefox in the programs list. If it’s not in the immediate menu, browse to it. (default location should be “C:\Program Files\Mozilla Firefox\firefox.exe“
-When Firefox launches, click button in the dialog box that says “Install Now”.
-After the install of the plug-in, hit the “Restart Firefox” button when you see it.

Okay, so, now that you have the tools, we’re going to record a simple test in Selenium IDE. Even though we want to eventually run the test in Selenium RC, Selenium IDE is a great tool for getting the basic test recorded since it’s all UI point and click.

• Start up Firefox.
• Click on the “Tools” menu and click on “Selenium IDE”
• (Take your time to explore the Selenium IDE form)
• Notice that it’s already recording.  The red circle button on the right side is pressed in:
• Type “www.google.com” the main Firefox browser address bar and navigate to it.
• In the search box, type in “thetestingblog.com” and hit the “I’m Feeling Lucky” button.
• Notice in your Selenium IDE window that steps are being recorded.
• With any luck, you hopefully landed on this blog’s home page.
• Highlight the text “Daniel Brown” on the page, right click.  You should select the option that says “VerifyTextPresent Daniel Brown”.  See below:

• Okay, we got our test recorded.  Now let’s run it to make sure it passes.
• Unclick the red circle button to turn off recording.
• Click on the green play triangle with the single bar to run the test

• All green, it passed.  Good.
• Now we want to convert this test to C#.
• Select “Options ->Format -> C# – Selenium RC” from the Selenium IDE window.

• Next select all the C# code and copy it.

• Open up Visual Studio
• Select “File ->New ->Project”
• In the New Project window under project types, select “Visual C# ->Windows”
• Select the “Console Application” Template and name it something like “TestingBlogSeleniumTest” and click “Ok”.
• It will create the project along with a file called “Program.cs”. Paste your SeleniumIDE test into Notepad. You’re going to take pieces out of the test and put it in “Program.cs”.

1. Add the using references from the SeleniumIDE test.  Replace any that are there.
2. Add the “ISelenium” and “StringBuilder” object references inside the Main() method.  As you see, I removed the “private” access control prefixes.  We don’t need them.
3. Add the body of the SetupTest() method after the object references.  (I put a comment to designate that it was part of the setup.)
4. Change “*chrome” to “*iexplore” for Internet Explorer.  Use “*firefox” for Firefox.  (There are other browser options, but those work best.)
5. Change the URL to the site you start your test on.  In this case, change that URL to “http://www.google.com/” since we’re starting from Google.

6. Paste body of the TheUntitledTest() method into the Main() method.

7. Paste body of the TeardownTest() method into the Main() method.

8.  Get rid of the Try/Catch blocks on Teardown.

9. Get rid of the StringBuilder references.

10.  Get rid of the Try/Catch block in the test body, replacing it with an if statement that writes to the console window if true (See my code below).  Notice that I have to cast the Selenium.IsTextPresent statement as a boolean.  I don’t understand why it doesn’t work without it because the return type for that method should be boolean.  But whatever.  I’m just dummy tester; what do I know?

• Next we need to add all the references so this thing will run.
• Go to the solution explorer, and right click on the “References” and select “Add Reference…”
• Hit the “Browse” tab and navigate to where you unzipped your SeleniumRC files.  Look in the dotnet client driver folder.  Here’s the path to mine:

C:\selenium-remote-control-1.0.1\selenium-dotnet-client-driver-1.0.1\

• Once you navigate to where the proper *.dlls are, select all of them and click “OK” to add them to the project.  (I know you can probably get away with adding less, but let’s keep it simple.)

• Build the project to see if it works.  (Mine worked.  Hopefully yours did too.)

• In order to run the Selenium RC test, you first have to have the Selenium RC server running.  The Selenium RC server folder should be in the files you unzipped.  Normally, I have a batch file on my desktop created to run the Selenium Server, but for our purposes, we’ll just run this from the “Start -> Run” commandline, using this command:

java -jar C:\selenium-remote-control-1.0.1\selenium-server-1.0.1\selenium-server.jar

• You should see some a dos window popup, and you will see something like the screenshot below.  DON’T CLOSE IT WHILE TESTING; IT NEEDS TO STAY RUNNING.  If you don’t have Java installed, this won’t work.  But most people should already have Java.

Next you need to select Debug from your Visual Studio menu and select “Start Without Debugging”. (I’m choosing this option because it doesn’t close the console window after the end of execution.  The debug option does.)

You should see the Selenium window(s) pop-up.  And it should do it’s thing.  When it’s finished, you should see something like this:

So, where is my test executable?  Well, it’s in the bin\debug folder of the project.  For me, the path to it looks like this:

C:\Documents and Settings\dbrown\My Documents\Visual Studio 2008\Projects\TestingBlogSeleniumTest\TestingBlogSeleniumTest\bin\Debug\

So, what do I need to run the program?  Just take the executable file and the dll’s.  Make sure the dll’s are in the same folder as the test when you execute it, otherwise you will get a big fat error.  If you send the executable to someone, make sure you send the dll’s too.

Remember if you port that executable to another computer, it needs to be a Windows box with the .NET framework installed and Selenium server needs to be running.  I hope this actually helps someone.  I’m going to post some tricks to “sex up” your console app.  Be looking for them shortly.

Sorry, I still haven’t figured out Selenium GRID good enough to post on it.  Maybe Marisa could post on it?  She’s awesome like that.

Yesterday I started working on an automated test (or, if you prefer, collection of checks) that needed to verify data in a result set, so I started learning about the Slim Query table. The query table example and corresponding fixture code presented in the FitNesse user guide are fairly straightforward. The part that I really had to wrap my head around was the statement “The fixture class must have a query method that returns a list of rows. Each row is a list of fields. Each field is a two-element list composed of the field name and its string value.” Say whaaat?

Now, maybe for those of you who have more of a programming background than I do, you might have read that statement and immediately pictured what the Query results “look” like. I didn’t. I’m used to thinking of query results as a table, not as lists of lists. I had to create a visual.

Given this result set:

Starting at the lowest level, we have “Each field is a two-element list comprised of the field name and its string value.” OK, so for row 0 column 0 we’d have:

For row 0 column 1 we’d have:

etc.

Let’s go up a level. “Each row is a list of fields.” That makes sense now. For row 0 we’d have:

Finally, the Query method “returns a list of rows.” Ohhh now it makes sense.

Hmm…but then I had to figure out how to translate a DataTable result set into the list of this format. It was actually pretty simple; iterate through the rows and columns to create what I call column-value pairs, collect each row’s column-value pairs into a list, then collect the rows into a list.

I’ve provided my solution below (sorry about the formatting – this theme’s code tag sucks).

It’s a method that takes a DataTable and creates the list of lists of column-value pairs that will work with Slim Query tables. So, in each Slim Query Fixture class, I just call off to the method and pass in the DataTable of results, then return the list of lists of column-value pairs from the Query() method.


using System;
using System.Collections.Generic;
using System.Data;
using System.Text;


namespace Your.Namespace.Here
{
public static class SlimQueryResultsHelper
{
// Convert a result set in DataTable format to the format required by the Slim Query test table
public static List ConvertDataTableToObjectList(DataTable theTable)
{
List queryResults = new List();
foreach (DataRow row in theTable.Rows)
{
List rowListOfColumnValuePairs = new List();
foreach (DataColumn col in theTable.Columns)
{
List columnValuePair = new List();
object val = row[col];
columnValuePair.Add(col.ColumnName);
columnValuePair.Add(val == DBNull.Value ? "null" : val);
rowListOfColumnValuePairs.Add(columnValuePair);
}
queryResults.Add(rowListOfColumnValuePairs);
}
return queryResults;
}
}
}


Because I love my readers and aspiring software testers, I’m going to give a dummy tutorial on how to get Selenium RC up and running for C#.NET users using NUnit. I’m going to cover every painful detail. I’m confident once you have this example running, you should be able to do a whole lot of things (assuming you have some basic coding skills).

First off, you need to get all these things (if you don’t have them):

• Selenium RC
• NUnit
• Firefox
• Selenium IDE
• Java Runtime Environment (You should already have this)

Selenium RC
-No installation necessary. Just find a permanent location and unzip it there. I unzipped mine at C:\ just to make things simple

NUnit, Firefox, and Java
-Just run the respective installs with all defaults selected.

Selenium IDE
-To install, right click on the file and select “Open With…”
-Select Firefox in the programs list. If it’s not in the immediate menu, browse to it. (default location should be “C:\Program Files\Mozilla Firefox\firefox.exe“
-When Firefox launches, click button in the dialog box that says “Install Now”.
-After the install of the plug-in, hit the “Restart Firefox” button when you see it.

Okay, so, now that you have the tools, we’re going to record a simple test in Selenium IDE. Even though we want to eventually run the test in Selenium RC, Selenium IDE is a great tool for getting the basic test recorded since it’s all UI point and click.

• Start up Firefox.
• Click on the “Tools” menu and click on “Selenium IDE”
• (Take your time to explore the Selenium IDE form)
• Notice that it’s already recording.  The red circle button on the right side is pressed in:
• Type “www.google.com” the main Firefox browser address bar and navigate to it.
• In the search box, type in “thetestingblog.com” and hit the “I’m Feeling Lucky” button.
• Notice in your Selenium IDE window that steps are being recorded.
• With any luck, you hopefully landed on this blog’s home page.
• Highlight the text “Daniel Brown” on the page, right click.  You should select the option that says “VerifyTextPresent Daniel Brown”.  See below:

• Okay, we got our test recorded.  Now let’s run it to make sure it passes.
• Unclick the red circle button to turn off recording.
• Click on the green play triangle with the single bar to run the test

• All green, it passed.  Good.
• Now we want to convert this test to C#.
• Select “Options ->Format -> C# – Selenium RC” from the Selenium IDE window.

• Next select all the C# code and copy it.

• Open up Visual Studio
• Select “File ->New ->Project”
• In the New Project window under project types, select “Visual C# ->Windows”
• Select the “Class Library” Template and name it something like “TestingBlogSeleniumRc” and click “Ok”.
• It will create the project along with a file called “Class1.cs”.  Paste your SeleniumIDE test into that file.

1. Rename “Class1.cs” to “FirstTest.cs”.
2. In the code, change the namespace from “SeleniumTests” to “TestingBlogSeleniumRc”:
3. Modify the class name from “Untitled” to “FirstTest”.  It’s good to change this so that if you add other tests, there won’t be any duplicated class names in the namespace (This has gotten me before, causing compile errors.).
4. We also have to modify the Selenium instantiation to launch Internet Explorer.  Where it has “*chrome”, change it to “*iehta”.
5. Where it has “http://change-this-to-the-site-you-are-testing/”, change it to “http://www.google.com”.

• Next we need to add all the references so this thing will run.
• Go to the solution explorer, and right click on the “References” and select “Add Reference…”
• Hit the “Browse” tab and navigate to where you unzipped your SeleniumRC files.  Look in the dotnet client driver folder.  Here’s the path to mine:

C:\selenium-remote-control-1.0.1\selenium-dotnet-client-driver-1.0.1\

• Once you navigate to where the proper *.dlls are, select all of them and click “OK” to add them to the project.  (I know you can probably get away with adding less, but let’s keep it simple.)

• Build the project to see if it works.  (Mine worked.  Hopefully yours did too.)

• In order to run the Selenium RC test, you first have to have the Selenium RC server running.  The Selenium RC server folder should be in the files you unzipped.  Normally, I have a batch file on my desktop created to run the Selenium Server, but for our purposes, we’ll just run this from the “Start -> Run” commandline, using this command:

java -jar C:\selenium-remote-control-1.0.1\selenium-server-1.0.1\selenium-server.jar

• You should see some a dos window popup, and you will see something like the screenshot below.  DON”T CLOSE IT WHILE TESTING; IT NEEDS TO STAY RUNNING.  If you don’t have Java installed, this won’t work.  But most people should already have Java.

• Next you need to launch the NUnit GUI to run the test from.  Since you installed NUnit, you should be able to navigate to it “Start -> All Programs -> NUnit 2.4.7 -> NUnit GUI (.NET 2.0)”
• Once you open the NUnit runner, Click “File -> Open Project…” and navigate to the compiled dll from your test project.  For me, the path looks like this:

C:\Documents and Settings\dbrown1\My Documents\Visual Studio 2008\Projects\TestingBlogSeleniumRc\TestingBlogSeleniumRc\bin\Debug\TestingBlogSeleniumRc.dll

• Once it loads, click on “Run” to start the test.

If you got all green, congrats!  You just got your first passing Selenium RC test!  From here we can definitely get more complex. I’m sorry if I was too verbose, but I didn’t want to leave out any important details.  If you have any specific issues, please post a question; I’ll be happy to answer it.  Thanks for reading!

UPDATE  10/14/2009:

Thank you for your comment, Ron.  You pointed out something that I completely spaced on.  There’s one more step you need to complete before your test is ready.  You must get rid of the try-catch exception block.  As long as you have that block in your test, your test will never fail.  Nunit counts on an exception being thrown when an assert statement fails.  If that exception gets caught, then Nunit thinks that the test passed.  Why does Selenium IDE put that try-catch in when coverting the test to C#?  I have no idea.  I just know that it doesn’t need to be in there in our example.

BEFORE Try-Catch is removed:

AFTER Try-Catch is removed:

Just because I want to prove that it works, I will show you an example where I have two tests in the file.  One looks for the text “Jello Pudding”, and one looks for the text “Daniel Brown”.  Here’s code for the tests:

(Helpful Note to Nunit beginners: when Nunit runs, it executes the [SetUp]  method before each [Test] and [TearDown] method after each [Test] .)   After running the two tests, I have very contrasting results (as I figured I would).  The test that looks for “Daniel Brown” passed like a charm, but the test for “Jelly Pudding” failed:

The bar is solid red because one failure at any level of the tree is a failed test run.  THANKS RON!  Sorry to leave out something so important.

Another order of business is I didn’t mention in my article, as granular as it is, that you need Visual Studio!  THANKS DAWN!  I guess I just kind of assumed that you might have it if you are interested in running Selenium in C#.  Well, I hate to assume so if you don’t have Visual Studio, you can download a free copy of Visual C# 2008 Express Edition: http://www.microsoft.com/express/download/ I believe you can run this example using the express edition, but I haven’t tested it.  If you do test it, give me some feedback.  I’d like to hear how it works.  Also, instead of the Visual C# Express Edition, you may also be able to use the Microsoft Visual Web Developer, but again I haven’t tested it.  I don’t know for sure.

I’m working on automating tests for a web app that has a lot settings configuration features, and many settings are managed in the UI via interactive grid controls. As the details of this app are proprietary, I will of course use a fake example. Since I’m on a Beatles kick, let’s imagine (har  har sorry for the bad pun and yes I know that was Lennon solo) I’m testing an album ranking control that looks like this:

If I record my interactions with the grid using Selenium IDE, the output is something like this:

Notice anything? I notice a couple things.

1. Aside from the fact that I see “grdBeatlesAlbums” in the locator string, there is no context to the rest of it – “//tr[4]” doesn’t appear to correspond to Abbey Road which was the album I was moving up in the ranks.
2. Yikes – am I really going to have to figure out the table row number for the album I want to act on in every test involving this grid?

My first inclination was to write a method that would determine the row position of an album with a given name. Then I realized that would involve parsing HTML. Yuck. I thought to myself, there has to be an easier way.  I went back to the Selenium docs to read about control location strategies a little more closely and saw that xpath would probably do the trick.  As so succinctly put on w3school’s page, “XPath is used to navigate through elements and attributes in an XML document.”

Now my problem was figuring out how to write an xpath expression that would locate a control in a column “next to” a given album name.

I wrote an almost-plain-English sentence to describe what I needed.

Version 1: Locate the “Move Up” link in the column adjacent to “Abbey Road.”

I thought “adjacent” would be a difficult concept but after Googling for a bit, I found information about the following and preceding xpath axes.

Version 2: Locate the “Move Up” link following the cell containing the string “Abbey Road.”

OK, so now to translate this to xpath – the Move Up link is just an HTML link, a.k.a. an “a” element.

a[.='Move Up'] translates to: a element whose inner text exactly matches ‘Move Up‘

And the cell containing “Abbey Road” – well, that’s just a table cell, or “td” element.

td[.='Abbey Road'] translates to: td element whose inner text exactly matches ‘Abbey Road‘

Next step: relate the two.

td[.='Abbey Road']/following::a[.='Move Up'] translates to: a element whose inner text exactly matches ‘Move Up‘, which follows a td element whose inner text exactly matches ‘Abbey Road‘

We just need one more thing: since the td element is not the first element on the page, we need to prepend the statement with // to indicate that the entire document (page source) should be searched to find the element.

//td[.='Abbey Road']/following::a[.='Move Up']

Even though the end result isn’t exactly plain English, it’s a lot more clear to me than “/tr[3]/td[4]/a” which is what the IDE recorded. It could also be reusable, providing the album name is passed via a variable instead of hard-coded.

Using xpath, you can also locate elements whose attributes (e.g. id or name) match certain criteria, and there are some handy functions like contains() to help in situations where a control’s full id or name is not known beforehand.

I’d still prefer the convenience of using control IDs, but there are just some situations where that either not possible or practical – xpath is now my fallback for tricky UI.

Update 8/28/09

Since I didn’t include this originally, the way to write an expression to find an element based on attribute values is

elementtype[@attributename='attribute value']

e.g. //table[@id='someTableID']

An example with contains():

elementtype[contains(@attributename, 'attribute substring')]

Now on to the real topic…I’ve been working with a software development team to guide them in their implementation of agile and automated testing techniques for about a month now. At the first sprint demo (yes – the team practices Scrum), I wanted to show the team a couple of examples of simple automated tests I had created for one of their applications. I demonstrated a GUI-driven test and a sub-GUI test. The demo went off without a hitch and the team seemed eager to start automating their tests in these manners.

The next day, I was to conduct a more detailed meeting with the testers regarding my suggestions for implementing automated tests to cover a particular application. Again, all was going well until I said something (I thought to be) pretty benign – something along the lines of “Well, I’m hoping that the architecture of the app is such that most of these test actions and verifications can be performed sub-UI.”

I heard brakes screech.

All of a sudden, a debate ensued about how driving the application some other way than through the UI was not a “real” acceptance test (after all, “real” users have no choice but to use the application via the UI). I stumbled and fumbled to explain; I was so used to automating 90% of a test suite at the sub-UI level that I actually had to stop and ask myself the question: “Well, why DO we want to automate test actions and verifications primarily at the sub-UI level?”

Here are my primary reasons:

• A verification being performed on the UI may not actually be verifying what the tester intended to verify.

  For example, if I want to verify that a folder monitoring process has swept a test file and read the data from the file correctly, verifying that the file name appears in a “Processed OK” list on a screen only tells me that the file name is appearing in that list – NOT that the file was processed correctly (whatever “correctly” means with respect to the AUT).

  Ask yourself the question “How can I confirm that X action produced Y results?”

• Verification that information and data are presented correctly in the UI doesn’t mean the data stored in the database is correct!

  I actually once found a bug where the UI populated the Last Name text box correctly, but this was because both the insert and retrieve stored procedures were referencing the Middle Name column. In other words, the Last Name data was being stored in and retrieved from the Middle Name column, so it “looked like” the Last Name data was being saved and retrieved correctly.

  Ask yourself the question “Where are all of the ‘outputs’ of action X?”

• (Hopefully) it is not the UI’s job to “do” most of the application processing, but rather to transfer information to and fro the business logic layer.

Ask yourself the question: “Am I testing the code that’s actually responsible for action X?”

There are other reasons commonly posited, such as:

• The UI is fragile and therefore UI tests are more likely to break
• UI tests are much slower than sub-UI tests (and if we have a lot of slow tests, we’re less likely to want to run them)
• If we have to automate a test via the UI, we have to wait until the UI is done, and typically the UI is the last component to be completed (even within a sprint/iteration)

Do I still automate UI tests? Yes, but typically only to confirm that UI behavior is as expected – e.g. buttons are enabled/disabled given certain circumstances, fields or data are hidden for certain users, etc. I also automate 1 or 2 “all the way through” tests – from UI to backend – but leave detailed business logic tests for the sub-UI suite.

Those are most of the reasons why I typically prefer to automate most integration/acceptance tests at the sub-UI level – but don’t take my word for it – the Google Testing blog has a thorough and interesting post on this same topic.

There we have it. My first blog post. Ever. Please, respond to the poll if appropriate and comment as you wish (unless you disagree with me). Just kidding of course.