Posts Tagged ‘test automation’

Retrofitting Cucumber to an existing codebase

Thursday, December 2, 2010

The new job (well, not so new now, but in any case my latest job) uses quite a bit of tech I’d not dug into previously. As far as source control goes, I’m used to centralised SC like SVN (or even CVS in a pinch), so getting my head around a distributed model (Git) took some doing (more work required there, but that’s fodder for a different post). Working with a new language and framework as well – I hadn’t touched Ruby on rails up to now, which was stupid of me – I’m finding it pretty freaking awesome for the most part.

It’s taken me longer than I thought to come up to speed. I’ve been scraping the rust off my unix-fu and with unfettered access to the codebase and the command line, I feel like a long-lost missing limb has been reattached. Between that and the whole ‘learning new stuff’ thing, I’ve not had much time for anything else. There seems to be light at the end of the tunnel though. I’m grokking more of the code base and starting to make some useful code contributions.

One of the things that I’m trying to do at the moment is put a little bit of structure in the way that the business group and the development group communicate. Right now there is lots and lots of communication, which is great. The confusion comes in where the lines blur between the business telling dev what they want and going to the extent of speccing out everything in such minute detail that there’s not much design work left to be done by the dev group. That might not really sound like a problem to some, but in practice I’ve found that it can make the dev guys feel micromanaged or like their opinion is not required/valued – especially when design-level decisions are architecturally difficult to implement. If non-tech design reaches too far and gets it wrong, it can mean wasted time and effort either for them when the developer rejects it, or for both if the developer codes it in order to prove the point. The thing I love about my current place is that there are no passengers, and this stuff tends to get worked out, but I think I can do more.

This is where I am hoping that Cucumber will help out. If I can put together a series of tests that step out the current behaviour with a DSL that both the business side and tech side can build their communication around, then I suspect we will find it a little easier to step through the desired behaviour in enough detail to be clear about functionality while leaving the dev guys to work out the implementation. It’s not the only reason I want to use Cucumber, but I do have high hopes in this area. It probably won’t hurt that people will be able to see this stuff turn into checks that can be executed automatically. The key is still in the communication. I’m looking to make the clarity of that communication easier.

Cucumber is new tech to me as well. Conceptually I get it, but it’s different when you start to get your hands dirty. After a lot of reading, I decided to go with Cucumber, Capybara and at this stage, I think a combination of Selenium and TestUnit (Ruby’s in-built unit testing framework). I’m a Capybara noob as well, so I’m up to my elbows in new stuff and loving it. Mostly. Sorta. Apart from the frustrating bits.

Almost the entirety of the literature I’ve found on this setup is for BDD – which is to be expected, given that’s what its for. All of the setup and how-to stuff is for building stuff that doesn’t exist yet. Great if you’re building from scratch, but not super helpful if you have an existing code base you want to fit tests to.

I had an utter bastard of a time getting Cucumber and Capybara to work at all. Granted that it’s probably because I’m working with code that does a few unusual things, but I didn’t expect quite as much screwing around as I had to do – so, I’m going to detail it here in case there are any other poor bastards out there who are trying something similar. Hopefully this will save you some frustration.

My setup is (as of writing) Ubuntu 10.4, using rails 2. I already have selenium installed, and the installation of that is not arduous, so I’m going to omit that part.

I installed the following gems:

libxml2 libxml2-dev libxslt1-dev
gherkin cucumber cucumber-rails nokogiri capybara database-cleaner

I’ve listed some that should be pulled in automatically by other gems’ dependencies. For instance, gherkin should be installed when installing cucumber, but for some reason I had to do it manually. YMMV.

After installing the requisite gems, it should be a simple matter of generating all of the cucumber goodness from the root of your rails project

(ruby) script/generate cucumber –capybara
(you can add other options if you want to have them built in)

If all your bits and pieces are installed correctly, you should see a bunch of files and directories get created.

Here are some things that tripped me up:
I thought I might do some work with rspec as well, so I pulled those down. If you’re thinking along the same lines, then Cucumber seems to barf if you install anything beyond rspec 1.3.3
I also had to mess around with the dependencies for the json gem in my project as cucumber seems quite particular about wanting 1.4.6

By default, cucumber will clean the database after each run (as is generally considered good automation practice). In my case however, I have enough stuff in the db that it doesn’t make sense right now to load from scratch or mock the bits I need. Maybe down the track but for the moment I’ve just switched this off in the environment file (more on this shortly).

Anyway, moving on. From root, there should now be a features directory. edit features/support/env.rb
You can either make your settings changes in here, or in another file and require it from here. This file is automatically generated, so if you plan on making a lot of changes here or you think you’ll have cause to regenerate this file, the latter might be a better option.

Here are the changes I put in

Capybara.default_selector = :css
Capybara.run_server = false
Capybara.app_host = <url of my testing area>
Capybara.javascript_driver = :selenium

One of the things that really screwed me around was not realising that Cucumber starts its own webserver and runs on that. I tried pointing it at the server instance that Rails kicks off, only to find that things were falling over and there was nothing in my logs to tell me why. I felt like a complete moron once I did find out, but then again, the docco in this area appears to be awfully light so I don’t feel too bad. Anyway – be warned. If you don’t want Cucumber to use its own webserver, you need to tell it so explicitly.

As I said, there’s bugger all out there right now on retrofitting scripts for an existing code base. It sounds like it’d be easy, right? The code and functionality is already there, so skip the red/green/refactor bit and go straight to green, right? Not really.

Rather than conceptually pre-fabricating your functionality, you get to look through your existing functionality, decide what is most important to test, work out where you need to retroactively install test hooks and build use cases that serve a business purpose, while not getting lost down a rabbit hole around functionality that currently exists, but perhaps doesn’t work quite how it should. Yay! Given that there is the potential for test to ‘just light up green’ when you write them, I’ve also found it a good idea to write checks that fail so you know they’re actually doing something, then correcting them – it sort of keeps the spirit of BDD/TDD even if you’re not driving any development just at this point.

Retrofitting Cucumber tests to your site is not as simple as it might sound. I am hopeful that it will be rewarding. It’s a little too early to tell just yet, but the signs are positive. If there’s any interest, I may follow up with a post about some more of the specifics of the actual hands-on retrofitting bit. If you have questions, fire away.

Tags: , , , , , , ,
Posted in Everything, Software Testing | 3 Comments »

Data driven Selenium in JUnit via @Parameters

Friday, July 9, 2010

I played around a bit with iteration using JUnit today. I have some generic tests that behave differently depending on the values fed to them. I don’t want to have iteration code living alongside each test (maintenance nightmare), so I wanted to use JUnit’s @Parameters tag to pull in my test data via a Preferences file and do the iteration for me.

It took me longer than I expected to get right, mostly because static methods annoy me and trip me up a bit. (Your parameters method must be static in order for it to work).

However, get it working I did. On the off chance that it saves someone else some time, here’s how it works:

import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Enumeration;
import java.util.Properties;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class paramTest
{
	final static String dataFile = "/dataDriver.txt";
	private String words;

	public paramTest(String words)
	{
		this.words = words;
	}

	@Test
	public void verifyThing() throws Exception
	{
		System.out.println("key: " + words);
	}

	@Parameters
	public static Collection<Object[]> data()
        {
             Collection<Object[]> returnList = new ArrayList<Object []>();
             InputStream dataStream = paramTest.class.getResourceAsStream(paramTest.dataFile);
             Properties dataProperties = new Properties();
             try{
                       dataProperties.load(dataStream);
             } catch(Exception e)
             { System.out.println(e);}
             Enumeration<?> e = dataProperties.propertyNames();
             while (e.hasMoreElements())
             {
                       returnList.add(new Object[]{dataProperties.getProperty((String) e.nextElement())});
             }
             System.out.println(returnList.toString());
             return returnList;
       }
}

Interestingly enough, the data comes back in the reverse order that it exists in the preferences file. Something to keep in mind if you want your tests to run in a certain order.

Tags: , , , ,
Posted in Software Testing | 1 Comment »

My Selenium setup so far

Wednesday, July 7, 2010

I’m having a lot of fun with this Selenium gig (shh, don’t tell anyone). There have been a few yaks shaved and I’ve seen the inside of some rabbit holes, but there seems to be a lot of good material online to guide you.

At this point, I’m running my tests directly from eclipse, though we’ll be looking to kick them off with Ant in the near future. I’ll jump off that bridge when I get to it.

LoggingSelenium
I’ve set up loggingSelenium to log test execution. I’m not sure what its current development / support status is over at SourceForge. The documentation it comes with is not real flash, but there’s a decent example here (via here). I have made a couple of custom modifications to the source of loggingSelenium as there are a few gripes that I have with it.
Some of the pluses are:

  • it will automatically grab screenshots for you when your scripts bork.
  • it color-codes your asserts and failures
  • it can wrap JUnit asserts and log them alongside your selenium commands.

Some of the not-so-good things are

  • it requires a hardcoded path for writing output
  • it looks like it’s geared to non-windows systems, given that out of the box, the screenshot links in html output don’t work.
  • the color of failed selenium asserts is different to JUnit asserts, the former being an easy-to-miss beige color.

The mods I made were quick and dirty hacks, one to fix the screenshot link and the other to make the beige color more noticeable. If I find myself with a spare second I want to get rid of the need to use a hardcoded path, or at least make it more nicely configurable. I also want to add a color selector so you can customise your output.

Autodocumentation
I’ve set up doxygen to generate documentation from code comments. I’ve not hooked it up to anything yet, but either an SVN pre-commit hook or getting ant to kick it off shouldn’t be a problem.

Test Structure
JUnit 4 does not require you to extend tests from a test class, which is quite nice, actually. Instead, I have written a parent class for my tests to extend from. The parent holds all of the setup and most of the methods that drive the tests themselves.

Currently I’m writing one test per class. The class contains a single method containing calls to methods in the parent and nothing else, so it ends up looking like a set of test case steps. The parent has the potential to get quite big, so as test case numbers grow, it’s likely that I’ll add further abstract classes between the current parent and the tests to group them more reasonably.

I’m importing data via Properties as a resource stream. I have data that is used to drive the tests and data that is variable, but predictably constrained. For instance, I use a vehicle class number as a driver. The class number refers to a class of vehicle whose data is stored against that number. Depending on the number selected, different vehicle data is populated during setup.

On the subject of Properties – If you’re developing in a language that uses double-byte characters (such as Japanese), if you are generating your Properties files as flat files, you’ll need to run native2ascii on them to convert your double-byte code to unicode. Otherwise, they will not be properly interpreted and things will fall over.

I have just started playing around with the @Parameterised tag and @RunWith(Parameterized.class) to get iteration working with data variation. I haven’t decided quite how I’ll run with that yet. I could do this at a suite level and control iterations of tests by feeding data to them, or I can have each test take care of its own iteration by setting up parameterisation within the tests’ parent class.

Browser setup
I’m doing mobile emulator testing at the moment, so I’ve set up a firefox profile specifically for it. I’m using the user.js file to alter the plugin config so that the browser starts up with the plugin activated and a particular device switched on.

When starting seleniumRC, I specify this browser profile specifically so it is automatically used when firing up the browser. I was hoping to be able to change device type on the fly by mucking around with chrome urls, but I’ve not found an elegant way to do this yet. Instead, I am swapping the user.js file out whenever I need to change. Essentially, you can load up the user.js file with calls to your plugins to switch them on/off and configure them based on what configuration the plugin makes available.

It’ll end up looking something like

user_pref(“<plugin.configItem1″, “value1″);
user_pref(“<plugin.configItem2″, “value2″);

You can hunt down config items by typing about:config into your browser. Any config item that is not set will be invisible however, so I ended up looking through the plugin’s jar file to see what else I could see.

So at this point, I have a loose collection of parts that are not quite all tied together but it’s getting there.
Other things I’m contemplating in the not too distant future is a web front-end so non-technical testers can select test suites and kick them off at will.

While I haven’t gone into code-level specifics, the links should provide a fair amount of detail. Let me know if there’s something I’ve not explained so well. If you have comments or questions re any of this, I’d love to hear it.

Tags: , , , , , ,
Posted in Software Testing | No Comments »

Adventures in GUI Automation – Basic Hierarchy

Thursday, May 6, 2010

On the suspicion that my last post amounted to a bunch of word salad for a few, let me see if I can add some clarity by describing the basic hierarchy I’m using for my framework.

At the top level, I have a suite control script that looks like this:

doSetup   ‘Calls to set up data and environmental variables and such like

Call Script1
Call Script2
…
Call Script n

Each script that is called is some sort of major transaction that an end-user would recognise.

If you missed the update I made to my last post, I’ve moved a number of function libraries to classes to lessen the amount of coupling required between the ValidationManager and the scripts. There is a small overhead involved in that the relevant class has to be instantiated at the start of the script. For debugging purposes, I have also added a commented-out debug section that calls the setup function that the suite control script does. Each script looks a bit like this:

‘#begin debug
‘doSetup    ‘uncomment this if debugging this script
‘#end debug

Transaction = loadTransaction    ‘This is the instantiation of the class that holds the Transaction’s functions

Transaction.function1
Transaction.function2
SharedFunction
Transaction.function3
…
Transaction.function n

For functions that are shared across multiple transactions, I have left them as a standard function library. On the plus side, the Transaction namespace aids clarity and lets you know where your functions live. On the downside, you can no longer access them by using the right-click->go to function definition option. No biggie imho.

The transaction classes aren’t that much different from your standard function library other than the fact that they’re wrapped in a class definition and there’s a bit of setup stuff at the start.

Class Transaction

Dim VerificationManager

Private Sub Class_Initialize()
VerificationManager = loadVerificationManager   ‘Here’s where the verification manager is instantiated
EndSub

Public Sub Function1
do.something
VerificationManager.CalculateExpectedState(“someAction”)
do.someAction
VerificationManager.Verify(“someAction”)
do.somethingElse

End Sub

…

End Class

I’ve still not gotten the all-clear to post the verification manager code, but as mentioned previously, it is a class also. There are 4 methods; 1 for setup and 3 relating to verification.

CalculateExpectedState calls a file called “someAction”. someAction calls a setup routine that creates a new dataSheet, also called someAction and populates it with expected data.
Back in the class method, the action takes place, at which point we return to the verification manager when Verify is called.

Verify calls the someAction file which calls a second routine that gathers data from the app and does any calculations required and stores the results in the someAction dataSheet. Verify then calls a diff function that compares the expected and actual results and reports. Finally, verify calls UpdateMasterData, which compares the actual data with the master data sheet and replaces anything that was changed.

At this point, there is a new data sheet created for each different call to the verification manager. This has the potential to get out of hand as the number of verification points increases. I’m thinking of using a single sheet and dumping the results to a file at the end of each verification. I’ll see what else occurs to me as I go.

I promised to talk a bit about what I’m doing with data. I still need to cover that. I’ve been having fun trying to define heuristics to govern where the data should drive the scripts and where I should handle variance programatically. I think I have something that works. We shall see.

Tags: , , , , , , , ,
Posted in Everything | No Comments »

Adventures in GUI Automation – getting my bearings

Wednesday, March 17, 2010

So I mentioned some of the inherent issues that I dislike about QTP 10. One of those appears to be that reusable actions intermittently fail. An action in QTP is a tool-defined item that collects a number of GUI manipulations (or function calls) into a named action. It sounds handy. It probably would be if it worked reliably, but it doesn’t – at least not for me. The other thing is that if you have reusable actions (let’s say for instance that you keep your common ones in a single file and call them from other files), you cannot modify those actions without first closing your current test file and opening the one where the action is defined.

Instead, I have organised my scripts into a series of user-centric actions, which I have defined as functions in function libraries. To be clearer, each function executes a series of steps in the gui that an end-user would recognise as an action. e.g. log in, navigateToQuotationScreen and so on. This seems to work pretty well. I end up with a QTP screen that looks like a high-level checklist of how to execute a particular workflow. You do not have the editing restrictions that you do with reusable actions, as you can have as many function libs open as you like.

Neato. So I’ve been using the record tool to get something that works. Mostly because I dislike VBScript. I think it’s the Orcish of programming languages (Maybe not even that. Maybe it’s Kobold </nerd>). The recording tool creates a bunch of objects and sticks them in the object repository. That’s fine, saves me having to do it myself. Drawbacks to this are that for dynamically created objects, you may end up with a bunch of identical objects with different names. Not too hard to figure out, given that you should be naming your objects something sensible anyway. You can weed out the dupes when you do this and update your script accordingly. I’m definately going to check out Albert Gareev‘s suggestion of dynamically loaded object repositories. That sounds quite handy. I will say though that at the moment I’m being assisted by some not-so-technical testers who I am weaning away from the keyword view and the active screen.

The recording tool doesn’t always use the best code for the job. It’s the same old story – want the job done right? Gotta do it yourself. After the initial script is in place, I’ve been reading through the code and separating it into user-centric actions. At the same time, I look for:

  • places where the code can be optimised
  • places where data should be dynamic
  • any fluff commands that QTP put in
  • potential duplicates in the OR

At this point, what I’ve ended up with is having a series of user-centric ‘tests’ – I’ll call them transactions. Each one is an action (as every QTP test has 1 action by default), but consists of a series of calls to function libraries.  Each function is named for a step that a user might consider part of the transaction. For example, if we were talking about a program that provides a quote for car maintenance, the list of function calls might look like this:

  • enterVehicleDetails
  • selectServiceTypeAndLocation
  • selectPreferredDates
  • completeQuotation

Each of these functions lives in an associated function library. These functions may in turn call helper functions (such as date/time functions to enter date ranges in the future). Each transaction hides the ugly details of what is going on under the hood and while you don’t get to use the ‘keyword view’ anymore, you do have a clearly labeled set of steps.

So at this point, I have some scripts. They work. They’re not robust and they don’t do any data verification to speak of, but it is a beginning. Next up – parameterization and making use of the data table.

Tags: , , , , , ,
Posted in Everything, Software Testing | 6 Comments »

Adventures in GUI automation

Friday, March 5, 2010

So I’ve been hacking away at QTP for the last couple of weeks (no, I didn’t make the decision to purchase it, yes I have to use it as the political consequences of not doing so are more painful).

I’m not a massive fan of QTP, mostly because I think that it’s really bloody cheeky for a company to charge six figures for a product because of its feature richness, then charge 5 figures for training because it’s so complex that no one can use the damn thing, not to mention that all the official written how-to information is safely tucked away behind a walled off site, available for those who pay their support fees. For the same price HP charge for basic QTP training, I could fly any world-class tester out here for a week to hang out with me and my team. I know which option I’d get more value from.

Luckily there are plenty of useful sites out there maintained by QTP users, which is handy because if your support fees are not paid up, you get bugger all from the source.

QTP 10 has some inherent bugs that annoy me and I’m still dubious about how well the shared object library will work for me in the long run, but leaving the ranting and bitching aside, I should say that it does seem to be doing what I tell it to most of the time.

I’m attempting to implement Michael Hunter’s Logical Functional Model (which I think is really bloody clever, btw) as best I’m able. It’s still early days yet, but we’ll get there. I do run into problems a lot, largely because if my inexperience with the tool, but perhaps some good will come of it if I share my pain with you :) I’m going to post bits and pieces that I discover, just in case they’re useful to others. I don’t know how well QTP will fit the LFM, but I’ll see what I can do about documenting my progress with that as well.

For the moment, here are a few links that I have found useful:

There are a few others out there, but they’re full of ISTQB ads, so I don’t really feel like linking to them. I’m sure you understand.

Tags: , , , ,
Posted in Everything | 5 Comments »