The typical track record for test automation

I've been in a couple of companies that have some experience in test automation, and below in very short sentences I'll show you their typical track record.

They,

1. Hear about test automation
2. Think: Holy Grail !!
3. Call test tools sales people
4. Listen to sales
5. Believe sales (who doesn't?)
6. Only remember 'record/playback'
7. Go to management
8. Make them cum in their pants talking about ROI and total automated test coverage usable by business people
9. Hire cheapest junior test engineer in the universe
10. Tell him to automate everything (Don't forget to tell him about record/playback off course)
11. wait 6 months
12. See automation in action and feel like God
13. Wait for new release
14. Notice that the tests do not work anymore
15. Discover maintenance
16. Call test tools sales people again
17. Yell at them for not mentioning maintenance
18. Estimate size of maintenance
19. Stay at home with depression for six weeks
20. Come back to work
21. Throw out automated test suite
22. Get yelled at by management for not delivering on promises
23. Do research
24. Discover: stability of UAT, GUI dependency, Data control, suitable test cases, ...
25. Discover framework
26. Start over
27. Hire senior test automater
28. Make a plan of attack
29. Make a test framework
30. Start with most suitable test cases
31. improve framework
32. Maintain framework for new versions
33. Have a maintainable, usefull test suitethat's beneficial for your company!! woohoo!

You would think that it's easy to skip the first 22 steps, but one way or another you see it again and again. There's a million whitepapers on the net about test automation, please look them up. It will save you money (lots) and headache (even more).

Happy automating!!

Test automation for terminal emulators

This is an article for beginning Quick Test Pro users. Quick Test Pro is a test automation tool from HP (former mercury). It is sold as a record/playback tool for making test cases. I’ll describe how to automate terminal based applications. Why? Because there’s a good way and a bad way. The problem is you’ll only discover exactly how bad the bad way is after you see the good way. So if no senior automation engineers are present there’s a very good change you’ll be going the bad direction.

Forget record and playback. I’ll repeat: FORGET RECORD AND PLAYBACK. But you’ll say: “but it is working for me!”. Well yes, it is, in the beginning. Standard use of QTP works by using a repository in which a objects in your AUT (application under test) are stored, so that when replaying QTP can access them. The problem is not making the tests. It will work perfectly, but the problem lies in maintaining the tests. Most beginning engineers will have a script for each terminal emulator screen and a central repository (I do hope you’re using a central repository, otherwise you can look forward to a full-time job in test maintenance.) and a datasheet (excel).
Now here’s the problem: when a field is added to a screen, you’ll have to update the repository, the script for this particular screen and the datasheet. So you’ll need to uses the application, QTP and excel. Sounds like a lot of work? Well it is!

So what can we do? The good thing about terminal emulators is the fact that objects are very clearly and easily defined. So would it be nice if we could skip the whole repository, skip the ‘one script per screen’ and be able to customize our complete flow and screen setup from within excel. Wouldn’t that mean we could have business users write the tests? Yes it would! J

How? Make a terminal emulator framework in QTP! The explanation below is a very very very basic flow of how we could do this, but it will give you an idea:

** Make an excel sheet. Make 1 tab that will contain the flow, make one more for each screen in your application. In the first tab, have a test case on each row, so make each row a series of screens (each column is a screen). On the other tabs (screens) each row contains the data for the fields on that screen related the the row of your flow, the names of the columns are the field names of the screen in your UAT.

Tab 1 – row 1: screen1 screen2 screen3
Tab 2 (name=screen1) – row1: dataforfield1 dataforfield2 dataforfield3

** Now where going to make a driver script in QTP. In it’s most basic form it’ll do this:

Loop1 – loop every row on tab 1 (every test case)
Loop2 – loop every column on tab 1 (every screen)
Goto the right tab
Loop3 – loop every column on the screen tab (field in AUT)
Find the field in the application
Fill in the field
Submit
(check reply)
End loop3
End loop2
End loop1

** How are we going to make the driver script in QTP? Descriptive programming!! Descriptive programming is not nearly used enough in QTP. For Terminal emulators it’s perfect. With descriptive programming there’s no need for a repository anymore, you can access any object from within the script. A little example syntax:

TestObject("PropertyName1:=PropertyValue1", "..." , "PropertyNameX:=PropertyValueX")

Look it up in the help file if you’ve never heard of it. It’ll change your life. The important thing is to keep your PropertyValues out of your script and into your datasheet (for example as the names of the columns on your screen tabs).

So that’s about it. Implement the loops, add some checking code and error handling and there you have it: your automated solution in 500 lines of code or less for terminal emulators that’s practically maintenance free and can be updated by adding data,tabs and columns in an excel sheet.
I’d say have a go at it, a very basic test could be easily implemented in a day, a full blow 1000 testcase suite in a week.

Virtualized Test lab

Virtualization in IT is big big business, but for some reason virtualization is still very little known when it comes to testing. At the clients I’ve been in my career, only one used virtualization in the test environment and it was best described as a first step. Some observations about virtualizing in test environments:

- Virtual test data

Every test professional knows the problem with test-data. There’s a test database which is very nice after the first set-up, but over some months or years it gets to be a real garbage can of good data, bad data, corrupt data, …, you know what I’m talking about. So .. make it virtual. Make it so that your updates are non persistent. If you reboot the machine it will be back in it’s original state. Now you’ll probably say, “but we need to create new test data that’s persistent for our new tests”. Off course you need to do that. So here’s how you do it. Make two test databases. One that allows persistent updates and one virtual that doesn’t. when you need to create persistent test data, connect to the first, and add your data. Schedule a job at night that makes a non persistent virtual DB from this DB and the next day you’ll have your VM up and running with your new data, do all your dirty test work on this one, if you screw up, restart and voila, you’re all set to go again.

- Virtualize the complete test lab.

You need to test your applications on a whole array of setups? You have a whole room of PC’s that’s hardly ever used? Virtual machines are your answer. Make one for every setup that you can imagine. Dump them on a fileserver and your test lab is ready to go. Your test engineers enter the office choose their machine of choice and are happily burning test cases in no time. No more hardware that’s hardly used at all. You can make them persisten, non persistent, anything you want, it will save a lot of setup time.

-Performance testing

Do not use virtualization when doing performance tests. Although the technology is getting really mature, there will always be a performance penalty. And even if there’s almost none, prediction of scaling in an production environment will be quite impossible

So in short: When hearing 'test lab' or 'test environment', consider virtual machines, it won’t always be possible but test labs are made for this technology.