woensdag 6 februari 2008

The typical track record for test automation

I've been in a couple of companies that have some experience in test automation, and below in very short sentences I'll show you their typical track record.

They,

1. Hear about test automation
2. Think: Holy Grail !!
3. Call test tools sales people
4. Listen to sales
5. Believe sales (who doesn't?)
6. Only remember 'record/playback'
7. Go to management
8. Make them cum in their pants talking about ROI and total automated test coverage usable by business people
9. Hire cheapest junior test engineer in the universe
10. Tell him to automate everything (Don't forget to tell him about record/playback off course)
11. wait 6 months
12. See automation in action and feel like God
13. Wait for new release
14. Notice that the tests do not work anymore
15. Discover maintenance
16. Call test tools sales people again
17. Yell at them for not mentioning maintenance
18. Estimate size of maintenance
19. Stay at home with depression for six weeks
20. Come back to work
21. Throw out automated test suite
22. Get yelled at by management for not delivering on promises
23. Do research
24. Discover: stability of UAT, GUI dependency, Data control, suitable test cases, ...
25. Discover framework
26. Start over
27. Hire senior test automater
28. Make a plan of attack
29. Make a test framework
30. Start with most suitable test cases
31. improve framework
32. Maintain framework for new versions
33. Have a maintainable, usefull test suitethat's beneficial for your company!! woohoo!

You would think that it's easy to skip the first 22 steps, but one way or another you see it again and again. There's a million whitepapers on the net about test automation, please look them up. It will save you money (lots) and headache (even more).

Happy automating!!

dinsdag 5 februari 2008

5 reasons why John Rambo would make a good test engineer


(and 5 why he wouldn't)

John Rambo would make an excellent test engineer. Here's why:

1. If the sytem has one more flaw left, Rambo would find it, exploit it, and tear the whole system appart. Not only the sytem, probably the whole computer room and the building with it (managers and developers included). Rambo tests until there's nothing to test anymore.

2. Being in forest and swamps all of his life, the man is not afraid of bugs, he probably eats them for breakfast. He doesn't see annoyance, he sees high energy snacks. Bugs rivitalize him. And not only those easy to find functional bugs, no, you know those pesky little memory leaks. Ha!, when Rambo looks at his screen these memory leaks would ask for a defect tracker account and report themselves!!

3. Performance testing. That's a no brainer, the man is his own virtual army. He embodies at least a million virtual users. If Rambo, presses 'enter' after filling in his URL bar in firefox, all major internet DNS servers go down and the google headquarters explode. The whole internet would vaporize under the load he would generate (and that's when he's pressing enter with his 'pinky')

4. Test automation. Did you see how he shot that helicopter in Rambo II. come on doesn't that say enough, this walking machine of death uses tools like no-one used them before. This guy could automate your regression suite by writing down "test automation tool" on a piece of paper and waving with it. Your regression suite could run every testcase made since the big bang.

5. Deadlines. You can put your deadlines anywhere you want, Rambo will save the day. 'Failure' is just not an option. According to him, it is not even a word. (it must be noted however, that there are multiple words that Rambo apparently has not heard about, but this not an issue here)

So far an excellent test engineer ... but then again .... here's why things could get ugly:

6. Rambo does not do process. He has lost his TMap somwhere in the jungle (or tossed it away, he doesn't quite remember), he doesn't do V-model, he is the original (upper body) V-model, the alpha and omega of software testing. Any use of process would just cramp his style of exploratory testing and plain tearing stuff appart.

7. Rambo does not report defects. As there is no software flawless, while reporting a defect, Rambo would find a bug in the defect tracker, exploit, an tear the whole .... you know what I mean, see point 1 if you don't remember. Rambo would actually have to be forbidden of touching the defect tracker.

8. There would be nothing left to fix. After Rambo is done testing, well, ..., that's it. There would be no fixing of found bugs or finetuning of performance, there would only be 'aftermath', there would be an emergency UN meeting, and some memorial somewhere but there would not be any bug fixing.

9. Social skills. Ok, seriously, the man has the social skills of a bucket of carrots. He would get in arguments with developers and bussiness people about 5 minutes into the job. This wouldn't be so bad if it weren't for the fact that he'll probably kill them all, which would probably fall bad with the stock holders.

10. Rambo does not like a desk job!!

Would he be an excellent test engineer or not? I'm undecided, if anyone knows the fact that would tip the scale one way or the other, let me know!!

How to become a test professional

So you want to be a test professional? Here’s my become a test professional in 1 month recipe.

Ingredients:
- One college educated applicant
- Computer/IT mindedness (IT education not really important)
- Analytical skills
- Communication skills
- A little cash

Recipe:

So you have a higher education, want to do something in IT because you are IT minded but don’t really have a degree in IT. No problem, you could be the perfect functional test engineer.

Functional test engineers are somewhere in between business people and technical people. Your job is to verify technical systems versus technical and business needs. So what’s important when doing this job. Analytical skills, ability to use technical products en very good communication skills because you will be talking to both ends of a very wide spectrum.

Step 1. Make sure you really want to do this. Make up your mind!

Step 2. Learning the skills:

- Go to the bookstore and buy:

One book about TMap – Tmap is the main test process used in the world, all other test processes are essentially rehashes of TMap. In this you’ll find the extremely important concept of the V-Model. Read it – Learn it. It should take you about a week, it’s not terribly difficult.

One book about SQL. The standard query language for databases. You want to access a DB directly? You’ll need SQL. Nobody in IT can do without. It’s not difficult so three days should be enough to learn it.

One book about UML (unified modeling language – a method of analysis for applications, it’s not the only one, but it’s an important one) - 1 week

- Goto: http://en.wikipedia.org/wiki/Software_testing - Read it! - 3 hours

- Learn about:

Test Techniques: ways to make test cases. Not used very often, but you will be asked about them. Make some execises. They are more difficult to do then to explain! 1 day

Testing Tools: Read up on testing tools, especially test and defect trackers.
If you have the opportunity to actually install and use a test tracker or or test tool, do so! Read up on automation and performance tools (not your biggest concern right now, but you need to know about them) – 2 days
(tip: Mercury/HP has about 80% of the market when it concerns test tooling so start there – Quality Center/Quick Test Pro/Loadrunner)

- If you have the cash get certified:

Look into ISEB (http://www.bcs.org/server.php?show=nav.6942) or
ISTQB (http://www.istqb.org/).

Don’t be afraid of certification, you basically have to be a retard to fail at these exams. If you study for 4 or 5 days, there’s no way you can fail this and it improves your chances of finding a job greatly.

Step 3: Put your resumé online, apply for some jobs, go for an interview and get a job!

If You can talk about all these subjects and understand them, there is no way that you can’t find a job as junior test engineer!

Happy cooking!!

Test automation for terminal emulators

This is an article for beginning Quick Test Pro users. Quick Test Pro is a test automation tool from HP (former mercury). It is sold as a record/playback tool for making test cases. I’ll describe how to automate terminal based applications. Why? Because there’s a good way and a bad way. The problem is you’ll only discover exactly how bad the bad way is after you see the good way. So if no senior automation engineers are present there’s a very good change you’ll be going the bad direction.

Forget record and playback. I’ll repeat: FORGET RECORD AND PLAYBACK. But you’ll say: “but it is working for me!”. Well yes, it is, in the beginning. Standard use of QTP works by using a repository in which a objects in your AUT (application under test) are stored, so that when replaying QTP can access them. The problem is not making the tests. It will work perfectly, but the problem lies in maintaining the tests. Most beginning engineers will have a script for each terminal emulator screen and a central repository (I do hope you’re using a central repository, otherwise you can look forward to a full-time job in test maintenance.) and a datasheet (excel).
Now here’s the problem: when a field is added to a screen, you’ll have to update the repository, the script for this particular screen and the datasheet. So you’ll need to uses the application, QTP and excel. Sounds like a lot of work? Well it is!

So what can we do? The good thing about terminal emulators is the fact that objects are very clearly and easily defined. So would it be nice if we could skip the whole repository, skip the ‘one script per screen’ and be able to customize our complete flow and screen setup from within excel. Wouldn’t that mean we could have business users write the tests? Yes it would! J

How? Make a terminal emulator framework in QTP! The explanation below is a very very very basic flow of how we could do this, but it will give you an idea:

** Make an excel sheet. Make 1 tab that will contain the flow, make one more for each screen in your application. In the first tab, have a test case on each row, so make each row a series of screens (each column is a screen). On the other tabs (screens) each row contains the data for the fields on that screen related the the row of your flow, the names of the columns are the field names of the screen in your UAT.

Tab 1 – row 1: screen1 screen2 screen3
Tab 2 (name=screen1) – row1: dataforfield1 dataforfield2 dataforfield3

** Now where going to make a driver script in QTP. In it’s most basic form it’ll do this:

Loop1 – loop every row on tab 1 (every test case)
Loop2 – loop every column on tab 1 (every screen)
Goto the right tab
Loop3 – loop every column on the screen tab (field in AUT)
Find the field in the application
Fill in the field
Submit
(check reply)
End loop3
End loop2
End loop1

** How are we going to make the driver script in QTP? Descriptive programming!! Descriptive programming is not nearly used enough in QTP. For Terminal emulators it’s perfect. With descriptive programming there’s no need for a repository anymore, you can access any object from within the script. A little example syntax:

TestObject("PropertyName1:=PropertyValue1", "..." , "PropertyNameX:=PropertyValueX")

Look it up in the help file if you’ve never heard of it. It’ll change your life. The important thing is to keep your PropertyValues out of your script and into your datasheet (for example as the names of the columns on your screen tabs).

So that’s about it. Implement the loops, add some checking code and error handling and there you have it: your automated solution in 500 lines of code or less for terminal emulators that’s practically maintenance free and can be updated by adding data,tabs and columns in an excel sheet.
I’d say have a go at it, a very basic test could be easily implemented in a day, a full blow 1000 testcase suite in a week.

Virtualized Test lab

Virtualization in IT is big big business, but for some reason virtualization is still very little known when it comes to testing. At the clients I’ve been in my career, only one used virtualization in the test environment and it was best described as a first step. Some observations about virtualizing in test environments:

- Virtual test data


Every test professional knows the problem with test-data. There’s a test database which is very nice after the first set-up, but over some months or years it gets to be a real garbage can of good data, bad data, corrupt data, …, you know what I’m talking about. So .. make it virtual. Make it so that your updates are non persistent. If you reboot the machine it will be back in it’s original state. Now you’ll probably say, “but we need to create new test data that’s persistent for our new tests”. Off course you need to do that. So here’s how you do it. Make two test databases. One that allows persistent updates and one virtual that doesn’t. when you need to create persistent test data, connect to the first, and add your data. Schedule a job at night that makes a non persistent virtual DB from this DB and the next day you’ll have your VM up and running with your new data, do all your dirty test work on this one, if you screw up, restart and voila, you’re all set to go again.

- Virtualize the complete test lab.


You need to test your applications on a whole array of setups? You have a whole room of PC’s that’s hardly ever used? Virtual machines are your answer. Make one for every setup that you can imagine. Dump them on a fileserver and your test lab is ready to go. Your test engineers enter the office choose their machine of choice and are happily burning test cases in no time. No more hardware that’s hardly used at all. You can make them persisten, non persistent, anything you want, it will save a lot of setup time.

-Performance testing


Do not use virtualization when doing performance tests. Although the technology is getting really mature, there will always be a performance penalty. And even if there’s almost none, prediction of scaling in an production environment will be quite impossible

So in short: When hearing 'test lab' or 'test environment', consider virtual machines, it won’t always be possible but test labs are made for this technology.

maandag 4 februari 2008

Communication at multi-lingual companies

I work at a tri-lingual company right now. Everyday there’s a mix of Dutch, French and English to be heard here. Needless to say, there’s a lot of confusion about how to actually have a conversation, write an e-mail or conduct a meeting. So after living in a tri-lingual country all my life and working at companies where there’s often as much as 4 languages spoken, I think I can give some general guidelines for communicating.

Verbal communication.
When meeting somebody new. Introduce yourself in your own language, he will likely do the same. If you speak the same language you’re all set to go (mostly accompanied by a mutual sigh of relief and happy smile).
If both speak a different language but you both understand each other, each should speak his own. I speak Dutch but I understand French (and speak it also), so I’ll speak Dutch while the person I’m talking to can speak French.
If one of the two does not understand the other, he or she should politely ask the other to speak in his language. If this is not possible a middle ground should be found, which is, in most cases, a third language. I say a third language but I could just as well say English.
The exception to the rule above would be talking to somebody higher in hierarchy where you’ll try to speak his language. If you fail miserably he’ll probably switch to yours but at least you’ve tried.
Written communication
If you start the communication, write in your own language, if the reply comes in your own language everybody is happy and the communication can continue. If the reply come back in his/her language just continue switching languages. If the reply comes in a third language (mostly English) continue in English.
Except when you’re initiating a communication with a group, start in the third language so nobody starts complaining. Again when communicating with somebody higher in hierarchy try to use his/her language. Easy huh!
When writing company wide documents, there will usually be a company policy, so that’s easy, just use the policy.
Meetings
When starting to talk to a group of people, just ask what languages they speak. In reality a third language will be mostly be used (in my case English). An exception seen a lot is when senior managers speak to the employees. In that case they mostly use a mix between the two main languages (in my case Dutch and French) which is quite funny to hear. They start with five sentences in Dutch, switch to French for the next five and so on. It’s strange to hear (because the message is not repeated in both languages!) but it is extremely difficult to do.

So these are my ideas about communicating in multi-lingual companies. Don’t know if it’s of any use to anybody because you need to live in a country as Belgium to have this kind of situation, but if you find yourself in a situation like this, be careful about communication as it can cause great problems and tension when done wrong.

Knowing what you’re worth

Try to find a new job every year. Yes, you read it right, try to find a new job every year. Let me clarify. I’m not saying change jobs every year (that would be a very bad idea actually), but see if you can find a new job.
This helped me a lot in my career. Even if you’re very happy in your current job, try to find a new one. Put your resume online or send it to a couple of companies and see what happens. The advantages of doing so are:

- You should be able to find another job in one month. If no company reacts, something is wrong. Maybe your skill set is not what it used to be, maybe the sector you’re working in is not doing so well. So if there’s no reaction, try to find out why, so you can act on it, so you can focus your efforts in the right direction.

- Know what you’re worth. Maybe you’re very happy with your salary, but then again maybe you very underpaid and you just don’t know it. By looking for another job you can get an idea of the salary you should be given.

- It boosts your self esteem and sense of security if you know you can find another job without too much hassle. It’s good to know that you can walk away if you should.

- Your resume stays up to date and your interview skills stay sharp.

But remember never walk away from a job too soon. Stay until your work is finished. Nobody likes a job-hopper. Just keep track of your possibilities!
Stay safe and work hard!
PP