It's nice to see you all here. This is session 310, A Practical Approach to Software Testing. I'm Doug Simons from Redstone Software in Colorado. And in case you're not interested in software testing or you're only interested in impractical approaches to software testing, this slide is your cue to head for the exit.

Otherwise, I suppose you're here for a reason, and I just wanted to give you a quick overview of what this talk is about. First of all, we're looking at maximizing the value with your testing resources. It's a chronic problem that most testing organizations, the testing department in most organizations, tends to always be a little understaffed, underfunded, and overworked. So if you find yourself in that position, don't feel alone.

So it's really important to get the maximum value out of the testing resources that you've got, and we'll try and give you some practical approaches to that here. We'll mostly be looking at testing software from the user's perspective. This is not a session about unit testing of code, which is also very important, but we're looking at the experience that you're going to be delivering to your customers when you deliver your product to them. I'll try and focus a lot on the what and how of software testing. We'll have a little bit of theory and some slides along the way, but I also want to give you some real concrete practical examples here.

So what I hope you'll learn is some of the benefits of automation and software testing. Automation can be a huge help to you in carrying out your software testing since you have limited resources and limited time. It can be a real boon to you. And we'll talk about how to get started because sometimes it can be hard for people to know how to get going. They know they want to test their software, they know they would like maybe to automate some of that testing, but it can be a little daunting sometimes to know how to begin.

And I'll try and give you a couple of ideas about where you might focus your testing efforts. And along the way I'll introduce six principles of practical testing. This is some key take-home points that I wanted to give you. And throughout the demos you'll see some specific techniques of how to carry out your testing.

So I thought I should introduce myself first, let you know who I am. I'm a developer. I don't have a PhD in software testing. I'm not a testing expert, but I've been developing software for a long time. And in particular, for the last four years, I've been developing a product called Eggplant, which is a software testing tool that hopefully some of you have heard of. And in the process of developing Eggplant and in working with our customers, of course, most of whom are software testers, I've learned a fair amount about software testing. So I wanted to share some of that with you.

Now, who are you? I'm kind of guessing that most of you are maybe not software testing experts either. You're probably software developers, people who are doing some testing maybe or looking to do some more testing, and hoping to give you a little bit of a boost here in your software testing efforts as you move towards becoming a practical tester here.

Doug Simons One of the first questions that people are likely to ask when they get into doing some testing is, "Well, what should I test?" If you explore the testing literature at all, you're likely to come across a lot of different terms. Functional testing, regression testing, performance, compatibility, use cases, integration load. There's a lot of stuff out there, and it can be a little bewildering at first.

You may also be looking at your software. You probably have some complex software you want to test and wondering whether you should be testing the inputs or the outputs or the menus or the windows. The whole thing may be a little overwhelming at first. I just wanted to give you a brief word of advice in the immortal words of Douglas Adams from A Hitchhiker's Guide to the Galaxy, don't panic. You may be feeling like there's no way that you're going to be able to test the software. You may be feeling like there's no way that you're going to be able to test everything. Relax, it's okay because you can't.

There's no way you're going to test everything that you'd like to test. You've got limited resources, as I said, and even if your organization has plenty of staff and plenty of budget for testing, there's always one resource that you're going to be short on, and that's time. The system that you're testing is likely to be a moving target. Your developers keep developing things, and there's always more to be tested. Taking advantage of the technology. Taking advantage of some of these approaches and using automation is probably something that can really help you out.

So the thing to do, of course, since you can't test everything, is you want to focus your attention on the areas that are going to give you the greatest return. So where do you start? Well, my advice is to start somewhere. It's always good to get going at least and see if you can get going. If you're not sure where to start, I thought I'd toss out a couple of possibilities that occurred to me. One is to validate bug fixes.

Most of us, as we're developing software, we have bugs now and then. And so if you have a bug, this is a great opportunity to do some testing. You can create a test first to verify that, in fact, you do have a bug, that something isn't working correctly. And then once you've fixed the bug, you can create a test to validate that that bug no longer exists.

So that can be a great place to start and can help you to build up a library of things as you go. Clearly, one of the places that you want to focus your attention is on the central features of your application. What is it about your application that makes it special? What are your users using it for? And you want to focus on those areas the most.

Use cases is something that many organizations, certainly those who have more of a formal development process, use. If you're creating use cases as part of your requirements process, which describe how your customers will be using the product, that's a great thing to use as a basis of a test.

Risk is another area that you might take a look. If there's some area of your software that's a little shaky and you've had problems there in the past, that may be a good part of your application to focus some of your testing resources on to make sure that you don't have more problems going down the line. Another category of risks would be risks to your customers. Hopefully you're not doing anything that's going to physically damage them, but you may have areas that could lose data and that sort of thing, which we'll discuss in a little bit.

Risk is another area that you might take a look at. If there's some area of your software that's a little shaky and you've had problems there in the past, that's a good part of your application to focus some of your testing resources on to make sure that you don't have more problems going down the line.

Another category of risks is risk is an area that you might take a look at. If there's some area of your software that's a little shaky and you've had problems there in the past, that's a good part of your application to focus some of your testing resources on to make sure that you don't have more problems going down the line. Another category of risks is risk is an area that you might take a look at. If there's some area of your software that's a little shaky and you've had problems going down the line.

I'd also like to suggest that you consider automating from the start. Now, a lot of people look at automation and, you know, they're busy testing their software and they think, I don't have time to figure out automation and to get this stuff to work. And, you know, I can understand that, but it's kind of interesting. A friend of mine put it this way. It's like, suppose you're bicycling along, you're heading for your destination, you're pedaling away, and a friend of yours comes by in a car and says, hey, I'm heading for your destination.

I'm heading that way. Would you like a ride? And you say, oh, man, are you kidding? I'm pedaling as fast as I can. And, you know, a lot of us feel that way a lot of the time. We're pedaling as fast as we can. We don't have time to get off and get in the car. But sometimes it's a good idea to stop and consider whether what you're doing, you know, whether you're doing the same thing over and over again and expecting some sort of different results.

It might be a good idea to take the time and get off the bike and get in the car. Now, I recognize that automation takes some effort. It's not something that is without some costs there. But on the whole, the payback can be tremendous. So something that you should take a look at. So let's take a quick look at what are some of the benefits of automation that you might want to consider.

For one thing, automation provides constant feedback. If you've developed some tests-- let's say you're testing your application. You've created a test for some particular feature. You say, great, that feature works. And so a little later, you create some other tests, or you do some testing. But your application keeps evolving.

And as it changes, sometimes things that you've previously tested stop working. And so if you can repeat the same tests that you did before, you're that much better off. Now, when you're doing manual testing, you lots of times don't have time to go back and test the things that you've already tested.

With automation, you've got the computer there to do that work for you. And so it's easy to repeat those tests and make sure that you're not introducing new bugs in areas that you've already tested. Automation is great for doing repetitive tasks, things that a person is not going to want to do, doing the same thing over and over again. And of course, automation gives you consistent and reliable results which can be really valuable.

You can run regression tests after every build, so each time you have a new build of your software, you can rerun those tests again to make sure that things aren't breaking. You can perform tests repeatedly with different data. Testing with large quantities of data can be really valuable depending on what your application is, and automation is a great way to go about doing that.

Automation can also be helpful for reproducing some of those intermittent failures, the things that show up every once in a while and that are really hard to nail down when you're testing things manually. Every time you go to try that part, it works fine, and then one of your customers runs into it later. Those can be the most frustrating things to find and track down.

Well, God eliminates human error up here. Of course, automation doesn't eliminate human error. We're all human and we keep making mistakes, but if you can turn some of the task over to the computer, then at least hopefully it's not going to make those same mistakes. And the bottom line is that automation is going to increase your efficiency to save you time and effort.

So I said I was going to introduce some principles of practical testing. The first one I want to throw out to you here is to test the user experience, not the code. What do I mean by this? As developers, we oftentimes kind of lose sight of the application for the code. We get so involved with the code that we're focused on, and as we're developing it and working with it, perhaps you're doing unit testing of your code, which is great.

But keep in mind that no matter how much you test individual modules of your code, that doesn't guarantee that the application, the thing that your end user is working with, is going to work. So it's really important to test the final product. And so I'm going to emphasize this point again, testing from the user's point of view, because that's really where the value is to your customers.

So before we get into the demos, I want to tell you just a little bit about Eggplant so you'll understand what you're seeing. Eggplant is a two-computer system. It runs on a Mac. It's installed here on a Mac OS X system. I've got kind of an old picture there. And it connects across the network to a computer that's running something called VNC. Now, VNC is an open source product that does remote control of another system.

So some of you probably use this. It's great because you can access your Mac at home when you're at work or vice versa. The OS X VNC product that runs on OS X is actually maintained by our company. So if any of you are using that, it's free. If you're not using it, go get it. It's a great tool to use to remotely control another computer.

And what VNC does is serves up a picture of the screen. It's a great tool to use to remotely control another computer. And what VNC does is serves up a picture of the screen to your local computer and allows you to control the keyboard and the mouse. Now, what Eggplant adds to this is the ability to create scripts to automate things on the remote system.

And Eggplant is kind of unique in the way it goes about doing scripting because its scripts are visual. Eggplant, all it knows about the remote system is that it has a lot of pixels there on the screen. So in order to interact with that, it's looking for things visually on the screen, controlling the keyboard and mouse, and this, as it turns out, gives us a tremendous amount of flexibility to allow us to drive any software written in any language. It doesn't matter if it's Carbon or Cocoa or Java or PC. It doesn't matter if it's PHP or what. And it can be running on any OS.

So going back here to our principles, again, just a reminder, testing the user experience, not the code. And also, our next principle here is to start simple. This is a principle borrowed from agile programming approaches. Begin with the simplest test and build up as you go. And let's see how we're going to do that. So if we could switch to the demo machine. Demo? So the first thing we're going to do in Eggplant is open a connection to another machine.

Actually connecting to the laptop over here, which is-- I've got a screen scaled down to a nice small resolution there so we can fit it on the demo screen here. Eggplant likes a nice big screen. Since you're controlling another computer, it's nice to be able to see the other system. So it's a great excuse to go buy a big monitor.

But anyway, here we've got another system that we're controlling. You can see I'm moving things around on the desktop over there. And I've got the address book icon out there, because for our demos today, we're going to be creating some tests to test Apple's address book. Now the first thing I want to do for creating these tests is create a new suite here. And since we're testing the address book, let's just call this suite Address Book.

And a suite is just a collection of scripts and images and various things. We're going to go ahead and create a new script here. Now, as I said, we want to start very simple. So what's the simplest thing we can do to test the address book? Well, what we're going to do is simply launch it.

OK? Make sure that it launches. So I'm going to create a script here called Launch Address Book. This is a script editing window and I can just type a script over here. What I'm going to do is go over here and tell Eggplant what I want it to do with the system.

In dealing with Eggplant, if you think of it as a willing and eager, but not terribly bright assistant, you kind of have the right idea. I mean, this is a purple vegetable we're dealing with here. So we can't just tell Eggplant, oh, go launch the address book. It doesn't understand what that means. But instead, we have to say things like, oh, Eggplant, would you look on the screen and see the icon for the address book? Please double click on it.

So that's what we're going to do. And so we're going to select this area of the screen over here, kind of point that out to Eggplant and say, that's what the address book looks like, and I want you to double click on it. So I'll select double click on the toolbar here. Now, you can see that we're capturing a picture of that icon, and I need to give that a name in order to make our script readable. So I'm just going to call that AB icon here.

Now, when I clicked on Save there, three things happened. The first was that we actually did capture that image and save it over here in our suite. We also automatically created a command in the script that says double-click the address book icon. And the third thing that happened was that Eggplant carried out that command. It went ahead and executed that command.

And when it did that, Eggplant looked on the screen, searched wherever it needed to on the screen to find that address book icon, so it could have been anywhere, and it double-clicked on it. This is great because now we're ready to go on with the next step of our test. Oh, but that's all our test was, was to launch the address book.

Keep in mind that whenever you're testing something, it's a good idea to verify that what you're automating has actually taken place. So we don't want to end our test quite yet. First, we want to make sure that the address book is actually launched. And we're going to do that by looking for the title bar here. We'll just tell Eggplant we want to wait for that. So let's look for the address book title bar here.

And the wait for command, in this case, it's waiting up to eight seconds here for the title bar to show up. It's a good idea to allow a little bit of time at certain points in your script when things might take a little bit of time. When you're launching an application, quite often it may take a few seconds for it to come up and be initialized. Eggplant, if you don't kind of tell it to hold on a little bit, it's going to just charge right ahead with the next step. So these wait for commands can be really valuable.

The other thing that I think we'd like to do here in this test is kind of finish things up, put it back to the state that things were in. So we'll go ahead and quit out of address book as well. I'll do that by just typing Command-Q here.

And so now you can see that we're back to our initial state, which allows us to run our script here and make sure that it works. Just going to run it here, and you can see that it's launching the address book and quitting-- oh, it didn't take long. But there we are. Can we switch back to slides, please? Slides? Starting simple is a really great way to get going, building up tests as you go. And we're going to build some more tests on the address book in just a minute.

Another point to make here is to complete the cycle. So as you saw, we started off in a well-known state. Now this is important. To make your life easier as you're going about testing, it's really helpful if you can start with your system in a known state. If you can have a machine that's dedicated to testing, or if not a machine, then perhaps at least a separate user account that is used for testing, so that you know that things are more or less in a known condition. You don't have people tweaking different parts of the interface and all that.

It'll make your life a little easier. So start in a well-known state, perform your test, verify that your test has actually done what it's supposed to do, and then at the end, return to the initial condition again. This allows you to then repeat the cycle later if you kind of end up back where you started.

The next principle is a really valuable one: make it modular. If you can create some basic building blocks, then you're able to reuse those as you go on and create more complex scripts. Let's take a look at how we might do that. Can we go back to the demo machine, please? For our next test here, we're going to create a new card. Let's just go ahead and do that. We're going to create a new script here. Call it our new card test.

How's that? In this test, let's see, the first thing we need to do is to launch the address book. Well, it happens that we've got a nice script out there that can do that. If I just drag that in here, it says launch address book, and this will actually allow us to call one script from another. Let's bring that script up, and the only thing is that our launch address book script also quits the address book. We don't really want that to happen yet, so I'm just going to change that script now. We probably won't want to use it all by itself again.

We can verify that this works by just running our very short script right now. Sure enough, Eggplant has launched the address book for us, so now we can go on with the next step of our test. Now the thing that we want to do next is to create a new card. We can do that by typing a command Q, not a command Q, command N.

And so that's created a new card. It's now waiting for us to type in the first name. So we'll type some text here. And eggplant. Now the next thing we want to do is actually tab over to the next field. I'm just going to put type text command directly in here to tab over there.

Of course, when I'm typing in a script like this, Eggplant doesn't automatically execute that. We have a kind of a handy feature here called Run Selection that allows us just to select part of a script and run that. So I'm going to go ahead and run a selection here to just execute that one line.

And I could go through the toolbar items to type in our text and that sort of thing. I think it might be a little simpler to just copy and paste some of this code. So I'm going to do that, and we'll put in support here. And I think we need two tabs after that one. So let's have it do that much and see where we're getting here. So we're now ready to type in a phone number. So we'll put in the phone number.

And I'll just run that a little bit. We're kind of building the script and executing pieces of it as we go here. And the next thing we want to do is type in an email address. We need all these tabs here now. Just put in support at Redstone Software.

And there we go. So how much of that do we still need to run? We need to put in our-- So now we've entered some information on our card here, and the next thing I want to do is end editing. So I'm going to come back over here and to do that we're going to capture another image. The edit button down at the bottom there that's pressed in, we need to click on that to end editing. So I'll just call that end editing.

And so there we go. We've created a new card. As I mentioned, it's a good idea after doing something to verify that in fact whatever you've done has taken place. Why don't we do that by doing a little search for this card. We can do that by putting another type text command in here to look for eggplant support.

You may notice that the search field is already selected there after ending editing, so all we have to do is type in this name and we'll see that it's found that card now and we just want to verify that it's shown up. We're going to look for the word eggplant support, kind of all selected there. Once again, we'll tell eggplant just to wait for that to show up.

We'll call that eggplant support selected, I guess. So here we go. Now, the final step that we want to take is to kind of clean things up and quit out of here. So why don't we go ahead and delete this card. We can do that by-- I'm just going to put a comment in here to kind of-- oops. Remember where we're at. We're going to verify here. Now we're going to delete the card again. We're going to go ahead and delete the card. Go through the edit menu here. Just click on the edit menu. And click on delete.

There we go. Our card is deleted and we'll go ahead and quit out of Address Book. It would be nice to do this in a modular fashion like I was suggesting, so why don't we just create a command called Quit Address Book. So that our script is nice and readable, we haven't actually created that script yet, so we'll go ahead and do that. All this one needs to do is type a command "q".

We've got our complete script here now. Why don't we try running this one and see how that works out. This is kind of a nice point for you to go and get a cup of coffee and sit back and relax while eggplant does your work for you. And there it is. It's tested it. We know that everything is working great there. So let's go back to the slides from it.

Adding modularity is really important. We're going to see a little bit more of that in the next demo as well. Another principle that's really valuable in testing, particularly in automation, is repeat. To take advantage of the capability of the computer to do things repetitively. Some of the benefits of repeat are working with large quantities of data, for example. If you can feed your application a lot of different data, this can be really a good way to stress it out and make sure that it can handle all of the data.

You may also want to test boundary conditions. What do we mean by boundary conditions? Well, odd values, things like putting zeros into a field where it's expecting a quantity, or maybe even a negative number, or putting in a name where it's expecting a number, things like that, to kind of test some of the edges. And so you want to test a lot of different kinds of conditions, and being able to iterate over a lot of different sets of data can be really valuable.

A similar idea is testing with different combinations of data. Maybe you're testing a catalog, for example, or a clothing catalog, and you've got an item that comes in five different colors and five different sizes. Well, you can test each of your five sizes and each of your five colors. That's great, but what about the different combinations? So if you can kind of nest some repeat loops and test all of the different combinations, this can be a really powerful thing to do in case there are certain combinations that lead to issues. Cheers.

Intermittent failures is kind of an interesting one. I mentioned this briefly before. This is something that we actually ran into ourselves. Some people ask whether we use Eggplant to test Eggplant, and yeah, we do. We have a suite of tests that we call the Eggplant Squared Test, in which Eggplant drives another machine running Eggplant and driving another machine to create scripts and run those scripts and all of that.

So it's a nice test that we've got. At one point a couple of years ago, we had a bug, something was happening every once in a while, and we'd go through and try and see what was going on, and it would never fail when we were looking for it. So we ran our Eggplant Squared Test overnight. It ran about 100 times in the course of the night, and we came back the next morning to see what the results were and found that it had failed twice.

So this is the kind of thing that's really hard to track down, but by examining the logs in the cases where it had failed and looking through that in detail, we were able to figure out exactly where the problem was and ultimately fix it. So let's do another demo here, if we can go back to the demo machine, and take a look at how we might do some repetition.

So I have a data file out here, some valuable data that we want to iterate over and create a number of cards in the address book this time. So let's start off by creating a new test. We'll call this the "Many Cards Test." And as you can imagine, the first thing we're going to want to do is to launch the address book. I'll just drag that script in.

And at the end, we're going to want to quit. So here I've got the beginning and end of our script written already. That's nice. In between, we're going to want to do some repeating with some of this data in here. And so I'll put it in a repeat loop. But what do we want to do? We're going to repeat. Let's give a name record here. And we're going to set that to be each line of this file. Now, I don't know where that file is exactly. I'm just going to drag that in here.

Okay, so there it is. We're going to repeat with each line of this file. And now we've got something called record that's being set successively to each of the lines of the file. Now the lines in the file, as you can see, have several different items of information.

There's the first name and the last name, the phone number and the email address. They're all separated by tabs. So I don't really want to deal with that as a whole. I need to split that up. So I want to take and turn around and put...

And so this is a way of very quickly accessing all the data in the file, but... What are we going to do with that? Well, we want to create an address card for each of these.

So we just happen to have a script out here that does a pretty good job of creating a new card. Let's see if we can borrow some of our previous work. I'm just going to copy that part of it there and create another new script called "Create new card, let's say. Okay. And we'll just put all of that in there.

To call that script, let me just drag this in over here, create new card, and we're going to need to pass it some information. So we've got these four items of information that we want to pass over to that other script. So we'll do it like that. Over in this script we need to be able to receive those items, so I'll put them over there as well.

Now, what we've got of course is some code that we copied over from another script that's going to type the word eggplant in here, which is not the name that we want. So we'll just replace those with some of these other Terms here, first name, the last name, and we'll just put phone in here, no email, phone, and then email.

Okay, so there we've got that and yeah, that looks like it ought to be pretty good. So why don't we try, let's see where we're at now with our script. I don't think we need to look at this anymore. Maybe we could just run that script, all except for actually quitting things here. We'll just run the first part of it and see how that works.

So it's launched the address book and you can see that it's kind of going through, entering the information here in the script. If we wanted to, here in our run window, we could pause the script and kind of step through one at a time, take a look at some of our variables. We want to put the phone here and see what number it's working with. So we've got some ability to interact there. Let's just go ahead and continue, let it finish entering this information in there.

That's nice. Now we've got a bunch of cards out there with a lot of data in it. The next step that we want to do, of course, is to verify that it's all there, and we're also going to want to kind of clean things up. I'm a great believer in being lazy here, so instead of typing this all again, I'm just going to copy and paste. Now we've got another loop to go through the same set of data again. This time, instead of creating a new card, we will verify. and delete the card.

Okay, so we need another little script here then to do that, making things nicely modular. So I'll just copy that name, create a new script by that name. Now this script is going to want to do what's more or less the last part of our original script out there, the verifying and the deleting, right? So if I just kind of copy that code over here. Once again, we can get some data names here for our parameters.

So, Rather than typing "eggplant support," of course, we're going to want to type the first name and last name that came from the data file. This will be typing in the search field at this point. There's a little quirk that I've discovered in trying this out, that if you don't clear out the search field first, you end up putting a lot of names in there together. So it's a good idea to do something to clear that out. We'll do a type command A here first. That will select the entire contents of the search field before we start typing so that we're not just adding names to each other.

So, and there we go. Now, we've also got a line that we copied here from the other script that says we're going to look for eggplant support selected. Well, that's not one of our data items, and certainly it's not all of our data items, so we can't do that quite the same way in our more general script here. Look for-- oh, I left something out of the other script. It's going too fast. After we deleted the menu item, you know what we're going to do? We're going to build this on the fly here.

Here's what we're going to do. We forgot to verify that, in fact, when we had deleted our card, that it was gone. So I just want to add some of that in here.

There's something that shows up. It's called no cards selected. We haven't actually captured this image yet.

We'll do that in a moment. If that image is found, then we know that something went wrong because, in fact, we have not found the thing that we were looking for. And so in that case, we're going to want to log some kind of a warning here to indicate that there's been a problem.

On the other hand, if it was found, then we can go ahead and delete it here. So the first thing we need to do is capture that image that we kind of missed out on before. Let's go into live mode over here. This is what's going to show up in the case where we're searching for something that's not found. Let me just capture an image of that. I'm going to capture mode here. I'm just going to capture an image called "No Matching Cards." So that's the image that we'll be looking at. Oh, no, no card selected. Well, it's called no matching cards.

Sorry, I got a little confused there. But there we are. And once we're done, we want to also verify that when we've clicked the delete menu item, that in fact, that image is there. So we're going to put in our wait for-- give it eight seconds again, I guess-- for the no matching cards here.

That was the little step that got left out of our other script. So let's take a look now and see if this is going to work. We've got all of our cards in here still. We've got a little repeat loop. Let's just kind of run that as a selection and see if I made any other mistakes there. Looks like something's not quite working right. Verify and delete the card.

Okay, we'll just step through that real briefly.

That would make a difference. We were calling an empty script because it had never been saved. So if we run this again, maybe we'll get a little better results there. So you can see it's typing in Harry Potter here, searching for him, deleting him out of the address book, and so forth.

Going through, verifying that each of those cards is there, and getting rid of those cards. And once that's done, we were just running a little selection there again. Let's go ahead and let it finish by quitting out of the address book. And let's see if we can run our script in its entirety one more time.

Once again, being able to repeat through a quantity of data is really a valuable thing, something that you're not going to want to sit and type all these names in by hand. Eggplant is actually working through the user interface just like a user would put these cards in. In some situations, you may have cases where you could load a lot of data into a database directly using some back-end tool, and that can be a valuable thing to do if you need to have a lot of data in there that you want to test against.

But there may be situations where your application is going to fail if somebody sits and enters hundreds of data items directly through the user interface. So you may want to do both kinds of testing, and that's one of the values that we've got here with automation. So let's switch back to slides, please.

So once again, leveraging repeat is a great thing to do. Once you've done some automation, this is one of the things that's going to give you a great benefit. About this point, you may be thinking, OK, well, great. We've got started. We've got some tests going. I've got a little bit of momentum here.

And what should I test next? Well, here's a few little ideas, at least. One is to kind of grow as you go. So what does that mean? Well, as you're creating your application, you're going to be continuing to add new features. So it makes sense to add the tests for those features as you go. And this is a great way to build up a larger library of automated test scripts that you can run after each build.

Another thing, of course, is to focus on high-value areas of your application, because you really want to maximize the value that you're getting out of your testing here. So you want to focus on the things that are important to your customers and also take a look at those high-risk areas again, things like bug regressions.

What do we mean by bug regressions exactly? Well, when you have a bug and you fix it, that's great, but something that happens sometimes is that as you go on and add other features in your application, it occasionally happens that those same bugs occur again or something in the same area of the program. So it's a good idea to have tests in place that will guard against your application regressing in that way. Crashes, obviously, or other areas of reapplication where there's potential for data loss for your customers. And once again, the boundary conditions, testing a variety of different data there.

You also may want to focus on the perceived value to your users. So what are your users looking for in your application? They need the usability there. They want to be able to use your application easily. There are certain features of your application that are really important to them, and this is really critical. Another area, lots of times, is performance.

You care about how well your application performs. If it's getting really sluggish, your users aren't going to be really happy. Let's just talk about that a little bit more, because performance testing gets kind of a lot of attention sometimes. As I see it, there's a couple of different ways to look at performance. One is that you can look at it in terms of the code and your algorithms. As developers, we love to get in there and tweak things and make them run faster, and that can be great.

Sometimes this provides an enormous benefit to your users. But at other times, you may speed something up by ten times, but if it only accounts for one percent of what's going on in your application, the actual benefit to your users may not be all that great. So it's important to look at also the user experience. How does the performance appear to the user? Because their satisfaction is really based on that. So for example, let's say you've got some kind of a search that you're doing in your application.

So you might have a little script that clicks on a search button and then waits for, in this case, up to 30 seconds for whatever the next thing is to show up on the screen that's going to indicate that the search is done. Well, this is fine. An eggplant will come along and do this, and it'll wait for up to 30 seconds for the search to finish.

But your user may not be too happy if they're waiting around for 30 seconds. So you might want to add a little extra code in here to record the start time. Before you start, you might want to start the search and then afterwards subtract the start time from the current time to see if it's greater than five seconds. Then that may be something that you want to be aware of. So this little bit of code here would log a warning that'll show up in your result log that you can see later then to know that things aren't quite up to par there.

So it's important to validate what's valuable to the user, because ultimately what you're really trying to do here-- I know I've said this quite a few times already-- but really to deliver a quality experience to your customers. And so this is a great way to focus your testing when you really can't test everything. Let's go back to the demo machine again here.

Now, about this point, some of you may be wondering, well, how do you know if all these scripts are succeeding or if they're failing and what's going on here? I've kind of referred briefly to a results log in here. Let's go back and take a look at our suite that we created. We've got a number of scripts out here now. We've also got a number of different images that we've captured in the course of developing these scripts.

We also have something called the results tab here, and that is where we list the results of the different tests. So our first test was called Launch Address Book, and we can see that it's got a run that's listed under here in green because when we ran that test, it succeeded. And if you look down here, we can see exactly what it did, when it started, what it did next.

It double-clicked on that address book icon. You'll see that it's recorded here exactly where it found that icon on the screen. You probably don't really care, but once in a while, it may make a difference where it was that the eggplant found it and the exact time that that occurred, down to the millisecond if you need it. And so you can see each of the steps that it took as it went through and that ultimately it was a success with an execution time of one second.

So in the case of our many cards test, that was a little more interesting. We have quite a bit more going on there. It was going through, again, launching the address book and then typing some various data in the fields and so forth. And down at the end here, it went through and checked out some-- checked to make sure that everything was there and deleted it all and finished up. So again, that's nice. But there's still this question of, well, what happens if something goes wrong? So let's see if we can make something go wrong. Here's our new card test.

And-- This is one where we were just creating a single card, and then we went on and verified that it was there. And we verified by looking for the word eggplant support. Maybe we changed that to something else so that it doesn't find it. I'll put a one second wait in there because I found that sometimes eggplant is going ahead so fast that it'll start looking for the next thing before it's actually got this all typed in.

So if we change this now, let me just run that one more time before we do that. I'll kind of comment that out there. Run the script again so that you can remember what it was that it did. It goes through, it creates the card, enters all the information in here, and--

that that image wasn't found on there.

And if we go up into our results here, we can see that, in fact, the most recent run of that script has failed. And if we click on that one, you'll notice that Eggplant has actually captured a picture here. Whenever a script fails, Eggplant will capture a full-screen image of what was going on on the system at the time to help you go back and find out later exactly what was happening and where things might have gone wrong.

So we can see that somebody has typed in Vista support here. Well, that doesn't exist, I guess. And so it wasn't able to find that. So... We can also track back through here still in our results and see everything that the script did do, what it found, where it went, and so forth, up until the point where, well, this is what it was looking for, was eggplant support, and of course it didn't find that on the screen. But if I double click on that, it'll bring us right back to that line in the script, and that'll help us to figure out how to resolve the issue. So that's just a little bit about results. And let's switch back to slides once again.

So once again, it's really important to validate the things that are valuable to your users, focus on what their interests are, and you'll be delivering a quality experience for them. Working with the results course to understand which of your tests are passing and which ones are failing is really important to your overall testing.

I'd like to talk just a little bit about one other point here, the idea of system integration testing, or what I call system integration testing. Because your application, well, you know, I already said, you know, you don't want to focus just on your code and testing that. You should test the application.

Well, I'm not going to say that testing the application isn't really enough either. You need to test your application in a broader environment because your application, most of the time, is not going to be living all by itself on the system. Most software these days is going to be interacting with other software on the system. It's got things going on there, either exporting data or importing data or interacting in some other way with other elements of the system. So it's really important to do some testing also in the overall context where it's going to be.

So you may also want to test a complete process. There are times when you may have data that's flowing through into your application and then out into some other application and so forth. And it can be really nice to be able to follow that through a complete process, not just within a single application.

You may also have different configurations of your software, different options that could be turned on or off, that sort of thing. If you've got an automated test that's all set up to test one configuration, it shouldn't be too hard to adapt that to test another one. Or if you're testing on different platforms, maybe your application runs on Mac OS 10.1 and 10.2 and 10.3 and 10.4, you're going to want to be able to test it on all these different platforms. And having a single set of tests that can test it consistently the same way on each platform can be really great.

So clearly there's some benefits to automation here. Consistency across different configurations, also across different platforms, can be really valuable. Having repeatable results is always nice. Being able to run the same test in exactly the same way. Having those logs that you can go back to to see exactly what was done. This is an issue that comes up from time to time in organizations. The developers have spent a lot of effort creating a great application and they think it's wonderful.

They hand it off to the testers and the testers say, "Oh, it's all broken. This thing, it doesn't work when I do this." And the developers go, "Yeah, yeah, right." And so if you have that kind of communication issue between your developers and your testers, it can be really nice for the testers to have a tool to say, "Oh yeah, this is what I did. And here's the log that shows exactly what went on and exactly what the results were." So that can be a really valuable communication tool to go back from the testing team to the development team.

And of course, automation is going to give you more efficient repetition of complex tasks, just as it does with some of the simpler tasks. So let's go back once more, the last time here, to the demo machine. And take a look at an integration test. First I want to go in here in the address book, delete the card that didn't quite get deleted there.

Back to our initial state here. Now I've got a little script that I wrote this afternoon actually. Let me show you what this does a little bit. First thing in here is it's going to slow some things down for the demo. Eggplant has a lot of different timing parameters that you can adjust.

If you're testing a system that's running a little slower, you may need to slow Eggplant down a little more. Or if you're, whatever, if you're testing a web application, oftentimes you need even more time there for the web application to respond. So you can adjust some of your general timing parameters.

Then this script is going to go through, launch the address book, create a new card, enter some information in the card, finish the editing. It's going to export a vCard to the desktop. So it's going to export the information from that card onto the desktop. Then it's going to delete the card in an address book, import the information again from the vCard by dragging that back in. Then it's going to send email. Well, it's not actually sending an email. There's a command in Eggplant, actually, to send email. If you're running a test overnight and you want to have it send you the results in the morning, you can do that.

In this case, we're testing the sending email feature of the address book. So it's going to try to send, generate an email message to the person who's in the address card there. And then it's going to actually type some information into the mail message, including putting a picture in there. And then finally it goes through and kind of cleans things up.

So let me just show you. Again, I've got this slowed down a little bit so you can can see what's going on here. I think I've got the script animation turned on here, so I'll just leave this window up so you can kind of watch what's going on a little bit there as it's running.

So it's going through and you can see that it's launched the address book there. It's creating a new card, typing in some information. I've got it typing nice and slow here. Sometimes it's nice to be able to have things run slowly, either because you want to do a demonstration or for whatever reason, to be able to kind of watch what's going on a little better because eggplant tends to run pretty fast. It responds to things as quickly as it can as they come up on the screen.

So it's now exporting the V-card there. You can see that on the screen on the desktop there. And it's deleted the card. Now it's dragging the card nice and slowly there back into the address book window. So it's imported the data again. Now it's going to go send email here. So now it's launching the mail application. And mail, of course, has come up with that address in the to field. Now it's composing a mail message. I'm not trying to have it actually send the email message, but you certainly could do that kind of thing.

Doug Simons Eggplant has the ability also to connect to a different machine in the middle of a test if it needs to. So you can send an email from one machine, go to another machine, open it up, see if the mail has arrived. Here we've got a picture of Aunt Aubergine here reclining on the divan. And there's our mail message. And of course, it's going to go through and clean things up and close that all out afterwards to put things back to where they started, including dragging the V card down into the trash.

So that's our little demo. Can we switch back to slides, please? You may have some ideas at this point that there are some other possible things that you could do with an automation tool like this besides just testing software. In fact, a number of our customers have found some rather creative things to do with Eggplant.

But some of the things that I thought I would just toss out here as ideas to kind of get your imagination going would be, for example, to monitor and test deployed applications. Mostly we think of software testing in terms of testing software that we're developing. So as you're creating it, you want to test it and you're going to be delivering that.

But what if you have a web-based application and once you've created it and tested it, you're actually going to deploy it on the web and it's going to be up running 24 by 7. It might be nice to have a system that can go out and call up that website and run through and kind of make sure that things are actually up and running and validate the application while it's running live.

Another option would be to validate some third-party software. Eggplant is totally independent of the system that it's testing. It doesn't know anything about the software that's out there. So it can test software from other companies as well as it can test your software. And we've had situations where we've had customers who wanted to be able to validate that, for example, different versions of the applications that they use at their company are still going to work with the new version of an operating system when the new operating system rolls out. So that's a good idea. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

got an interface, then it's something that can be scripted and you can create tests through the software interface to the hardware. We also have a capability in Eggplant to generate movies. So while your script is running it can be capturing a movie of what's going on on the remote system.

By slowing things down a little bit like I did there in the last demo, you can adjust the timing and create a movie of something that might be used for documentation or for training purposes to show people certain aspects of your application or to use in your marketing. So those are just a few ideas that I wanted to throw out there.

So what you've learned, or what I hope you've learned, just getting back on those same points again, the benefits of automation and software testing, because this can really help you to get a lot more testing done, test more thoroughly things that you could never test manually, but also to test the things that you can test manually more efficiently.

I hope you've seen that getting started with automation doesn't have to be terribly hard. We've created two or three good tests there in a very short period of time, and it's not all that hard. I, of course, know eggplant pretty well, but you can get there, too. It's really not terribly difficult to get started with. Doug Simons And where to focus your testing areas, clearly, I've said it several times, focusing on the areas that bring the most value to your users.

We looked at a few different techniques there as we went through the demos. We were doing what are called functional GUI tests, you know, testing functionality through the graphical interface. We played around with some modularity there. We did quite a bit of that actually in the demo today. Having scripts that call other scripts and those we were able to use to build up our later tests. And you can get quite a lot of value out of your tests that way if you make them in a modular fashion so that you can reuse parts of them.

Hello? Okay. Couldn't hear myself anymore. I'm getting used to this loud voice. So, data-driven tests, we, of course, played around with those as well. One of the great advantages of automation. So, and of course, we introduced the six principles of practical testing. Testing the user experience, not the code, because you really want to test things from the user's point of view.

Starting simple and building up from there as you go. Completing that testing cycle, which makes it a lot easier as you're developing a number of tests if you can always end where you started so that you're ready to run your next test. Creating things modularly, as I said. Taking advantage of repeat and validating the things that are valuable to your users so that you can deliver some really great software to your customers, which is what we'd all like to see.

Please check it out. I'm Doug Simons. Emma Campbell is here also, our Director of Sales. I'd just like to invite her up for the Q&A. And we've got documentation and sample code and other resources on our website there at redstonesoftware.com. And if you'd like to go download a free trial version of Eggplant, give this thing a try and see how it works for you. So there's a URL there too, or you can just find it on our website.