Experience report: testing dojo with entire dev team

Last spring I worked as a test lead/quality coach for 3 teams that did their own testing. I experimented with different techniques to help them further improve their testing skills. I wrote this experience in March but I didn’t get around to publishing it then which I’m doing now.

I want to share with you another way of combining testing, learning and fun

At the Agile Testing Days in Potsdam/Berlin I accidentally ended up in a testing dojo session. For an hour, 4 pairs of testers tried their skills at a buggy piece of software and received feedback about their testing. It became immediately clear to me that this was a great opportunity to improve testing skills and I decided to try it at home with my teams.

I work as the sort of test lead who provides inspiration and encouragement for 3 teams of programmers who do their testing themselves. For our domain, web development, this works well. We have developed a testing strategy together and I also help them improve their testing skills. They are awesome, committed to continuously delivering value to our customers and eager to do a good job.

I planned a testing dojo of an hour and promised candy and laughs. The response was, to my relief, positive. I wasn’t sure that they would want to spend an hour of precious programming time doing testing. I chose an hour so that it wouldn’t be too long and it was easier to find a room and a time slot.

The preparations took a while because I needed to decide on a suitable piece of software and read up on dos and don’ts for testing dojos.

Finally, the software I picked was the one I had tested in Potsdam. It was crawling with bugs and this meant that everyone would find some. I thought this would be good for a first session to make everyone comfortable. It was also small enough to be constraining but big enough to allow people to try different areas. I also wanted to have something which no one in the teams had written themselves so that there wouldn’t be any awkward situations. This meant finding external software.

The parking rate calculator – object under test http://adam.goucher.ca/parkcalc/index.php

Format for the dojo
We had the 3 roles described in the testing dojo document. Every 5 minutes we rotated. We ended with a 10 minute debrief to discuss what we observed and what was good. http://www.testingdojo.org/tiki-index.php?page=Roles

Setting up the environment
I created a few posters which I put up on the walls. They detailed the format of dojo and the parking rates for everyone to see.

I explained carefully the purpose of the dojo. I put the emphasis on the fact the purpose was to learn from each other. This means that both observers and testers learn and we should be gentle to each other. It’s not easy to sit down a new computer and start testing in front of everyone, there needs to be humility from the audience for this. And on the other hand, active and good observers are key for learning.

How was it?
First of all, we had fun! The overwhelming buginess of the software created a lot of reactions: surprise, confusion, entertainment, frustration and joy.

The programmers were a bit overwhelmed by the amount of bugs. This is the downside of using this test object. In a normal situation I would just send it back after 2 minutes, but this isn’t a normal situation. I encourage splitting the debrief into two parts: “what did you think of the software?” and “what did you observe about the testing that we did?” or even say “let’s agree that the software is crap, but what did you observe about the testing?”.

It was clear that this was an ad hoc session. There was no plan and a lot of randomness. A few people started trying to be systematic but bugs caused them to lose focus. We tried a bit of everything, here and there.

This was a good thing though. For the group to observe this randomness was interesting. It shows well of you can spend an hour testing without knowing much more than when you started. When answering the question “what would you do differently if you did it again?” the group answered that they would be more systematic or organized. We also tried to highlight some of the things that the participants had done successfully.

What now?
We will do it again. This time I want to start with creating a plan together and see the difference in an organized approach. After this I think we’re ready for code closer to our domain or maybe even our own code.

Conclusion
I strongly recommend doing this kind of exercise with your team or your peers. It’s fun, interesting and a great opportunity to pick up new skills.

 

Talk about bug hunt

In May I spoke at the Smart Bear user conference MeetUI about “Commiting to quality as a team”. In particular, I took the example of the bug hunt to describe how a team can kickstart its collaboration.

You can see the talk (about 15 minutes) here: http://www.soapui.org/Community/meetui-videos.html

And the slides can be found here: http://www.prezi.com/user/ulrikamalmgren

Smart Bear is the awesome company behind the open source software SoapUI och the load testing tool LoadImpact.

Bug hunt experiment

One of the action points from our regression test retrospective was to hold a Bug Hunt. We wanted to find some bugs, enhance the team spirit and learn from each other.

Results

The results were mixed. On one hand this was really fun, but on the other hand, the bugs that we found weren’t particularly interesting. Almost half of them were rejected because we weren’t able to reproduce them or they weren’t real bugs (working as intended). Also out of those that made it into Jira, most were trivial or “cosmetic”. As a team activity though, this was a lot of fun!

Retrospective

Inventory of strategies used

We held a retrospective afterwards were we first collected the different strategies that each pair had used, making an inventory on the whiteboard. We then talked a bit about the hunt in general. It seemed that everyone had enjoyed it and the discussion was going on about “when we do it again then…”, which is nice.

We then grouped the bugs (printed out on sheets of paper) in categories and read them out loud for each other. This made us discuss why there were so many trivial bugs: people felt stressed by the lack of time. We discussed point systems, having more time, etc…

We then opened up for a more free discussion and I recorded it using a mind map on the white board.
Ideas

• One interesting semi-related proposal was to arrange cross team testing at the end of the sprint. Since we are two teams, this would increase our knowledge of the other team’s features and also help us look at things from another perspective.
• Our new Scrum Master was of the opinion that the competition part of the activity had to go. At first I disagreed but when I think about it there could be ways to work together instead. Let’s say that we for example work together to try to find 4 major bugs in one hour. What do you think?

Resources

http://sqa.stackexchange.com/questions/2726/bug-hunts-and-possible-alternatives
Details

People
We were 7 pairs during the hunt and two judges. The pairs were formed in order to get a good mix of people from the two teams, scrum masters and support. The stakeholders were invited but did not participate.

Time
We used 1½ hour for the event in total with introduction, jury deliberation and reward ceremony.
The hunt went on for 1 hour.

Results
We had 44 bugs posted on the whiteboard at the end of the hunt.
This resulted in 26 bugs entered into Jira (we expect some of these to get rejected when we start working on them)

Rejected bugs
The rejected bugs did not make it into Jira because:
– we were not able to reproduce them
– functionality was working as intended

Accepted bugs
16 Minor
10 trivial

You’re stuck with manual regression testing, do something with it!

While holding a retrospective about regression testing, we improved how we work with quality during the sprints. By analyzing and discussing the bugs we found during the regression we were able to discover some things we could easily improve with test automation, find process improvements and some patterns of bugs that we missed.

I believe that all of us who have ever had to do manual regression testing can relate to the agony of it. I’ve seen different types but in each case it has felt like monkey work, checking things that could be checked by a computer. Nevertheless, this is the reality for a lot of teams and change can be slow.

In the mean time, this is no excuse to not improve our ways of working and we decided to do a regression test retrospective. In particular I wanted to do a root cause analysis for one or more bugs. The goal was double: to work more efficiently during regression and during the sprints.

I asked for advice on the Agile Testing Mailinglist on how to do a root cause analysis and got some good ideas from there. Here is how we proceeded.

Preparation

In preparation for the retro we used Jira to print out all of the 50 or so bugs we found during this period, each on a piece of paper. We used a conference room with a big table and removed the chairs. During the regression test period we had asked everyone who submitted a bug to mark it with “test specification” if it had been found while following the steps of a test specification and “exploratory” if it had been found during exploratory testing so that we would know the technique used to find the bug.

Sorting by time

I asked the participants, programmers and testers from our teams, to sort the bugs on date. We then had a timeline with the amount of bugs per day visible. We decided to mark the major bugs so that we could distinguish them easier. We talked a bit about what we saw and realised that there was nothing surprising. The days were a lot of people were involved in testing, we found more bugs. The major bugs were evenly distributed in time. This could seem as a waste of time because there was nothing surprising, but I don’t regret spending time on it because there could have been something interesting there. Had we found all of our major bugs in the last day, for example, we might want to do something about that.

We talked a bit about how we would do this in the future when the sales process forces us to release even less often and thus increasing the time between regression test periods and thought of trying bug hunts a few times.

Bugs sorted by category

Sorting by category : old/new functionality

After a break, I asked the participants to sort the bugs according to categories. The categories we decided on were “bugs in functionality which is new with this release” and “bugs in old functionality”.

We decided to keep the timeline and put the new bugs on the top and the old bugs in the bottom. We could tell that about 2/3 of the bugs were related to old functionality.

We then decided to sort those into “old functionality affected by new functionality” and “just plain old bugs”.

Here, things got interesting. We could see for example that we had new bugs, found using a test spec. Did this mean that the test spec wasn’t run when the functionality was implemented or that it had become obsolete already?

We could also reflect on the fact that some of the old bugs, we already knew of, but none of us had bothered filing them until a new tester joined the team.

During the discussions we identified that we wanted to do more pairing in the teams, that we wanted to brainstorm a test plan together at the start of each user story and that we needed to talk more about the value of our test specs.

Root cause analysis of one bug

Finally, I wanted to do a root cause on one of the bugs to see if there was any specific action we could take.

I chose the “5 Whys” to do the analysis mostly just because I had to pick something. This was my first time doing any kind of root cause.

We looked at why the bug was introduced first and then at why didn’t find it earlier in testing. We realized that this was a very complex part to test because of the immense amount of possible combination. The programmers said that this particular bug could have been found using unit tests and we decided that we would implement those. This made me very happy, because this part of the application in particular is one which feels very tedious to test manually.

Conclusion
For me, doing this kind of workshop meant that we worked together as a team to talk about testing. We could identify at least one area where we could automate tests. I also believe that us testers realized we could get some help from our programmers with these kind of problems and that the programmers better understood that we need their help.

Talking about a regression was also a way to discuss how quality work and test work can be done during the sprints in order to not create these bugs or to catch them earlier.

So, I can recommend doing this kind of workshop even if (or especially when?) you’re stuck in a low release frequency, manual regression test only organisation because there are benefits of talking about testing together.

 

 If you’re interested in testing in an agile context, you should check out my upcoming full-day class on agile testing. Only in Swedish, for now.

Checklist for planning

We’ve started using a checklist during sprint planning in order to keep the things that usually go wrong in mind while talking about how to solve the user stories.

Some items in the list are very specific for the product that we’re developing and I’m leaving those out but here are a few more generic things:

• Should a log event be generated when the function is performed?
• Is the function affected by time zones, and if so, which time zone shall be used?
• Should the function be accessible via WEB-Services?
• What rights are required to perform the function?
• Can there be any concurrency issues?
• Do we need any special test data?
• What views are affected? Would mockups be helpful?
• What about performance?

My favorite is of course “Do we need any special test data?”. Setting up tests can be costly and it needs to be thought of during planning. We might need to get information from customers, order hardware, change the test environment to have data that will allow us to test the feature. But also, thinking about how the testing will work might have an impact on how the feature will be designed. It’s design can make testing harder or easier.

Does this dress make me look fat?

Don’t you just know if you have good quality or not?

I bet that if you spend lots of time worrying about how to measure your quality then you’re probably feeling insecure. On the other hand, if you’re confident about having a high quality product then you won’t be bothered as much.

“Wow! I look fabulous in this dress!”

Howto: do basic things using ProjectWhite/White

Howto:
– launch an application
– navigate into a submenu
– get a hold of a MDIChild-window.

Public Class Test1

Private Const path As String = "C:\Path\myprogram.exe"

Public Sub TestMethod1()
Dim application As Application = application.Launch(path)

//Find the form
Dim main_window = application.GetWindow("name of window", InitializeOption.NoCache)
Assert.IsNotNull(main_window)

//Find the Menubar and its choices
Dim menu As MenuItems.Menu = window.MenuBar.MenuItem("Menu choice")
menu.Click()
Dim openSubMenuChoice = menu.SubMenu("Submenu choice")
openSubMenuChoice.Click()

//Find the MDIChild-window
Dim child_window = main_window.MdiChild(SearchCriteria.ByControlType(ControlType.Window))
Assert.IsNotNull(child_window)

application.Kill()
End Sub
End Class

Where does quality come from?

At work I was asked, together with a colleague, to come up with a plan on how to improve/introduce automated testing on our two systems.

It didn’t take us long to identify that integration testing was the way to go. We presented that track. And we were met with questions such as “And how are you going to measure the amount of defects we have?”

Well, would the quality of our systems improve if we measured the amount of defects they had? Would it tell us anything more than the amount of faults they used to have? (Or still have if the mentality is to tolerate and not fix the bugs)

Our plan was completely different.

Quality is something that you build into the system and testing is a tool that can help keep focus on it.

People are fond of TDD because the quality of the code can improve when you use it. It helps you keep track of the intent of your code, make it more structured and hopefully keep the complexity down.

In the same way, by using BDD and integration testing, you can keep track on the goal of the system. What business value does your code add? Who will use it? What is the desired effect?

Even though they are testing techniques, they are also development tools. And an interesting part in introducing quality.

Howto: add logging to White (continued)

I wrote a while back about how I managed to add logging with log4net to my ProjectWhite tests in Visual Studio. However, after being forced to move my source code it suddenly didn’t work anymore eventhough I followed my amazing instructions. After trying a whole bunch of things out, one of them worked.

For log4net to work, the log4net.config file needs to be copied into the TestResult folder created by Visual Studios testrunner. In order to achieve this, enter the “Test” menu in VS, select “Edit Test Run Configurations”, “Deployment” and there, add your log4net.config file.

This made things work for me.

Note: I’m using the built-in testrunner in Visual Studio to drive my tests and not NUnit right now.
Note2: The first thing I did was setting it up as I did in my previous post, I haven’t tried to see if just adding the file to deployment is enough, it might not be.

The 7 plagues of Software Testing

James A. Whittaker tells us about the 7 plagues of Software Testing on the Google Testing Blog.

1. The plague of Aimlessness – The lack of communication between testers means that we keep repeating each other, creating the same tests for the same things. An exchange of knowledge, lore, is necessary.
2. The plague of Repetitiveness – Tests need to be diverse, even the automated ones.
3. The plague of Amnesia – The problem you are working on has probably been solved before.
4. The plague of Boredom – When forgetting the tactical aspects of testing such as writing test cases, testing becomes boring and uncreative.
5. The plague of Homelessness – You need to live in a house for a while before you realize that the dishwasher should have been placed a bit more to the right, or that there is leak somewhere. A tester cannot do this.
6. The plague of Blindness. A software is untangible we must rely on our less concrete senses to for any feedback about our effort.
7. To be announced.