Are you getting a positive ROI from testing?
Last night I watched a movie called The Tourist with Angelina Jolie and Johnny Depp. Despite the promising previews, it was a huge turd. The movie was chalked full of awful dialog, worse acting, and cheesy clichés. If you watch enough movies, you might come to the conclusion that there’s no petty crime in Europe. All European criminals are rich playboy savants that pull off only the most grandiose of crimes. They are all philosophical and very civilized, yet there’s a brutal streak to them. I can recall several instances of the “gentleman” gangster killing someone with his bare hands and then instantly returning to civility? In these movies, there’s almost always a self-reflective explanation of their plans or motives. And somehow, the police in Europe are either corrupt or powerless to stop these kingpins. Basically, almost every European villain since 1964 is Goldfinger.
The accepted, overplayed archetypes present in the Tourist made me think about testing. Just like the movie clichés, we’re so stupid in freely accepting bullshit testing patterns and practices. MY definition of a bullshit testing pattern or practice: one that doesn’t work for your situation. Testing methods that might be pure gold to your project might be pure garbage to another. (We can’t all be Charlie Sheen, converting “tin cans into pure gold”). I’ve heard it said that there are no best practices in testing, only good ones. The worst thing you can do as a QA group is do work that doesn’t somehow help the quality of your software. When doing anything in testing, you need to ask yourself, what’s my return on investment (ROI)? Is the testing that we’re doing telling us anything about the code?
One area that seems to be rife with bullshit is test automation. YES, I SAID IT!!! In fact, if you’re using off-the-shelf test automation tools (like Mercury products), then there’s a 90% chance (with a +-10% margin of error) that your test automation is complete and utter bullshit! Before you get mad at me, ask yourself, how many bugs have you found by running that behemoth suite of tests? How has it helped your product actually get better? Are you getting any assurance that your product is up and running? Was it worth the oil sheik fortune that you paid for it?
Let me give you a real world example on automation bullshit. There’s this product team that I have intimate knowledge of. This product team has both unit tests (JUnit) and Selenium UI tests being written against their code base. The unit tests are being run with each build, and the build won’t complete if any tests break. The Selenium tests are scheduled to run nightly in Hudson at various intervals so they don’t overlap. Except for a one hour reserved maintenance window, the Selenium tests keep the servers busy all night. Yet, with all this “testing” that is being done, the product still often arrives to the QA team egregiously broken. Why is that? Don’t the unit tests find problems? The offshore team writing the Selenium tests told me that they have 700+ tests running at regular intervals. He went on to say that they have no bugs to report from running those tests, but when they do, they will let me know. Hmm, so you’ve spent months creating these suites, and you aren’t finding any bugs? On the other hand, you’ve paid another team to write unit tests on code they didn’t write, and major problems are still getting through? You have these Cobetura reports show that you’re getting close to complete coverage on the code base. Complete coverage and there’s still problems. YOU GOT TO BE KIDDING!!! WHAT A STEAMING LOAD OF BULLSHIT!!! One team is writing Selenium tests. One team is writing JUnit tests. And there are no bugs to show for it. There are no red flags showing us potential defects. There is no confident assurance that anything is working or not working. I think it’s safe to say that the company is getting a negative return on their testing investment. And that, in this case, automation is a complete waste of time.
Obviously, JUnit, Cobertura, and Selenium aren’t bad tools. They just happen to be bad tools in the wrong hands–in the wrong situations. They are used to create all these “tests”. Instead of just accepting these “tests”, teams should be testing that the tests do something meaningful. They should be looking to the yield of the fruits of their labor, not how much work they’ve done. For every automated test written, you should prove how it can fail by making it fail. Think about it! Test it! Have others think about it too!
Let’s get away from automation because my point isn’t to trash automation. Here’s another example. Given Agile’s popularity, I think the full manual regression of a product is very passé. The full regression of a big product could take weeks or months. If your dev team is adding code everyday, then it’s not pragmatic to take on a full regression. You need to be more surgical with your testing, more precise. Focus on testing the parts that are actually effected by change. The full regression is one of those archetypes that’s accepted because it’s familiar, but it’s unrealistic, antiquated, and pretty stupid in a continuous integration world. However, you may be in an industry that requires that kind of attention to every detail. And in your case, that might not be a bullshit testing practice.
My point is not to tell you what you’re doing to test your product is bullshit. My point is to wake you up to practices that aren’t helping your product. I understand that not all tests are created to find problems, but some testing is there to assure you don’t have problems (regression testing). However, are you sure that your tests will reveal a problem when there is a problem? My point is to be skeptical. Don’t just do something because you saw it done like this when you worked at IBM or AT&T. Don’t just do something because that’s the only thing you know to do. Do something because it’s going to make the code better. Start with a goal for the testing in mind. If your goal is to tell you X, then come up with scenarios where that X would be revealed. Then execute those scenarios! Just like we don’t trust the developer code handed to us. Don’t let your left hand trust the test case that your right hand created. Remember, at the end of the day, it’s all about the return on investment. Just like developers deliver value by coding something useful, testers need to deliver value by uncovering problems or telling the true story of the product.
Read Full Post | Make a Comment ( 1 so far )
Marisa Roman
Daniel Brown
Josh Carroll
