Vinayak has noticed that testers he's worked with are more apt to use experience, intuition, and analytical skills in place of formal test case designs methods. Read this article to find out why Vinayak believes such ad-hoc testing habits could be the downfall of a tester's credibility.
Test case identification is the essential skill that every test engineer must possess. A test case is "a set of inputs, execution conditions, and expected results developed for a particular objective." It is also the "smallest entity that is always executed as a unit from beginning to end." Ideally executing a program using every possible input or ensuring all the possible paths of the program are executed it could be said that 100% of the program has been tested. But generally it has now been established that exhaustive input testing or exhaustive path testing is not possible.
To overcome these challenges many test design techniques/methods/approaches are available that systematically narrow down the number of test cases in an effective way allowing the broadest testing coverage with the least effort. A number of books and articles have been written on various test design techniques. In fact nowadays test design techniques seem to be an actively researched area. As a result a lot of knowledge base is being made available to the software testing practitioners. But are the practitioners making use of this? I think not. Myers wrote in his book that " . . . there is no guarantee that a person has used a particular methodology . . . properly and rigorously." Dustin reiterated the same in his book saying that "While test techniques have been documented in great detail, very few test engineers use a structured test-design technique."
Most of the time the activity of test case identification could be seen as an ad-hoc activity done by a group of so-called experienced testers or by new comers in this field under the guidance of the former. Testers are found to be using their experience, intuition and analytical skills to derive the test cases. In numerous instances when asked to a tester "How do you identify test cases?" I get a plain answer "By using Functional Specification." In a few cases they would add other specification names too. Then I try to be more specific and ask "I want to know how from those specifications the test cases are identified?" Here the answer comes: "After reading the specifications if there are any doubts and clarifications, we send them across to the spec writer and once the clarifications are okay the test cases are identified after reading those specifications."
That was my experience conducting some interviews. But in my on-job experience till now I have never seen any colleague using any formal test case design method. At least I never saw them drawing those Cause-Effect graphs, State-Charts, Activity Diagrams etc. May be they did all this thought-process in their mind. But considering the kind of complex modules sometimes they were handling I feel it was just impossible to do this kind of analysis in mind.
What could be the reasons for not using the formal test design methods? Are they not useful? Are they so complicated that it is difficult to use? Does the inputs required for utilizing these methods not as expected? Or are they not known?
They are useful as number of studies and examples could be found proving this. Some methods could be complicated but that again depends on the level of training and the motivation an individual has in using it. But yes, I have found in my interviewing experience that the methods are not known or even if known they are known just because they might have read some interview tips. And in many cases it could also be the case that the inputs like specification documents may not be conducive for applying these techniques or the processes of designing tests are not as clear cut as the processes of software design and testing
Ad-hoc identification of test cases without following any formal test design techniques has every possibility of them being unreliable, redundant, and having inadequate test coverage. So what could be done to ensure that test design techniques are applied in analyzing the specifications and this analysis is used to derive high-yield test cases? How do we bring that rigor, discipline, commitment in the test team?
Even today Software Testing is not given due importance in the curriculum. That means most individuals are not trained in using the test design techniques. Also it can be noticed that Software Testing is not the first choice or at least voluntary choice among Computer/IT graduates. This also should explain the low motivation in learning the test design techniques. In fact, at least in Hyderabad, India there are a number of private courses run by different institutes on software testing. But all of them deal with some kind of automation tool training. None of the available courses teach formal test design techniques and their practical application.
Because the practitioners have little training and experience in using these techniques practically, their utility is hardly seen by them. Moreover over the period of time as they are put into one project and then another, designing test cases just by skimming through the specifications (and sometimes by playing with the software when specifications are not present) quickly and intuitively becomes a habit. Even they start to think that this ad-hoc method works and gives them a false feeling of being expert. They start justifying their method in the name of exploratory testing, error-guessing, experience, etc.
So
- Test engineers should be trained in practical application of known test design techniques.
- It should be made mandatory to explicitly use the applicable test design techniques in the test design phase.
- Every test engineer working on test case identification should be asked to document and present its analysis.
- During presentation a list all known test design techniques should be present. It could be a chart or it could be written on white-board.
- Techniques not used by the Test Engineer should be identified and then its applicability should be discussed. Test Engineer should be able to justify its non-applicability.
- If no applicable technique is found, Test Engineer should be able to prove the completeness of the test cases identified demonstrating the rigor with which the technique is applied to derive the test cases.
