I believe that boundary testing is in the calculator exercise in Testing Computer Software, which is also known as "The Bible" among software testers. Despite the fact that most professional testers know that we should be testing for boundaries, I'm finding that often, these tests aren't well performed, and even when we do perform the tests, the bugs aren't fixed! Here is a true story about several different bugs found in one day, just by testing ONE boundary.
A bug was found that when a certain number or more characters were entered, truncation did not happen. This error was handled, and an error dialog was brought up, alerting the user. Unfortunately, this application is expected to be run without a person watching (without a UI, or also known as headless in some circles). This meant that for all practical purposes, the application was frozen. A fix was put in place, but I was curious to test all of the boundaries. The team created an exploratory testing charter.
When the charter was run, we found MANY diverse behaviors depending on where we exceeded the expected characters.
1. In some cases, the application would crash. We wrote a bug on this straightforward issue with ease.
2. In other places, truncation happened, leaving unusable and incorrect email addresses and ages. These are easy to report, but difficult to know what is "expected" or what would be desirable in that case.
3. The most interesting issue we found was a silent failure. The entire entry would be skipped, and the log would report for example--Success! 10 entries imported. 9 Correct 0 Skipped 0 Errors. Subtle silent failure is the riskiest issue we found in my opinion, because if you are dealing with many enterprise solutions, you might have millions of imported pieces of data per day, and nothing to check that any are ignored, truncated, or missed.
A few things to think about in terms of boundaries. It isn't just how many characters, or what characters, or is "replace with nothing" or "null" allowed, or can you delete nothing that concerns me. The better question is, on what scale are these issues fixed? Will the headless system ever be fixed so that NO error dialogs will put it into a useless state again? It is easy to test for the error conditions that we handle. It is the error conditions that we don't handle and don't expect that we need to test for. This is why unless we have tested the boundaries and error handling ourselves, we should not assume that because it is basic that it is covered, or that because in one part of the application it is handled, that it is handled everywhere.
A bug was found that when a certain number or more characters were entered, truncation did not happen. This error was handled, and an error dialog was brought up, alerting the user. Unfortunately, this application is expected to be run without a person watching (without a UI, or also known as headless in some circles). This meant that for all practical purposes, the application was frozen. A fix was put in place, but I was curious to test all of the boundaries. The team created an exploratory testing charter.
When the charter was run, we found MANY diverse behaviors depending on where we exceeded the expected characters.
1. In some cases, the application would crash. We wrote a bug on this straightforward issue with ease.
2. In other places, truncation happened, leaving unusable and incorrect email addresses and ages. These are easy to report, but difficult to know what is "expected" or what would be desirable in that case.
3. The most interesting issue we found was a silent failure. The entire entry would be skipped, and the log would report for example--Success! 10 entries imported. 9 Correct 0 Skipped 0 Errors. Subtle silent failure is the riskiest issue we found in my opinion, because if you are dealing with many enterprise solutions, you might have millions of imported pieces of data per day, and nothing to check that any are ignored, truncated, or missed.
A few things to think about in terms of boundaries. It isn't just how many characters, or what characters, or is "replace with nothing" or "null" allowed, or can you delete nothing that concerns me. The better question is, on what scale are these issues fixed? Will the headless system ever be fixed so that NO error dialogs will put it into a useless state again? It is easy to test for the error conditions that we handle. It is the error conditions that we don't handle and don't expect that we need to test for. This is why unless we have tested the boundaries and error handling ourselves, we should not assume that because it is basic that it is covered, or that because in one part of the application it is handled, that it is handled everywhere.
