Software Testing 10 Rule

1. Test early and test often.
2. Integrate the application development and testing life cycles. You’ll get better results and you won’t have to mediate between two armed camps in your IT shop.
3. Formalize a testing methodology; you’ll test everything the same way and you’ll get uniform results.
4. Develop a comprehensive test plan; it forms the basis for the testing methodology.
5. Use both static and dynamic testing.
6. Define your expected results.
7. Understand the business reason behind the application. You’ll write a better application and better testing scripts.
8. Use multiple levels and types of testing (regression, systems, integration, stress and load).
9. Review and inspect the work, it will lower costs.
10. Don’t let your programmers check their own work; they’ll miss their own errors.

How to Write a Useful Bug Report

Useful bug reports are ones that get bugs fixed. A useful bug report normally has two qualities:

1. Reproducible. If an engineer can’t see the bug herself to prove that it exists, she’ll probably stamp your bug report “WORKSFORME” or “INVALID” and move on to the next bug. Every detail you can provide helps.

2. Specific. The quicker the engineer can isolate the bug to a specific area, the more likely she’ll expediently fix it. (If a programmer or tester has to decipher a bug, they may spend more time cursing the submitter than solving the problem.)

Description: Please provide a detailed problem report in this field. Your bug’s recipients will most likely expect the following information:

Overview Description: More detailed expansion of summary.

-Drag-selecting any page crashes Mac builds in NSGetFactory

Steps to Reproduce: Minimized, easy-to-follow steps that will trigger the bug. Include any special setup steps.

1) View any web page. (I used the default sample page,
resource:/res/samples/test0.html)
2) Drag-select the page. (Specifically, while holding down
the mouse button, drag the mouse pointer downwards from any
point in the browser’s content region to the bottom of the
browser’s content region.)

Actual Results: What the application did after performing the above steps.

-The application crashed. Stack crawl appended below from MacsBug.

Expected Results: What the application should have done, were the bug not present.

-The window should scroll downwards. Scrolled content should be selected. (Or, at least, the application should not crash.)

Build Date & Platform: Date and platform of the build that you first encountered the bug in.

——Why testing is needed Real Example——

A digital camera was released in the market and after sometime a customer came back complaining that …….

Problem :

The camera’s software was crashing continously while he tried to capture images.

FIR :

They first asked some basic information from the customer and made an attempt to reproduce the issue but it was not reproducible.

Analysis :

* Software is crashing – so they listed out all possible scenarios where the software could crash.
* Tried putting the camera to all listed possibilities.
* Giving a report to the management that it was unable to be reproduced.

Further Analysis :

Taking this as a challenge , a great tester had the following concern -

* He was concerned what festival was it ?
* He was concerned what was the customer was trying to capture ?
* He was concerned what is the frequency of the crash the customer noticed ?
* He was concerned about all other surrounding factors like temperuature , place where the festival was celebrated … etc ?

Outcome of Analysis :

* The festival was Diwali or Deepavali !
* The customer was trying to capture fire crackers , explosion , fireworks display in an open ground at around 22 00 hours ( night ).

What a fantastic tester was he !

He identified the problem and told the developers

“The camera software is crashing when the viewfinder is black ( because of dark night ) and a sudden gush of light through the firecracker explosion is putting the DSP ( digital signal processor ) to a load/stress beyond its boundaries to display the image. And at this point of time when the user tries to capture it simply crashes”

What he said ?

was right !

Finally they fixed it and now today its a robust digital camera !

This is why Testing is Important

Summary of Story

* Collect real time data for testing a product/application.
* When you are unable to reproduce an issue , think out of the box , think out of the black box.
* When you are unable to reproduce an issue call someone who can dig deeper than you.
* Don’t assign test case draft to the person who is going to test it, he may omit complicated cases to bring down his/her complexity in executing it.
* Review the cases with a non team member/customers/domain specialist/tester network.
* When you release a product : anticipate and be prepared to face such situation.
* Hate the product while testing and love the product after its release – emotional testing ( ha new concept introduced by me , i do it , to be frank )
* Good Testers not only report bugs , they suggest a solution ! (more to come about this )
* Keep learning to be a good tester!
* Test all your products in India , its Testing Bowl of the world !
* Dont terminate an issue just because you are unable to reproduce , it could kill your company !

Bug Taxonomy (the science dealing with the description, identification, naming, and classification)

When you get toward the end of a product, you have to make the tough calls about what gets fixed and what doesn’t. Most bugs fall into one of two obvious buckets:

* Must Fix – Something is really bad here. We cannot ship the product without fixing it.
* Won’t Fix – The problem is minor, hard to get to, etc. It won’t be fixed for this release.

However, there are a few bugs which don’t clearly fit these buckets. There is no clear call to make on them.  Once you have started changing things and have to redo your test pass, you might as well include those nice-to-have issues too.

???????? Testing – Y in Client/Server Technology Validation is very Important,,,

The javascript validation is used by many developers for the the client side validation of the data fields and other checks. Does this mean that there is no need for the data validation on the server side, the answer is NO. Client side data validation is used for speeding up the things so that number of requests made to the server are reduced. Think of a simple banking application where I can transfer the money from one account to another. This application has the check on the client side for the amount to be transfered should be non-negative and greater than zero, and on the server side there is just one check whether the amount to be transferred is less than the current balance amount or not. So the server side code will be like:

if(transfer_amount > available_balance)
return

BUG_CODE

else
MyAvailable.balance= MyAvailable.balance-transfer_amount;
CurrentAccount.balance= CurrentAccount.balance + transfer.amount;

So if user enters the amount to be transfered to his friends’s (or may be his enemy’s ) account is negative then client side validation fails and gives an error, but think of case wherein the user disables the client side validation by say commenting out the javascript code. Now user will enter the amount as -1000, which goes to the server and from the code as you can see you will be richer by 1000 $ and your friend (or enemy) will be poorer by same amount. I have seen one shopping site which has this kind of thing where there is no server side validation on the number of items, so if user gives -5 items, assuming cost of each item is 100$ the total amount payable is -500 $. SO the client side validation is just for increasing the speed and is not substitute for server side validation.

This are called as Test Cases —- it is Security Testing

Automation Testing versus Manual Testing Guidelines

I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?

Pros of Automation

* If you have to run a set of tests repeatedly, automation is a huge win for you
* It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
* It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nighlty)
* Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.

Cons of Automation

* It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
* Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.

Pros of Manual

* If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
* It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.

Cons of Manual

* Running tests manually can be very time consuming
* Each time there is a new build, the tester must rerun all required tests – which after a while would become very mundane and tiresome.

Other deciding factors

* What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.
* Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?

Criteria for automating

There are two sets of questions to determine whether automation is right for your test case:

Is this test scenario automatable?

1. Yes, and it will cost a little
2. Yes, but it will cost a lot
3. No, it is no possible to automate

How important is this test scenario?

1. I must absolutely test this scenario whenever possible
2. I need to test this scenario regularly
3. I only need to test this scenario once in a while

If you answered #1 to both questions – definitely automate that test

If you answered #1 or #2 to both questions – you should automate that test

If you answered #2 to both questions – you need to consider if it is really worth the investment to automate

What happens if you can’t automate?

Let’s say that you have a test that you absolutely need to run whenever possible, but it isn’t possible to automate. Your options are

* Reevaluate – do I really need to run this test this often?
* What’s the cost of doing this test manually?
* Look for new testing tools
* Consider test hooks