This is why formalized testing is not just the role of a QA professional, nor should a QA professional be confined to the walls of a “Testing” department.
Organizational Structure Level 0: Putting the Cart Before the Horse
This is the obvious, and usually first choice for management to divide the organizational structure into.Generally organizations will have the development department submit their finished product to the testing department who will then queue the work for testing. This unfortunately divides the two roles, and creates a gap between them, reducing overall communication and increasing the time lag between communication events when they occur.
The steps taken become very disjointed:
- The developer assesses the expectations
- The developer implements those expectations
- The developer considers it “done” and puts it in Quality Assurance’s inbox for testing
- Quality Assurance assesses the expectations
- Quality Assurance implements validation checks, some of which fail
- QA places the work back in Development’s inbox, with the validation checks attached.
- Goto Step 1
Organizational Structure Level I: Having a Plan
One obvious solution to the redundant effort noted is to place the test definition prior to the production. By defining the validity checks prior to the development process, developers do not risk developing against faulty expectations. The expectations of success are clearly defined prior to them writing the first line of code.The drawback to this model is that as development occurs, new ideas get discovered, new ways of looking at things become apparent. Effectively, this is the waterfall method of development where developers are told what they will produce, and have no mechanism for offering feedback, except to put the item right back into the tester’s queue.
Organizational Structure Level III: Unity
Rather a more vertical and integrated approach should be taken. In the end, we are all on the same team.With this configuration, the workflow is much tighter and the feedback generated by both QA and the Developers allows them to react to one another’s suggestions immediately, reducing the need for massive adjustments later on.
Not only can the feedback cycle be reduced, experts are better able to learn from one another. When automating a test, who could be a better resource to ask for advice than a developer who sits across from you?
In fact, this is not limited to just the developers and the Testers. As the tests are implemented, feedback from the results may challenge the Business Analyst’s expectation of the anticipated workflow. Why not bring them into the mix? How about Technical Writers? Surely code documentation could benefit from a professional editor and clearly written test summaries can influence the approach taken by a junior tester.
Every member of the team is responsible for every aspect of quality.
Quality Assurance across departments
At the very least, it is important for every member to have a basic understanding of the teams work in order to understand how to integrate and work together.| Tester | Dev | Analyst | Writer | more... | |
|---|---|---|---|---|---|
| Testing | Writes Automated tests | ||||
| Dev | Debug Code/Interpret code capabilities | ||||
| Analysis | Determine Dangerous System Usages | Implementation Decisions | Explain Need to Users | ||
| Writing | Defines Tests | Creates Documentation | Writes Need Assessments | ||
| more... |
Quite often a System Analyst is expected to take a broad view of their system, and assure that it is behaving well from the customers point of view. However, this means testing the system across several areas of expertise that are generally assigned to specialized areas. Taking a system perspective, we can ask questions that span multiple areas of expertise to identify potential issues:
System Quality
InfrastructureWhen investigating "System Quality", we are generally asking questions about the performance of the underlying system. A thousand little moving parts are associated with large systems, and each of them must perform to deliver the expected service. Many times a single failure of any part of underlying infrastructure that we take for granted can result in unexpected and catastrophic failures.
Many years ago, a friend of mine was working an IT Help-desk. He was sitting in the break room when someone ran in and demanded "all hands on deck". The phones were suddenly backing up.
Nobody was sure what had exactly happened, but all customers in the Texas area had lost their internet connectivity. It didn't matter whether they were business or personal, they were all phoning in at once. Details were a little sketchy, something to do with a squirrel, a power line, and a data junction. Known facts: fixed, dead squirrel, unhappy customers.
What can we take away from this little story? Systems fail, unpredictably and unexpectedly. If you have begun wondering why a squirrel had access to the delicate area, you are missing the point. Remember the objective is not to ascribe blame, but to solve problems. Generally, it is not possible to predict how something is going to fail (if we could predict it, we would handle the scenario).
Questions as simple as "Is the system available?" are highly critical to be monitoring.
Frequency
Tests like this should be run real-time. Depending on the system real-time could mean as infrequently as daily though could also mean as frequently as every millisecond. Naturally, the cost/benefit comparison needs to be considered of having real-time checking versus overloading the system with checks.Sample Questions
As with most tests, it is best to ask the most generic question first and work down to most specific ones. To use the example of the squirrel and the power-line, asking "Has an armadillo chewed through the lines" is too specific, asking "has an animal chewed the power-lines" is better, asking "do we have power?" catches all the scenarios. The most general question we can ask is "Is the system available?"In a web based environment this is as simple as opening one of the web pages and will answer 99% of your "is it up" questions. Once we have answered this question there are many others that can help identify problems before they occur.
- Is the system available?
- Is the system responding in an acceptable time limit?
- What percentage of RAM is the server consuming?
- How much Hard drive space is remaining?
- How much bandwidth is being consumed?
If this system had a monitor that simply checked the speed at which forms loaded, my colleague would have been notified that something was wrong much earlier. While she may not have been aware of what the problem was, she would have initiated an investigation significantly earlier.
The more eyes that are on a problem, the faster it gets solved. This is the principle of transparency
Brainstorming ideas of monitors
- things like alarms, monitors
- Ping the server every 5 minutes
- Web app? Check the front page. Did it load?
- Critical service? Is it loaded in memory and running? Did the time-stamp on the log change in a reasonable amount of time ago?
- system scans for known issues are always useful. Sometimes you end up with a known bad data state. You understand what data causes the problem, you even know how to fix it when it occurs. For the life of you, you can't figure out how it gets that way. While you are investigating the root cause, why not run periodic system scans for the bad state. It would at least allow you to respond to problems much quicker.
Functional Behaviour
- does it do what we expect it to do?
- regression testing is important - when I change one thing, there is a risk of “bumping” other parts of the system
Business Expectations
- are the specs meaningful?
- we can test the specs by rewriting them
Standards Conformance
- are all of the website’s links valid?
- are content authors using using effective keywords
- spell-checking
