I am subscribed to several mailing lists. One of them is Software Test Professionals (http://www.softwaretestpro.com). They organize training and lectures, and in November it was offered one with the title "My Testing Metrics Allergy" (by Dawn Haynes, more information at http://www.softwaretestpro.com/Item/5332/?utm_source=Email&utm_medium=email&utm_content=111611-ONLINE-SUMMIT-&utm_campaign=TRAINING). The title and the topic stroke a chord. I have been in too many meetings where BA, PM, SME and others with a tester hat on their head uttering ridiculous sentences like "testing is 78.12% completed". Why this sentence is ridiculous? Well, for several reasons:

a) The two digits precision have no sense. This simply shows the organization has spend a lot of money and effort in a software program that keeps tally of how many test scripts have been executed versus the ones that have not been executed. Given the nature of software testing (see below) this precision is meaningless. In most of the cases the most can be said is "Testing is approximately 25/50/75 % completed".

b) The statement implicitly assumes all test scripts are equal. They are not. Let's use the following example:
- Test script to validate the correct position of the corporate logo.
- Test script to measure the time to respond for a user, when there are already 1000 users in the system

the first is straightforward to execute and set up. The second... well, if nothing else will require several measurements (to get some statistics), some data analysis and correlation with other performance indicators of the system. In addition the set up will be nothing but straightforward.To give to both test scripts the same weight when calculating the percentage of completion of testing is to have a skewed view of reality.

c) The percentage does neither account for missing requirements more requirements that have some, but not the entire necessary test to cover them. In theory it should not happen, but in reality it does.

d) Usually test scripts may cover more than one requirement. If mapping between requirements and test cases is not perfectly done, the percentage of completion will be misleading.

e) When you test, you venture into an unknown territory where you may discover surprises that anybody expected. Surprises that were not hinted by any requirement. How these surprises are reflected in the percentage of completion value? The answer is that they are not.


These are a few of the problems a very precise percentage of completion involves. In other words, the error bars around the number are much larger than 0.01%, or even larger than 10%

So, how you can evaluate where you are in your testing effort? Well, it depends on the project and/or your experience with similar projects. In my case, I like to use the bug convergence and zero bug bounce milestones (check http://technet.microsoft.com/en-us/library/bb497042.aspx for clarification of these concepts in the context of the stabilization phase of a software project). Based on my experience, these milestones provide, in the context of an integration project, the 30% and 60% completion marks. It is less impressive that being able to say 29.78% or 61.07%, but, in my opinion, it is much more honest.

In this moment I have changed my tester hat for a project manager, or better said, general contractor hard hat. I am building what I hope will be my future house.

It's a lot of excitement, a lot of frustration, and a lot of opportunities to learn. And one of the things I have learnt is that professions are not so different from one another.

A few weeks ago the structural engineer -the guy that make sure the architect design will hold in case of storm, high winds, earthquake and so on- presented me the structural drawings. He has a reputation of preferring to overdue things. However, he was very frank: the structure he designed will ensure that in case of earthquake -the most significant hazard in my corner of the woods- the house will not collapse. However, after an earthquake, the house can not be inhabitable. To continue being useful after an earthquake (think of hospitals, fire stations and so on....) you need to spend more money and time.

This is a situation identical to software testing: to execute all possible testing paths and areas (functionality, performance, stress, security, usability....) will require an enormous effort that you may choose not to do. The important aspect is the decision needs to be made rationally and even more important, documented. All the stakeholders should be aware of the limitations of the testing effort and the trade off that have been made.

And trade off are a part of life, and part of software testing. Unless you are testing a flight control system, for example...

A few months ago, I posted the following question to a test discussion group

Is Agile Testing oxymoron?
Does Agile make life for testers more difficult?
Agile was though by developers to solve specific developers' issues. However, for other trades like QA Agile has done not much, and arguably has complicated our life a bit more, if nothing else creating false expectations: integration testing, performance testing, stress testing and so on can not follow an Agile model, despite some project managers wish it could.

I would to start a discussion around this topic, if nothing else to gauge the opinion of fellow QA engineers. If nothing else may provide some food for thought for project managers that think that vague user stories with changing details are enough to develop a good test script!

Unfortunately, only Jason M. Morgan replied to my post. His comments are

My most recent employer was using Scrum. The QA activities related to the new features being developed in the scrum team fit easily into the sprints. But there were larger testing activities such as regression testing that were to big for one team or even one sprint. So I had to break down the regression testing into small pieces that could be assigned to each team within a sprint. It was a lot of overhead for no benefit other than following the scrum model.

So, clearly, this topic didn't awake the imagination of testers. However, I still think that Agile, as currently formulated and applied, doesn't make life easier for tester and I will argue that to other trades involves in the software development process.

I think that Agile contains a lot of good principles and it makes a lot of sense to use it in most projects, specially the small/middle size. But I would like to see Agile taking into account the needs of Business Analyst, Technical Writers, Trainers, Infrastructure, QA and QC. For example, it is very difficult for a Technical Writer to put a story into words without really knowing how the other stories have been implemented. The same for a tester: it is possible to test the flow of a single story, but it is much harder to imagine how this story interacts with others if they exist only in concept (think about processing a payment as one story, but the create payment story still is under development). It is in this context that I say Agile has shifted work from development to the other areas. Now the question is not to throw Agile away, but how to extend its benefits to everybody that contributes to the completion of a project.

Lessons learned from Performance Testing a complex application

After two years too busy to write any blog, I have now a bit more time to share some of the lessons I have learned while doing Performance Testing for a complex application (Enterprise-level, several millions code lines kind of application).

- Complex application usually mean lots of different components: network, hardware, software... When producing performance data, you need to gather data about how each component behaves. It is not useful to say "the application is slow" you need to say "the application has a bottleneck in the disk I/O".

- You will be able to gather pretty much all the information you need about how your system is working using the performance counters embedded into your OS (MS, Linux...).

- Use only the performance counters you need, as they consume resources. Once you realize the values provided by a counter are "good", you can stop using it.

- When the application is far from being optimized, run all the counters and make a list of the worse one grouping them by logical areas (data repository, UI, business processes...)

- With the previous list, test individual components that are described by the counter. For example you may focus on the Data repository layer, looking for the disk I/O, for hard/soft page faults etc.

- Based on the previous results, concentrate then in a single counter. Using the previous example you may focus in the disk I/O counter if the other counters look Ok.

- Focus first in the low hanging fruit: most performance problems are related to network bottlenecks, lack of memory, disk I/O and pinning CPUs. Once these are removed, you can focus in secondary problems: too many threads, unoptimized queries, unmaintained databases etc. After that, you do not have so many options. The most available would be to throw more hardware to the problem. If the Performance is still unsatisfactorily, then rearchitecting your application is usually the only option left

- Unless there are egregious errors, refactoring your code helps very little if at all.

- Use automation to create load. To measure response time to user's actions use a human with a stopwatch

This last point may be surprising for some. My experience is that introducing timers in the code doesn't work. First it is not always possible, second, in practice it is pretty much impossible to capture certain events: you can capture when an image starts to be displayed in the screen, but not when it is shown entirely. If you want to measure the user experience, better to do it by hand, assuming the role of your user.

To automate or not automate, this is the question...

Whether 'tis nobler in the mind to suffer
The slings and arrows of manual testing
Or to take arms against a sea of automated test
And by developing them. To die, to pain--

.....


Anybody that has tried automation knows that automation needs to be done carefully. Managers, convinced by automated tools salespeople and Start Treck voice activated computers will try to force you to automate everything. The arguments will be the usual ones of increase productivity, results reliability and speed. However the land of automated tests is littered with corpses and failed projects.

Why? Well, based on what I have seen the foremost reason is that test automation are projects that need to be though, managed, staffed and planned as any other software development project. To be useful they need to answer the following questions:

• Will the test be done a limited number of times or it is a test that needs to be executed regularly?
  
• The requirements for the application under test, will change soon? change with time? are still undefined?
  
• Is the code under test architected in such a way that a change in a component will not affect our automated test?
  
• Can the code under test be automated with an off-the-shelve software?
  
• Have the code under test be instrumented? Does it have hooks that can be used?
  
• Is the automated test code modular?
  
• Is the automated test code easy to maintain?
  
• There is time allocated to document your automated test?
  
• Are you testing your automated tests?
  

These are basic considerations that need to be answer before proceeding with an automation project. However, before this considerations it is fundamental to consider the first point Ihave made: Any automation exercice as a project in itself. This means it need to have the appropriate level of staffing, project management supervision, methodology... Failure to plan the automation project appropiately will incur the same risk as any other project: overworked staff, last minute problems, cost/time over runs...

So, my sincere recommendation to anybody trying to automate the testing of their new piece of software: treat it seriously as a subproject of your main project

Do not give [free] honest advice/assesment in job interviews

Ok, this is a rant, so feel free to skip it!

About a year ago I had a job interview for a contract position. The project was complex, ill organized and with high probability of failure. A brief description of the project is:

Company A starts a job for a customer. They were going to adapt their exisiting product, based on US business processes, to the Canadian business processes. After a year, and no progress, company B purchases company A. Another year pass with nothing to show. It is now two years after the project has begun and the customer still has nothing. When company B realizes the mess, decides to cut their loses and hires company C to finish the project. Customer sets up a team to speed the project that will coordinate internal resouces with company C, who incidentlly are located in a different time zone.

At this stage I interviewed for the QA position in the coordinating team. Among other responsibilities, it includes testing the code that Company C is producing. There were no procedures, vague specs and the group of developers used the word Agile methodoly to describe what is a cowboy approach. The PM, working for the customer, has no carrot no stick over company B or company C. My assesment of the situation was somehow bleak, and I proposed measures like change control, regular risk meetings, clearly defined [and achievable] goals for each iteration... I know I sounded a bit negative, but definitely I didn't wanted to sound like all was fine: it wasn't.

A year later, and because Vancouver is an small IT town, I have learnt they hired another person that had 'Don't worry, be happy' as mantra. Six month after the hiring the PM realized it was not working. So, they started to do risk meetings, put in place a change control and each iteration has a small number of features that had to be implemented. It has took over three year for the customer to have something that is operational and can be put into production.

Conclusion: next time I want the job I need to smile, assuage all fears with 'It's going to be fine' and send a bill for any advice I may give during the interview.

Adding/modifying work items in Team Foundation Server -Part 2

In the previous blog I explained how to create a 'Steps to reproduce' control. Lets now add this control to default Bug Work Item (WI). In this case, I want to add 'Steps to reproduce' as a new tab.

If you go to the Layout tab, you will get something like the following image

It is easy to read. The TabGroup refers to tab structure that previously I have qualified as not meeting the tester’s needs. Now that we have created the 'Steps to Reproduce' control, we can add it to the layout.

To add a new tab element, right Click the ‘TabGroup’ folder and choose ‘New Tab Page’. A new ‘Tab Page-New Tab Page’ element will appear at the bottom of the ‘TabGroup’ list. On the right pane, change the label to something more meaningful, for example ‘Steps to Reproduce’. Note: this simply adds a new container, in this case a tab, where we can place control.

Now, we are going to add the previously created 'Steps to reproduce' control we created in the first part of this tutorial to this new tab. To add this control to the tab, right click the ‘TabPage- Steps to Reproduce’ and choose ‘New Control’. The property pane appears, change the label to ‘Steps to Reproduce’, and choose the Visiphor.Steps we created for the field Name field.

Accept everything, and if you click the ‘Preview Form’ button, you will get the ‘Steps to reproduce’ tab added to the tabs section

If you want to move this tab to be the first, right click the ‘TabPage – Steps to Reproduce’ folder and ‘Move Up’ as many times you want.

At the end, you can build a bug work item like

That fits much better my needs as a tester!

Notice the new tab 'Steps to reproduce' contains the control 'Steps to reproduce'

Hope it will work well for you too.

Adding/modifying work items in Team Foundation Server -Part 1

As most of you in the trade will know, Microsoft released in 2006 Team Foundation Server (TFS), a new product/tool that allows testers, developers and architects to work together using Visual Studio as its GUI. In this way, a seamless environment will be created where all these roles can work together, or at least this is what the Sales&Marketing department would like us to believe. In the practice, this level of integration is some iterations away.

TFS out of the box include two approaches: CMMI and Agile. In both of them it has the concept of work items (WI), among them, it has a WI named 'Bugs'. Unfortunately, I find the Bug WI included in these templates not very useful. So, if you want to create a new Bug WI in TFS with the product as it is out of the box, you will get the following

Enlarging the tab section (apologies for the bad quality of some of the images...)

As a tester I am surprised to see how 'developer' oriented is this interface. For example: I couldn't care less about the fix, however, however, I think it is very important to describe the steps to reproduce the bug. This is a single example... there are several more fields/tabs I would like to see that are not there by default!.

So, how you manage to customize TFS to your needs? This blog will guide you to the process of creating/editing work items.

First you need to have Visual Studio 2005 on your machine. Then, you will need to download the following tools from Microsoft:

http://go.microsoft.com/?linkid=6270225
http://msdn2.microsoft.com/en-us/vstudio/aa718351.aspx

The tool properly is the second link, but you need the components included in the first.

Once is installed, you will be able to access the documentation, that is pretty clear. You access the documentation from Start>All Programs>Microsoft Team Foundation Server Power Tools>Microsoft Visual Studio Team System Process Editor>PEUserGuide.doc

If you want to modify a WI, from Visual Studio you select the menu item Team>Process Editor>Work Item Types>Open WIT from Server

You will get the following interface

In this case, 'Add' a new control (controls in the .NET sense, like text boxes, drop down...) with type Plain Text. The reference name used in this case is Visiphor.Steps, the first part is the namespace used internally and the second part anything you want. For this example, I have created a new control named 'Steps to reproduce.

The second part of this article will refer of how to add this control to a WI.

Second part of the list, as promised! I feel a bit less strongly about the recommendations in this second list. In some cases, I would advise a 'menu approach' that will allow the team to pick and choose.

As before any omments, suggestions and real life stories will be most certainly welcomed!

Action	Description	Who is responsible?	Phase
Code Coverage	The new Visual Studio suite offers the possibility to explore all the possible paths of a piece of code. If all the lines of code are executed at least once in a test environment, it is less probable that they will contain critical errors. Also, if there are areas of the code that are never accessed, it would point to areas where maintenance can be compromised.	Developers	Development
Tracing requirements to Use Cases/Test Cases	This would add overhead to the QA operation, however, it may help to keep all the requirements in sight and also limit their scope	Tester/PM	Planning, Development, Stabilization
Sign on at each milestone.	My proposal would be that at each milestone the PM MUST send an email to the team announcing it. This means that a change in design (Planning) after the ending of the Planning phase will not affect developing. The change will affect next iteration (Thinking in MSF context). This may seem draconian, but may help architects, PM and Customer to be more conscious when they are breaking the process.	PM	End of each phase inside the iteration
No switching of resources	Every time that resources are brought in/out of the project, there is a learning curve and information that is lost. A project with a big resource turn-over risks to have a larger amount of undetected issues	PM	All phases
Representative Test environment	Sometimes, due to the complexity and expense of the production system, the test environment is significantly different from the Production Environment. If the environment are similar, the test effort twill be much more accurate	PM, System Engineering, Testing	Stabilization
Risk Management	Core to the MSF, it should include the customer/customer representative. The extra overhead is well compensated by the managed approach to problem when, if they, appear.	PM, Everybody	All phases
Real and represenative test Data	If the customer supplies real and representative test data, it is easier and more probable to spot problems/code deficiencies during the test process	PM, Testing	Development, stabilization
Customer agreement on quality	Customer should take an active role defining what they expect of the product.	PM	Envisioning
Formal Security Review (if applicable)	For project that require a high level of security, in addition to the usual test procedures –encryption, use of certificates, users rights…- it also should be submitted to deliberate attacks to try to break into the application	Sys Eng/Testing	Planning, stabilization, deployment
Cross disciplinary Peer review of documents	This could identify issues that are more obvious to another team.	Everybody	All phases

Despite the name, most of QA groups/departments in the software industry are not QA departments at all. They are a testing group that design, implements and execute test plans, test cases and test scripts. Nothing wrong with that indeed!. However, the next step to improve the quality of the software is to go beyond the testing phase and create a culture of quality among all the participants of the software development process.

How is possible to begin this culture of quality? Ideally you will implement any of the existing models (CMMI, ISO...), however, given the amount of effort and time (so, money!) involved, it is usually difficult to get committed support from senior management. A group that incidentally looks at testing and QA as non-core/non-fundamental software development process activity.

Here at Visiphor the Test Group is reinventing itself into a QA group by pushing the following list of actions that are easy to implement. Incremental change is powerful if steady. You do not need to apply all the items, you can pick and choose or add your own. However consistency is important. If you pick one of the items, you need to keep it during all the development cycle: it doesn't make sense to peer review version 1, 2 and 4 of a document but not versions 3, 5 and 6 of the same document!

Here is the list. Comments, suggestions and real life stories gladly accepted. I would love to read your comments and what has worked (or not!) for you.

Action	Description	Who is responsible?	Phase
Peer review of documents (architects)	Each document is reviewed at least by another member of the Architects team Architects	Architects	Planning
Peer review of documents (developers)	Each document is reviewed at least by another member of the developers team	Development	Planning, Development
Peer review of documents (tester)	Each document is reviewed at least by another member of the testing team	Testing	Planning, development, stabilization
Peer review of documents (systems)	Each document is reviewed at least by another member of the systems engineer team	Sys. Eng.	Planning, development, stabilization, deployment
Solution Setup (developers)	A senior developer creates the Visual Studio solution(s), project structure, and namespace framework.	Development	Planning or Development
Source Control Repository Setup	A senior developer adds the solution(s) to the source control repository.	Development	Development
Code Reviews (developers)	Similar to the peer reviews of documents, the code for the application is regularly reviewed by another member of the team or by an automatic tool.	Development	Development
Kick off meeting. Also known as Launch Meeting.	Everybody involved in the project needs to be present. The purpose of this meeting is to fulfill several of the MSF principles of common goals, vision, foster communication….	PM	Planning
Setup and use of Issue tracking system	An Issue Tracking System needs to be set up for every project, in addition, it needs to be the central collaboration tool that each team member uses to identify, record, triage and solve bugs and change requests.	Testing	Planning
Sign on at each iteration/Marking Iteration boundaries	The beginning of each new iteration (as defined in the MSF process) should be marked by a team meeting with the same purpose as the Launch meeting. Iteration ends should at least be marked by an email from the PM, and in some cases by a post-mortem-like team meeting. The goal is to keep everybody informed, and with the same objectives.	PM	Through the lifecycle of the process
Dedicated Test environment	The Test environment should be used exclusively by testers and not shared with development.	PM/Sys Eng/Testing	Development, stabilization
Unit test	Developers, should unit test each piece of their code either manually or using automated tools	Development	Development, stabilization
Define the Scope for every project.	It is important to define what is in the project and what is out. Gray areas will always exist but need to be minimized.	PM, Development, Testing	Envisioning
Customer involvement in the testing effort of User Interfaces	The rationale of this point is based in the premise the customer is the best to judge the user interface: both the information contained, the user-friendliness, the work flow and the business logic behind it . In this context, if the customer is involved in the testing, the quality of the end product will be better.	PM	Stabilization, deployment
Project will proceed in a continuous stretch with not time gaps in between.	When a project stops, and then restarts, a lot of the knowledge accumulated is lost. Projects done as a continuum will have better chances to reflect better the customer original intentions.	PM	Through the lifecycle of the process
Post-mortem meetings a the end of each iteration	In a multi-phase or multi-iteration project, this practice should help to improve process and correct problems that have appear in previous phases	PM	End of the project
Setup and use of Change Request Procedure	The team, PM and customer should set up a procedure to deal with new/changed requirements. I should include a risk assessment for the requests.	Testing, everybody	Planning
Setup and use of Requirements System	A centralized requirements repository will reduce the dispersion of information and ensure all the member of the team share a common list of requirements. It may allow the possibility to generate metrics and be able to show to the customer all the requirements changes.	BA, Testing, PM	Planning
Data dictionary	A spreadsheet or document that describes the name, meaning, type and other information about the fields of a database and how they transform (if applicable, as not all the projects may recommend a data dictionary)	BA, Development, testing	Planning, Development
Monitoring of the systems for unauthorized changes	This monitoring should be associated with clearly defined consequences for intentional unauthorized changes	Everybody	All phases

Soon I will post a second list of recommendations that are a little bit harder to implement or less applicable to all projects.

Note: The "Phase" is in the context of the Microsoft Solutions Framework (MSF), and are orientative. Some of the tasks do not have a clearly defined phase in which they need to be done.

I have read the interesting article about Michael Porter's ideas titled 'Why Do Good Managers Set Bad Strategies' (
http://knowledge.wharton.upenn.edu/article.cfm?articleid=1594). The answers are interesting, although I have some concerns about the premise: Excellent managers can be very bad strategist and vice-versa (In WWI Germany was very well managed, but its long-term strategy proved disastrous, while the new and chaotic Soviet Union became a World power).

However, it is his statement of "Strategy has to do with what will make you unique" what make me scratch my head. In the current context of the software development industry, specifically in the context of a small professional services firm like the one I am working on, what we can bring that is unique?

Perhaps it is a self-interested answer. However, it is clear for me that currently that major differentiator among our competitors is Quality, used in the broadest possible terms. Quality understood as the set of processes that will be able to deliver in time in budget a solution a business need in accordance to the customers' expectations. Notice this would include managing the customer's expectations!

After more than ten years in the software industry, still surprises me that so few are trying to exploit this niche as differentiator. It still surprises me that many managers, visionaries, captains of industry and so on see QA as a luxury or fat to trim. In most of the cases, it is a simple marketing investment.

In an ad from Mercury, they say, "80% of applications are deployed untested. 100% of customers really hate that.” So, if this is the state of the industry, let me return to my suggestion and to the point of this lucubration: what about developing a strategy that will focus in the QA of your product as a way to make it unique?

The last but not the least. The article also includes the quote "If you don't pursue a direction for two or three years, it's meaningless." Putting the quote in the QA context: you should not waste energy and efforts implementing a lukewarm QA strategy. Do not set a policy and then constantly break it. For QA professionals it is frustrating and you will not achieve any good.

I have been reading the massive article

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/mtf.asp

posted in the Microsoft web site with the title of "Testing .NET Application Blocks–Version 1.0".

I can not say that I liked it. I consider it useful, I think that will take me some time to fully digest it contents, however, I would say that it could be written in a less bone-dry tone. For example, I should be excited to read the article after reading the introduction, not to consider that it could be appropriate to read it...

In addition, there are tendentious comments :-). For example
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/mtf_ch02.asp
"Testing Methodologies" is not an honest report about the different testing methodologies. It is a naked attempt of Agile apologetics. After the authors, Agile has no disadvantage at all! What about scalability for example? The more I know about Agile, the more attractive I find some aspects. However, you should not forget all the strings this methodology has attached: a competent, committed and engaged customer for starters....

In any case, if you have lots of patience, are not sleep deprived and have some time, it is a reading that I recommend.

As the lead tester at Visiphor Consulting Services, I find myself in the position of having to expose my opinion even in case in which I do not really have an opinion!

In this blog I will post some of my though about the art of testing, specially in the brave new world of SOA, .NET, Web services, integration platforms and hints of Agile methodology.