In this moment I have changed my tester hat for a project manager, or better said, general contractor hard hat. I am building what I hope will be my future house.

It's a lot of excitement, a lot of frustration, and a lot of opportunities to learn. And one of the things I have learnt is that professions are not so different from one another.

A few weeks ago the structural engineer -the guy that make sure the architect design will hold in case of storm, high winds, earthquake and so on- presented me the structural drawings. He has a reputation of preferring to overdue things. However, he was very frank: the structure he designed will ensure that in case of earthquake -the most significant hazard in my corner of the woods- the house will not collapse. However, after an earthquake, the house can not be inhabitable. To continue being useful after an earthquake (think of hospitals, fire stations and so on....) you need to spend more money and time.

This is a situation identical to software testing: to execute all possible testing paths and areas (functionality, performance, stress, security, usability....) will require an enormous effort that you may choose not to do. The important aspect is the decision needs to be made rationally and even more important, documented. All the stakeholders should be aware of the limitations of the testing effort and the trade off that have been made.

And trade off are a part of life, and part of software testing. Unless you are testing a flight control system, for example...

A few months ago, I posted the following question to a test discussion group

Is Agile Testing oxymoron?
Does Agile make life for testers more difficult?
Agile was though by developers to solve specific developers' issues. However, for other trades like QA Agile has done not much, and arguably has complicated our life a bit more, if nothing else creating false expectations: integration testing, performance testing, stress testing and so on can not follow an Agile model, despite some project managers wish it could.

I would to start a discussion around this topic, if nothing else to gauge the opinion of fellow QA engineers. If nothing else may provide some food for thought for project managers that think that vague user stories with changing details are enough to develop a good test script!

Unfortunately, only Jason M. Morgan replied to my post. His comments are

My most recent employer was using Scrum. The QA activities related to the new features being developed in the scrum team fit easily into the sprints. But there were larger testing activities such as regression testing that were to big for one team or even one sprint. So I had to break down the regression testing into small pieces that could be assigned to each team within a sprint. It was a lot of overhead for no benefit other than following the scrum model.

So, clearly, this topic didn't awake the imagination of testers. However, I still think that Agile, as currently formulated and applied, doesn't make life easier for tester and I will argue that to other trades involves in the software development process.

I think that Agile contains a lot of good principles and it makes a lot of sense to use it in most projects, specially the small/middle size. But I would like to see Agile taking into account the needs of Business Analyst, Technical Writers, Trainers, Infrastructure, QA and QC. For example, it is very difficult for a Technical Writer to put a story into words without really knowing how the other stories have been implemented. The same for a tester: it is possible to test the flow of a single story, but it is much harder to imagine how this story interacts with others if they exist only in concept (think about processing a payment as one story, but the create payment story still is under development). It is in this context that I say Agile has shifted work from development to the other areas. Now the question is not to throw Agile away, but how to extend its benefits to everybody that contributes to the completion of a project.

Lessons learned from Performance Testing a complex application

After two years too busy to write any blog, I have now a bit more time to share some of the lessons I have learned while doing Performance Testing for a complex application (Enterprise-level, several millions code lines kind of application).

- Complex application usually mean lots of different components: network, hardware, software... When producing performance data, you need to gather data about how each component behaves. It is not useful to say "the application is slow" you need to say "the application has a bottleneck in the disk I/O".

- You will be able to gather pretty much all the information you need about how your system is working using the performance counters embedded into your OS (MS, Linux...).

- Use only the performance counters you need, as they consume resources. Once you realize the values provided by a counter are "good", you can stop using it.

- When the application is far from being optimized, run all the counters and make a list of the worse one grouping them by logical areas (data repository, UI, business processes...)

- With the previous list, test individual components that are described by the counter. For example you may focus on the Data repository layer, looking for the disk I/O, for hard/soft page faults etc.

- Based on the previous results, concentrate then in a single counter. Using the previous example you may focus in the disk I/O counter if the other counters look Ok.

- Focus first in the low hanging fruit: most performance problems are related to network bottlenecks, lack of memory, disk I/O and pinning CPUs. Once these are removed, you can focus in secondary problems: too many threads, unoptimized queries, unmaintained databases etc. After that, you do not have so many options. The most available would be to throw more hardware to the problem. If the Performance is still unsatisfactorily, then rearchitecting your application is usually the only option left

- Unless there are egregious errors, refactoring your code helps very little if at all.

- Use automation to create load. To measure response time to user's actions use a human with a stopwatch

This last point may be surprising for some. My experience is that introducing timers in the code doesn't work. First it is not always possible, second, in practice it is pretty much impossible to capture certain events: you can capture when an image starts to be displayed in the screen, but not when it is shown entirely. If you want to measure the user experience, better to do it by hand, assuming the role of your user.