A few months ago, I posted the following question to a test discussion group

Is Agile Testing oxymoron?
Does Agile make life for testers more difficult?
Agile was though by developers to solve specific developers' issues. However, for other trades like QA Agile has done not much, and arguably has complicated our life a bit more, if nothing else creating false expectations: integration testing, performance testing, stress testing and so on can not follow an Agile model, despite some project managers wish it could.

I would to start a discussion around this topic, if nothing else to gauge the opinion of fellow QA engineers. If nothing else may provide some food for thought for project managers that think that vague user stories with changing details are enough to develop a good test script!

Unfortunately, only Jason M. Morgan replied to my post. His comments are

My most recent employer was using Scrum. The QA activities related to the new features being developed in the scrum team fit easily into the sprints. But there were larger testing activities such as regression testing that were to big for one team or even one sprint. So I had to break down the regression testing into small pieces that could be assigned to each team within a sprint. It was a lot of overhead for no benefit other than following the scrum model.

So, clearly, this topic didn't awake the imagination of testers. However, I still think that Agile, as currently formulated and applied, doesn't make life easier for tester and I will argue that to other trades involves in the software development process.

I think that Agile contains a lot of good principles and it makes a lot of sense to use it in most projects, specially the small/middle size. But I would like to see Agile taking into account the needs of Business Analyst, Technical Writers, Trainers, Infrastructure, QA and QC. For example, it is very difficult for a Technical Writer to put a story into words without really knowing how the other stories have been implemented. The same for a tester: it is possible to test the flow of a single story, but it is much harder to imagine how this story interacts with others if they exist only in concept (think about processing a payment as one story, but the create payment story still is under development). It is in this context that I say Agile has shifted work from development to the other areas. Now the question is not to throw Agile away, but how to extend its benefits to everybody that contributes to the completion of a project.

Lessons learned from Performance Testing a complex application

After two years too busy to write any blog, I have now a bit more time to share some of the lessons I have learned while doing Performance Testing for a complex application (Enterprise-level, several millions code lines kind of application).

- Complex application usually mean lots of different components: network, hardware, software... When producing performance data, you need to gather data about how each component behaves. It is not useful to say "the application is slow" you need to say "the application has a bottleneck in the disk I/O".

- You will be able to gather pretty much all the information you need about how your system is working using the performance counters embedded into your OS (MS, Linux...).

- Use only the performance counters you need, as they consume resources. Once you realize the values provided by a counter are "good", you can stop using it.

- When the application is far from being optimized, run all the counters and make a list of the worse one grouping them by logical areas (data repository, UI, business processes...)

- With the previous list, test individual components that are described by the counter. For example you may focus on the Data repository layer, looking for the disk I/O, for hard/soft page faults etc.

- Based on the previous results, concentrate then in a single counter. Using the previous example you may focus in the disk I/O counter if the other counters look Ok.

- Focus first in the low hanging fruit: most performance problems are related to network bottlenecks, lack of memory, disk I/O and pinning CPUs. Once these are removed, you can focus in secondary problems: too many threads, unoptimized queries, unmaintained databases etc. After that, you do not have so many options. The most available would be to throw more hardware to the problem. If the Performance is still unsatisfactorily, then rearchitecting your application is usually the only option left

- Unless there are egregious errors, refactoring your code helps very little if at all.

- Use automation to create load. To measure response time to user's actions use a human with a stopwatch

This last point may be surprising for some. My experience is that introducing timers in the code doesn't work. First it is not always possible, second, in practice it is pretty much impossible to capture certain events: you can capture when an image starts to be displayed in the screen, but not when it is shown entirely. If you want to measure the user experience, better to do it by hand, assuming the role of your user.

To automate or not automate, this is the question...

Whether 'tis nobler in the mind to suffer
The slings and arrows of manual testing
Or to take arms against a sea of automated test
And by developing them. To die, to pain--

.....


Anybody that has tried automation knows that automation needs to be done carefully. Managers, convinced by automated tools salespeople and Start Treck voice activated computers will try to force you to automate everything. The arguments will be the usual ones of increase productivity, results reliability and speed. However the land of automated tests is littered with corpses and failed projects.

Why? Well, based on what I have seen the foremost reason is that test automation are projects that need to be though, managed, staffed and planned as any other software development project. To be useful they need to answer the following questions:

• Will the test be done a limited number of times or it is a test that needs to be executed regularly?
  
• The requirements for the application under test, will change soon? change with time? are still undefined?
  
• Is the code under test architected in such a way that a change in a component will not affect our automated test?
  
• Can the code under test be automated with an off-the-shelve software?
  
• Have the code under test be instrumented? Does it have hooks that can be used?
  
• Is the automated test code modular?
  
• Is the automated test code easy to maintain?
  
• There is time allocated to document your automated test?
  
• Are you testing your automated tests?
  

These are basic considerations that need to be answer before proceeding with an automation project. However, before this considerations it is fundamental to consider the first point Ihave made: Any automation exercice as a project in itself. This means it need to have the appropriate level of staffing, project management supervision, methodology... Failure to plan the automation project appropiately will incur the same risk as any other project: overworked staff, last minute problems, cost/time over runs...

So, my sincere recommendation to anybody trying to automate the testing of their new piece of software: treat it seriously as a subproject of your main project