Agile Business Change Blog Thoughts on Agile Strategic Business Change and Agile Delivery

Tweet

Mail

Doing non functional testing early in a project has obvious advantages. It’s also a challenge of course to be able to get meaningful outputs based on the current state of the project. This has created an interesting issue on several projects that I am currently working on.

OK, so let’s be more specific here. I was initially looking specifically at performance issues and not looking at stress/load, security and soak testing and any other NFT issues we could discuss.

So my challenge to the testers was how can we undertake some performance testing in early sprints? They went away to have a think about it and came back saying that although they thought it was a brilliant idea to do this early, it was impossible. The dedicated team that ran the automated tool kit to perform these services was unavailable and the system wasn’t in a state to test yet because it wasn’t on a production equivalent platform and only a fraction of the software was developed. So the quandary was they wanted to do it, but didn’t see how they could do anything but 100% of it. Not only that, in their view, the results would be worthless because they would not be representative of a production system.

All these things may be true to a lesser or greater extent, but that didn’t help me understand if we were developing a system that would not be fit for purpose performance wise and were heading at high speed towards the no-go wall with the risk of significant rework.

So what we did was firstly move the goal line. I wasn’t after definitive performance metrics against which I could ‘prove’ my system. I was after green, amber and red flags that could be waved to say if we thought there may or may not be an issue. Secondly, we agreed that a relative performance estimate against a known baseline would meet my requirements. Getting a baseline wasn’t too hard as the project was a package based solution. So, the package vendor could supply an estimated response time for a running a ‘standard’ transaction on representative production kit. The actual time wasn’t important here, all I needed to know was this would represent an acceptable transaction time. Of course, it was possible that no transactions would run fast enough on the production system. However, as this package was installed in thousands of companies and as we were using roughly the recommended production environment, I thought this was a reasonable assumption to make, although I accepted I had been bitten by this kind of problem before. From this we generated a baseline we could compare against. We could then run other custom transactions through our software and see how many times slower they performed compared to the baseline. So if a test runs in one or two times the baseline time, its probably OK, but if its 1000 times slower it looks like we may have a problem.

It’s reasonably crude, but does give an insight into possible performance issues for very little effort, and that was all I was after. Its also pretty easy to automate as well. I explained my reasoning to the test team and as soon as I explained I wasn’t that interested in the numbers and only general indicators, I got a mixed response. Some simply wouldn’t accept the value of the results, but a few got my angle and went off to set something up. Great, the first hurdle was cleared and I started to think what else could be achieved and chose concurrent access.

This one was relatively easy to get past the testers who usually used a big test package to simulate 100 concurrent sessions. We just got a simple script running on 5 machines. Nothing special there.

In fact there wasn’t anything special at all in anything we achieved. This was the kind of thing we did before these heavyweight tools were available and although the results we got were simplistic, it was all we had at the time.

What these modern NFT tools can simulate is pretty amazing, and then can give some much needed confidence before you turn the key. But they can be expensive to set up (and maintain) and are often a resource that’s only available late in the project. Late in the project in my eyes is often too late and an early indicator is needed.

Sometimes there is value in regressing to something more primitive, at least temporarily! As the Hitchhikers Guide to the Galaxy says” And we'll be saying a big hello to all intelligent life forms everywhere. And to everyone else out there, the secret is to bang the rocks together, guys”. I am happy to be selecting the type of rock to use knowing early results are better than no results. So what do you think, granite or limestone?