Without a new approach to testing, incidents like the recent NASDAQ trading glitch and last summer’s Knight Capital mess will inevitably keep popping up and weakening the strength and reputation of our markets. Exchanges need to adopt an approach that tightly integrates testing into development. An iterative process in which testing and development happen in parallel is the only way to ensure that systems are thoroughly tested, and that errors are found and fixed before production.

Typically, testing that happens in exchanges follows a “develop now, test it at the end” approach. Concrete dates are set months in advance that promise specific functionalities for new systems. Teams dedicate a certain number of months to development and the rest is left for testing. For example, if a team plans for a system to roll out in six months, they may plan to spend four of those months developing and two of those months testing. Realistically, for any number of reasons, development takes longer than anticipated and testing time is cut down. In this scenario there is probably five months of developing, which leaves only one month – four small weeks – for testing.

So what exactly happens when an exchange gets ready for testing? First, test cases have to be created to test the system, issues are found, and then fixed. Then, this process repeats itself until no more issues pop up. Sounds simple enough, but when using this approach it is hardly possible to catch five months of development issues within the small time frame of one month. Since the system is being delivered on a strict deadline, there is simply not enough time to test the whole system or to test it thoroughly, which is where most problems arise from. Roll out a system that hasn’t been thoroughly tested, and once in production, you could be surprised by problems at the wrong time.

Another problem is that errors that may have popped up early in development – say in week 1 or week 2 – are not found until close to the end of the project when testing actually gets underway – past week 16 or even week 20. The code that caused an issue is much past the point of “fresh” in the developers mind and it can take a lot of time to re-familiarize them with the problem. Furthermore, issues that pop up early in development can cause a whole new string of errors that could otherwise be prevented had the initial error been caught right away.

If an error halts trading, it prevents a release. Are exchanges doing the level of testing they need to in order to find errors? The answer is no.

If the quality of the markets is to improve, a parallel approach to testing must happen and technologists must view the testing process as a critical piece of the development process. Currently, there are better tools for testing and feedback, but these tools need to be adopted by the exchanges. It is up to the technologists at the exchanges to get the tools into the process so teams aren’t scrambling at the end, but rather testing all the way through. Exchanges need to adopt a process that would include a test every time development made a change to the system; then they would know immediately when something is broken.

To be more comprehensive, automated testing needs to happen early on. It must start at the beginning of the process – before development happens – so that the quality of the system can be visualized throughout. Value comes the sooner testing is automated, so it has to be done as soon as possible.

We must look at the current and past issues and realize that something needs to be done. A fresh perspective on the development process needs to be put in place. Without a proper resolution, these glitches will continue to happen and our markets will continue to weaken. 

The opinions and writing contained in this article are of the author alone and do not necessarily represent those of HFTReview.com.