When you Can’t Drop the Database
Like high-level requirements, live testing goals appear deceptively simple. “Exercise the trading API implementation with my trading partner” is a single statement that to the trained eye reveals numerous scenarios. Ideally, you already considered most scenarios and wrote automated tests to verify functionality. But, unlike sandbox testing, when something goes wrong in a live testing scenario, clearing a database probably won’t fix anything.
Consider the situation I found myself in as I was testing my trading API implementation. I am on a conference call with my trading partners preparing to manually submit trades. I submit a trade that should not be filled when I am told, “That order shouldn’t have been filled, but you were executed. Close your position immediately!” I need to act quickly. Real money is the on the line and people from another company are awaiting my response.
I admit this is a dramatic example. In live testing scenarios, I tend to be a bit tense and nervous. Even with a suite of automated tests, I still fear something will go wrong. I don’t do my best thinking under-the-gun, so I prepare plans for handling unexpected outcomes in advance. This helps keep me calm and in the event something goes wrong, I will hopefully have a fallback available.
How to Prepare
Reflecting on the different areas of preparation led me to the following categories: communication, flexibility, and instrumentation. Let me detail questions I answer in each category to determine readiness.
Communication
Have I explicitly stated the test scope?
While the purpose of a test may seem obvious to you, the third party may be unaware of special cases you wish to exercise. Express your test cases up-front to avoid discovering during the test that a subsequent test is needed for a special case.
Has the third party confirmed that it can take corrective action in the event of X?
X is a disaster scenario where you lose control to fix the situation. For example, if an order erroneously executes and you lose network connectivity, it is imperative that the third party can close a position on your behalf.
Flexibility
How easily can my test harness test each known scenario?
Using the trading example, it is helpful to be able to interactively specify all order properties on-the-fly. It becomes increasingly more important to have a flexible test harness when live tests are hard to coordinate.
In the event of an emergency, what is the fastest way to deploy a new build?
Ideally this option is rarely or never used. But, in the event that you need to build a kludge for testing purposes, knowing the deployment process will minimize delays.
Instrumentation
When I perform test case X, what information will the third-party want me to confirm?
Asking this question of each test scenario is a good way to ensure all the requisite information will be on-hand.
If X goes wrong, what information would help me debug?
When a test scenario goes awry, it is extremely discomforting to discover a piece of relevant information is not visible.
Are log levels configured appropriately?
This sounds trivial, but I’ve missed this item before and nullified all my efforts to view my debug statements.
And a useful tip for testing on remote machines: use Screen! This way when you lose network connectivity at the worst possible moment, your foreground process continues running and you can reconnect and resume as if nothing happened.
I learned some of these lessons the hard way. Learn from my experiences so you don’t fall prey to the same mistakes!
Software Jockey 