The Mobile Testing Nightmare
Challenge: How to communicate the value proposition of a highly-focused testing technology for mobile apps? After two years of NIST-funded R&D, mVerify shifted its focus to commercialization. The early stage of mobile platforms meant that few development organizations would consider even superficial testing. The branding strategy and its campaign had to reach those early adopters and make optimal use of a very tight bootstrapping budget.
Solution: Recruited a legend in IT marketing and then collaborated to develop a core message: the Mobile Testing Nightmare. This resulted in a memorable intersection of emotional impact and technical truth. Developed a trade show stage and script to showcase this. Designed high-impact collateral to amplify the message. Conducted high-impact demos at carefully selected trade shows. Deployed Salesforce.com to manage the pipeline.
Results: For a total cost of about $40,000, generated a pipeline of 632 qualified opportunities, resulting in 7 closed orders.
No glitches allowed: 3 million transactions per hour, 15 billion dollars per day.
Challenge: Perform all testing on a new “screen-based trading” system designed to process millions of transactions per hour to make markets for hundreds of thousands of complex regulated derivatives. The exchange’s legacy test process averaged about one test case per hour. This was a non-starter: even with 10 times more budget, it couldn’t have covered 1/1000th of the complex derivative scenarios the new system offered or evaluate its performance under realistic and varying loads.
Solution: Devised and developed a completely original model-based testing system using discrete-event simulation and AI business rule evaluation. With this, generated over 200 completely realistic trading sessions (each with hundreds of thousands of transactions) as tests. The transactions were automatically submited to trader screens and the distributed object interface for high-volume program trading. Then, the results of every trade were automatically checked.
Results: Huge improvement in test productivity and effectiveness. Revealed about 1,500 bugs over two years, of which 5% were showstoppers. The last pre-release test run applied 500,000 fresh test cases in two hours, with no failures detected. Post-release, no failures were reported in the first six months of operation. The system was later used to prove that integration with another exchange was feasible and scalable, leading to a multi-billion dollar venture for joint operations.