I discussed the following inventory of open source and free model-based testing tools in a recent QUEST panel session.
After checking each tool, it is clear there’s wide variation in maturity, stability, and provisioning. I see five groups
Almost all of the following is from provider’s web sites – YMMV. The “Stack” column indicates the source code language and/or run-time environment(s); “*nix” indicates a tool written to run on most Linux/Unix variants. See end for a revision list.
| Tool | Description | Stack | License |
|---|---|---|---|
| JUMBL 5.0 | The Java Usage Model Builder Library (JUMBL) supports automated, model-based, statistical testing of systems. Includes command-line and GUI interfaces. | Java. Provided as Jar file (byte code). Windows and *nix. | University of Tennesse |
| Spec Explorer 3.5 | A Visual Studio add-in with support for modeling software behavior, graphical visualization of model explorations, and model checking. Users write model programs in C# and define usage scenarios. The model programs can use a built-in combinational (e.g. pairwise) data generation feature. Generates ready to run test code from model programs. NModel was a prototype. | C#, Visual Studio. Windows. | Microsoft Visual Studio 2010 Power Tools |
| Torx 3.10.5k | Supports conformance testing of reactive software; checks the correct behaviour of a real implementation during its execution based on its formal specification. Can execute a Torx specification as SUT. | *nix, maybe Windows+Cygwin | Apache 2.0 |
| Uppaal TRON 1.5 | An integrated tool environment for modeling, validation and verification of real-time systems modeled as networks of timed automata, extended with data types (bounded integers, arrays, etc.) | *nix, maybe Windows+Cygwin | Uppaal license. Non-commercial use only. |
| Tool | Description | Stack | License |
|---|---|---|---|
| NModel 1.0.21029.0 | A model-based testing and analysis framework. User writes model programs in C#. NModel generates tests from your model program in advance or on-the-fly as the test run executes. Users may add custom test generation strategies in C# to achieve any coverage criteria. SUT not limited to Windows. | C#. Requires .NET, but not Visual Studio. | Microsoft Open Source License |
| TopCased 5.1 | Model-driven development and testing environment for critical hw/sw embedded systems; Uses formal methods, SysML, UML, OCL. | Java/Eclipse | Eclipse Public |
| Tool | Description | Stack | License |
|---|---|---|---|
| Graphwalker 2.5.16 | Generates offline and online test sequences from state machines modeled with GraphML -- no UML to learn. User selects any of seven built-in coverage (stopping) criteria for test generation. Tool may be integrated with a Java test harness or called with SOAP as web service from the developer's server. | Java | MIT |
| IFX Omega 2.0 | A compiler/simulator/model-checker for a rich subset of UML 2.2, SysML 1.1, based on the VERIMAG IF model checker. May be used to validate and verify asynchronous timed system models with complex dynamics. Uses rigorous semantics suited for critical real-time embedded systems. | Java/Eclipse | Not Specified |
| ModelJunit 2.0 | Adds model-based testing to Junit. Models are written as Java classes. Uses the annotations feature of Java 5.0. Automatically generates Junit test suites from these models. | Java, JDK 1.6 or higher. | GPL |
| Tool | Description | Stack | License |
|---|---|---|---|
| ATS4 AppModel | Supports app specification, model based testing, and test script generation. Simple interface to manage complex models. Developed at Nokia for handheld testing. | Java | BSD |
| AutoFocus 0.3.0 | Supports modeling and analyzing the structure and behavior of distributed, reactive, and timed computer-based systems. | Java/Eclipse | Apache 2.0 |
| Fastest 1.3.6 | Generates tests from models expressed in the Z notation, following the Test Template Framework proposed by Phil Stocks and David Carrington. Supports unit testing. | Java, JDK 1.6 or higher. | GPL |
| GUITAR 1.0 | Automatically extracts structure and widget APIs from Java Foundation Classes (JFC) and web GUIs. Generates an Event-Flow Graph (test model), generates test cases from the model, evaluates tests with built-in oracle. | Java | None specified |
| JTorx 1.10.0 | Generates tests to check that the ioco testing relation holds between a labelled transition system expressed in Aldebaran or GraphML and an IUT. Derived from Torx. | Java | BSD |
| MISTA 1.0 | Uses lightweight high-level Petri nets as a visual modeling notation.Test models can be animated and verified.Generates executable test code fom a test model in Java, C, C++, C#, VB, or HTML and/or test engine JUnit, NUnit, Selenium IDE1, or Robot Framework. Several test generation strategies are supported. | Java | None specified |
| TEMA 3.2.1 | A prototype tool and methodology for online test generation for smartphone GUI testing. Based on the GUI keywords/action word testing strategy. | Java | MIT |
| Tool | Description | Stack | License |
|---|---|---|---|
| Graphwalker 2.5.16 | Generates offline and online test sequences from state machines modeled with GraphML -- no UML to learn. User selects any of seven built-in coverage (stopping) criteria for test generation. Tool may be integrated with a Java test harness or called with SOAP as web service from the developer's server. | Java | MIT |
| IFX Omega 2.0 | A compiler/simulator/model-checker for a rich subset of UML 2.2, SysML 1.1, based on the VERIMAG IF model checker. May be used to validate and verify asynchronous timed system models with complex dynamics. Uses rigorous semantics suited for critical real-time embedded systems. | Java/Eclipse | Not Specified |
| ModelJunit 2.0 | Adds model-based testing to Junit. Models are written as Java classes. Uses the annotations feature of Java 5.0. Automatically generates Junit test suites from these models. | Java, JDK 1.6 or higher. | GPL |
| OSMO Tester 2.1.1 | Open Source Modelling Objects (OSMO) includes tools and libraries to model software behaviour with extended state machines in Java for online/offline test generation and non-deterministic online models. | Java | LPGL |
| Parteg 1.3.3 | Partition Test Generator: generates tests meeting control flow-based coverage criteria (e.g., MC/DC) and boundary-based coverage criteria. | Java Eclipse | Eclipse Public |
| Date | Action |
|---|---|
| 31 May 2012 | Added Fastest. |
| 25 April 2012 | Formatting, clarifications. |
| 20 April 2012 | Added GUITAR. |
| 15 April 2012 | Initial release. |
The WordPress plugin WebSimon generated the above tables — easy to use, looks great.
That is an interesting list. Thanks for compiling it! What about the GUITAR model based GUI testing framework?
http://sourceforge.net/apps/mediawiki/guitar/index.php?title=GUITAR_Home_Page
I will add it. I’ve been following GUITAR research for quite awhile and I’m glad to see that it has finally been released.
Thanks for compiling the list. You may also add Fastest. It’s available from http://www.fceia.unr.edu.ar/~mcristia/index-en.html. Fastest is a MBT tool for the Z notation. Regards.
Thanks for this — I’ll add it to the list.
Which of the tools given above under ‘Binaries’ category used for Black box testing?
All of them. A “binary” here refers to a tool that is released as an executable only file – the source code is not public.
Hi,
Did you already saw the DTM tool? For more information, please visit http://www.dtmtool.com.
Cheers,
Silvio
Hi Silvio
Thanks for the suggestion. DTM looks interesting, especially in the way you’ve oriented it for Agile development.
However, it doesn’t seem like it qualifies as open source.