GRU testing framework: user manual

Bernard AMADE

Revision History
Revision 1.02011-10BA

who and why

Author (other contributing authors needed!)

Paul Bernard AMADE
Laboratoire APC
10, rue Alice Domon et Leonie Duquet
75205 Paris Cedex 13
bamade@in2p3.fr
(33) 1 57 27 69 14

GRU

This is a presentation of the GRU testing framework developped specifically for the LSST project. The tool is rather general and probably could be presented as a separate open source project.

GRU is a D.S.L (Domain Specific Language) based on Groovy. Its main purpose is to design, run and report various tests on Java code. Users should have a minimum knowledge of groovy since the gru scripts use many groovy features (but a very simple usage of groovy will be explained in this document).

The document is incomplete since the framework is still under development (and waiting for advices from users).

An introductory chapter will try to answer this question: "why create a specific framework instead of using one out of the shelf?". To cut a long story short: the goals are different from those of JUnit.

Table of Contents

1. What to expect from a testing framework ?
1.1. "Carpet bombing"
1.1.1. Test data sets
1.2. "Carpet bombing" drawbacks
1.3. Tests specifications: where and when?
1.4. Test execution
1.5. Test specifations and reports
1.6. Report handling
2. Getting started
2.1. Script execution
2.2. Introduction to simple test data and to simple groovy
3. Testing constructors
3.1. Single argument calls
3.1.1. special arguments
3.2. Multiple argument calls
3.3. Beans
3.4. Other features
3.4.1. setup/teardown codes
3.4.2. assertions
3.4.3. variables
3.4.4. exporting data
3.5. Testing static factories
3.6. Simplified syntax shortcuts
4. Testing methods
4.1. instance methods
4.2. instance factories
4.3. static methods
5. Testing "in situ" objects through proxies
5.1. Starting a Modular SubSystem through groovy
5.2. Proxy class and instances for testing
6. Handling results
7. General language layout
7.1. Tests descriptions
7.2. Monitoring
8. What’s next ?

Chapter 1. What to expect from a testing framework ?

Not all expectations for a testing framework can be met through the use of a single tool. But stating overall goals is important to understand some features of this particular tool. (Note: this chapter is rather abstract so you may skip it and read it later!).

Testing comes in many flavours : we are investigating "carpet bombing". It means that we keep predefined lists of values and that those "check lists" are used to spread values over parameters of constructors and methods.

Unless otherwise specified those values are not picked up at random. We want to keep lists of values that may have specific properties. Setting up and maintaining such lists requires experience and intuition. For instance try to set up a list of String objects that may have specific properties in a general context : you will be suprised by the sheer number you will find! Then you can add more Strings for a specific context.

1.1. "Carpet bombing"

Programmers are lazy! Simple tests should be written quickly and easily all the time: during detailed design, during implementation, and any time afterwards. If simple tests specifications are not simple enough we will get "lip service" testing (that happens all the time in projects!).

1.1.1. Test data sets

Often java constructors and method calls should be tested against a set of values.

If we have a single parameter for a call it may be important to test against a whole range of values for this parameter.

What could those "sets" be? Some examples:

  • double values: positives/zero/negatives , "normal" values, small/big values, imprecise values (such as 0.1), values that may cause rounding problems, exotic values( NaN, infinite, etc.) , …
  • String values: "normal" strings, strings with i18n chars, with unicode chars from other planes, null/empty strings, Strings with spaces or non printing chars, "exotic" strings (with slashes, colons and so on …)
  • size values : 1K (plus/minus 1), 2k, 5K … (found a bug in a messaging system using this: a message of exactly one of these sizes crashed the system)

Since we cannot have the same expectations for the whole set of possible values for a type we should group those into subsets.

We need a naming strategy: names for all kinds of sets and, in fact, a symbolic name for each value we use. Each test "instance" should also be labeled : if the test produces data then this data is associated with this key.

So we are going to use named values that may be specified as generated by a test or simply as being assigned a value of some type (e.g using a litteral assignment).

Note that we could also apply a method on a whole set of instances.

1.2. "Carpet bombing" drawbacks

The combinations of values tested that way could grow exponentially! (for simple unit test on a simple class you may end up with hundreds of tests -if not thousands!-).

Though we should not be concerned by performances it is not necessary true that more tests yield better tests! Moreover the analysis of results could be obfuscated : for instance the very same bug will produce many negative results that have all the same cause.

Assigning expectations to a whole set of values may not be trivial: subsets should be cleverly designed.

So caution and balanced choices should prevail : one one hand it is important to test again varied ranges of values in a check list; on the other hand over-testing may not be a viable option.

1.3. Tests specifications: where and when?

There should be different places for test specifications : detailed design documents, source code itself, external files (for codes we did’nt write), maintenance documents ,…. Those specifications should be extracted and then the tests executed.

This means that build software should be aware of the corresponding dependencies and that different "test documents" should be linked together.

Tests could be run when code is incomplete : some code is still waiting to be written but test spec is already here. This lead to an interpretative nature of test execution: if the corresponding java code is not yet here a special report is issued.

Since the programmer may need to write complex code to design a complete test there may be a clash between the need for java code and the need of an interpretative test specification.

To alleviate the programmer learning curve and have the best compromise between simplification and flexibility the choice of a script language (such as groovy) looks promising (note that groovy is not exactly an interpretative language -though it has many dynamic features-)

1.4. Test execution

Let’s call a "testing unit" (not to be confused with "unit test") a set of related documents that describe a set of tests (for a given java class for instance).

Some units have a special status: they just declare set of values to be used by other tests. Otherwise execution of a testing unit should go through successives phases : optionnal pre-test code execution , test collecting, execution, optionnal post-test code, reporting .

  • tests specifications (and related codes) are first gathered.
  • actual tests are generated (this is due to the combinatorial nature of tests descriptions)
  • once the collection is complete the order of execution is determined: first tests for constructors and factories then method calls on the ensuing instances. In each bundle the order of execution is picked up at random: this is the default strategy but it could be specified to execute sequentially (parallel tests also possible). Each time a test is executed a report is built and kept: only logging information is issued on the fly as the tests execute.
  • when all tests have been executed the reports are issued in a specific order: since a key is assigned to each test the reports are sorted according to the alphabetical order of the keys. (this is important to compare different batch of tests results).

Since other codes may be linked to code execution the syntax of a test language needs to link precisely those codes to execution phases.

1.5. Test specifations and reports

A "testing unit" will contain various codes (and imports) and many "test specification".

Due to the set description for instances and arguments used, each test specification could result in many effective tests.

The result of each actual test could be complex data. Here is the present enumeration of results used in GRU:

/**
 * A simple value that summarizes the result of an execution.
 * It is "raw" because  the analysis of test results may produce more sophisticated
 * reports (for instance a FAILED test may be annotated with BEYOND_LIMIT to indicate
 * that this is not an error : it is a feature!)
 */
public enum RawResult {
    /**
     * the tests specification may be erroneous (or the testing tool itself failed)
     */
    TOOL_OR_CODE_ERROR,
    /**
     * the test failed
     */
    FAILED,
    /**
     * the test failed because some needed data could not be evaluated (usually because
     * a previous test failed and did not produced this data). Usually the data is null
     * without being tagged with a NULL* name.
     */
    MISSING_DATA,
    /**
     * not used: this is a programmatic mark, values superior to this one are considered
     * to mark a test that did not fail.
     */
    OK_MARK,
    /**
     * the requested class or method is not yet implemented
     */
    NOT_YET_IMPLEMENTED,
    /**
     * the test was not evaluated
     */
    NOT_EVALUATED,
    /**
     *  not used: this is a programmatic mark: values superior to this are
     *  considered executed and not failed
     */
     OK_DONE_MARK ,
    /**
     * the test succeeded but with warnings
     */
    WARNINGS,
    /**
     *  no advice on fail or succeed, just a trace.
     */
    NEUTRAL,
    /**
     *  success: expectations met
     */
    SUCCESS ;
}

The test "expectations" description may contain :

  • nothing : the fact that the code runs is in itself a success
  • result or exception description : if a value or an expected exception is produced it is a success.
  • various assertions: each assertion is executed regardless of previous results, a FAIL is issued if one of these failed. All assertions will be traced.
  • a "pending" annotation: evaluation of test will be made later (e.g. by report analysis).

So a test description may contain:

  • a set of instances on which it is operating (for method tests)
  • which constructor or method is tested
  • for this batch there may be a list of :

    • Keyword associated to test (or Keyword "seed" so that each test generates a unique keyword)
    • parameters (special notation for set combination)
    • code that reports a final result and/or execute code for testing values (assertions) and/or code to join some data to the test report (mostly strings). Special variable names should be determined for current instance and for code execution result.
  • optionally code snippets to be executed before and after each test or batch of test.

An exemple of code snippet could be to test for scalability: a code is executed in a loop an time measured : if time tend to grow exponentially there may be an algorithmic snag (in fact numerous bugs have been found that way!)

One important feature is that once a testing unit has run it could "export" generated objects for other testing units (that could import those).

Again these dependencies should be described for the build system.

Each time a test is executed the corresponding report is generated and stored (logging information is also produced on the fly).

1.6. Report handling

The reports are accumulated and sorted. After all tests completed they are passed to report handlers (according to the alphabetical order of the tests' key).

Those reports handlers are pluggable: that is one can write and register any code that is fit for reporting.

So , for instance, reports could be registered and compared to previous runs. If there had been results pending, verified after the previous run, then the new results could be considered verified.

Chapter 2. Getting started

2.1. Script execution

Tests are specified in a text file (example tztMyClass.gru). This code being edited with the tesxt editor of an IDE or generated from special comments in a java or groovy source file.

To execute this code you need:

  • library jars in your CLASSPATH:

    • Java
    • groovy
    • the GRU libraries
    • libraries of the codes you are testing and those refered to by these codes.
  • proxy to codes referenced by your script:

    • groovy classes that declare list of values used as standard in your tests (see later examples such as _java.lang.Double_s)
    • codes "exported" by previous test runs ( .gruh files: not implemented yet)

So modify the grush shell script provided to fix all these parameters

If your script file is named tztMyClass.gru just call : grush tztMyClass (without .gru suffix)

2.2. Introduction to simple test data and to simple groovy

Tests need data for constructors and method parameters. An important convention is that (almost) all data used in this context should be "tagged" That is there should be a name describing the role of this data.

All parameters handled by the software are of type TaggedObject. But you do not have to worry about that; here is a way to declare such variables in your gru script:

// note the String syntax in Groovy: simple quotes
_vars SIMPLE_STRING: 'hello'

println _this.SIMPLE_STRING

// yields-> SIMPLE_STRING: hello

Here:

  • the _vars function builds a TaggedObject with key SIMPLE_STRING and value hello
  • SIMPLE_STRING becomes also the name of a variable you can use in the script. At this place you’ll have to use a special context (binding in groovy parlance) named _this but it won’t be necessary inside a test description.

Now you can declare many tagged variables in a row:

_vars LONG_STRING: 'para_dimethyl_aminobenzene_azobenzene' , NEUTRAL_STRING: 'aName' ,
        EMPTY_STRING: '', NULL_STRING: null as String

Here some remarks about Groovy syntax:

  • the argument of _vars is a Map litteral (a comma separated sequence of key: value).
  • Groovy is a line oriented language (statement ends at line end, unless …. -here the comma indicates that the Map litteral is not finished-)
  • as is a cast operator.

And some gru conventions:

  • a null value for a test should be named NULLsomething : it indicates to the tool that this is not missing data but something you explicitly want!
  • an object named NEUTRALsomething has also a special status: it indicates to a test that this value is always ok and could be used whenever we want to get rid of too many possible values.

A group of TaggedObjects can be refered by a simple variable name :

def strings = _vars LONG_STRING: 'para_dimethyl_aminobenzene_azobenzene' , NEUTRAL_STRING: 'aName' ,
        EMPTY_STRING: '', NULL_STRING: null as String

println strings.toString()
// a groovy bug here: normally "println strings" would suffice

// -> [ EMPTY_STRING:  ,LONG_STRING: para_dimethyl_aminobenzene_azobenzene ,NEUTRAL_STRING: aName ,NULL_STRING: null ]

Though the strings variable is without an explicit type (def groovy feature) its actual type is TaggedsMap (note that the objects are sorted using their tag).

Those TaggedsMap could be manipulated with groovy style:

strings << [SHORT_STRING: 's', SPACE_STRING: ' a string']

Here a Groovy Map litteral is added to the existing Map.

In some (dire?) circumstances you can add an "untagged" object: it will be "autotagged" (the code will try to get a getName or a getKey or a getId method on the object or else use toString).

// an empty groovy Map
def untaggeds =  _vars()
untaggeds << 'accents=éà'

long time = System.currentTimeMillis()
// an evaluated GString
untaggeds << "date=$time"

// a Groovy List
untaggeds << ['menel' , 'tecel', 'pares']

You do not need to create sets of data each time you create a test script. A better practice is to define reusable sets in groovy code that could be shared.

So here is some groovy code to be reused by gru scripts.

package _java.lang

class Double_s {
    static def  positives = [
          NEUTRAL_DBL: 12.12 as double ,
          SMALL: 0.02 as double ,
          ZERO : 0 as double ,
          VERY_SMALL : 0.000000000037 as double ,
          BIG: 1273747576.46 as double ,
          VERY_BIG: 12345678973747576777879000.45 as double ,
          //do with small and big in scientific notation
          // with prime values
          // with imprecise double values
      ]
    //....

This groovy class is here to provide predefined sets of values for testing doubles. Note the name of the package (a bogus _java.lang name with underscore at the beggining).

In groovy a litteral such as 12.12 is not of type double (hence the cast).

Another (more complex) example:

package _java.lang
class String_s {
     static def  numeric = [
            positives: [
                    NEUTRAL_POS_STR: '12.12' ,
                    ZERO: '0' ,
                    //....
            ] ,
            negatives: [
                    //....
            ] ,
            //...
    ]
    static def plain = [
        //...
    ]
    static def i18n = [
        //...
    ]
    //...
}

And now its use from a gru script

def  posNumStr = _vars String_s.numeric.positives

(note the syntax: the name positives is used as if it were a member of numeric)

Another facility to create a TaggedsMap is to use its subclass FlatMap this way:

FlatMap String = [ String_s.plain, String_s.i18n, String_s.empties]

For those who do not know Groovy some details will be discovered later.

Now we are ready to specify tests.

Chapter 3. Testing constructors

For demo purposes let’s start with that version of a Java Class:

public class MovingThing  {
    public MovingThing(String name) {
       //...
    }

    public MovingThing( String name, double radius){
        //...
    }

3.1. Single argument calls

Now a gru test specification :

// imports at the beginning of script
import org.lsst.ccs.testers.testdata.MovingThing

// strings is a TaggedsMap defined before

_withClass MovingThing _group {
    _test GENERAL: strings
}
[Note]Note

This supposes that the class exist.

For a completely dynamic behaviour (test specification while the class has not yet been written) use:

_withClass 'org.lsst.ccs.testers.testdata.MovingThing' _group {
    _test GENERAL: strings
}

Running the test will produce as many test reports as there are members in the strings Map.

Let’s have a look at a simple string representation of one of these results:

##result
        testName: GENERAL [NEUTRAL_STRING]
        rawDiagnostic: SUCCESS
        advice:
        methodName: <ctor>
        className: org.lsst.ccs.testers.testdata.MovingThing

        data: MovingThing : name=aName;radius=0


    #messages-----------------------------------
         []
    #end----------------------------------------

    #assertions-----------------------------------
         []
    #end----------------------------------------

    #caughts-----------------------------------
         []
    #end----------------------------------------

##end

Here:

  • The testName tells about which test was performed with which parameters (using tags)
  • the result is supposed to be SUCCESS (but the field advice could be edited to change that!)
  • the method called is a constructor (<ctor>)
  • data is here the result of toString being called on the generated object (normally it IS the generated object)
  • no other information is available ….

Now inspection of results may let you discover an abnormal thing:

 ##result
        testName: GENERAL [NULL_STRING]
        rawDiagnostic: SUCCESS
        advice:
        methodName: <ctor>
        className: org.lsst.ccs.testers.testdata.MovingThing

        data: MovingThing : name=null;radius=0
        ...

This should not have been a proper result if we wanted to avoid null string as an argument!

So let’s try another version:

_withClass MovingThing _group {
    _test GENERAL: strings _xpect {
        if(_args[0] == null)  _okIfCaught RuntimeException
    }
}

The _xpect clause specify expectations: if the first argument of the call (_args[0]) is null a RuntimeException should have been fired !

(Note: to simplify the example we have adopted here a java-like syntax accepted by groovy. A more "grooviesque" syntax is possible -and shorter-)

Now we can spot that our program does not work properly! (The object had been generated without firing a RuntimeException!)

 ##result
        testName: GENERAL [NULL_STRING]
        rawDiagnostic: FAILED
        advice:
        methodName: <ctor>
        className: org.lsst.ccs.testers.testdata.MovingThing

        data: MovingThing : name=null;radius=0

    #messages-----------------------------------
         []
    #end----------------------------------------

    #assertions-----------------------------------
         [assert: not found class java.lang.RuntimeException -> FAILED]
    #end----------------------------------------

    #caughts-----------------------------------
         []
    #end----------------------------------------

##end

The constructor is modified to check for null values…

    public MovingThing(String name) {
        if(name == null) {
            // specs says NullPointerException
            // should be a specific exception  InvalidNullArgumentException
            throw new IllegalArgumentException("null argument");
        }
        this.setName(name);
    }
    // there is a getName() method!

Another gru specification (_it is the generated object):

def legalStrings = _vars LONG_STRING: 'para_dimethyl_aminobenzene_azobenzene' , NEUTRAL_STRING: 'aName' ,
        SHORT_STRING: 's', SPACE_STRING: ' a string'

def illegalStrings = _vars EMPTY_STRING: '', NULL_STRING: null as String

_withClass MovingThing _group {
    _test LEGAL: legalStrings _xpect { _message "Name = ${_it.getName()}" }
    _test ILLEGAL: illegalStrings _xpect {  _okIfCaught RuntimeException }
}

Results extracts:

a report with a message. 

        testName: LEGAL [SPACE_STRING]
        rawDiagnostic: SUCCESS

    #messages-----------------------------------
         [Name =  a string]
    #end----------------------------------------

yet another failed test. 

        testName: ILLEGAL [EMPTY_STRING]
        rawDiagnostic: FAILED

    #assertions-----------------------------------
         [assert: not found class java.lang.RuntimeException -> FAILED]
    #end----------------------------------------

test succeeded. 

        testName: ILLEGAL [NULL_STRING]
        rawDiagnostic: SUCCESS

    #assertions-----------------------------------
         [assert: java.lang.IllegalArgumentException: null argument -> SUCCESS]
    #end----------------------------------------

    #caughts-----------------------------------
         [java.lang.IllegalArgumentException: null argument]
    #end----------------------------------------

3.1.1. special arguments

Now what if you want to test a constructor with no arg?:

_withClass MovingThing _group {
    _test NULL_ARG: _NO_ARG
}

Do use the specific variable _NO_ARG!

now you could get a success (if the no arg contructor is present) or:

       testName: NULL_ARG
       rawDiagnostic: NOT_YET_IMPLEMENTED

(as a general policy a missing constructor or method yields the result NOT_YET_IMPLEMENTED : gru is optimistic!)

3.2. Multiple argument calls

_withClass MovingThing _group {
    _test LEGAL: legalStrings, LEGAL2: [legalStrings, positiveDoubles] _xpect {
        _message "Name = ${_it.getName()}"
    }
    _test ILLEGAL: NULL_STRING _xpect {  _okIfCaught RuntimeException }
}

Here the test specification:

  • declares two test batch (one named LEGAL and the other LEGAL2)
  • the second batch (LEGAL2) uses the two argument constructor and calls it with a combination of arguments from each list (legalStrings, positiveDoubles). Beware: the number of actual tests could grow quickly so may be you could choose a NEUTRAL argument for one of the parameters!
  • the ILLEGAL test uses only one object (not a list of objects).

An example of a corresponding report:

 ##result
        testName: LEGAL2 [SPACE_STRING, BIG]
        rawDiagnostic: SUCCESS
        advice:
        methodName: <ctor>
        className: org.lsst.ccs.testers.testdata.MovingThing
        data: MovingThing : name= a string;radius=1273747576

    #messages-----------------------------------
         [Name =  a string]
    #end----------------------------------------

3.3. Beans

Now if MovingThing is a bean (no_arg constructors + accessors and mutators) you can also do things like that :

_vars A_BEAN: [name: 'moving', radius: 3.14 as double]

_withClass MovingThing _group {
    _test BEANS: A_BEAN
}

Here a Map with names of bean properties was used (MovingThing has methods setName and setRadius).

(only drawback: you may end up with reports bearing the same name; this is going to be changed in future versions).

[Warning]Warning

for "one-liners" fans: this is possible (but not necessarily easier to read!)

_withClass MovingThing _group {
    _test BEANS: _vars (A_BEAN: [name: 'moving', radius: 3.14 as double])
}

3.4. Other features

A complex thing to consider is: when are the codes of our test specification executed?.

There are three phases during the execution of a gru script:

Test data collection

this is the code that is executed when gru "reads" the script. So this is "level 1" code such as:

def illegalStrings = _vars EMPTY_STRING: '', NULL_STRING: null as String

this code is in the scope of the script.

Code inside the _group block is also executed during this phase:

_withClass MovingThing _group {
    // possible code in this block scope
    _test ILLEGAL: illegalStrings _xpect {  _okIfCaught RuntimeException }
}

A function like _test does not execute code: it gathers data at this stage.

You can have code in the block: it is going to be immediately executed but will be in the block scope. Such a block of code is called a closure. There are special context rules for the code inside.

The closure that is _xpect argument is not executed now: it is just stored for future use.

Test execution
during this phase the closures that have been stored are executed. (this is not the case of the _group closure which is executed before).
Test reporting
the code for this phase is in ResultReporter implementations that can be hooked to the tester. Reporting can be fired explicitly or by default is started at the end of the script.

3.4.1. setup/teardown codes

Test execution may need to set up special conditions before the test is run and that this setup is dumped after the test (setup/teardown).

So for a group of tests that need specific conditions (not to be shared by other test groups):

_withClass MovingThing _group {
    // possible code executed at "build" time
    _pre {
        //setup code executed at "test run" time
        // before the tests
     }
    _test COMMONS: values
    _post {
        //teardown code executed at "test run" time
        // after the tests have been run
    }
}

But it could be as well:

_withClass MovingThing _group {
    // possible code executed at "build" time
    _pre {
        //setup code
     }
    _post {
        // teardown code
    }
    _test COMMONS: values
}

Since those setup/teardown code are in different closures sharing data could be through the _this global binding or through a specific _thisG group binding (remember: a binding is a context tool in groovy)

(since these functions just "gather" code to be executed later, the order of description is unimportant).

[Warning]Warning

setup/teardown is not yet implemented for groups!

There can be the same pre/post code for specific tests runs:

_withClass MovingThing _group {
    _test COMMONS: values _pre {/**/} _xpect {/**/} _post {/**/}
}

The order of function calls is also unimportant.

The closures can use _this, _thisG bindings plus a specific _thisL binding local to the test.

3.4.2. assertions

Code in the _xpect closure has proxy to the current object and to a report object where some fields can be modified. The result in the report is set by default by execution of the test (for instance SUCCESS if nothing special happens; FAILED , TOOL_OR_CODE_ERROR, MISSING_DATA, NOT_YET_IMPLEMENTED if something wrong happens,…). This can be modified by assertions and messages can also be added to the report.

The behaviour of gru's assertions is peculiar: the assertions results are stored, all assertions are evaluated (even if a previous one failed) and, in the end, the global result if the "worst" assertion result.

[Warning]Warning

There are exceptions to this rule: a global result could be forced.

Up to now the _okIfCaught assertion forces the result (but this specification needs to be re-evaluated with finer analysis).

List of functions you can use in _xpect context:

  • _message(String message) : not an assertion, it just adds a message in the message list of the report. Better use it with a Groovy GString (such as "name = ${it.getName()}")
  • Serializable _reportData(Serializable data): not an assertion, may be used to add some sata to the report (such as performance measure). The function returns its argument.
  • _neutral(String message, Closure code) : executes the code in the closure; if all is well adds an AssertionReport in the report with the message, the result of code execution and an overall result NEUTRAL; else reports an exception and result is TOOL_OR_CODE_ERROR.

    Another version of this function is _neutral(String stringExpression) : stringExpression being probably a GString that is evaluated as groovy code (DO NOT USE YET: does not work properly).

  • _failIf(String message, boolean booleanExpr) : if booleanExpression is true an AssertionReport is issued with the message and result FAILED, otherwise result is NEUTRAL.

    _failIf(String stringExpression) : (DO NOT USE YET: does not work properly).

    _failIfNot(String message, boolean booleanExpr) and _failIfNot(String stringExpression) are doing the same if condition is false.

  • _warnIf and _warnIfNot have the same signatures and result is WARNING instead of FAILED
  • _okIfCaught(Class<? extends Throwable> throwClass) : if a subclass of throwClass has been thrown during execution the overall result is forced to SUCCESS otherwise FAILED

3.4.3. variables

Some predefined variables are defined in the context of the test run:

  • global Binding _this, group Binding _thisG, local binding _thisL
  • the current object : _it (means the object generated by the constructor or the object supporting the method when the test is about instance methods).
  • when testing methods yielding a result it is called _result
  • _args is the array of call arguments
  • _argNames is the array of arguments tags
  • _className and _methodName may be used in some dynamic context.

To do : proxy current fired exception.

3.4.4. exporting data

During the gru script execution the test code is generated from the descriptions. Tests are run and reported only at the end of the script. The order of execution is: constructors, static factories, instance factories, instance methods, free code (notion not yet defined). In each category the order of test runs should be random (not yet implemented).

Sometimes it may be useful to run or to run and report earlier.

This can be done in different ways:

_withClass MovingThing _group {
    _test BEANS:  A_BEAN
} _run()

The group of tests will be run immediately

_withClass MovingThing _group {
    _test BEANS:  A_BEAN
} _runReport()

The group of tests will be run and reported immediately

Now for testing instance methods you may need objects created by a constructor test:

def neededObjects = _vars()

_withClass MovingThing _group {
    _test BEANS:  A_BEAN
} _fill neededObjects

Here the tests in the group will be run and those properly generated will go into the set neededObjects.

A further extension of this notion will be to export such objects for other ensuing gru scripts (this generation of .gruh files is not yet implemented).

3.5. Testing static factories

Here a variation of object creation testing

_withClass TaxTool _factory 'getInstance' _group {
        _test CREATION: locales
}

NOT YET IMPLEMENTED

3.6. Simplified syntax shortcuts

In many cases it is possible to write a simpler description of tests when there is no need for all the features .

A simplified group :

_withClass MovingThing {
    _test BEANS:  A_BEAN
}

A simplified expectations specification :

_withClass MovingThing  {
    _test COMMONS: values , {/* expectations */}
}

Chapter 4. Testing methods

4.1. instance methods

The _group block is the same but the header is different:

// neededObjects is a TaggedsMap
// TAGGED is a single Tagged Object

_withObjects  neededObjects, TAGGED _method 'toString' _group {
    _test TOSTRINGS: _NO_ARG
}

Here:

  • the arguments for _withObjects is a sequence of one or more TaggedObjects and/or TaggedsMaps.
  • the _method argument is the name of a method. (this argument should be a groovy string)
  • the instances on which this method is going to be called are not necessarily of the same type! (they just need to have the same method).
  • if the method supports overloading you can specify tests with different arguments combinations.

Here are some reports extracts:

    ##result
        testName: TOSTRINGS
        rawDiagnostic: SUCCESS
        advice:
        methodName: toString
        className: MovingThing
        supportingObject: LEGAL [NEUTRAL_STRING]
        data: MovingThing : name=aName;radius=0

And

 ##result
        testName: TOSTRINGS
        rawDiagnostic: SUCCESS
        advice:
        methodName: toString
        className: BigDecimal
        supportingObject: 345.56
        data: 345.56

4.2. instance factories

NOT IMPLEMENTED

4.3. static methods

NOT IMPLEMENTED

Chapter 5. Testing "in situ" objects through proxies

GRU can also test objects already deployed in an application by adding assertions and observation code to an existing object.

Right now the implementation is clumsy since we need to write subclasses that act as proxies on the current object. An A.O.P implementation could be more elegant but has not yet been carried out.

Most examples in this section are specific to the LSST/CCS project (juste browse or skip if you are not in this context).

5.1. Starting a Modular SubSystem through groovy

The current example is a script for starting Single Filter test.

Instead of being managed through Spring the code is started by a groovy script (BTW this helps for use of a debugger).

Instead of using a JMS server as transport this code uses the JGroups library (communication is through topic-like "groups" on top of multicast addresses and thus do not need a central server)

File name is sft.gru

statements for infrastructure. 

// Using Jgroups Messaging: no server needed!
// this deployment configuration should change and use ServiceLoader
System.setProperty("lsst.messaging.factory","org.lsst.ccs.bus.jgroups.JGroupsMessagingFactory" )

// trying to set up listening appender (not sure this is correct)
// see Log4J documentation
System.setProperty("log4J.appender.busAppender","org.lsst.ccs.bus.Log4JBridge" )
System.setProperty("log4j.rootlogger","INFO, busAppender" )


// A SubSystem independent from Spring
// this class was created for this very purpose
// note the groovy syntax way of building the object
BasicModularSubSystem system = ['single-filter-test']

creating beans and modules. 

Filter filter = [name: 'dummyFilter']

SftMainModule mainsft = [name: 'mainsft', dummyFilter: filter]

SimuLatch simuCarouselLatch = [ name: 'simuCarouselLatch', locked: false]

SimuCarouselMotor simuCarouselMotor = [ name: 'simuCarouselMotor', tickMillis: 17 ,
    serialNumber: 'CCS-FCS-simu-CarouselMotor-20100718-1', nominalVelocity: 21.0, maximalVelocity: 21.8 ]

SimuNumericSensor sensorFXplus0 = [ value: 0]

SimuFilterClampModule clampXplus0 = [name: 'clampXplus0', tickMillis: 1000 ,
    filterPresenceSensor: sensorFXplus0  ]

SimuNumericSensor sensorFXminus0 = [ value: 0]

SimuFilterClampModule clampXminus0 = [name: 'clampXminus0', tickMillis: 1000 ,
        filterPresenceSensor: sensorFXminus0  ]

SimuCarouselSocket socket0 =  [  position: 0.0 , standbyPosition: 0.0 ,
        clampXminus: clampXminus0 , clampXplus: clampXplus0  ]

SimuActuatorModule  clampsActuator = [ name: 'clampsActuator', tickMillis: 1000]

SimuAutoChangerMotor simuAutoChangerMotor = [name: 'simuAutoChangerMotor',
    tickMillis: 15, serialNumber:'CCS-FCS-simu-AutoChangerMotor-20110110-1' ,
        nominalVelocity:10.0 , maximalVelocity: 20.0 , position: 30.0 ]

SimuLatch simuStandbyLatch = [ locked: false]

SftAutoChangerModule autochanger = [ name: 'autochanger' , tickMillis: 1000 ,
        trucksPositionOnline: 50 , trucksPositionAtStandby: 0 , trucksPositionSwapout: 30 ,
        motor: simuAutoChangerMotor, standbyLatch: simuStandbyLatch ]

SftCarouselModule carousel = [
        name: 'carousel', tickMillis: 1000 ,
        carouselMotor: simuCarouselMotor , clampsActuator: clampsActuator ,
        latch: simuCarouselLatch , nbSockets: 1 ,
        sockets: [socket0]
]

The statements are all the same (initializing bean data with properties) and though we could have cited an excerpt of this file we can also notice it is a long description process. We could probably generate it automatically from the existing xml description file (if needed!).

listening structure between modules. 

autochanger.listens simuAutoChangerMotor, simuStandbyLatch
clampXplus0.listens autochanger
clampXminus0.listens autochanger
carousel.listens simuCarouselMotor , simuCarouselLatch , clampXminus0 , clampXplus0

setting up the subsystem. 

// Adding modules to subsystem
// THIS CAN BE AUTOMATISED (by a "module" declaration function)

system.addModules  carousel, simuCarouselLatch ,simuAutoChangerMotor, autochanger ,simuCarouselMotor , clampXminus0,  clampXplus0 ,clampXminus0 ,clampsActuator, mainsft

system.start()

creating a BusMaster to issue commands. 

// a specific BusMaster written for this purpose
BusAccess cmd = new BusAccess()
// default: does not show log
// does not show status
cmd.handleStatus = false

issue commands. 

cmd.invoke 'single-filter-test/carousel' , 'getPosition'

/// .... other commands

Thread.sleep 50000
cmd.shutdown()
system.shutdown()

We can create stress tests simply:

def val = 45 as double

println '-------------------------------------------------------------'
for (int ix = 0 ; ix < 5 ;ix++) {
    cmd.invoke 'single-filter-test/carousel' , 'rotate', val
}
println '-------------------------------------------------------------'

We can now modify the type of one of the modules in the architecture to trace and test its behaviour.

5.2. Proxy class and instances for testing

Here is a subclass created for the purpose of "decorating" calls to methods of its superclass.

class SftCarouselMonitor extends SftCarouselModule {
    Gruse grus = new Gruse(this)

    public String rotate(double angle) {
        grus._monitor rotate: {super.rotate(angle)} _xpect true , {
            _message("should fail: not implemented")
            _okIfCaught(RuntimeException)
        }
    }
}

A instance of SftCarouselMonitor will replace an instance of SftCarouselModule in our deployment script.

There:

  • each instance of this proxy should create a Gruse object that will support monitoring code.
  • when overriding a method we want to monitor the last statement in the code should be a call to the _monitor method of the Gruse field. This _monitor + _xpect combination will deal with returning the normal data returned by the super method (or propagate the exception thrown by the super method).
  • (We thought of creating a macro for that but for the time being we choose to let explicit code be written -though it is always the same-).
  • the _xpect clause is particular :

    • It takes an additional boolean argument: if true a report will be generated each time the method is called, otherwise a report will be generated only if additional data has been requested or if the result is different from NEUTRAL or SUCCESS. (so when the method is called often then only "special" reports are issued: this helps alleviate the report handling)
    • It does not have proxy to usual variables: _it, _args, _argNames, _thisG, _thisL, otherwise the usual functions used in the context of _xpect can be used.
  • a _post {/*closure*/} code can be specified as a teardown code but should be specified before the _xpect clause (NOT implemented yet!)

Now in the gru script we can change the type of an object:

SftCarouselMonitor carousel = [
        name: 'carousel', tickMillis: 1000 ,
        carouselMotor: simuCarouselMotor , clampsActuator: clampsActuator ,
        latch: simuCarouselLatch , nbSockets: 1 ,
        sockets: [socket0]
]

Chapter 6. Handling results

Chapter to be completed (see "what’s next" chapter).

What to expect from result handling?

How to create, handle, deploy ResultReporters and ResultFormatters ?

Right now the SimpleResultFormatter just creates a String version of the result and it is printed to standard output.

Chapter 7. General language layout

This is not a complete grammar description but could help.

7.1. Tests descriptions

testSpecification:

testConstructor

testStaticFactory

testStaticMethod

testInstanceFactory

testInstanceMethod

testConstructor:

_withClass classLitteral groupSpecification

_withClass className groupSpecification

testStaticFactory:

_withClass classLitteral _factory factoryMethodName groupSpecification

_withClass className _factory factoryMethodName groupSpecification

static factories are not yet implemented.

testStaticMethod:

_withClass classLitteral _method methodName groupSpecification

_withClass className _method methodName groupSpecification

static methods are not yet implemented.

testInstanceFactory:
_withObjects listTaggeds _factory factoryMethodName groupSpecification

instance factories are not yet implemented.

testInstanceMethod:
_withObjects listTaggeds _method methodName groupSpecification
groupSpecification:

_group nameOfGroupopt groupClosure runFunctionopt

groupClosure runFunctionopt

NameOfgroup (optionnal) could be used to name a group: this will be useful to export tests results (the list of data generated will bear the name of the group). If no name is specified one will be automatically generated.

runFunction: one of

_run

_runReport

_fill taggedsMapRef

_xport

_xport not implemented yet! be cautious if using _fill and _xport with instance mehods.

groupClosure:
{ listOfGroupClosureElements }
groupClosureElement:

groovyCode

_pre closure

_post closure

testSpecification

_pre and _post should be called once at most.

testSpecification:

_test listOfTestArgs setupCodeopt expectationsCodeopt teardownCodeopt

_test listOfTestArgs expectationsClosure

testArg:
NameOfTest: arg
arg:

_NO_ARG

taggedObjet

taggedsMap

[ listOfTaggeds ]

A listOfTaggeds (list of TaggedObject and/or TaggedsMap) is to be used when a call needs many parameters.

setupCode :
_pre closure
teardownCode :
_post closure
expectationsCode :
_xpect expectationsClosure

The expectationsClosure uses specific functions such as gru assertions

7.2. Monitoring

Code in a method "decoration" (should be the last statement of the overriding method)

methodMonitoring:
gruseInstance._monitor methodName: { superMethodCall } teardownCodeopt _xpect expectationsClosure

Chapter 8. What’s next ?

GRU is under development.

In fact this first version needs a complete rehaul since it is a prototype.

We need your advice :

  • does the application provide the services you need?
  • is the DSL clear enough or obscure?
  • how can we update the user manual?
  • what about bugs and/or strange or missing features?

Our plans:

  • Complete missing features (factories, static methods, A.O.P ?)
  • Write functions to deliver more sophisticated services such as performance monitoring.
  • Write a Result Editor. Tests results could be stored somewhere (database? file?) then edited for comments (the programmer checked a result after the test was run, the bug is cornered for future code modification, the bug is not going to be corrected -the corresponding values are not going to be provided in a real situation-,…).

    The "commented" results are going to be stored. Then after next test run the new results are going to be compared with the commented one. An automatic analysis will provide clues about addition, regressions, and will yield a synthetic result.

    This is a complex matter that should be experimented.

  • Write a filter for reading tests specifications directly in the Java code (javadoc-like feature).
  • Deal with dependencies: a test run could provide new objects for other ensuing tests.
  • Integration with IDE?
  • If proven useful and efficient publish this project outside LSST as open source ? (lsst.org being "owner" of the application)

For the moment the tests are run on a single thread (and gru does not stand against multi-threaded tests). This is going to be a more complex extension that needs more thorough specifications (what do we want to achieve? how?).