Author (other contributing authors needed!)
Paul Bernard AMADE Laboratoire APC 10, rue Alice Domon et Leonie Duquet 75205 Paris Cedex 13 bamade@in2p3.fr (33) 1 57 27 69 14
This is a presentation of the GRU testing framework developped specifically for the LSST project. The tool is rather general and probably could be presented as a separate open source project.
GRU is a D.S.L (Domain Specific Language) based on Groovy.
Its main purpose is to design, run and report various tests on Java code.
Users should have a minimum knowledge of groovy
since the gru scripts
use many groovy
features
(but a very simple usage of groovy
will be explained in this document).
The document is incomplete since the framework is still under development (and waiting for advices from users).
An introductory chapter will try to answer this question: "why create a specific framework instead of using
one out of the shelf?". To cut a long story short: the goals are different from those of JUnit
.
Table of Contents
Not all expectations for a testing framework can be met through the use of a single tool. But stating overall goals is important to understand some features of this particular tool. (Note: this chapter is rather abstract so you may skip it and read it later!).
Testing comes in many flavours : we are investigating "carpet bombing". It means that we keep predefined lists of values and that those "check lists" are used to spread values over parameters of constructors and methods.
Unless otherwise specified those values are not picked up at random. We want to keep lists of values that may have specific properties. Setting up and maintaining such lists requires experience and intuition. For instance try to set up a list of String objects that may have specific properties in a general context : you will be suprised by the sheer number you will find! Then you can add more Strings for a specific context.
Programmers are lazy! Simple tests should be written quickly and easily all the time: during detailed design, during implementation, and any time afterwards. If simple tests specifications are not simple enough we will get "lip service" testing (that happens all the time in projects!).
Often java constructors and method calls should be tested against a set of values.
If we have a single parameter for a call it may be important to test against a whole range of values for this parameter.
What could those "sets" be? Some examples:
Since we cannot have the same expectations for the whole set of possible values for a type we should group those into subsets.
We need a naming strategy: names for all kinds of sets and, in fact, a symbolic name for each value we use. Each test "instance" should also be labeled : if the test produces data then this data is associated with this key.
So we are going to use named values that may be specified as generated by a test or simply as being assigned a value of some type (e.g using a litteral assignment).
Note that we could also apply a method on a whole set of instances.
The combinations of values tested that way could grow exponentially! (for simple unit test on a simple class you may end up with hundreds of tests -if not thousands!-).
Though we should not be concerned by performances it is not necessary true that more tests yield better tests! Moreover the analysis of results could be obfuscated : for instance the very same bug will produce many negative results that have all the same cause.
Assigning expectations to a whole set of values may not be trivial: subsets should be cleverly designed.
So caution and balanced choices should prevail : one one hand it is important to test again varied ranges of values in a check list; on the other hand over-testing may not be a viable option.
There should be different places for test specifications : detailed design documents, source code itself, external files (for codes we did’nt write), maintenance documents ,…. Those specifications should be extracted and then the tests executed.
This means that build software should be aware of the corresponding dependencies and that different "test documents" should be linked together.
Tests could be run when code is incomplete : some code is still waiting to be written but test spec is already here. This lead to an interpretative nature of test execution: if the corresponding java code is not yet here a special report is issued.
Since the programmer may need to write complex code to design a complete test there may be a clash between the need for java code and the need of an interpretative test specification.
To alleviate the programmer learning curve and have the best compromise between simplification
and flexibility the choice of a script language (such as groovy
) looks promising (note that groovy
is not exactly an interpretative language -though it has many dynamic features-)
Let’s call a "testing unit" (not to be confused with "unit test") a set of related documents that describe a set of tests (for a given java class for instance).
Some units have a special status: they just declare set of values to be used by other tests. Otherwise execution of a testing unit should go through successives phases : optionnal pre-test code execution , test collecting, execution, optionnal post-test code, reporting .
Since other codes may be linked to code execution the syntax of a test language needs to link precisely those codes to execution phases.
A "testing unit" will contain various codes (and imports) and many "test specification".
Due to the set description for instances and arguments used, each test specification could result in many effective tests.
The result of each actual test could be complex data. Here is the present enumeration of results used in GRU:
/** * A simple value that summarizes the result of an execution. * It is "raw" because the analysis of test results may produce more sophisticated * reports (for instance a FAILED test may be annotated with BEYOND_LIMIT to indicate * that this is not an error : it is a feature!) */ public enum RawResult { /** * the tests specification may be erroneous (or the testing tool itself failed) */ TOOL_OR_CODE_ERROR, /** * the test failed */ FAILED, /** * the test failed because some needed data could not be evaluated (usually because * a previous test failed and did not produced this data). Usually the data is null * without being tagged with a NULL* name. */ MISSING_DATA, /** * not used: this is a programmatic mark, values superior to this one are considered * to mark a test that did not fail. */ OK_MARK, /** * the requested class or method is not yet implemented */ NOT_YET_IMPLEMENTED, /** * the test was not evaluated */ NOT_EVALUATED, /** * not used: this is a programmatic mark: values superior to this are * considered executed and not failed */ OK_DONE_MARK , /** * the test succeeded but with warnings */ WARNINGS, /** * no advice on fail or succeed, just a trace. */ NEUTRAL, /** * success: expectations met */ SUCCESS ; }
The test "expectations" description may contain :
So a test description may contain:
for this batch there may be a list of :
An exemple of code snippet could be to test for scalability: a code is executed in a loop an time measured : if time tend to grow exponentially there may be an algorithmic snag (in fact numerous bugs have been found that way!)
One important feature is that once a testing unit has run it could "export" generated objects for other testing units (that could import those).
Again these dependencies should be described for the build system.
Each time a test is executed the corresponding report is generated and stored (logging information is also produced on the fly).
The reports are accumulated and sorted. After all tests completed they are passed to report handlers (according to the alphabetical order of the tests' key).
Those reports handlers are pluggable: that is one can write and register any code that is fit for reporting.
So , for instance, reports could be registered and compared to previous runs. If there had been results pending, verified after the previous run, then the new results could be considered verified.
Tests are specified in a text file (example tztMyClass.gru
).
This code being edited with the tesxt editor of an IDE or generated from special comments
in a java
or groovy
source file.
To execute this code you need:
library jars in your CLASSPATH:
proxy to codes referenced by your script:
_java.lang.Double_s
)
.gruh
files: not implemented yet)
So modify the grush
shell script provided to fix all these parameters
If your script file is named tztMyClass.gru
just call : grush tztMyClass
(without .gru
suffix)
Tests need data for constructors and method parameters. An important convention is that (almost) all data used in this context should be "tagged" That is there should be a name describing the role of this data.
All parameters handled by the software are of type TaggedObject
.
But you do not have to worry about that; here is a way to declare such variables in your gru script
:
// note the String syntax in Groovy: simple quotes _vars SIMPLE_STRING: 'hello' println _this.SIMPLE_STRING // yields-> SIMPLE_STRING: hello
Here:
_vars
function builds a TaggedObject
with key SIMPLE_STRING
and value hello
SIMPLE_STRING
becomes also the name of a variable you can use in the script. At this place
you’ll have to use a special context (binding
in groovy parlance) named _this
but
it won’t be necessary inside a test description.
Now you can declare many tagged variables in a row:
_vars LONG_STRING: 'para_dimethyl_aminobenzene_azobenzene' , NEUTRAL_STRING: 'aName' , EMPTY_STRING: '', NULL_STRING: null as String
Here some remarks about Groovy syntax:
_vars
is a Map
litteral (a comma separated sequence of key: value
).
Map
litteral is not finished-)
as
is a cast operator.
And some gru
conventions:
null
value for a test should be named NULL
something : it indicates to the tool
that this is not missing data but something you explicitly want!
NEUTRAL
something has also a special status: it indicates to a test
that this value is always ok and could be used whenever we want to get rid of too many possible values.
A group of TaggedObject
s can be refered by a simple variable name :
def strings = _vars LONG_STRING: 'para_dimethyl_aminobenzene_azobenzene' , NEUTRAL_STRING: 'aName' , EMPTY_STRING: '', NULL_STRING: null as String println strings.toString() // a groovy bug here: normally "println strings" would suffice // -> [ EMPTY_STRING: ,LONG_STRING: para_dimethyl_aminobenzene_azobenzene ,NEUTRAL_STRING: aName ,NULL_STRING: null ]
Though the strings
variable is without an explicit type (def
groovy feature)
its actual type is TaggedsMap
(note that the objects are sorted using their tag).
Those TaggedsMap
could be manipulated with groovy style:
strings << [SHORT_STRING: 's', SPACE_STRING: ' a string']
Here a Groovy Map litteral is added to the existing Map.
In some (dire?) circumstances you can add an "untagged" object: it will be "autotagged"
(the code will try to get a getName
or a getKey
or a getId
method on the object or else use toString
).
// an empty groovy Map def untaggeds = _vars() untaggeds << 'accents=éà' long time = System.currentTimeMillis() // an evaluated GString untaggeds << "date=$time" // a Groovy List untaggeds << ['menel' , 'tecel', 'pares']
You do not need to create sets of data each time you create a test script. A better practice is to define reusable sets in groovy code that could be shared.
So here is some groovy code to be reused by gru scripts
.
package _java.lang class Double_s { static def positives = [ NEUTRAL_DBL: 12.12 as double , SMALL: 0.02 as double , ZERO : 0 as double , VERY_SMALL : 0.000000000037 as double , BIG: 1273747576.46 as double , VERY_BIG: 12345678973747576777879000.45 as double , //do with small and big in scientific notation // with prime values // with imprecise double values ] //....
This groovy class is here to provide predefined sets of values for testing doubles
.
Note the name of the package (a bogus _java.lang
name with underscore at the beggining).
In groovy a litteral such as 12.12
is not of type double (hence the cast).
Another (more complex) example:
package _java.lang class String_s { static def numeric = [ positives: [ NEUTRAL_POS_STR: '12.12' , ZERO: '0' , //.... ] , negatives: [ //.... ] , //... ] static def plain = [ //... ] static def i18n = [ //... ] //... }
And now its use from a gru script
def posNumStr = _vars String_s.numeric.positives
(note the syntax: the name positives
is used as if it were a member of numeric
)
Another facility to create a TaggedsMap
is to use its subclass FlatMap
this way:
FlatMap String = [ String_s.plain, String_s.i18n, String_s.empties]
For those who do not know Groovy some details will be discovered later.
Now we are ready to specify tests.
For demo purposes let’s start with that version of a Java Class:
public class MovingThing { public MovingThing(String name) { //... } public MovingThing( String name, double radius){ //... }
Now a gru
test specification :
// imports at the beginning of script import org.lsst.ccs.testers.testdata.MovingThing // strings is a TaggedsMap defined before _withClass MovingThing _group { _test GENERAL: strings }
![]() | Note |
---|---|
This supposes that the class exist. For a completely dynamic behaviour (test specification while the class has not yet been written) use: _withClass 'org.lsst.ccs.testers.testdata.MovingThing' _group { _test GENERAL: strings } |
Running the test will produce as many test reports as there are members
in the strings
Map.
Let’s have a look at a simple string representation of one of these results:
##result testName: GENERAL [NEUTRAL_STRING] rawDiagnostic: SUCCESS advice: methodName: <ctor> className: org.lsst.ccs.testers.testdata.MovingThing data: MovingThing : name=aName;radius=0 #messages----------------------------------- [] #end---------------------------------------- #assertions----------------------------------- [] #end---------------------------------------- #caughts----------------------------------- [] #end---------------------------------------- ##end
Here:
testName
tells about which test was performed with which parameters (using tags)
SUCCESS
(but the field advice
could be edited to change
that!)
<ctor>
)
data
is here the result of toString
being called on the generated object (normally it IS the generated object)
Now inspection of results may let you discover an abnormal thing:
##result testName: GENERAL [NULL_STRING] rawDiagnostic: SUCCESS advice: methodName: <ctor> className: org.lsst.ccs.testers.testdata.MovingThing data: MovingThing : name=null;radius=0 ...
This should not have been a proper result if we wanted to avoid null string as an argument!
So let’s try another version:
_withClass MovingThing _group { _test GENERAL: strings _xpect { if(_args[0] == null) _okIfCaught RuntimeException } }
The _xpect
clause specify expectations: if the first argument of the call (_args[0]
) is null
a RuntimeException
should have been fired !
(Note: to simplify the example we have adopted here a java
-like syntax accepted by groovy.
A more "grooviesque" syntax is possible -and shorter-)
Now we can spot that our program does not work properly!
(The object had been generated without firing a RuntimeException
!)
##result testName: GENERAL [NULL_STRING] rawDiagnostic: FAILED advice: methodName: <ctor> className: org.lsst.ccs.testers.testdata.MovingThing data: MovingThing : name=null;radius=0 #messages----------------------------------- [] #end---------------------------------------- #assertions----------------------------------- [assert: not found class java.lang.RuntimeException -> FAILED] #end---------------------------------------- #caughts----------------------------------- [] #end---------------------------------------- ##end
The constructor is modified to check for null values…
public MovingThing(String name) { if(name == null) { // specs says NullPointerException // should be a specific exception InvalidNullArgumentException throw new IllegalArgumentException("null argument"); } this.setName(name); } // there is a getName() method!
Another gru
specification (_it
is the generated object):
def legalStrings = _vars LONG_STRING: 'para_dimethyl_aminobenzene_azobenzene' , NEUTRAL_STRING: 'aName' , SHORT_STRING: 's', SPACE_STRING: ' a string' def illegalStrings = _vars EMPTY_STRING: '', NULL_STRING: null as String _withClass MovingThing _group { _test LEGAL: legalStrings _xpect { _message "Name = ${_it.getName()}" } _test ILLEGAL: illegalStrings _xpect { _okIfCaught RuntimeException } }
Results extracts:
a report with a message.
testName: LEGAL [SPACE_STRING] rawDiagnostic: SUCCESS #messages----------------------------------- [Name = a string] #end----------------------------------------
yet another failed test.
testName: ILLEGAL [EMPTY_STRING] rawDiagnostic: FAILED #assertions----------------------------------- [assert: not found class java.lang.RuntimeException -> FAILED] #end----------------------------------------
test succeeded.
testName: ILLEGAL [NULL_STRING] rawDiagnostic: SUCCESS #assertions----------------------------------- [assert: java.lang.IllegalArgumentException: null argument -> SUCCESS] #end---------------------------------------- #caughts----------------------------------- [java.lang.IllegalArgumentException: null argument] #end----------------------------------------
Now what if you want to test a constructor with no arg?:
_withClass MovingThing _group { _test NULL_ARG: _NO_ARG }
Do use the specific variable _NO_ARG
!
now you could get a success (if the no arg contructor is present) or:
testName: NULL_ARG rawDiagnostic: NOT_YET_IMPLEMENTED
(as a general policy a missing constructor or method yields
the result NOT_YET_IMPLEMENTED
: gru
is optimistic!)
_withClass MovingThing _group { _test LEGAL: legalStrings, LEGAL2: [legalStrings, positiveDoubles] _xpect { _message "Name = ${_it.getName()}" } _test ILLEGAL: NULL_STRING _xpect { _okIfCaught RuntimeException } }
Here the test specification:
LEGAL
and the other LEGAL2
)
LEGAL2
) uses the two argument constructor and calls it
with a combination of arguments from each list (legalStrings
, positiveDoubles
). Beware: the number
of actual tests
could grow quickly so may be you could choose a NEUTRAL
argument for one of the parameters!
ILLEGAL
test uses only one object (not a list of objects).
An example of a corresponding report:
##result testName: LEGAL2 [SPACE_STRING, BIG] rawDiagnostic: SUCCESS advice: methodName: <ctor> className: org.lsst.ccs.testers.testdata.MovingThing data: MovingThing : name= a string;radius=1273747576 #messages----------------------------------- [Name = a string] #end----------------------------------------
Now if MovingThing
is a bean (no_arg constructors + accessors and mutators)
you can also do things like that :
_vars A_BEAN: [name: 'moving', radius: 3.14 as double] _withClass MovingThing _group { _test BEANS: A_BEAN }
Here a Map
with names of bean properties was used
(MovingThing
has methods setName
and setRadius
).
(only drawback: you may end up with reports bearing the same name; this is going to be changed in future versions).
![]() | Warning |
---|---|
for "one-liners" fans: this is possible (but not necessarily easier to read!) _withClass MovingThing _group { _test BEANS: _vars (A_BEAN: [name: 'moving', radius: 3.14 as double]) } |
A complex thing to consider is: when are the codes of our test specification executed?.
There are three phases during the execution of a gru
script:
this is the code that is executed when gru
"reads" the script.
So this is "level 1" code such as:
def illegalStrings = _vars EMPTY_STRING: '', NULL_STRING: null as String
this code is in the scope of the script.
Code inside the _group
block is also executed during this phase:
_withClass MovingThing _group { // possible code in this block scope _test ILLEGAL: illegalStrings _xpect { _okIfCaught RuntimeException } }
A function like _test
does not execute code: it gathers data at this stage.
You can have code in the block: it is going to be immediately executed
but will be in the block scope.
Such a block of code is called a closure
. There are special context rules
for the code inside.
The closure
that is _xpect
argument is not executed now: it is just stored
for future use.
closure
s
that have been stored are executed. (this is not the case of the _group
closure which is executed before).
ResultReporter
implementations that can be hooked to the tester.
Reporting can be fired explicitly or by default is started at the end of the script.
Test execution may need to set up special conditions before the test is run
and that this setup is dumped after the test (setup/teardown
).
So for a group of tests that need specific conditions (not to be shared by other test groups):
_withClass MovingThing _group { // possible code executed at "build" time _pre { //setup code executed at "test run" time // before the tests } _test COMMONS: values _post { //teardown code executed at "test run" time // after the tests have been run } }
But it could be as well:
_withClass MovingThing _group { // possible code executed at "build" time _pre { //setup code } _post { // teardown code } _test COMMONS: values }
Since those setup/teardown code are in different closures
sharing data could be through the _this
global binding or
through a specific _thisG
group binding (remember: a binding
is a context tool in groovy)
(since these functions just "gather" code to be executed later, the order of description is unimportant).
![]() | Warning |
---|---|
|
There can be the same pre/post code for specific tests runs:
_withClass MovingThing _group { _test COMMONS: values _pre {/**/} _xpect {/**/} _post {/**/} }
The order of function calls is also unimportant.
The closure
s can use _this
, _thisG
bindings plus a specific
_thisL
binding local to the test.
Code in the _xpect
closure has proxy to
the current object and to a report object where some fields can be modified.
The result in the report is set by default by execution of the test
(for instance SUCCESS
if nothing special happens; FAILED
, TOOL_OR_CODE_ERROR
, MISSING_DATA
,
NOT_YET_IMPLEMENTED
if something wrong happens,…).
This can be modified by assertions and messages can also be added to the report.
The behaviour of gru
's assertions is peculiar: the assertions results are stored,
all assertions are evaluated (even if a previous one failed) and, in the end,
the global result if the "worst" assertion result.
![]() | Warning |
---|---|
There are exceptions to this rule: a global result could be forced. Up to now the |
List of functions you can use in _xpect
context:
_message(String message)
: not an assertion, it just adds a message in the message list of the report.
Better use it with a Groovy GString
(such as "name =
${it.getName()}")
Serializable _reportData(Serializable data)
: not an assertion, may be used to add some sata to the report
(such as performance measure). The function returns its argument.
_neutral(String message, Closure code)
: executes the code in the closure; if all is well
adds an AssertionReport
in the report with the message, the result of code execution and an overall result
NEUTRAL
; else reports an exception and result is TOOL_OR_CODE_ERROR
.
Another version of this function is _neutral(String stringExpression)
: stringExpression being probably
a GString
that is evaluated as groovy code (DO NOT USE YET: does not work properly).
_failIf(String message, boolean booleanExpr)
: if booleanExpression
is true an AssertionReport
is issued with the message and result FAILED
, otherwise result is NEUTRAL
.
_failIf(String stringExpression)
: (DO NOT USE YET: does not work properly).
_failIfNot(String message, boolean booleanExpr)
and _failIfNot(String stringExpression)
are doing the same if condition is false.
_warnIf
and _warnIfNot
have the same signatures and result is WARNING
instead of FAILED
_okIfCaught(Class<? extends Throwable> throwClass)
: if a subclass of throwClass
has been thrown during execution the overall result is forced to SUCCESS
otherwise FAILED
Some predefined variables are defined in the context of the test run:
_this
, group Binding _thisG
, local binding _thisL
_it
(means the object generated by the constructor or
the object supporting the method when the test is about instance methods).
_result
_args
is the array of call arguments
_argNames
is the array of arguments tags
_className
and _methodName
may be used in some dynamic context.
To do : proxy current fired exception.
During the gru script
execution the test code is generated from the descriptions.
Tests are run and reported only at the end of the script.
The order of execution is: constructors, static factories, instance factories, instance methods, free code
(notion not yet defined).
In each category the order of test runs should be random (not yet implemented).
Sometimes it may be useful to run or to run and report earlier.
This can be done in different ways:
_withClass MovingThing _group { _test BEANS: A_BEAN } _run()
The group of tests will be run immediately
_withClass MovingThing _group { _test BEANS: A_BEAN } _runReport()
The group of tests will be run and reported immediately
Now for testing instance methods you may need objects created by a constructor test:
def neededObjects = _vars() _withClass MovingThing _group { _test BEANS: A_BEAN } _fill neededObjects
Here the tests in the group will be run and those properly generated
will go into the set neededObjects
.
A further extension of this notion will be to export
such objects for other ensuing gru scripts
(this generation of .gruh
files is not yet implemented).
Here a variation of object creation testing
_withClass TaxTool _factory 'getInstance' _group { _test CREATION: locales }
NOT YET IMPLEMENTED
In many cases it is possible to write a simpler description of tests when there is no need for all the features .
A simplified group :
_withClass MovingThing { _test BEANS: A_BEAN }
A simplified expectations specification :
_withClass MovingThing { _test COMMONS: values , {/* expectations */} }
The _group
block is the same but the header is different:
// neededObjects is a TaggedsMap // TAGGED is a single Tagged Object _withObjects neededObjects, TAGGED _method 'toString' _group { _test TOSTRINGS: _NO_ARG }
Here:
_withObjects
is a sequence of one or more
TaggedObject
s and/or TaggedsMap
s.
_method
argument is the name of a method. (this argument should be a groovy string)
Here are some reports extracts:
##result testName: TOSTRINGS rawDiagnostic: SUCCESS advice: methodName: toString className: MovingThing supportingObject: LEGAL [NEUTRAL_STRING] data: MovingThing : name=aName;radius=0
And
##result testName: TOSTRINGS rawDiagnostic: SUCCESS advice: methodName: toString className: BigDecimal supportingObject: 345.56 data: 345.56
GRU
can also test objects already deployed in an application
by adding assertions and observation code to an existing object.
Right now the implementation is clumsy since we need to write subclasses that act as proxies on the current object. An A.O.P implementation could be more elegant but has not yet been carried out.
Most examples in this section are specific to the LSST/CCS project (juste browse or skip if you are not in this context).
The current example is a script for starting Single Filter test.
Instead of being managed through Spring
the code is
started by a groovy script
(BTW this helps for use of a debugger).
Instead of using a JMS server as transport this code uses the JGroups
library (communication is through topic-like "groups" on top of multicast addresses and thus
do not need a central server)
File name is sft.gru
statements for infrastructure.
// Using Jgroups Messaging: no server needed! // this deployment configuration should change and use ServiceLoader System.setProperty("lsst.messaging.factory","org.lsst.ccs.bus.jgroups.JGroupsMessagingFactory" ) // trying to set up listening appender (not sure this is correct) // see Log4J documentation System.setProperty("log4J.appender.busAppender","org.lsst.ccs.bus.Log4JBridge" ) System.setProperty("log4j.rootlogger","INFO, busAppender" ) // A SubSystem independent from Spring // this class was created for this very purpose // note the groovy syntax way of building the object BasicModularSubSystem system = ['single-filter-test']
creating beans and modules.
Filter filter = [name: 'dummyFilter'] SftMainModule mainsft = [name: 'mainsft', dummyFilter: filter] SimuLatch simuCarouselLatch = [ name: 'simuCarouselLatch', locked: false] SimuCarouselMotor simuCarouselMotor = [ name: 'simuCarouselMotor', tickMillis: 17 , serialNumber: 'CCS-FCS-simu-CarouselMotor-20100718-1', nominalVelocity: 21.0, maximalVelocity: 21.8 ] SimuNumericSensor sensorFXplus0 = [ value: 0] SimuFilterClampModule clampXplus0 = [name: 'clampXplus0', tickMillis: 1000 , filterPresenceSensor: sensorFXplus0 ] SimuNumericSensor sensorFXminus0 = [ value: 0] SimuFilterClampModule clampXminus0 = [name: 'clampXminus0', tickMillis: 1000 , filterPresenceSensor: sensorFXminus0 ] SimuCarouselSocket socket0 = [ position: 0.0 , standbyPosition: 0.0 , clampXminus: clampXminus0 , clampXplus: clampXplus0 ] SimuActuatorModule clampsActuator = [ name: 'clampsActuator', tickMillis: 1000] SimuAutoChangerMotor simuAutoChangerMotor = [name: 'simuAutoChangerMotor', tickMillis: 15, serialNumber:'CCS-FCS-simu-AutoChangerMotor-20110110-1' , nominalVelocity:10.0 , maximalVelocity: 20.0 , position: 30.0 ] SimuLatch simuStandbyLatch = [ locked: false] SftAutoChangerModule autochanger = [ name: 'autochanger' , tickMillis: 1000 , trucksPositionOnline: 50 , trucksPositionAtStandby: 0 , trucksPositionSwapout: 30 , motor: simuAutoChangerMotor, standbyLatch: simuStandbyLatch ] SftCarouselModule carousel = [ name: 'carousel', tickMillis: 1000 , carouselMotor: simuCarouselMotor , clampsActuator: clampsActuator , latch: simuCarouselLatch , nbSockets: 1 , sockets: [socket0] ]
The statements are all the same (initializing bean data with properties) and though we could have cited an excerpt of this file we can also notice it is a long description process. We could probably generate it automatically from the existing xml description file (if needed!).
listening structure between modules.
autochanger.listens simuAutoChangerMotor, simuStandbyLatch clampXplus0.listens autochanger clampXminus0.listens autochanger carousel.listens simuCarouselMotor , simuCarouselLatch , clampXminus0 , clampXplus0
setting up the subsystem.
// Adding modules to subsystem // THIS CAN BE AUTOMATISED (by a "module" declaration function) system.addModules carousel, simuCarouselLatch ,simuAutoChangerMotor, autochanger ,simuCarouselMotor , clampXminus0, clampXplus0 ,clampXminus0 ,clampsActuator, mainsft system.start()
creating a BusMaster to issue commands.
// a specific BusMaster written for this purpose BusAccess cmd = new BusAccess() // default: does not show log // does not show status cmd.handleStatus = false
issue commands.
cmd.invoke 'single-filter-test/carousel' , 'getPosition'
/// .... other commands
Thread.sleep 50000
cmd.shutdown()
system.shutdown()
We can create stress tests simply:
def val = 45 as double println '-------------------------------------------------------------' for (int ix = 0 ; ix < 5 ;ix++) { cmd.invoke 'single-filter-test/carousel' , 'rotate', val } println '-------------------------------------------------------------'
We can now modify the type of one of the modules in the architecture to trace and test its behaviour.
Here is a subclass created for the purpose of "decorating" calls to methods of its superclass.
class SftCarouselMonitor extends SftCarouselModule { Gruse grus = new Gruse(this) public String rotate(double angle) { grus._monitor rotate: {super.rotate(angle)} _xpect true , { _message("should fail: not implemented") _okIfCaught(RuntimeException) } } }
A instance of SftCarouselMonitor
will replace an instance of SftCarouselModule
in our deployment script.
There:
Gruse
object that will
support monitoring code.
_monitor
method of the Gruse
field.
This _monitor
+ _xpect
combination will deal with returning the normal data returned by the super method
(or propagate the exception thrown by the super method).
the _xpect
clause is particular :
boolean
argument: if true
a report will be generated
each time the method is called, otherwise a report will be generated only if additional data
has been requested or if the result is different from NEUTRAL
or SUCCESS
.
(so when the method is called often then only "special" reports are issued: this helps alleviate
the report handling)
_it, _args, _argNames, _thisG, _thisL
,
otherwise the usual functions used in the context of _xpect
can be used.
_post {/*closure*/}
code can be specified as a teardown code but
should be specified before the _xpect
clause (NOT implemented yet!)
Now in the gru script
we can change the type of an object:
SftCarouselMonitor carousel = [ name: 'carousel', tickMillis: 1000 , carouselMotor: simuCarouselMotor , clampsActuator: clampsActuator , latch: simuCarouselLatch , nbSockets: 1 , sockets: [socket0] ]
Chapter to be completed (see "what’s next" chapter).
What to expect from result handling?
How to create, handle, deploy ResultReporter
s and ResultFormatter
s ?
Right now the SimpleResultFormatter
just creates a String version of the result
and it is printed to standard output.
This is not a complete grammar description but could help.
- testSpecification:
testConstructor
testStaticFactory
testStaticMethod
testInstanceFactory
testInstanceMethod
- testConstructor:
_withClass
classLitteral groupSpecification
_withClass
className groupSpecification
- testStaticFactory:
_withClass
classLitteral_factory
factoryMethodName groupSpecification
_withClass
className_factory
factoryMethodName groupSpecification
static factories are not yet implemented.
- testStaticMethod:
_withClass
classLitteral_method
methodName groupSpecification
_withClass
className_method
methodName groupSpecification
static methods are not yet implemented.
- testInstanceFactory:
_withObjects
listTaggeds_factory
factoryMethodName groupSpecification
instance factories are not yet implemented.
- testInstanceMethod:
_withObjects
listTaggeds_method
methodName groupSpecification
- groupSpecification:
_group
nameOfGroupopt groupClosure runFunctionoptgroupClosure runFunctionopt
NameOfgroup (optionnal) could be used to name a group: this will be useful to export tests results (the list of data generated will bear the name of the group). If no name is specified one will be automatically generated.
- runFunction: one of
_run
_runReport
_fill
taggedsMapRef
_xport
_xport
not implemented yet! be cautious if using _fill
and _xport
with instance mehods.
- groupClosure:
{
listOfGroupClosureElements}
- groupClosureElement:
groovyCode
_pre
closure
_post
closuretestSpecification
_pre
and _post
should be called once at most.
- testSpecification:
_test
listOfTestArgs setupCodeopt expectationsCodeopt teardownCodeopt
_test
listOfTestArgs expectationsClosure
- testArg:
- NameOfTest
:
arg
- arg:
_NO_ARG
taggedObjet
taggedsMap
[
listOfTaggeds]
A listOfTaggeds (list of TaggedObject
and/or TaggedsMap
) is to be used when a call needs
many parameters.
- setupCode :
_pre
closure
- teardownCode :
_post
closure
- expectationsCode :
_xpect
expectationsClosure
The expectationsClosure uses specific functions such as gru assertions
GRU is under development.
In fact this first version needs a complete rehaul since it is a prototype.
We need your advice :
Our plans:
Write a Result Editor. Tests results could be stored somewhere (database? file?) then edited for comments (the programmer checked a result after the test was run, the bug is cornered for future code modification, the bug is not going to be corrected -the corresponding values are not going to be provided in a real situation-,…).
The "commented" results are going to be stored. Then after next test run the new results are going to be compared with the commented one. An automatic analysis will provide clues about addition, regressions, and will yield a synthetic result.
This is a complex matter that should be experimented.
javadoc
-like feature).
lsst.org
being "owner" of the application)
For the moment the tests are run on a single thread (and gru
does not stand against multi-threaded tests).
This is going to be a more complex extension that needs more thorough specifications (what do we want to achieve? how?).