Update: Part III of this posts available.
In the previous WLA testing post I gave an introduction to what the platform provides for unit testing.
We also mentioned that it was lacking some features and that I rebuilt the testing framework.
So, I'll just list what the current (rebuilt) framework supports and then start showing code of how to make your own.
Here is the list of procedures available for testing:
procedure TestExInitializeBattery (TEST_NAME, TEST_SUMMARY)
procedure TestExGetBatteryResults ()
procedure TestExErrorIfDoesNotContainArray (STRING, REGEXARRAY)
procedure TestExCompareString (EXPECTEDSTRING, ACTUALSTRING)
procedure TestExCompareStringNotEqual (EXPECTEDSTRING, ACTUALSTRING)
procedure TestExCompareObjects (EXPECTEDOBJECT, ACTUALOBJECT)
procedure TestExCompareObjectsNotEqual (EXPECTEDOBJECT, ACTUALOBJECT)
procedure TestExCompareQuery (REGEX, QUERY)
procedure TestExCompareQueryNotEqual (REGEX, QUERY)
procedure TestExCompareQueryArray (REGEXARRAY, QUERY)
procedure TestExCompareQueryArrayNotEqual (REGEXARRAY, QUERY)
procedure TestExMatch (DOMAIN, QUERY)
procedure TestExMatchNotEqual (DOMAIN, QUERY)
The first two procedures define a new test battery and stops it (and shows the results in the conversation window). a simple template for defining a test battery could be this:
+ test_name_procedures
call TestExInitializeBattery("TESTNAME", "TESTSUMMARY")
// TODO: Fill with tests
call TestExGetBatteryResults()
So we could just launch it by typing to the agent "test_name_procedures".
The procedure highlighted in red should be ignored, as it is used internally most times.
We have procedures to assert equal or not equal strings, objects, queries (conversational patterns and their responses), queries with response arrays (very very useful for multi-response or random-response patterns!), and finally for testing the matching domain (under which language domain the query/pattern it is executed).
Like in NUnit, everything follows the (Expected,Actual) pattern for parameters.
To develop a basic framework, we will need three variables to hold the current test battery data:
variable TESTNAME = ""
variable TESTSUMMARY = ""
variable TESTRESULTS = ""
Next, we write the initialize battery procedure:
procedure TestExInitializeBattery(TEST_NAME, TEST_SUMMARY, WRITE_TO_XML)
TESTNAME = TEST_NAME
TESTSUMMARY = TEST_SUMMARY
- -----------------------
if (TESTSUMMARY == "")
- Begin testing: TEST_NAME
else
- TESTSUMMARY
TESTRESULTS.xml = (WRITE_TO_XML eq true)
TESTRESULTS.Summary = TESTSUMMARY
TESTRESULTS.TestName = TESTNAME
TESTRESULTS.Ellipsis = 500
TESTRESULTS.NbTests = 0
TESTRESULTS.NbErr = 0
The fragments in red could be deleted as they are for a future XML output feature unused right now. Nothing complex here.
Here it is the end battery procedure:
procedure TestExGetBatteryResults()
PASSEDTESTS = TESTRESULTS.NbTests - TESTRESULTS.NbErr
if (TESTRESULTS.xml)
- <unfiltered><tests>
<name>TESTRESULTS.TestName</name>
<count>TESTRESULTS.NbTests</count>
<passed>PASSEDTESTS</passed>
<failed>TESTRESULTS.NbErr</failed>
</tests></unfiltered>
else
- Done testing TESTRESULTS.TestName
nop
- Total tests: TESTRESULTS.NbTests Passed tests: PASSEDTESTS Failed tests: TESTRESULTS.NbErr
if (TESTRESULTS.NbErr)
- Test battery failed
else
- Test battery passed
- -----------------------
Nothing complex here neither, just writing the results.
We will have a simple counter for keeping track of the tests:
TESTRESULTS.NbTests = TESTRESULTS.NbTests + 1
A HTML escape function:
function EscapeHtmlChars(HTML_STRING)
ESCAPED = StringSubstitute(HTML_STRING, "<", "<")
ESCAPED = StringSubstitute(ESCAPED, "&", "&")
return ESCAPED
A base procedure to mark a test as failed:
procedure TestExError(TESTMETHOD, ERRORDESCRIPTION, RETURNEDOUTPUT)
TESTRESULTS.NbErr = TESTRESULTS.NbErr + 1
- Test: \"TESTMETHOD\" failed.
nop
- ERRORDESCRIPTION
nop
- Returned: \"EscapeHtmlChars(ellipsis(RETURNEDOUTPUT, TESTRESULTS.Ellipsis))\"
And two more elaborate failure-checking procedures:
procedure TestExErrorIfContains(TESTMETHOD, STRING, REGEX)
if (Contains(REGEX, STRING))
ERROR_DESC - Expected not to return: \"EscapeHtmlChars(REGEX)\"
call TestExError(TESTMETHOD, STRING, ERROR_DESC)
procedure TestExErrorIfDoesNotContain(TESTMETHOD, STRING, REGEX)
STRING_NO_NEWLINES = StringSubstitute(STRING, "\n", "X")
if (!Contains(REGEX, STRING)) && ((STRING_NO_NEWLINES eq STRING) || (!Contains(REGEX, STRING_NO_NEWLINES)))
ERROR_DESC - Expected: \"EscapeHtmlChars(REGEX)\"
call TestExError(TESTMETHOD, ERROR_DESC, STRING)
Quite simple too, they just stores the calling method name (TESTMETHOD) and search for a given string (STRING) inside another variable (REGEX), calling TestExError() if the condition is met.
Having this, we can actually build some basic string-check tests:
procedure TestExCompareString (EXPECTEDSTRING, ACTUALSTRING)
- Test: \"TestExCompareString\"
call IncrementTotalTests()
if (StringLowercase(EXPECTEDSTRING) != StringLowercase(ACTUALSTRING))
ERROR_DESC - Expected string: "EXPECTEDSTRING". Actual string: "ACTUALSTRING"
call TestExError("TestExCompareString", ERROR_DESC, ACTUALSTRING)
procedure TestExCompareStringNotEqual (EXPECTEDSTRING, ACTUALSTRING)
- Test: \"TestExCompareStringNotEqual\"
call IncrementTotalTests()
if (StringLowercase(EXPECTEDSTRING) == StringLowercase(ACTUALSTRING))
ERROR_DESC - Expected not equal string "ACTUALSTRING".
call TestExError("TestExCompareStringNotEqual", ERROR_DESC, ACTUALSTRING)
We output the test name, increment the test counter, check whenever the string is contained or not, and setup the error description and call TestExError() if the condition is not met. We could have used TestExErrorIfContains() and TestExErrorIfDoesNotContain(), but notice that we're not searching for a partial match (contains, "AAB" contains "AB") but for a full match ("AAB" is not equal to"AB").
Instead of a dummy sample, I'll create another test assertion, comparing objects, which can look hard but actually is easy to do using a small trick:
procedure TestExCompareObjects(EXPECTEDOBJECT, ACTUALOBJECT)
- Test: \"TestExCompareObjects\"
call IncrementTotalTests()
if (!IsObject(EXPECTEDOBJECT))
ERROR_DESC - "EXPECTEDOBJECT" not an object
call TestExError("TestExCompareObjects", ERROR_DESC, EXPECTEDOBJECT)
exit
S_EXPECTEDOBJECT = ObjectToString(EXPECTEDOBJECT)
if (!IsObject(ACTUALOBJECT))
ERROR_DESC - "ACTUALOBJECT" not an object
call TestExError("TestExCompareObjects", ERROR_DESC, ACTUALOBJECT)
exit
S_ACTUALOBJECT = ObjectToString(ACTUALOBJECT)
if (S_EXPECTEDOBJECT != S_ACTUALOBJECT)
ERROR_DESC - Expected equal objects. Actual: "S_ACTUALOBJECT"
call TestExError("TestExCompareObjects", ERROR_DESC, S_EXPECTEDOBJECT)
procedure TestExCompareObjectsNotEqual(EXPECTEDOBJECT, ACTUALOBJECT)
- Test: \"TestExCompareObjectsNotEqual\"
call IncrementTotalTests()
if !IsObject(EXPECTEDOBJECT)
ERROR_DESC - "EXPECTEDOBJECT" not an object
call TestExError("TestExCompareObjectsNotEqual", ERROR_DESC, EXPECTEDOBJECT)
exit
S_EXPECTEDOBJECT = ObjectToString(EXPECTEDOBJECT)
if !IsObject(ACTUALOBJECT)
ERROR_DESC - "ACTUALOBJECT" not an object
call TestExError("TestExCompareObjectsNotEqual", ERROR_DESC, ACTUALOBJECT)
exit
S_ACTUALOBJECT = ObjectToString(ACTUALOBJECT)
if (S_EXPECTEDOBJECT == S_ACTUALOBJECT)
ERROR_DESC - Expected not equal objects. Actual: "S_ACTUALOBJECT"
call TestExError("TestExCompareObjectsNotEqual", ERROR_DESC, S_EXPECTEDOBJECT)
I've painted in green the first appearance of the trick. If we "serialize" the object as a string, we can then check if two objects are equal (they might not be the same, though, just have the same properties and values ;)
The restriction/limitation here is that you have to be very strict and have the properties always in the same order, or the serialization (which is secuential) will create "different objects" (same properties in different order, that will mean strings not equal).
The post has a lot of code, so I'll stop here and leave for further posts adding more functionality to our testing framework.
Here it is a sample of testing the framework current assert procedures:
package TestUtilitiesAddon.pkg
+ test_testexcomparestring_procedures
call TestExInitializeBattery("TestExCompareString() Test Battery", "")
STRING1 = "string 1"
STRING2 = "string 2"
STRING1_1 = "string 1"
call TestExCompareString(STRING1, STRING1)
call TestExCompareString(STRING1, STRING1_1)
call TestExCompareStringNotEqual(STRING1, STRING2)
call TestExGetBatteryResults()
+ test_testexcompareobjects_procedures
call TestExInitializeBattery("TestExCompareObjects() Test Battery", "")
NONOBJ1 = "not object #1"
NONOBJ2 = "not object #2"
OBJ1 = ""
OBJ1.ID = 1
OBJ1.TEXT = "sample object 1"
OBJ1COPY = ""
OBJ1COPY.ID = 1
OBJ1COPY.TEXT = "sample object 1"
OBJ2 = ""
OBJ2.ID = 2
OBJ2COPY.TEXT = "sample object 1"
OBJ3 = ""
OBJ3.ID = 1
OBJ3COPY.TEXT = "sample object 3"
// This 6 commented tests were written to test detecting non-objects
//call TestExCompareObjects(NONOBJ1, NONOBJ2)
//call TestExCompareObjects(OBJ1, NONOBJ2)
//call TestExCompareObjects(NONOBJ1, OBJ1COPY)
//call TestExCompareObjectsNotEqual(NONOBJ1, NONOBJ2)
//call TestExCompareObjectsNotEqual(OBJ1, NONOBJ2)
//call TestExCompareObjectsNotEqual(NONOBJ1, OBJ1COPY)
call TestExCompareObjects(OBJ1, OBJ1)
call TestExCompareObjects(OBJ1, OBJ1COPY)
call TestExCompareObjectsNotEqual(OBJ1, OBJ2)
call TestExCompareObjectsNotEqual(OBJ1, OBJ3)
call TestExGetBatteryResults()
Tags: Testing
Windows Live Agents Testing: Building the Framework article, written by Kartones
. Published on