Summary: Discusses the anti-patterns of testing private methods and mocking class under test methods. It emphasizes the importance of focusing on behavior, refactoring when necessary, and testing in line with production conditions.
Summary: Discusses their experience with Elastic Beanstalk, Docker, and CircleCI while working on a new project. Provides notes and references for each tool and discuss alternatives to Elastic Beanstalk.
The idea of this post and part of the contents are from the fantastic book The Pragmatic Pogrammer.
We don't usually test for most of the following resources:
* An example of the wall clock time is pre-Service Pack 1 Vista's file copy, which was in theory faster and better but actually took longer to copy the same file than in XP for a normal user (more info about it can be found here).
This is part three and last on my WLA Testing posts. Read parts one and two if you haven't because we will go into more complex and advanced tests for our testing framework.
In the previous post we built a basic framework containing better summary reports, base error procedures to mark a test as failed, and Assert methods to compare strings and objects.
While all of these are useful, writing tests to test conversational patterns is still a bit difficult, requiring some code for a simple "check if this is the output of that", so we're going to build Asserts to compare arrays, and then use it to build Asserts that compare conversation patterns (with multiple responses) and simple conversation dialogs (example: if user says A then ask B and if says B then answer C).
This are the procedures we will build:
declare procedure TestExErrorIfContainsArray(TESTMETHOD, STRING, REGEXARRAY)
declare procedure TestExErrorIfDoesNotContainArray(TESTMETHOD, STRING, REGEXARRAY)
declare procedure TestExCompareQuery (REGEX, QUERY)
declare procedure TestExCompareQueryNotEqual (REGEX, QUERY)
declare procedure TestExCompareQueryArray (REGEXARRAY, QUERY)
declare procedure TestExCompareQueryArrayNotEqual (REGEXARRAY, QUERY)
declare procedure TestExDialog(EXPECTEDRESULTS,DIALOGRESPONSES,DIALOGQUERY)
We'll start with the Errors:
// Tests if a string contains at least one of the regular expressions in an array
procedure TestExErrorIfContainsArray(TESTMETHOD, STRING, REGEXARRAY)
CONTAINS = false
STRING_NO_NEWLINES = StringSubstitute(STRING, "\n", "X") // This is to make "." match on new lines, since Contains doesn't support the "s" modifier yet
for value REGEX in REGEXARRAY
if (Contains(REGEX, STRING)) && ((STRING_NO_NEWLINES eq STRING) || (Contains(REGEX, STRING_NO_NEWLINES)))
CONTAINS = true
if (CONTAINS)
ERROR_DESC - Expected: \"EscapeHtmlChars(ObjectToString(REGEXARRAY))\"
call TestExError(TESTMETHOD, ERROR_DESC, STRING)
// Tests if a string does not contain at least one of the regular expressions in an array
procedure TestExErrorIfDoesNotContainArray(TESTMETHOD, STRING, REGEXARRAY)
CONTAINS = false
STRING_NO_NEWLINES = StringSubstitute(STRING, "\n", "X") // This is to make "." match on new lines, since Contains doesn't support the "s" modifier yet
for value REGEX in REGEXARRAY
if (Contains(REGEX, STRING)) && ((STRING_NO_NEWLINES eq STRING) || (Contains(REGEX, STRING_NO_NEWLINES)))
CONTAINS = true
if (!CONTAINS)
ERROR_DESC - Expected: \"EscapeHtmlChars(ObjectToString(REGEXARRAY))\"
call TestExError(TESTMETHOD, ERROR_DESC, STRING)
As you can see, we just search for a string but not against a string but an array, with a For-value-in.
Then, the simple query (one question - one answer) Asserts:
// Tests a query against a regular expression for matching
procedure TestExCompareQuery (REGEX, QUERY)
- Test: \"TestExCompareQuery\"
call IncrementTotalTests()
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfDoesNotContain("TestExCompareQuery", RETURNED, REGEX)
// Tests a query against a regular expression for not matching
procedure TestExCompareQueryNotEqual (REGEX, QUERY)
- Test: \"TestExCompareQueryNotEqual\"
call IncrementTotalTests()
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfContains("TestExCompareQueryNotEqual", RETURNED, REGEX)
Here the trick is calling ExecuteQuery() and storing it's output. This function works as if a user had typed something, processing it and returning the Agent's response. We catch the response and compare it as we did with a simple string Assert.
But what if we've built an agent rich in answers, that can give different responses to the same pattern? Here's where the array error procedures we've just written come to help:
// Tests a query against a regular expression array to see if contains at least one of its elements
procedure TestExCompareQueryArray (REGEXARRAY, QUERY)
- Test: \"TestExCompareQueryArray\"
call IncrementTotalTests()
if !IsObject(REGEXARRAY)
ERROR_DESC - "REGEXARRAY" not an array
call TestExError("TestExCompareQueryArray", ERROR_DESC, QUERY)
else
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfDoesNotContainArray("TestExCompareQueryArray", RETURNED, REGEXARRAY)
// Tests a query against a regular expression array to see if does NOT contain any of its elements
procedure TestExCompareQueryArrayNotEqual (REGEXARRAY, QUERY)
- Test: \"TestExCompareQueryArrayNotEqual\"
call IncrementTotalTests()
if !IsObject(REGEXARRAY)
ERROR_DESC - "REGEXARRAY" not an array
call TestExError("TestExCompareQueryArrayNotEqual", ERROR_DESC, QUERY)
else
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfContainsArray("TestExCompareQueryArrayNotEqual", RETURNED, REGEXARRAY)
What we do is ExecuteQuery() again and check it's value against an array of expected responses.
And finally, here's the code of a simple dialog Assert:
// Tests a menu and all the responses defined in its parameter
// Note: Responses and expected results should be in the same order in the arrys
procedure TestExDialog(EXPECTEDRESULTS,DIALOGRESPONSES,DIALOGQUERY)
- Test: \"TestExDialog\"
call IncrementTotalTests()
RESULTSNUM = SCount(EXPECTEDRESULTS)
if (RESULTSNUM != SCount(DIALOGRESPONSES))
ERROR_DESC - "EXPECTEDRESULTS" and "DIALOGRESPONSES" should have same number of elements
call TestExError("TestDialog", ERROR_DESC, RESULTSNUM)
exit
// Always trigger the dialog, then choose one option each time and check its response
for I in 0 to RESULTSNUM-1
MENURESPONSE = ExecuteQuery(DIALOGQUERY)
ANSWERRESPONSE = ExecuteQuery(DIALOGRESPONSES[I])
call TestExErrorIfDoesNotContain("TestExDialog", ANSWERRESPONSE, EXPECTEDRESULTS[I])
The code is quite self-explanatory, but basically it starts a dialog pattern with DIALOGQUERY, and then loops asking the agent each question (contained in DIALOGRESPONSES array) and each response against the EXPECTEDRESULTS array.
As the code for some examples of this asserts is quite big and I don't want a huge post, I've instead uploaded the full testing framework with a test DDL file so you can directly add them to a buddyscript project and launch the sample tests.
I hope you've learned how to better test your Windows Live Agents with this posts.
I will write just one more post and then leave the legacy to my good friend PedroAfa, who is still working on a daily basis developing WLAs.
Update: Part III of this posts available.
In the previous WLA testing post I gave an introduction to what the platform provides for unit testing.
We also mentioned that it was lacking some features and that I rebuilt the testing framework.
So, I'll just list what the current (rebuilt) framework supports and then start showing code of how to make your own.
Here is the list of procedures available for testing:
procedure TestExInitializeBattery (TEST_NAME, TEST_SUMMARY)
procedure TestExGetBatteryResults ()
procedure TestExErrorIfDoesNotContainArray (STRING, REGEXARRAY)
procedure TestExCompareString (EXPECTEDSTRING, ACTUALSTRING)
procedure TestExCompareStringNotEqual (EXPECTEDSTRING, ACTUALSTRING)
procedure TestExCompareObjects (EXPECTEDOBJECT, ACTUALOBJECT)
procedure TestExCompareObjectsNotEqual (EXPECTEDOBJECT, ACTUALOBJECT)
procedure TestExCompareQuery (REGEX, QUERY)
procedure TestExCompareQueryNotEqual (REGEX, QUERY)
procedure TestExCompareQueryArray (REGEXARRAY, QUERY)
procedure TestExCompareQueryArrayNotEqual (REGEXARRAY, QUERY)
procedure TestExMatch (DOMAIN, QUERY)
procedure TestExMatchNotEqual (DOMAIN, QUERY)
The first two procedures define a new test battery and stops it (and shows the results in the conversation window). a simple template for defining a test battery could be this:
+ test_name_procedures
call TestExInitializeBattery("TESTNAME", "TESTSUMMARY")
// TODO: Fill with tests
call TestExGetBatteryResults()
So we could just launch it by typing to the agent "test_name_procedures".
The procedure highlighted in red should be ignored, as it is used internally most times.
We have procedures to assert equal or not equal strings, objects, queries (conversational patterns and their responses), queries with response arrays (very very useful for multi-response or random-response patterns!), and finally for testing the matching domain (under which language domain the query/pattern it is executed).
Like in NUnit, everything follows the (Expected,Actual) pattern for parameters.
To develop a basic framework, we will need three variables to hold the current test battery data:
variable TESTNAME = ""
variable TESTSUMMARY = ""
variable TESTRESULTS = ""
Next, we write the initialize battery procedure:
procedure TestExInitializeBattery(TEST_NAME, TEST_SUMMARY, WRITE_TO_XML)
TESTNAME = TEST_NAME
TESTSUMMARY = TEST_SUMMARY
- -----------------------
if (TESTSUMMARY == "")
- Begin testing: TEST_NAME
else
- TESTSUMMARY
TESTRESULTS.xml = (WRITE_TO_XML eq true)
TESTRESULTS.Summary = TESTSUMMARY
TESTRESULTS.TestName = TESTNAME
TESTRESULTS.Ellipsis = 500
TESTRESULTS.NbTests = 0
TESTRESULTS.NbErr = 0
The fragments in red could be deleted as they are for a future XML output feature unused right now. Nothing complex here.
Here it is the end battery procedure:
procedure TestExGetBatteryResults()
PASSEDTESTS = TESTRESULTS.NbTests - TESTRESULTS.NbErr
if (TESTRESULTS.xml)
- <unfiltered><tests>
<name>TESTRESULTS.TestName</name>
<count>TESTRESULTS.NbTests</count>
<passed>PASSEDTESTS</passed>
<failed>TESTRESULTS.NbErr</failed>
</tests></unfiltered>
else
- Done testing TESTRESULTS.TestName
nop
- Total tests: TESTRESULTS.NbTests Passed tests: PASSEDTESTS Failed tests: TESTRESULTS.NbErr
if (TESTRESULTS.NbErr)
- Test battery failed
else
- Test battery passed
- -----------------------
Nothing complex here neither, just writing the results.
We will have a simple counter for keeping track of the tests:
TESTRESULTS.NbTests = TESTRESULTS.NbTests + 1
A HTML escape function:
function EscapeHtmlChars(HTML_STRING)
ESCAPED = StringSubstitute(HTML_STRING, "<", "<")
ESCAPED = StringSubstitute(ESCAPED, "&", "&")
return ESCAPED
A base procedure to mark a test as failed:
procedure TestExError(TESTMETHOD, ERRORDESCRIPTION, RETURNEDOUTPUT)
TESTRESULTS.NbErr = TESTRESULTS.NbErr + 1
- Test: \"TESTMETHOD\" failed.
nop
- ERRORDESCRIPTION
nop
- Returned: \"EscapeHtmlChars(ellipsis(RETURNEDOUTPUT, TESTRESULTS.Ellipsis))\"
And two more elaborate failure-checking procedures:
procedure TestExErrorIfContains(TESTMETHOD, STRING, REGEX)
if (Contains(REGEX, STRING))
ERROR_DESC - Expected not to return: \"EscapeHtmlChars(REGEX)\"
call TestExError(TESTMETHOD, STRING, ERROR_DESC)
procedure TestExErrorIfDoesNotContain(TESTMETHOD, STRING, REGEX)
STRING_NO_NEWLINES = StringSubstitute(STRING, "\n", "X")
if (!Contains(REGEX, STRING)) && ((STRING_NO_NEWLINES eq STRING) || (!Contains(REGEX, STRING_NO_NEWLINES)))
ERROR_DESC - Expected: \"EscapeHtmlChars(REGEX)\"
call TestExError(TESTMETHOD, ERROR_DESC, STRING)
Quite simple too, they just stores the calling method name (TESTMETHOD) and search for a given string (STRING) inside another variable (REGEX), calling TestExError() if the condition is met.
Having this, we can actually build some basic string-check tests:
procedure TestExCompareString (EXPECTEDSTRING, ACTUALSTRING)
- Test: \"TestExCompareString\"
call IncrementTotalTests()
if (StringLowercase(EXPECTEDSTRING) != StringLowercase(ACTUALSTRING))
ERROR_DESC - Expected string: "EXPECTEDSTRING". Actual string: "ACTUALSTRING"
call TestExError("TestExCompareString", ERROR_DESC, ACTUALSTRING)
procedure TestExCompareStringNotEqual (EXPECTEDSTRING, ACTUALSTRING)
- Test: \"TestExCompareStringNotEqual\"
call IncrementTotalTests()
if (StringLowercase(EXPECTEDSTRING) == StringLowercase(ACTUALSTRING))
ERROR_DESC - Expected not equal string "ACTUALSTRING".
call TestExError("TestExCompareStringNotEqual", ERROR_DESC, ACTUALSTRING)
We output the test name, increment the test counter, check whenever the string is contained or not, and setup the error description and call TestExError() if the condition is not met. We could have used TestExErrorIfContains() and TestExErrorIfDoesNotContain(), but notice that we're not searching for a partial match (contains, "AAB" contains "AB") but for a full match ("AAB" is not equal to"AB").
Instead of a dummy sample, I'll create another test assertion, comparing objects, which can look hard but actually is easy to do using a small trick:
procedure TestExCompareObjects(EXPECTEDOBJECT, ACTUALOBJECT)
- Test: \"TestExCompareObjects\"
call IncrementTotalTests()
if (!IsObject(EXPECTEDOBJECT))
ERROR_DESC - "EXPECTEDOBJECT" not an object
call TestExError("TestExCompareObjects", ERROR_DESC, EXPECTEDOBJECT)
exit
S_EXPECTEDOBJECT = ObjectToString(EXPECTEDOBJECT)
if (!IsObject(ACTUALOBJECT))
ERROR_DESC - "ACTUALOBJECT" not an object
call TestExError("TestExCompareObjects", ERROR_DESC, ACTUALOBJECT)
exit
S_ACTUALOBJECT = ObjectToString(ACTUALOBJECT)
if (S_EXPECTEDOBJECT != S_ACTUALOBJECT)
ERROR_DESC - Expected equal objects. Actual: "S_ACTUALOBJECT"
call TestExError("TestExCompareObjects", ERROR_DESC, S_EXPECTEDOBJECT)
procedure TestExCompareObjectsNotEqual(EXPECTEDOBJECT, ACTUALOBJECT)
- Test: \"TestExCompareObjectsNotEqual\"
call IncrementTotalTests()
if !IsObject(EXPECTEDOBJECT)
ERROR_DESC - "EXPECTEDOBJECT" not an object
call TestExError("TestExCompareObjectsNotEqual", ERROR_DESC, EXPECTEDOBJECT)
exit
S_EXPECTEDOBJECT = ObjectToString(EXPECTEDOBJECT)
if !IsObject(ACTUALOBJECT)
ERROR_DESC - "ACTUALOBJECT" not an object
call TestExError("TestExCompareObjectsNotEqual", ERROR_DESC, ACTUALOBJECT)
exit
S_ACTUALOBJECT = ObjectToString(ACTUALOBJECT)
if (S_EXPECTEDOBJECT == S_ACTUALOBJECT)
ERROR_DESC - Expected not equal objects. Actual: "S_ACTUALOBJECT"
call TestExError("TestExCompareObjectsNotEqual", ERROR_DESC, S_EXPECTEDOBJECT)
I've painted in green the first appearance of the trick. If we "serialize" the object as a string, we can then check if two objects are equal (they might not be the same, though, just have the same properties and values ;)
The restriction/limitation here is that you have to be very strict and have the properties always in the same order, or the serialization (which is secuential) will create "different objects" (same properties in different order, that will mean strings not equal).
The post has a lot of code, so I'll stop here and leave for further posts adding more functionality to our testing framework.
Here it is a sample of testing the framework current assert procedures:
package TestUtilitiesAddon.pkg
+ test_testexcomparestring_procedures
call TestExInitializeBattery("TestExCompareString() Test Battery", "")
STRING1 = "string 1"
STRING2 = "string 2"
STRING1_1 = "string 1"
call TestExCompareString(STRING1, STRING1)
call TestExCompareString(STRING1, STRING1_1)
call TestExCompareStringNotEqual(STRING1, STRING2)
call TestExGetBatteryResults()
+ test_testexcompareobjects_procedures
call TestExInitializeBattery("TestExCompareObjects() Test Battery", "")
NONOBJ1 = "not object #1"
NONOBJ2 = "not object #2"
OBJ1 = ""
OBJ1.ID = 1
OBJ1.TEXT = "sample object 1"
OBJ1COPY = ""
OBJ1COPY.ID = 1
OBJ1COPY.TEXT = "sample object 1"
OBJ2 = ""
OBJ2.ID = 2
OBJ2COPY.TEXT = "sample object 1"
OBJ3 = ""
OBJ3.ID = 1
OBJ3COPY.TEXT = "sample object 3"
// This 6 commented tests were written to test detecting non-objects
//call TestExCompareObjects(NONOBJ1, NONOBJ2)
//call TestExCompareObjects(OBJ1, NONOBJ2)
//call TestExCompareObjects(NONOBJ1, OBJ1COPY)
//call TestExCompareObjectsNotEqual(NONOBJ1, NONOBJ2)
//call TestExCompareObjectsNotEqual(OBJ1, NONOBJ2)
//call TestExCompareObjectsNotEqual(NONOBJ1, OBJ1COPY)
call TestExCompareObjects(OBJ1, OBJ1)
call TestExCompareObjects(OBJ1, OBJ1COPY)
call TestExCompareObjectsNotEqual(OBJ1, OBJ2)
call TestExCompareObjectsNotEqual(OBJ1, OBJ3)
call TestExGetBatteryResults()