Windows Live Agents Testing: Improving the Framework

This is part three and last on my WLA Testing posts. Read parts one and two if you haven't because we will go into more complex and advanced tests for our testing framework.

In the previous post we built a basic framework containing better summary reports, base error procedures to mark a test as failed, and Assert methods to compare strings and objects.

While all of these are useful, writing tests to test conversational patterns is still a bit difficult, requiring some code for a simple "check if this is the output of that", so we're going to build Asserts to compare arrays, and then use it to build Asserts that compare conversation patterns (with multiple responses) and simple conversation dialogs (example: if user says A then ask B and if says B then answer C).

This are the procedures we will build:

declare procedure TestExErrorIfContainsArray(TESTMETHOD, STRING, REGEXARRAY)
declare procedure TestExErrorIfDoesNotContainArray(TESTMETHOD, STRING, REGEXARRAY)
declare procedure TestExCompareQuery (REGEX, QUERY)
declare procedure TestExCompareQueryNotEqual (REGEX, QUERY)
declare procedure TestExCompareQueryArray (REGEXARRAY, QUERY)
declare procedure TestExCompareQueryArrayNotEqual (REGEXARRAY, QUERY)
declare procedure TestExDialog(EXPECTEDRESULTS,DIALOGRESPONSES,DIALOGQUERY)

We'll start with the Errors:

// Tests if a string contains at least one of the regular expressions in an array
procedure TestExErrorIfContainsArray(TESTMETHOD, STRING, REGEXARRAY)
CONTAINS = false
STRING_NO_NEWLINES = StringSubstitute(STRING, "\n", "X") // This is to make "." match on new lines, since Contains doesn't support the "s" modifier yet
for value REGEX in REGEXARRAY
if (Contains(REGEX, STRING)) && ((STRING_NO_NEWLINES eq STRING) || (Contains(REGEX, STRING_NO_NEWLINES)))
CONTAINS = true
if (CONTAINS)
ERROR_DESC - Expected: \"EscapeHtmlChars(ObjectToString(REGEXARRAY))\"
call TestExError(TESTMETHOD, ERROR_DESC, STRING)

// Tests if a string does not contain at least one of the regular expressions in an array
procedure TestExErrorIfDoesNotContainArray(TESTMETHOD, STRING, REGEXARRAY)
CONTAINS = false
STRING_NO_NEWLINES = StringSubstitute(STRING, "\n", "X") // This is to make "." match on new lines, since Contains doesn't support the "s" modifier yet
for value REGEX in REGEXARRAY
if (Contains(REGEX, STRING)) && ((STRING_NO_NEWLINES eq STRING) || (Contains(REGEX, STRING_NO_NEWLINES)))
CONTAINS = true
if (!CONTAINS)
ERROR_DESC - Expected: \"EscapeHtmlChars(ObjectToString(REGEXARRAY))\"
call TestExError(TESTMETHOD, ERROR_DESC, STRING)

As you can see, we just search for a string but not against a string but an array, with a For-value-in.

Then, the simple query (one question - one answer) Asserts:

// Tests a query against a regular expression for matching
procedure TestExCompareQuery (REGEX, QUERY)
- Test: \"TestExCompareQuery\"
call IncrementTotalTests()
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfDoesNotContain("TestExCompareQuery", RETURNED, REGEX)

// Tests a query against a regular expression for not matching
procedure TestExCompareQueryNotEqual (REGEX, QUERY)
- Test: \"TestExCompareQueryNotEqual\"
call IncrementTotalTests()
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfContains("TestExCompareQueryNotEqual", RETURNED, REGEX)

Here the trick is calling ExecuteQuery() and storing it's output. This function works as if a user had typed something, processing it and returning the Agent's response. We catch the response and compare it as we did with a simple string Assert.

But what if we've built an agent rich in answers, that can give different responses to the same pattern? Here's where the array error procedures we've just written come to help:

// Tests a query against a regular expression array to see if contains at least one of its elements
procedure TestExCompareQueryArray (REGEXARRAY, QUERY)
- Test: \"TestExCompareQueryArray\"
call IncrementTotalTests()
if !IsObject(REGEXARRAY)
ERROR_DESC - "REGEXARRAY" not an array
call TestExError("TestExCompareQueryArray", ERROR_DESC, QUERY)
else
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfDoesNotContainArray("TestExCompareQueryArray", RETURNED, REGEXARRAY)

// Tests a query against a regular expression array to see if does NOT contain any of its elements
procedure TestExCompareQueryArrayNotEqual (REGEXARRAY, QUERY)
- Test: \"TestExCompareQueryArrayNotEqual\"
call IncrementTotalTests()
if !IsObject(REGEXARRAY)
ERROR_DESC - "REGEXARRAY" not an array
call TestExError("TestExCompareQueryArrayNotEqual", ERROR_DESC, QUERY)
else
RETURNED = ExecuteQuery(QUERY)
call TestExErrorIfContainsArray("TestExCompareQueryArrayNotEqual", RETURNED, REGEXARRAY)

What we do is ExecuteQuery() again and check it's value against an array of expected responses.

And finally, here's the code of a simple dialog Assert:

// Tests a menu and all the responses defined in its parameter
// Note: Responses and expected results should be in the same order in the arrys
procedure TestExDialog(EXPECTEDRESULTS,DIALOGRESPONSES,DIALOGQUERY)
- Test: \"TestExDialog\"
call IncrementTotalTests()
RESULTSNUM = SCount(EXPECTEDRESULTS)
if (RESULTSNUM != SCount(DIALOGRESPONSES))
ERROR_DESC - "EXPECTEDRESULTS" and "DIALOGRESPONSES" should have same number of elements
call TestExError("TestDialog", ERROR_DESC, RESULTSNUM)
exit
// Always trigger the dialog, then choose one option each time and check its response
for I in 0 to RESULTSNUM-1
MENURESPONSE = ExecuteQuery(DIALOGQUERY)
ANSWERRESPONSE = ExecuteQuery(DIALOGRESPONSES[I])
call TestExErrorIfDoesNotContain("TestExDialog", ANSWERRESPONSE, EXPECTEDRESULTS[I])

The code is quite self-explanatory, but basically it starts a dialog pattern with DIALOGQUERY, and then loops asking the agent each question (contained in DIALOGRESPONSES array) and each response against the EXPECTEDRESULTS array.

As the code for some examples of this asserts is quite big and I don't want a huge post, I've instead uploaded the full testing framework with a test DDL file so you can directly add them to a buddyscript project and launch the sample tests.

I hope you've learned how to better test your Windows Live Agents with this posts.

I will write just one more post and then leave the legacy to my good friend PedroAfa, who is still working on a daily basis developing WLAs.