Evaluation Testing Processing
As part of the 2004 Email
Evaluation effort, the team produced a form version of our test script,
logged the responses to a file and created a script that would process
the log into an HTML table of testing results.
If other teams find this test processing useful, it's possible to
take advantage of the processing script.
- After the team has agreed on the test script, create a form for the script
to process the form and write the output into a log file. There are
certain form tag requirements:
Use your browser to view the source of the 2004 Email Evaluation team's test script.
|Required tag name
|The clients being evaluated should be presented as a
select list with an input tag name of client
<option value="">Select a client
<option value="Mulberry 2.1.2"> Mulberry 2.1.2
<option value="Eudora 5.1"> Eudora 5.1
<option value="Outlook 2000"> Outlook 2000
<option value="Entourage"> Entourage
|The operating systems being used should be presented as a
select list with an input tag name of os
<option value="">Select an OS
<option value="Mac OS 8.6"> Mac OS 8.6
<option value="Windows 2000"> Windows 2000
<option value="Windows XP"> Windows XP
<option value="Linux/Solaris"> Linux/Solaris
|Multiple choice questions, whether they are
radio buttons, checkboxes or select lists should have
an input tag name of the number of the question only.
<input type=radio name="1" value="-1" checked>Untested
<input type=radio name="1" value="0">0
<input type=radio name="1" value="1">1
<input type=radio name="1" value="2">2
<input type=radio name="1" value="3">3
|Questions that require longer answers,
whether they are input text boxes or textareas should have
an input tag name of Essay_ and
the number of the essay question.
<textarea name="Essay_1" cols=70 rows=3 wrap="virtual"></textarea>
- Create a feedback-forms configuration file that logs user input.
Besides the logging, there are other requirements that need to be included in your feedback forms configuration file.
View the source of the 2004 Email Evaluation team's feedback-forms configuration file.
|You will need to configure feedback-forms
to assign a sequential number to each entry that gets logged.
In this way, a number will be automatically assigned to each
user that fills out the test script.
|In your feedback-forms "Log:" directive you will need to
indicate the break between each user's input. The processing
script expects to see a line that begins with Tester - as
the first line of a new user's input. This is also where you will
use the sequential number assigned by feedback-forms.
Tester - $FF_acc
|The two required form elements, "client" and "os" have to be included in your output. It is advised that you should make these mandatory fields in your feedback-forms configuration.
Client OS: $Fos
|You need to list each response to a question on a
separate line that begins with the question number, like 1 -
1 - $F1
2 - $F2
3 - $F3
|Essay questions have to be handled in a different
way. Since textboxes do allow the user to input punctuation including line feeds, you need
to indicate to the script where the essay question starts and ends.
Use HTML comments that include the essay tag name on a line
by itself, the response to the essay question and then
HTML comments that include only the string <--End-->.
Essay questions will be written to a separate results file but there
will be links within the main test results page to the essay responses.
- Finally, you need to create a configuration file that will help the
test results processing script to create your test results. Unlike feedback-forms, the order in which you enter information in your configuration file is critical. You must enter elements in the specified order. Each entry in your configuration file should appear only one to a line and should start with the specified keyword, | (used as the field delimiter,) and then the value you are assigning.
View the source of the 2004 Email Evaluation team's test script configuration file.
Used to specify the header at the top of your page. This keyword is optional.
header|Email Evaluation 2004 - Test Form Results
Used to include an introductory paragraph prefacing the table
of test results. This keyword is optional.
comments|<b><nobr>Ranking: 0=Client doesn't have this functionality 1=Not very 2=Moderately 3=Very</nobr></b>
Used to list the hexadecimal values for the colors that you
wish to use for the columns in your test-results. This keyword is optional.
Used to list all clients that will be evaluated.
This keyword is required.
clients|Mulberry 2.1.2|Messenger 6.2.1|Messenger 4.79|Eudora 5.1|Outlook 2000|Outlook XP|Entourage
Used to list all opearting systems that will be used during the testing.
This keyword is required.
os|Mac OS 8.6|Mac OS 9.2.2|Mac OS X 10.1|Windows 98/ME|Windows 2000|Windows XP|Linux/Solaris
List all question numbers and then actual questions.
7|Can you configure the client to use an external editor when composing or reading messages?
Essay_3|Additional comments on configuration.
"UserRow" specifies that you wish to add a header row
of the users (acutally the numbers that represent the
users) to your test results matrix. This is useful if
your matrix is long and you need to be reminded of
which user gave which response.
This keyword can appear anywhere in your configuration where
applicable. This keyword is optional.
Calling the test processing script
If you'd like to have this script run for your evaluation effort, please send mail to firstname.lastname@example.org asking us to run the script. Make sure and include the following:
- base directory for your evaluation team
- log file, relative to the base directory
- test script configuration file relative to the base directory
- output file relative to the base directory
If you have essay questions, the script will automatically output
the essay questions to a second file based on your specified output file.
In this case, the script would write essay questions to
If there is interest, we will consider creating a web application
that will allow authorized team members to run the script themselves.