Evaluation Testing Processing

As part of the 2004 Email Evaluation effort, the team produced a form version of our test script, logged the responses to a file and created a script that would process the log into an HTML table of testing results. If other teams find this test processing useful, it's possible to take advantage of the processing script.

1. After the team has agreed on the test script, create a form for the script using feedback-forms to process the form and write the output into a log file. There are certain form tag requirements:

   Required tag name	Example
   The clients being evaluated should be presented as a select list with an input tag name of client	<select name="client"> <option value="">Select a client <option value="Mulberry 2.1.2"> Mulberry 2.1.2 <option value="Eudora 5.1"> Eudora 5.1 <option value="Outlook 2000"> Outlook 2000 <option value="Entourage"> Entourage </select>
   The operating systems being used should be presented as a select list with an input tag name of os	<select name="os"> <option value="">Select an OS <option value="Mac OS 8.6"> Mac OS 8.6 <option value="Windows 2000"> Windows 2000 <option value="Windows XP"> Windows XP <option value="Linux/Solaris"> Linux/Solaris </select>
   Multiple choice questions, whether they are radio buttons, checkboxes or select lists should have an input tag name of the number of the question only.	<input type=radio name="1" value="-1" checked>Untested     <input type=radio name="1" value="0">0     <input type=radio name="1" value="1">1     <input type=radio name="1" value="2">2     <input type=radio name="1" value="3">3
   Questions that require longer answers, whether they are input text boxes or textareas should have an input tag name of Essay_ and the number of the essay question.	<textarea name="Essay_1" cols=70 rows=3 wrap="virtual"></textarea>

   Use your browser to view the source of the 2004 Email Evaluation team's test script.

2. Create a feedback-forms configuration file that logs user input. Besides the logging, there are other requirements that need to be included in your feedback forms configuration file.

   Requirement	Example
   You will need to configure feedback-forms to assign a sequential number to each entry that gets logged. In this way, a number will be automatically assigned to each user that fills out the test script.	Accession: number.txt
   In your feedback-forms "Log:" directive you will need to indicate the break between each user's input. The processing script expects to see a line that begins with Tester - as the first line of a new user's input. This is also where you will use the sequential number assigned by feedback-forms.	Tester - $FF_acc
   The two required form elements, "client" and "os" have to be included in your output. It is advised that you should make these mandatory fields in your feedback-forms configuration.	Client: $Fclient Client OS: $Fos
   You need to list each response to a question on a separate line that begins with the question number, like 1 -	1 - $F1 2 - $F2 3 - $F3
   Essay questions have to be handled in a different way. Since textboxes do allow the user to input punctuation including line feeds, you need to indicate to the script where the essay question starts and ends. Use HTML comments that include the essay tag name on a line by itself, the response to the essay question and then HTML comments that include only the string <--End-->. Essay questions will be written to a separate results file but there will be links within the main test results page to the essay responses.	<!--Essay_1--> $FEssay_1 <!--End-->

   View the source of the 2004 Email Evaluation team's feedback-forms configuration file.

3. Finally, you need to create a configuration file that will help the test results processing script to create your test results. Unlike feedback-forms, the order in which you enter information in your configuration file is critical. You must enter elements in the specified order. Each entry in your configuration file should appear only one to a line and should start with the specified keyword, | (used as the field delimiter,) and then the value you are assigning.

   Configuration Keyword	Example
   header Used to specify the header at the top of your page. This keyword is optional.	header|Email Evaluation 2004 - Test Form Results
   comments Used to include an introductory paragraph prefacing the table of test results. This keyword is optional.	comments|<b><nobr>Ranking:   0=Client doesn't have this functionality     1=Not very   2=Moderately   3=Very</nobr></b>
   colors Used to list the hexadecimal values for the colors that you wish to use for the columns in your test-results. This keyword is optional.	colors|FF99999|CC99CC|CCFFFF|FFCCFF|FFFFCC|CCCCFF|CC9900
   clients Used to list all clients that will be evaluated. This keyword is required.	clients|Mulberry 2.1.2|Messenger 6.2.1|Messenger 4.79|Eudora 5.1|Outlook 2000|Outlook XP|Entourage
   os Used to list all opearting systems that will be used during the testing. This keyword is required.	os|Mac OS 8.6|Mac OS 9.2.2|Mac OS X 10.1|Windows 98/ME|Windows 2000|Windows XP|Linux/Solaris
   [question number] List all question numbers and then actual questions.	7|Can you configure the client to use an external editor when composing or reading messages? Essay_3|Additional comments on configuration.
   UserRow "UserRow" specifies that you wish to add a header row of the users (acutally the numbers that represent the users) to your test results matrix. This is useful if your matrix is long and you need to be reminded of which user gave which response. This keyword can appear anywhere in your configuration where applicable. This keyword is optional.	UserRow

   View the source of the 2004 Email Evaluation team's test script configuration file.

Calling the test processing script

If you'd like to have this script run for your evaluation effort, please send mail to www-help@isc.upenn.edu asking us to run the script. Make sure and include the following:

• base directory for your evaluation team
  e.g., /computing/eval/2004/email
• log file, relative to the base directory
  e.g., FF_logs/script-log.html
• test script configuration file relative to the base directory
  e.g., test-results.conf
  
• output file relative to the base directory
  e.g., test-results.html
  If you have essay questions, the script will automatically output the essay questions to a second file based on your specified output file. In this case, the script would write essay questions to test-results-long.html

If there is interest, we will consider creating a web application that will allow authorized team members to run the script themselves.