E-mail Task Force: Office Systems Working Group Product Evaluation

• March 1994
  • The Working Group selected a short list of products and then developed and distributed basic data about each to the working group.
  • We surveyed the campus very quickly to assess what products are in use, and
  • We also developed a standard product feature description methodology and a matrix to complete as we evaluate the products.
  • The Medical School performed a quick assessment of DEC TeamLinks and MailWorks, but ran into a few problems with the OpenVMS version of the MailWorks server, and abandoned the effort in favor of other activities.
• April-June 1994
  • Working Group members conducted an initial evaluation of products to gain consistent understanding of all product features, functions, and limitations.
  • These evaluations involved direct contact with the vendor, completing a short product feature checklist,
  • as well as performing a series of tests according to a "standard testing script."
• End of June 1994
  • The product list was narrowed based on results of initial evaluations. We were somewhat surprised by the common-ness of all the packages we were able to test.
    • All were able to do all the common, pedestrian things we require, but none was able to do all, or even most of the less pedestrian things we require if e-mail is to evolve beyond its current state.
    • We were struck by the fact that no product was superior to the current "campus-standard" email packages (elm and Eudora) on all three of the most important evaluation criteria -- SMTP/MIME support, support for location independent access, and integration with desktop computing environments (primarily Macs and Windows PCs). These three things are the three-legged stool of email. A variety of two-legged stools abound, but in a three-dimensional world, they are much harder to use than we desire.
    • We enjoyed presentations from ISA Corp on ECSmail, and from NCD/Z-Code on Zmail, concluding that these companies were in the process of building these three features into their products. Interaction with the other vendors proved very frustrating, as they fundamentally failed to understand these three key points.
    • We signed up to help beta-test both ECSmail 3.0 and Z-Mail in hopes of helping at least one of the companies succeed in developing a product the we can use.
  • Status report issued to the campus (appearing in the September Penn Printout)
• July-September 1994
  • Participation in the beta-testing of ECSmail 3.0 was less and less painful with each successive beta version. Unfortunately, much work is required before ISA Corp's three-legged stool is robust enough for our needs.
  • Z-Mail testing was similarly disappointing, as NCD/Z-Code has clearly not yet implemented much of the Zync server described in their white paper. As a result, Windows and Mac clients enjoy little more location independent access than POP clients like Eudora.
  • The Wharton School also launched a pilot of MS-Mail using the HP/OpenMail server to support a subset of their administrative population, understanding that support form MIME was scheduled for Spring of 1995, and that location independent access was no better than under Eudora.
• October-December 1994
  • While we had originally hoped to be pilot testing a small number of products over the summer, and making recommendations by the middle of the fall, even a three-month delayed version of this time-table was not possible as we waited for vendors to improve their products.
  • Although the Medical School had experienced problems with the DEC TeamLinks and MailWorks products, a visit by Joyce Graff, DEC's Senior Product Manager for Standards-Based Email convinced us that it was worth a second look. Discussions with developers and product managers leaves us hopeful that they will not only implement MIME in the first quarter of next year, but also port their server to third-party Unix systems, specifically Sun Solaris, HP HP-UX, IBM AIX and SGI IRIX. We are therefore beginning an evaluation of their products at this time, expecting it to be unique in already offering both desktop integration and location independent access.
  • We are also entertaining the idea that an interim, transitional strategy might be appropriate for people at the University who are using non-SMTP/MIME-compliant email. In short, this boils down to a suggestion that people convert to Elm or Eudora, depending on whether they need location independent access or not, and whether they have the requisite desktop hardware and network connections to use Eudora. This would then allow more incremental transitions to new email programs as they become available.