Some Preliminary Findings from the Blackboard Study
by John Noakes, John MacDermott, Elizabeth Scheyder and Jay Treat
Blackboard sites are becoming an increasingly prevalent component of courses in the School of Arts & Sciences. From 2000 to 2003, the number of courses at Penn with Blackboard sites has tripled, rising from 280 to 901, with about 77% of these courses offered in SAS. During an academic year, Penn students download over 2 million files from Blackboard sites. Most of these downloads, it seems, occur between 11 p.m. and 2 a.
It is easy to collect this kind of data about Blackboard use. The courseware system records dates and times and categorizes every hit on every site. Faculty members interested in how their course sites are being used by students will find a wealth of information by clicking on the "Course Statistics" link on their site's control board. But knowing how often or when a site is used is not enough. To evaluate Blackboard, we need to know how it affects teaching and learning at Penn. Those of us who discuss instructional technology with faculty members know that many use Blackboard in innovative ways. We also know faculty members who have yet to figure out how best to use it to aid the teaching and learning of their discipline.
With these concerns in mind, earlier this year the authors began an in-depth study of Blackboard use of a sample of 60 course sites from the fall 2003 semester. A brief description of how we determined our sample and established the boundaries of our usage categories can be found in the Appendix. After generating the list of courses randomly and obtaining the permission of the relevant faculty, we reviewed these sites in depth in an effort to develop a richer understanding of how faculty members use this technology in their courses.
Here we present a few early findings from our study. Central to several of our analyses is the variable "Blackboard usage," which we calculated by dividing the number of hits on a course site by the number of students enrolled in the course. Determined in this way, "Blackboard usage" gives us a measure of how integral a site is to a course.
Who uses Blackboard? One way to answer this is to examine the relationship between faculty rank (which we use as a rough proxy of academic generation) and Blackboard usage. There are two competing arguments one often hears about the relationship between instructional technology and faculty. One is that instructional technology is the domain of younger generations of professors who have grown up in a digital world. The other is that only tenured faculty have the time to invest in integrating instructional technology into their teaching. The results from our study do not lend support to either argument. Faculty members at all levels are spread roughly equally across usage categories. One statistic that jumps off this table is the number of faculty listed as "other." Most of those included in this category are instructors. This number may be unusually high because of the high number of foreign language courses with Blackboard sites. The Department of Romance Languages has the second highest total of Blackboard sites of any department in SAS trailing only the Department of Economics. Fourteen of the 60 courses in our sample are foreign language courses.
TABLE 1: Blackboard Use by Faculty Rank
Very High (12)
Below Average (12)
What are the most popular uses of Blackboard sites? To answer this question, we examined the content of the "Course Documents" folder in the 60 sites in our sample (see Table 2). No single use of Blackboard dominates. The most popular use is for e-reserve readings, but the 26 sites that included e-reserve readings represent a plurality, not a majority. Other popular uses include the posting of information on assignments and exams, and lecture notes.
TABLE 2: Contents of Course Documents Folder
Number of Sites
Percent of Sites
link to external material
How does Blackboard usage affect student evaluations of a course? Students in courses in which the Blackboard course site is integral rate these courses higher than courses in which it is less integral (see Table 3). SCUE evaluation scores are higher for courses in the "very high" and "high" usage categories for each of the three SCUE questions we included in our study than they are for courses with "average" or "below average" Blackboard usage. Curiously, this pattern is broken for courses with "low" Blackboard usage. One possible explanation of this last outcome is that "low" Blackboard usage courses are those in which students averaged less than 57 hits for the semester (see Table 4), suggesting that the course web site played little role in the student's experience of the class.
TABLE 3: Blackboard Usage by Student Evaluations
Quality of Instructor
Quality of Course
Ability to Stimulate Interest in Subject
Very High (12)
Below Average (12)
We welcome discussion of these findings. There is a still a great deal of research to be done. Most importantly, we are hoping to be able to move beyond numbers and begin conversations with faculty and students about how they use Blackboard and how it affects teaching and learning at Penn.
Our first step was to calculate the number of hits per student for each SAS course with a Blackboard site in fall 2003. Boundaries for our five usage categories were established by calculating one and two standard deviations from the mean number of hits per student. Those courses with average hits per student beyond two standard deviations from the mean were categorized as "very high" and "low" usage, depending on whether they were above or below the mean; those between one and two standard deviations from the mean were categorized as "high" and "below average"; those course whose average hits per students were within one standard deviation from the mean were categorized as average. The lower boundaries of the categories and the resulting number of hits per student in each category are listed in Table 4, below. The upper boundary of the "Very High" category is 2656.9, with four courses recording over 2000 hits per student.
Once we established the categories we took a stratified sample of 60 courses, randomly selecting twelve courses from each category. We then obtained permission from the faculty member who had established the site to included it in the study. On the rare occasion that we could not obtain permission, we added the next course on the list from that category to our study.
TABLE 4: Boundaries of Categories in Blackboard Study
Lower Boundary Hits Per Student
Number of Fall 2003 Course In Category
50 ( 7.9%)
John Noakes is Associate Director of the Center for Teaching and Learning. John MacDermott, Elizabeth Scheyder and Jay Treat are with SAS Computing.
This essay resumes the series that began in the fall of 1994 as the joint creation of the College of Arts and Sciences and the Lindback Society for Distinguished Teaching.