Click for Philadelphia, Pennsylvania Forecast
HOME ISSUE

CALENDAR

BETWEEN ISSUES ARCHIVE DEADLINES CONTACT US
 
 
Print This Issue
Front Page
Contents
Crimes
Directory
All About Teaching
Subscribe to E-Alamanc!
Staffbox
Guidelines
 

 

TALK ABOUT TEACHING AND LEARNING

Some Preliminary Findings from the Blackboard Study

 by John Noakes, John MacDermott, Elizabeth Scheyder and Jay Treat

Blackboard sites are becoming an increasingly prevalent component of courses in the School of Arts & Sciences. From 2000 to 2003, the number of courses at Penn with Blackboard sites has tripled, rising from 280 to 901, with about 77% of these courses offered in SAS. During an academic year, Penn students download over 2 million files from Blackboard sites.  Most of these downloads, it seems, occur between 11 p.m. and 2 a.

It is easy to collect this kind of data about Blackboard use. The courseware system records dates and times and categorizes every hit on every site. Faculty members interested in how their course sites are being used by students will find a wealth of information by clicking on the "Course Statistics" link on their site's control board. But knowing how often or when a site is used is not enough. To evaluate Blackboard, we need to know how it affects teaching and learning at Penn. Those of us who discuss instructional technology with faculty members know that many use Blackboard in innovative ways. We also know faculty members who have yet to figure out how best to use it to aid the teaching and learning of their discipline.

With these concerns in mind, earlier this year the authors began an in-depth study of Blackboard use of a sample of 60 course sites from the fall 2003 semester. A brief description of how we determined our sample and established the boundaries of our usage categories can be found in the Appendix. After generating the list of courses randomly and obtaining the permission of the relevant faculty, we reviewed these sites in depth in an effort to develop a richer understanding of how faculty members use this technology in their courses.

Here we present a few early findings from our study.  Central to several of our analyses is the variable "Blackboard usage," which we calculated by dividing the number of hits on a course site by the number of students enrolled in the course. Determined in this way, "Blackboard usage" gives us a measure of how integral a site is to a course.

Who uses Blackboard? One way to answer this is to examine the relationship between faculty rank (which we use as a rough proxy of academic generation) and Blackboard usage. There are two competing arguments one often hears about the relationship between instructional technology and faculty. One is that instructional technology is the domain of younger generations of professors who have grown up in a digital world. The other is that only tenured faculty have the time to invest in integrating instructional technology into their teaching. The results from our study do not lend support to either argument. Faculty members at all levels are spread roughly equally across usage categories. One statistic that jumps off this table is the number of faculty listed as "other." Most of those included in this category are instructors. This number may be unusually high because of the high number of foreign language courses with Blackboard sites. The Department of Romance Languages has the second highest total of Blackboard sites of any department in SAS trailing only the Department of Economics. Fourteen of the 60 courses in our sample are foreign language courses.

TABLE 1: Blackboard Use by Faculty Rank


Full
Professor

Associate Professor

Assistant Professor

Others

Very High (12)

2

3

2

5

High (12)

4

1

2

5

Average (12)

3

2

2

5

Below Average (12)

6

2

0

4

Low (12)

0

1

4

7

Total

15

9

10

26

What are the most popular uses of Blackboard sites? To answer this question, we examined the content of the "Course Documents" folder in the 60 sites in our sample (see Table 2). No single use of Blackboard dominates. The most popular use is for e-reserve readings, but the 26 sites that included e-reserve readings represent a plurality, not a majority. Other popular uses include the posting of information on assignments and exams, and lecture notes.

TABLE 2: Contents of Course Documents Folder


Number of Sites

Percent of Sites

E-reserve readings

26

43.33%

Assignments/exam info

18

30.00%

lectures/lecture notes

16

26.67%

syllabus

12

20.00%

reference material

11

18.33%

link to external material

 7

11.67%

video

 6

10.00%

audio

 5

  8.33%

How does Blackboard usage affect student evaluations of a course? Students in courses in which the Blackboard course site is integral rate these courses higher than courses in which it is less integral (see Table 3). SCUE evaluation scores are higher for courses in the "very high" and "high" usage categories for each of the three SCUE questions we included in our study than they are for courses with "average" or "below average" Blackboard usage. Curiously, this pattern is broken for courses with "low" Blackboard usage. One possible explanation of this last outcome is that "low" Blackboard usage courses are those in which students averaged less than 57 hits for the semester (see Table 4), suggesting that  the course web site played little role in the student's experience of the class. 

TABLE 3: Blackboard Usage by Student Evaluations


Quality of Instructor

Quality of Course

Ability to Stimulate Interest in Subject

Very High (12)

3.45

3.04

3.29

High (12)

3.50

3.14

3.22

Average (12)

3.26

2.97

2.86

Below Average (12)

3.07

2.81

2.74

Low (12)

3.34

2.99

3.00

Sample Averages  

3.32

3.00

3.00

We welcome discussion of these findings. There is a still a great deal of research to be done. Most importantly, we are hoping to be able to move beyond numbers and begin conversations with faculty and students about how they use Blackboard and how it affects teaching and learning at Penn.

Appendix

Our first step was to calculate the number of hits per student for each SAS course with a Blackboard site in fall 2003. Boundaries for our five usage categories were established by calculating one and two standard deviations from the mean number of hits per student. Those courses with average hits per student beyond two standard deviations from the mean were categorized as "very high" and "low" usage, depending on whether they were above or below the mean; those between one and two standard deviations from the mean were categorized as "high" and "below average"; those course whose average hits per students were within one standard deviation from the mean were categorized as average. The lower boundaries of the categories and the resulting number of hits per student in each category are listed in Table 4, below. The upper boundary of the "Very High" category is 2656.9, with four courses recording over 2000 hits per student.

Once we established the categories we took a stratified sample of 60 courses, randomly selecting twelve courses from each category. We then obtained permission from the faculty member who had established the site to included it in the study. On the rare occasion that we could not obtain permission, we added the next course on the list from that category to our study.

TABLE 4: Boundaries of Categories in Blackboard Study

Category

Lower Boundary
Hits Per Student

Number of Fall 2003 Course In Category

Very High

295.6

69 (10.7%)

High

216.2

50 ( 7.9%)

Average

136.7

86 (13.4%)

Below Average

  57.2

193 (30.1%)

Low

   0

244 (38.0%)

John Noakes is Associate Director of the Center for Teaching and Learning. John MacDermott, Elizabeth Scheyder and Jay Treat are with SAS Computing.

This essay resumes the series that began in the fall of 1994 as the joint creation of the College of Arts and Sciences and the Lindback Society for Distinguished Teaching.

See www.upenn.edu/almanac/teach/teachall.html for the previous essays.

If you would like to submit an article for this series, contact Larry Robbins, Director of the Center for Teaching and Learning, robbinsl@sas.upenn.edu.

 

 


  Almanac, Vol. 51, No. 7, October 12, 2004

ISSUE HIGHLIGHTS:

Tuesday,
October 12, 2004
Volume 51 Number 7
www.upenn.edu/almanac

 

Back to Contents page
HOME ISSUE CALENDAR BETWEEN ISSUES ARCHIVE DEADLINES CONTACT US