Pullout: Report of the Provost's Committee on Distributed Learning

 

SENATE Year End Committee Report

Report of the Faculty Senate Committee on Administration

April 17, 1998

 

A. Preface

The committee addressed four issues which were deemed to be important for the position and mission of the faculty. These were: the relation between budgetary policy and strategic issues, the adequacy of faculty consultation in the preparation of strategic plans in the schools, the appropriate recognition of faculty service to the University, and the teaching evaluation processes. Four subcommittees were therefore formed whose findings are presented below.

B. Subcommittee on Cost Containment

The major conclusion of this subcommittee report was that there has been a substantial relative shift over the past eighteen years away from the direct support of education and scholarship to other activities. The report was extensive and has been published separately in the March 24, 1998 issue of Almanac.

C. Subcommittee on Strategic Planning

The consultative process is an important vehicle by which faculty contributes to University decision making on strategic, policy and administrative matters. The Faculty Senate therefore needs to be aware of the degree to which this process succeeds in insuring adequate faculty participation in these matters. In particular, strategic planning can, and has, had serious consequences for departments and programs. The Subcommittee on Strategic Planning was therefore asked to examine the process by which each of the schools included faculty consultation in developing their strategic plans as requested by President Rodin in connection with the University's Agenda for Excellence (Almanac January 21, 1997). It did this by interviewing selected faculty members in the schools.

The difficulties in assessing the adequacy of consultation are great. The subcommittee realized that it had neither the time nor resources to address this issue for the School of Medicine so it was omitted from this study. Also, because of its size and diversity, it was not possible to obtain a comprehensive picture of the planning process in each department of the School of Arts and Sciences. In all schools, a relatively small fraction of the faculty was interviewed and it is possible that a larger sample would have uncovered problems of which the subcommittee in not now aware. The conclusions presented here must therefore be regarded as tentative and impressionistic. A description of the process in individual schools is:

1. School of Arts and Sciences

The School of Arts and Sciences is vital to the overall quality and reputation of the University and its strategic planning is critically important because the school is faced with serious and controversial problems. Financial stringency forces SAS to make difficult decisions on the growth, decrease or elimination of departments and programs. Some of these decisions were made prior to the formulation of the Agenda for Excellence and others are reflected in the strategic plan. The subcommittee's overall conclusions are twofold: First, the consultative process afforded faculty members in most departments the opportunity to communicate their views to the dean and her principle advisors if they wished to do so. Second, the process did not give the faculty the opportunity to review and comment on successive drafts of the plan.

The SAS plan was developed by deans in consultation with a committee of 20 faculty members (The Planning and Priorities Committee). The members of this committee were selected by the dean and represented most, but not all departments. In addition to advice from this committee, the dean attached considerable weight to past external and internal reviews, the departmental responses to these reviews and to self evaluations requested by the dean in connection with the strategic planning effort.

While communication from faculty to the school administration and Planning Committee was adequate, their seemed to be little reciprocal communication in the other direction. Members of the Planning Committee reportedly treated their deliberations as confidential and interested faculty members were unable to carry on a dialogue with the planners. Indeed, the faculty as a whole did not see the strategic plan, or any draft of it, before its submission to the President.

Faculty members from departments that were not favored in the plan were doubtful that their views were taken seriously even though they were formally considered. Some faculty members from other departments reported that they knew nothing of the planning process. However, letters of criticism to the President about specific features of the plan from other faculty members resulted in adoption of some of the changes recommended in the letters.

2. Law School

In preparing a draft strategic plan, the dean first met individually with each of the 30 faculty members and each of the major committees in the school. The draft was then submitted to the faculty for comments and an all-day retreat was held to discuss the draft and extensive reports by several departmental standing committees. The resulting revised draft was submitted to the faculty for approval before being sent to the President and Provost. This extensive faculty participation was possible, in part, because of the small size of the school, an absence of departmental structure and a long tradition of self-governance.

3. Annenberg School for Communication

An initial draft of the plan, prepared by the dean, was reviewed by the school's Executive Committee, which is elected by the faculty. The draft was then reviewed by the full faculty before being revised and sent to College Hall. Generally, faculty felt that they had been given full access to information and had ample opportunity to present and discuss their views.

4. Wharton School

By 1994, the Wharton School had already prepared a planning document as part of an ongoing process of planning and self-evaluation. Wharton drew heavily on this document in constructing its strategic plan for the Agenda for Excellence. The first draft was prepared by the dean and circulated to department chairs and the dean's Advisory Committee, a committee of senior faculty members. The resulting revision was circulated to the full faculty and discussed at a faculty meeting. The document went through nine revisions before being sent to College Hall.

It was clear to the subcommittee that the planning process in Wharton is highly consultative and participatory which includes the faculty and, on occasion, students, staff, alumni and even major recruiting organizations.

5. School of Veterinary Medicine

The School of Veterinary Medicine had a planning process in place which resulted in two planning documents before President Rodin's request with respect to the Agenda for Excellence. The first addressed the available options in response to the state-funding crisis of the early 90's and a later document dealt with fundamental institutional issues. These were the basis of the response to the President's request.

The planning process was led by a task force headed by a faculty member with representation from the school's four departments. While faculty members not on the task force did not contribute much to the final document, they clearly did so in the earlier stages over a period of five years.

6. School of Dental Medicine

The School of Dental Medicine also adapted a previously prepared planning document for the response to the Agenda for Excellence. This was done by an open and participatory process involving frequent faculty meetings. A unique feature in this school is a system for implementation of the strategic plan. For each of the goals specified in the plan, a committee of faculty, students and staff is responsible for monitoring and reporting on the progress of implementation. An oversight committee coordinates these reports and circulates them to the school faculty members which can then make comments on the progress or recommend changes to the strategic plan. The faculty is enthusiastic about this process.

7. School of Nursing

Formal strategic planning in Nursing dates back to at least 1993 when it was decided to expand the school's mission to include clinical practice. Several other retreats were held soon after and a Long-Range Planning Committee was formed. The Nursing faculty has met as a whole to determine which of the priorities in the Agenda for Excellence it could actively support. The Long-Range Planning Committee continues to meet monthly and retreats are held semi-annually to discuss planning issues. The standing faculty of the School of Nursing has been and continues to be heavily involved in strategic planning.

8. School of Engineering and Applied Science

The planning document in SEAS was prepared by an officer of the school's Development Office, was discussed with the seven departmental chairs and then brought to departmental faculties by the chairs. A revised document was then reviewed by the Faculty Council (elected by the faculty) who presented it for discussion at two faculty meetings.

9. Graduate School of Education

The GSE planning process was highly consultative. The dean held one hour meetings with each faculty member and drafted a first version of the plan. After discussion with the Executive Committee of the school, it was circulated to all faculty members for comments. The resulting revisions were discussed in faculty meetings and the final draft incorporated most of the faculty suggestions.

10. School of Social Work

The first draft of the planning document was prepared by the dean in consultation with some senior faculty members and reflected, in part, ideas contained in an accreditation self-evaluation study done in 1991-92. This draft was discussed at a faculty retreat, revised and then presented at later faculty meetings. Some faculty concur with the dean's statement that the planning process is "extraordinary in terms of faculty input and involvement."

11. Graduate School of Fine Arts

This school prepared its strategic plan under an interim dean. The interim dean circulated guidelines from an earlier planning document to the four department chairs and then prepared a draft strategic plan from their responses. This went through several stages of interactions between the interim dean and the chairs until a penultimate draft emerged. The present dean refined this document and sent it to College Hall. It is not clear to what extent the faculty, formally or individually, took part in this exercise. However, the major differences in the school follow departmental lines and the individual departments were well represented. It is probable that the dissatisfaction expressed by some city planners reflected issues of substance rather than process.

Generally, it appears that, with some reservations expressed below, faculty members had the opportunity to participate in the strategic planning process. No instances were noted in which faculty members who wished to participate were prevented from doing so by procedural obstacles.

In at least four schools, Wharton, Veterinary Medicine, Dental Medicine and Nursing, the Agenda for Excellence was the most recent phase in an ongoing planning process.

The planning process in the School of Arts and sciences did not give faculty members an opportunity to react to drafts of the planning document. Indeed, few faculty members, other than the 20 senior professors on the planning committee, even saw the strategic plan prior to its submission to the central administration.

The subcommittee received comments to the effect that the existence of a satisfactory consultative procedure does not guarantee that dissenting views will be taken seriously. The subcommittee could not determine whether this reflects a defect in the process or a disagreement with substantive decisions.

It is recommended that the faculty have the opportunity, collectively and individually, for input at all stages of strategic plan development. This should include the opportunity to review and react to interim drafts as well as the final version.

It is important to note that strategic planning is a continuing process and that plans are often modified as their implementation evolves. Mechanisms to monitor the processes of implementation should be encouraged.

D. Subcommittee on Service to the University

Increased pressures on the faculty to assume greater responsibilities, such as increasing outside support for research, is discouraging their service to the University. Although service to the University is recognized in various policy documents as a criterion for promotions, these are scattered among the schools and in different places in the Handbook on the Faculty and Academic Administrators. Significant differences obtain among the schools in both policy and practice on service to the University, ranging from positive affirmation to slight acknowledgment. At present it is generally regarded as a third, somewhat ambiguous consideration in evaluating faculty performance after research and teaching.

At the same time, the Faculty Senate continues to have a central responsibility to represent the faculty and contribute a faculty voice to University governance. As the University becomes more complex and claims on its resources multiply, and as the University reaches out to more constituents locally, nationally, and globally, this responsibility becomes more urgent.

The Faculty Senate has responded to these pressures by expanding volunteer efforts from individual faculty and seeking more information from the University administration. Although individuals have given much to the Senate and to its committees, it is clear that reasons for declining specific assignments are becoming more compelling while the need for them is increasing.

Because of the above considerations, the Senate Committee on Administration appointed a subcommittee to look into these issues. The subcommittee concluded that some initiatives should be taken to respond to the conflicting pressures of the need for more involvement by the faculty and the ever increasing demands on faculty time and recommended that the Faculty Senate take the following two steps to address this problem:

First, assemble existing policies on recognition of faculty service to the University into a single, consolidated statement in order to give it easy accessibility, consistency and the prominence it deserves. Second, appoint a subcommittee to identify an independent source of funds for the Faculty Senate that would provide the means and rewards for faculty involvement in the Senate as the main vehicle for the faculty to meet its obligation of service to the University.

E. Subcommittee on Teaching Evaluations

In response to the concern of many faculty as to the value and impact of the current course evaluation process, the objectives of this Subcommittee were to evaluate the current course and faculty teaching evaluation process, define the parameters for an "ideal" system, and develop conclusions and recommendations.

1. Concerns with the Current System

Evaluating the current system uncovered a number of concerns. While most of the faculty we discussed it with recognize the importance of course/faculty teaching evaluation, many did not like the current process. It was concluded that any system will have to address the following issues:

a. Who is the process for (faculty, administration, students)? Can a single instrument be used both to inform the students, serve as feedback to the faculty and for faculty promotion decisions?

b. Are students, at the time of taking the course, qualified to judge content?

c. Does the process have any negative effects on the rigor of education or on grade inflation?

d. How can the student ratings of quality be separated from the compounded effect of other variables such as showmanship, charisma, difficulty of material, grading policy, age, gender, reputation, etc.?

e. Should student evaluation be anonymous? (Will identifiable respondents lead to more responsible and consistent responses? Can identifiable responses lead to faculty retaliation? )

f. How complete or representative are the returns?

g. How to solve the problem that the current system "distances" the student from the "co-production" model of education by having "non-accountable evaluators"?

h. How to avoid the problem that reporting only averages can be misleading?

i. How can numerical scores be combined with constructive comment for self/course improvement?

j. When should the evaluation be conducted (mid-term, ongoing, end of term, end of program, a number of years after graduation)?

k. Should different processes be used for small classes?

l. Should the results be publicized? (Asymmetry-rights of faculty vs. rights of students.)

m. What is the value added of the Penn undergraduate course review guide?

2. Parameters of an Ideal System

The overall objective of a teaching and course evaluation system is to enhance the quality, appropriateness, effective delivery and learning of the course content. It should be feasible (cost effective and easy to administer) and the measures used should be reliable and valid. The system should meet the needs of three audiences: faculty, students and academic administration.

For faculty, the objective is to provide meaningful feedback for self and course improvement.

For students the objective is to provide an opportunity for engaging in constructive and reflective dialogue with the faculty, as well as appropriate input for improvement and selection of courses.

For the departments, schools and University academic administrators, the objective is to provide valid feedback on faculty teaching effectiveness and course quality.

A variety of options were considered for satisfying the above objectives ranging from keeping the status quo (which was rejected) through various modifications of the current system, to radical solutions including the option of eliminating all faculty evaluations and withdrawing the University's involvement with the undergraduate course review guide. Some of the more moderate options considered included:

  • Using an evaluation form focusing on open-ended questions that request constructive feedback for improvements of both faculty teaching and student learning;
  • Instituting a peer review of course delivery including evaluation of course materials;
  • Using outcome measures such as before/after testing, and correlation with career choice and success.

3. Conclusions

We found that there is no conclusive empirical evidence on the contribution of different course evaluation methods to enhanced learning. Yet, students have the right to reliable and valid information on the content, delivery, quality, and value of the courses offered.

Similarly, both faculty and the University administration can benefit from reliable and valid course evaluation feedback. Taking into consideration that course/faculty teaching evaluation processes have three different audiences-students, faculty, and the University-it should be of great concern that the current course evaluation system does not meet the needs of these groups.

We conclude that no single evaluation process can meet the needs of the three audiences and therefore recommend that separate (but interrelated) processes should be developed to address the needs of the three audiences.

Given differences among schools, the evaluation processes in different schools and departments are different. Each school is therefore encouraged to develop their own processes for evaluation, ideally guided by the principles outlined here and by the experiences of other schools.

4. Recommendations

We recommend that:

a. The Provost establish a committee representing the twelve schools to evaluate the current course/faculty teaching evaluation process. This committee should issue a report considering the issues presented above.

b. A University "clearinghouse" be created to accumulate, analyze and disseminate the experiences of the various schools and departments in various teaching/course evaluation processes. (This clearinghouse can be linked to a Senate committee for ongoing evaluation of the teaching evaluation process.)

c. The needs of the three audience groups be addressed in future evaluation systems.

d. We recommend that each school or department examine the possible establishment of a peer mentoring system that includes evaluation of course materials and measures of students' performance, i.e., the actual learning. This system can either be voluntary or required for faculty whose performance is deemed to be unsatisfactory.

e. Develop a Website with information about all course offerings.

f. Additional mechanisms for providing feedback to the faculty be considered such as periodic feedback for each class, and direct interactions with students similar to "Quality Circles" regarding the course and ways of improving it.

We believe that, if adopted, these recommendations will advance development of more reliable and valid measures of faculty teaching and their courses. This will be of value to students, faculty, and administration. It will further enhance a culture where students take responsibility for and, together with the faculty, are accountable for the quality of learning.

 

Submitted by the Faculty Senate Committee on Administration
Louis A. Girifalco (materials sci & engr), Chair
Frank Goodman (law)
Abba M. Krieger (statistics)
Joan Mollman (neurology/med)
Cynthia Scalzi (nursing)
Henry Teune (political science)
Jerry Wind (marketing)

ex officio:
Senate Chair Vivian C. Seltzer (social work)
Senate Chair-elect John C. Keene (city & reg plng)

Subcommittees

Subcommittee on Cost Containment
Solomon.R. Pollack (bioengr), Chair
Abba M. Krieger (statistics)
Louis A. Girifalco (materials sci & engr)

Subcommitee on Strategic Planning
Frank Goodman (law), Chair
Gregory S. Kopf (ob-gyn)
Cynthia Scalzi (nursing)
Neil Shubin (biology)

Subcommittee on Service to the University
Henry Teune (political science), Chair
Peter J. Freyd (mathematics)
Gino C. Segrè(physics)

Subcommittee on Teaching Evaluations
Jerry Wind (marketing), Chair
Martin Pring (physiology/med)
Christopher Looby (English)


Almanac, Vol. 44, No. 30, April 21, 1998

 FRONT PAGE | CONTENTS | JOB-OPS | CRIMESTATS | BETWEEN ISSUES | APRIL at PENN