Structure of Assessment Activities

As with the University’s recruitment strategies, assessment of undergraduate retention, graduation rates, and student satisfaction is carried out at the campus or school level, with central oversight provided by the Office of the Provost. In addition to regular central review and analysis by the Office of the Provost, members of the Council of Deans—which includes the presidents of the regional campuses—regularly review data related to retention, graduation, and students atisfaction rates; the Council of Deans also provides an ad hoc opportunity for senior academic administrators to share lessons learned regarding the success or failure of related initiatives.

On each of the five campuses, efforts to improve the undergraduate experience are coordinated through committees that include representation from all of the major units contributing to the undergraduate experience. These committees are responsible for identifying areas for improvement, recommending (and implementing) strategies to advance these goals, and assessing progress toward these goals. The committees also periodically review assessment processes and tools for effectiveness.

On the Pittsburgh campus, the Enrollment Management Committee (EMC) regularly reviews retention, graduation, and satisfaction data to monitor progress and to make recommendations for improving the student experience. EMC is co chaired by the vice provost and dean of students and by the vice provost for undergraduate studies. Members include the associate deans of the undergraduate schools; the director of the Office of Admissions and Financial Aid; the University registrar; and representatives from housing, residence life, and institutional research. Similarly, each regional campus has a presidential-level enrollment management committee focused on retention and graduation goals, with assessment making up a regular part of the committee’s agenda activities throughout the year. The committees are as follows: Enrollment Planning Task Force (Bradford), Advisory Committee on Enrollment (Greensburg), Enrollment Management and First Year Experience Task Force (Johnstown), and President’s Task Forcefor Enrollment Management (Titusville). Individual schools and campuses are held accountable for their efforts through annual reports to the Office of the Provost on strategic planning and progress as well as ad hoc reports.

Measures Used to Assess Retention, Graduation, Student Satisfaction, and Student Placement

Efforts to improve the undergraduate experience on each campus are assessed using data on retention, graduation, student satisfaction, and student placement. These data are collected through a combination of institutional data sources and student surveys discussed below. These data are used to identify areas for improvement and guide strategic planning and investments related to the student experience, to evaluate the effectiveness of these strategies, and to assess progress toward goals in an ongoing manner.

The primary measures used to assess progress in improving the student experience are first-to-second year retention rates, four-year and six-year graduation rates, and student satisfaction (overall, academic, and social). Student post graduation placements also are used to gauge success. Specific goals for retention, graduation, and student satisfaction are established through examinations of the current and past levels and, when possible, similar measures at peer and aspirational peer institutions. Progress toward these goals is assessed relative to past performance and to progress made at peer institutions, and goals are modified over time in response to progress. For example, in the late 1990s, the goal was to achieve peer averages for retention and for four-year and six-year graduation rates; annual goals were established with this longer-term goal in mind. On the Pittsburgh campus, these goals were achieved in the early 2000s, and achieving the median for aspirational peers became the new goal. The most recent benchmarks indicate significant progress toward that goal (see Figures10, 11, and 12). Similarly, early discussions of student satisfaction on the Pittsburgh campus centered on the percentage of students reporting being either “satisfied” or “very satisfied.” As this combined measure started to exceed 95 percent, the focus shifted to the percent of students who report to be “very satisfied.”

Data Sources

Data sources are described briefly below; all referenced reports and documents are available for review in the document room.

Retention and Graduation Rates

Retention and graduation rates are derived directly from institutional data and are reported annually in the campus-level Freshman Retention Report and Graduation Rates Report. Retention and graduation rates by race, gender, residency, and school also are monitored and included in these same reports. Since 2002, comparative data on these measures at other institutions has been collected from the Consortium for Student Retention Data Exchange and reported annually in the University Benchmarks report.

Student Surveys

Each campus also has in place a system for assessing student satisfaction with various aspects of the student experience and using the results to guide strategic planning and investments. Student satisfaction is gauged through a combination of home grown campus-level surveys, national surveys, and focus groups. The standard undergraduate student surveys used on the Pittsburgh campus are listed in Figure 13. See Appendix C17 for all student surveys.

Incoming Freshmen Survey

Information about the experiences and aspirations of incoming freshmen is collected, on the Pittsburgh campus, through the CIRP Freshman Survey, a national survey administered by the Higher Education Research Institute at the University of California, Los Angeles. Student responses to this survey are used to assess the admissions process; to establish baselines for some measures used to assess student growth and development; and as control variables formultivariate analyses of student retention, graduation, and satisfaction. Survey results are reported annually by the Office of Institutional Research and parsed and sent to the schools that admit freshmen.

Student Satisfaction Surveys

Several surveys are administered to enrolled students. Each campus has its own institution-specific survey(s) used to assess student satisfaction.

UCSUR Survey

On the Pittsburgh campus, this is a cohort based survey administered and analyzed by researchers in the University Center for Social and Urban Research (UCSUR). This survey/study began in 1997 with a random sampling of about 1,000 freshmen. Similar cohorts were chosen from subsequent freshman classes, and students in each cohort were surveyed in the spring of their freshman, sophomore, and junior years. Over time, the survey has transitioned from a phone-based to a Web-based survey and has been revised to include new questions and remove some that were not useful. However, the core questions and general structure of the survey have remained the same.

Student responses to satisfaction surveys are used to assess year-to-year improvements in overall satisfaction and satisfaction with the academic experience, social experience, facilities, and services. Trends over time and within specifics ubgroups such as class (freshman, sophomore, or junior), year of attendance, gender, race, residency, SAT scores, and school also are monitored. The UCSUR survey on the Pittsburgh campus also forms the basis of the comprehensive statistical studies of student retention and satisfaction discussed in Appendix C18. Annual survey results are provided to the Provost’s office and the Enrollment Management Committee.

Leavers Survey

Complementing the UCSUR Student Satisfaction Survey is a phone survey administered by UCSUR to all non returning students in the fall and spring of each year. Information from this survey is combined with information from those who do return to provide a more balanced view of student perceptions of the University. Annual results are reported to the Enrollment Management Committee.

Collegiate Learning Assessment (CLA)

CLA is administered to 100 students in the fall of their freshman year and another 100 students in the spring of their senior year with questions aimed at assessing student engagement with the academic experience both inside and outside the classroom. These data allow campuses to benchmark student satisfaction against other institutions. Results are reported to the Enrollment Management Committee.

Student Experience in the Research University (SERU)

In 2009, the Pittsburgh campus joined a consortium of public Association of American Universities institutions in administering a survey designed specifically for undergraduates at research universities. The SERU survey allows comparisons of student satisfaction and academic and social experiences with those atother participating institutions, including major to-major comparisons. The University replaced the previously used National Survey of Student Engagement with SERU to better align its external survey instrument with its institutional goals and aspirations. Results are reported to the Enrollment Management Committee.

The results of the SERU survey and the previously used NSSE have been used to help formulate goals related to student satisfaction and to assess progress toward these goals relative to peers. The SERU survey also provides feedback to individual departments on various aspects of their offerings, including student perceptions of faculty engagement, advising, and quality of program. Because many aspects of the academic experience are offered at the program level, this feedback promises to be useful in closing the loop between student satisfaction and programmatic development.

Graduating Senior, Recent Graduate, Alumni, and Employer Surveys

Campuses also administer their own senior surveys. These surveys usually are administered to students as they apply for graduation and typically focus on a few key aspects of the undergraduate experience and postgraduation plans. Alumni surveys also are used to collect information on student outcomes, including employment and education, as well as retrospective impressions that can be used as part of Pitt’s overall assessment of the undergraduate experience. Standard benchmarks include the percentage of graduates who go on to graduate or professional school, the percentage who enter the workforce within six months of graduating, and the percentage who believe that their University experience prepared them for their chosen career.

Data are reported annually (to the committees and councils whose work is influenced by the information) for the Pittsburgh campus in institutional research reports of the Senior Survey results and the Alumni Survey, which surveys former Pitt students two, five, and 10 years after graduation. Post graduation surveys have been conducted since the mid-1990s, and in 2008, the Office of Institutional Research and the Office of Student Employment and Placement Assistance (SEPA) collaborated to conduct a joint post graduation survey to collect placement as well as employer information from each graduating class. SEPA also works directly with employers to organize career fairs, networking events, and on-campus recruiting and conducts employer satisfaction surveys following all special events. These surveys also ask employers to provide feedback on the students they are recruiting at Pitt, including the quality of the candidate pool.

Groups and Expert Consultants

To supplement the data collection efforts discussed above, campuses also use focus groups to assess and explore specific topics and programs. Sample focus group reports are available in the document room. Campuses also engage expert consultants such as Noel-Levitz, Inc., and Eduventures, Inc., as well as individual experts to conduct targeted assessments of various aspects of the undergraduate experience. The University also has made good use of best practices reports from the University Leadership Council.