Examples of Using Assessment to Enhance the Student Experience

The Outside the Classroom Curriculum

For the past 15 years, one of the University’s goals has been to better align student experiences inside and outside the classroom in an effort to develop the whole student. First steps in this direction were creating the Enrollment Management Committee with membership from both the academic and student services are as and moving the dean of students into the Provost’s office as a vice provost. Initial efforts also included developing the Pitt Pathway program, which works to align career and academic advising; the First Year Experience program, which engages faculty and staff in orienting students to University life; and the Office of Undergraduate Research, Scholarship, and Creative Activity and similar programs, which promote undergraduate engagement in research, teaching, and service learning. Academic support services were structured to be more closely aligned with the academic units. These and dozens of other programs offered by the academic units and by the Division of Student Affairs helped to support efforts to develop the whole student. By 2006, these efforts had developed to the point where the Council of Deans formally approved a set of goals for Pitt graduates that would be supported by both the academic and student services units (see Figure 16).

During the following years, the vice provost and dean of students led members of the Enrollment Management Committee in a review of the extracurricular programming within the academic units and Student Affairs to determine which of these goals for student development were supported by each program. Programs that did not support any of the goals were revised or discontinued. This alignment of the programs and the goals for student development resulted in the creation of the Outside the Classroom Curriculum (OCC). This program was first introduced in 2007 as a pilot program open only to freshmen. After a comprehensive review and resulting modifications, the program was expanded to all students in 2008. Through this program, students participate in programs and activities appropriate to the stages in their academic careers to develop attributes the University sees as important for their success. Programs are assessed periodically by the OCC oversight committee to ensure that they continue to meet the goals of the curriculum.

A key feature of the OCC program is the OCC transcript, which records student participation in OCC activities (see Figure 17). This electronic transcript is populated when students attending programs or events swipe their ID cards. It serves as a record of participation and completion of the milestones of OCC. Attendance data collected through thes electronic records are used to assess the programs as well as the students. Currently, these data are being linked to the student self-assessment sembedded in ongoing surveys to assess the effectiveness of individual programs in advancing the goals of educating the whole student.

RealWorld Action Program at Pitt–Johnstown

A similar comprehensive program of student engagement was introduced at Pitt–Johnstown in 2009. In response to student surveys and consulting reports that indicated that a lack of student engagement was limiting the progress of some students on its campus, Pitt–Johnstown created the RealWorld Action Program to provide students with an effective structure for developing customized personal and professional action plans. Such efforts to increase student engagement, improve academic advising, and expand academic support have produced dividends with respect to retention rates. First year retention at Pitt–Johnstown (74 percent) now exceeds the national average of four-year institutions (67.6 percent).

Through the RealWorld Action Program, Pitt–Johnstown has increased efforts to educateand assist students in order to develop, evaluate, and implement career and educational plans.This increased outreach to students continually establishes counseling relationships with students. New programs and events strive to provide integrated career support, teaching students to articulate the value of what they study and how their education applies to the workplace.

Evidence of the increased impact of the RealWorld Action Program on campus and in the community is reflected in the fall 2010 numbers. A total of 2,139 students took advantage of various programs and services offered through or in conjunction with the RealWorld Action Program office, reflecting a 72 percent overall percentage of engagement. In comparison to fall 2009, this reflects an increase of 21 percent among students and 61 percent among alumni. Walk-in contacts also showed a 14 percent increase. The current job placement rate for Pitt–Johnstown graduates is 93 percent.

In addition, outreach to employers has been significantly expanded. In fall 2010, Pitt–Johnstown connected with 335 employers, a 47 percent increase in employer engagement and interaction from fall 2009. Much of this increase is attributed to the introduction of the Have You Hired a Pitt–Johnstown Graduate? campaign. Through increased outreach, Pitt–Johnstown has educated local and regional organizations about its services, the quality of its students, and how to match the needs of the employer with the unique talents of its graduates.

Using Assessment to Improve Academic Support Services

Pittsburgh Campus

The 2001 UCSUR study identified first term GPA as the strongest predictor of students’ withdrawing from the Pittsburgh campus among those considered. In response, the University developed a number of strategies to improve academic support services and used a variety of measures to assess success of these programs. The appendices on student satisfaction goals and strategies and on student services (Appendices C19 and C21) include documents detailing some of the programmatic changes made to improve academic support services. These included a complete restructuring of these services in 2003with the creation of the Academic Resource Center and the Math Assistance Center, which, along with the existing Writing Center, brought academic support services fully into the academic units, and the implementation of a new strategic plan for the Academic Resource Center that focused on success for all students in 2005–06. The effectiveness of academic support services is assessed in a variety of ways.

The effectiveness of the Academic Resource Center (ARC) is assessed directly through an annual review of the number of students placed on probation during their first term and the number of these who use the ARC and who subsequently are removed from probation. Student assessments of the effectiveness of academic resources are collected annually through the UCSUR survey, including a special module added in 2009 to gain better insights into student experiences with academic support (Appendix C17 on student surveys). These surveys showed that between 2008 and 2010, student satisfaction with academic support services increased six percentage points. Also in 2009, Noel-Levitz was engaged to conduct a full review of the ARC following the same model that was used to assess the Advising Center one year earlier. This review is nearly complete and a report is forthcoming.

Pitt–Bradford: TRIO Student Support Services

Early in his tenure, the president of Pitt–Bradford established improving student retention as a goal for that campus and established targets based on a review of the campus retention history and that of other similar institutions. This review highlighted the role of demographics in shaping the retention rate on that campus, inparticular the large numbers of first-generationand low-income students (between 35 and40 percent of the students at Pitt–Bradford receive Pell Grants). In response, the campus developed several programs to improve retention and graduation, including the federally funded TRiO Student Support Services program, a program aimed at supporting low-income and first-generation students. The TRiO Student Support Services program provides students with one-on-one academic counseling, a lending library to assist in reducing the burden of buying textbooks, a computer lab with practice software to increase the understanding of subject content, résumé writing support, career and graduate school exploration, and workshops that enhance academic study skills and personal development.

Between 2005 and 2009, this emphasis on enhanced academic support services resulted in a six percentage point increase in retention rates, and the number of students on academic probation fell from 17 to 10.5 percent. Building on the success of the TRiO program, the campus has created a new advising center that consolidates academic support services for the campus’ underserved and most-at-risk students. Success of this center, like that of the TRIO program, will be assessed by monitoring the success of the students by looking at retention and probation rates.

Pitt–Greensburg: MAP Works Program

Similarly, Pitt–Greensburg identified academic support services as an area for improvement based on an assessment of retention rates and support programs. Based on this review, it concluded that one of the reasons that students were not succeeding was that they were not taking advantage of the support services offered on the campus. Further investigation suggested that the students were not always aware that they needed assistance or what the appropriate support would be until it was too late. In response to this assessment, Pitt–Greensburg introduced the Making Achievement Possible (MAP) Works program in fall 2009. The MAP Works program provides a mechanism for surveying new students and identifying those who are experiencing academic and nonacademic difficulties in their adjustment to college life. It also provides a means for faculty and staff to communicate with students, direct them to appropriate services, and help them to establish contact with a mentor early in their college career. With the use of MAP Works, fall-to-spring attrition among first-year students in 2009–10 decreased by 3.3 percentage points to 5.8 percent.

Improving Registration and Access to Student Data: All Campuses

While student satisfaction was increasing throughout the University during the early to mid-2000s, one notable exception was in the area of student registration, a process that involved several different units, including the Office of the University Registrar, academic advisors, and Student Financial Services. Early efforts to improve student satisfaction in this area focused on coordinating these efforts and on customer service. Targeted customer service surveys in the various units indicated significant improvements as a result of these efforts, but student satisfaction with the overall registration process did not improve. In 2005, the University moved to a new student data system, the People Soft Student Information System, which led to the implementation of online student self-registration, and access to class schedules and grades was improved through the University’s enterprise portal. At the same time, there were enhancements to student services, billing, and financial aid and improved advising and academic support (see Appendix C21 on student services). Following these improvements, student satisfaction with the overall registration process improved dramatically, and the number of students on the Pittsburgh campus responding that they were “very satisfied” with the registration process increased 15 percentage points between 2008 and 2010.

Developing a Sense of Belonging: All Campuses

The Student Retention and Satisfaction Study of 2001 identified social integration—helping students to create a strong sense of belonging—as a major factor contributing to student satisfaction and retention on the Pittsburgh campus. Surveys on the regional campuses led to similar conclusions. In response, the University focused increased attention on the first-year experience. Individual schools and campuses developed freshman programs to integrate these students into the Pitt community. Programming was developed to enhance student academic experiences such as PITT ARTS and undergraduate research opportunities, and efforts were made to improve the quality of student life in residence life, campus recreation, and career advising (see Appendix C21 on studentservices). During this period, both student surveys and focus groups indicated that freshmen were feeling more integrated into the life of the University, and the percentage of students on the Pittsburgh campus indicating a strong sense of belonging on the UCSUR survey increased by more than seven percentage points. However, as noted earlier, these gains in student satisfaction with the social experience lagged behind the gains seen in student satisfaction with the academic experience.

A second phase of enhancing freshman integration into the community began in 2007 on the Pittsburgh campus with the introduction of a new set of First Year Experience (FYE) programs intended to create an environment for first‐year students that would better assist them in connecting to the University. The initiatives were designed to engage first-year students in activities outside the classroom, help them to build meaningful relationships with their cohorts, and provide them with every reasonable opportunity to persist into their sophomore year. One of the major components of the FYE program is New Student Orientation, which is designed to provide students with opportunities to connect with other students, to informs tudents and their families of the opportunities available at the University and in the city of Pittsburgh, and to educate students about the mission of the University and their responsibilities within this educational setting.

Several strategies have been implemented to assess the orientation program and make modifications to the program to ensure that the intended outcomes are reached. For example, following every New Student Orientation program, a student evaluation is administered to collect data on several key factors of the program, including participant satisfaction; learning outcomes; and participation levels for all programs, events, and activities. Data collected from the evaluations are reviewed by the orientation planning team and the associate dean of students and director of the Office of Student Life. Over the last five years, decisions have been made to either enhance or eradicate orientation programs and services to reach intended outcomes (see Appendix C20 on first-year retention).

The regional campuses also have invested in the development of FYE programs to improve student retention by successfully integrating new students into the campus community.  Regional campuses have devoted resources to achieving better advising and academic support, more attractive residential experiences, and greater availability of recreational facilities. All of the regional campuses have worked to more effectively integrate academic affairs and student life efforts through programming such as learning communities and to expand experiential opportunities for students in leadership, study abroad, internships, and research. They also have developed methods of assessing students’ responses to these programs.