Many departments and programs at KU have formally considered how well they are meeting the learning goals they set out for their students. This process is intended to capture the intellectual progress of students in their entire program, not as an evaluation of any individual course or instructor within the department. In this way, departments can identify the areas of their instruction that are effective in achieving their own goals and the areas that would benefit from future attention or development.

Over three years of working with faculty teams from over 30 units, we have identified some key components of the process that help determine how valuable the results are for each unit and for an overall representation of learning at KU as an institution.  While no one of these components is either sufficient or necessary, planning for evaluation of successful learning can be made easier by considering what steps have been taken by units that have found the process useful and valuable.  These are placed roughly in optimal order of occurrence, but each program should proceed in a way that suits local conditions.

1. The department chair and other leadership in the department need to be committed to the process. Leadership buy-in would be demonstrated by the periodic inclusion of this project on faculty meeting agendas (as needed) and in the service structure of the department. When chairs set a reasonable time line for the activities, with target dates, the process is most likely to continue.
   
   
2. In a conversation involving the whole faculty, the department should discuss what they want to know about the intellectual skills, knowledge, and understanding of students as they progress through the program.  This conversation includes recognizing that there will be a distribution of understanding among the students in a program. In many units this has already been accomplished through interaction with accrediting bodies and professional organizations, making the first step straightforward.  To make this work sustainable, the unit should identify and select a handful of the most important program goals, or central features, that a successful student would demonstrate in your department.  The process should focus initially on that subset.
   
   
3. A small committee (generally 2-4 people) should be appointed to articulate the sub-components of each selected goal, including knowledge, skills, and broad understanding. Furthermore, they would begin to describe what beginner, intermediate, and advanced understanding might look like for each component. This group, with the support of the chair, should then bring this structure back to the faculty for feedback and revision.  This framework provides a road map for the process, so it should be constructed with some care.
   
   
4. The same committee would then ask colleagues to identify which of these components are covered in their courses or at critical points in a graduate program.  For each covered component, the faculty member would describe briefly how students have an opportunity to demonstrate beginning, intermediate, or accomplished levels of achievement. The committee would use this information to build a map of how the curriculum or program serves the goals, which should be returned to the faculty for a conversation, as well as revision if needed.
   
   
5. Once the department has a map of the goals and the places in the curriculum where they might be demonstrated, the unit can collect examples of assignments and student work to build an archive for each of the course goals. This archive needs to be representative (with a minimum of ~20 examples, selected randomly), so it should include work from multiple courses and multiple instructors. The samples should be selected from assignments that give students an opportunity to show high level achievement. One possible finding is that some courses may not provide students an opportunity to demonstrate the highest levels of unit goals.
   
   
6. A larger group of department members (possibly including graduate students) constitute a reading committee to identify patterns of student performance in the unit archive. Using the identified criteria in a systematic way, the readers describe how the department’s student work is distributed across the framework of components and achievement levels. Questions are basic and straightforward, such as “What percentage of students meets each level of the goal?” This description is not based on grades from individual courses, but on an independent reading of the level of demonstrated skill and the quality of the student work. The description of department performance should not identify the particular courses or instructors who contributed the archive.  At this point, a summary document should be prepared and presented to the department for consideration.
   
   
7. Using the summary data, the department as a whole should then reflect upon what it tells them about how well they are meeting their goals. Some of the questions a department might ask themselves include: Are we happy with the distributions? Should we change the types of learning tasks that students are asked to complete? Do we need to give students more opportunities to do high quality work? Do we need to revise any of the course structures, do we need different courses, or do we need a different mix of courses?  If all, or a great majority, of the students are excelling at a particular skill, how can we raise the bar and ask more of our students? In this way, the department can periodically examine how well its collective teaching is accomplishing the unit’s own programmatic goals.

With good leadership and support for well-selected faculty members, a version of this sequence can be completed in a single academic year.  More typically it is taking two years for those groups who have completed the cycle and learned from their own observations.  The long-term goal is for each unit to continue recycling the process, perhaps selecting different subsets of goals for various years and/or perhaps designating alternate years for gathering and reflecting on evidence. 

Sometimes faculty members ask when they are done with the process, but the answer is the process is continuous and iterative.  Frequently observing and discussing the quality of students’ work is a part of being a responsible instructional program.  The key to sustainability is to gather a small but random and representative sample of work all the time, while only asking key questions on a subset of goals and in a subset of years.  Over time the pattern of evidence of learning will make sense, so it is better to be selective and steady than to do too much in a given year or every year.

(For a doc version of this information, click here.)