Whiteboards of the OAI web strategy

I say “web strategy” in the title rather than “web site” because the concept is a “red” section (photo of left portion of white board) that is the branded OAI website hosted by UnivPubs CMS. The Red Section, (nominally) a
one page site, provides links off-site to the Assessment Innovation “blue” “social” site where there is a membership feature, with the goal of developing a community space. A “Green” section (photo of right portion of
white board) is a public read space. Its content is muddled in this drawing, including both 3rd party blogs and RainKing Chronicles and other fully public read OAI content. Added later in out thinking an “orange” section below the
green one has read restricted OAI content (eg projects being worked with programs)

The other three images are from conversations with Theron and Joshua about the MRG reading from John Seely Brown Minds on Fire

The challenge is to figure out how to have JSB’s “legitimate peripheral participation” and a “social” site (a la Educause’s Ning site) and still have the “professional” qualities of a site linked to OAI. (See Data Flows photo)

Data flows to red and blue zones

The issue is, how to get the “authority” voices to take more central places without squashing the “novice” voices on the periphery.


Contexts (groups)

The reason to work on this I’m still framing, but it involves developing a “system of assessment” for NWCCU that I think requires a “system” of professional development for OAI and Liaisons as well as a “system” of sharing WSU work on the problem of Learning Outcomes assessment in ways that can gather help on the problem from distributed collaborators.

The model that we were looking at in these drawings is Huffington Post more than the old CTLT Common Underground page.

Later, Nils wrote this blog post trying to think thru the”Blue Zone” part of this strategy discussion.

From Student Feedback to University Accreditation

This post supports a free TLT Group webinar
The Harvesting Gradebook: From Student Feedback to University Accreditation

The event is now available as an archive made Friday, September 25, 2009  2:00 pm (ET)

Theron DesRosier, Jayme Jacobson, Nils Peterson, & Gary Brown
Office of Assessment and Innovation, Washington State University

This webinar is an extension of our previous thinking “Learning from the Transformative Gradebook.” Prior to (or following) the session, participants are invited to review a previous session and demonstration of these techniques being applied at the course level

During the session, participants will be invited to pilot our Assessment of Assessment rubric on a program-level accreditation report and to discuss the broader implications of the strategies proposed.

This hour long session will:

1. Review WSU’s model implementations of the Harvesting Gradebook that can be used to gather feedback on student work and the assignments that prompted it. (Background on Harvesting Gradebook)

2. Show how data from harvested assessments at the course level can flow up, providing direct evidence of learning outcomes at the program, college and university levels.

3. Demonstrate a systemic assessment process that applies a common assessment of assessment rubric across the all university’s assessment activities

4. Invite the audience to provide feedback on the Assessment of Assessment rubric by assessing an accreditation report. The goals of the hands-on activity are to:

  1. Gather feedback on the rubric
  2. Demonstrate time effective means of gathering feedback from a diverse community on assessment activities

5. New perspective on  Curricular Mapping, using harvested data.

Further Reading

The Prezi document used in the session (requires Adobe Flash).

Harvesting Gradebook in Production: We have been investigating the issues on the WSU campus surrounding taking the harvesting gradebook into production. While all the integrations with WSU Student Information Systems are not in place yet, we can see a path that automates moving student enrollments from the registrar to create a harvesting survey and moving the numeric scores from the survey back to the instructor where they might combine with other course scores to create the final grade that can be uploaded to the Registrar. A mostly automated pilot is being implemented Fall 2009.

Student Evaluations of Program Outcomes: The presentation references the idea of using student course evaluations to gather indirect evidence on the course’s achievement of the program’s learning outcomes. For several years, WSU’s College of Agriculture Human and Natural Resource Sciences (CAHNRS) has used a course evaluation college wide that asks students their perception of how much the course helped them developing skills in: thinking critically, writing, speaking, working on a team and other dimensions that align with university learning goals. We have explored gathering the faculty goals related to these skills and comparing them to the student perceptions.  To date, this data has not been systematically rolled up or used as evidence in accreditation self-studies.

Getting started with transformative assessment university-wide

Moving a university from a compliance mode of accreditation assessment to a transformative mode is a complex task, yet it is the task brought on by changing requirements of accrediting bodies. To get there, the university (viewed as a collection of learners) needs some scaffolding and some easy place to get started.

Background to the problem

Washington State University is accredited by the Northwest Commission on Colleges and Universities (NWCCU). The NWCCU is engaged in a process to review its standards. The process includes drafting some new standards and converting the review from a decennial calendar to a new septennial review schedule and adding a new catalog of types of reports that institutions must produce.

Regarding these new standards, NWCCU says:

“Standard Three requires evidence of strategic institutional planning that guides planning for the institution as a whole as well as planning for its core themes. Much like the current accreditation model, Standard Four requires assessment of effectiveness and use of results for improvement. However, unlike the current accreditation model, assessments are conducted with respect to the institution’s core themes, rather than its major functions.” [emphasis added]

It goes on to say:

“Goals and intended outcomes, with assessable indicators of achievement, are identified and published for [the institution’s] mission and for each of [its] core themes…

“A core theme is a major concentration, focus, or area of emphasis within the institution’s mission. Examples include, but are not limited to: Developmental education; workforce preparation; lower division transfer education; baccalaureate education; graduate preparation for professional practice; graduate preparation for scholarship and research; service; spiritual formation; student life; preservation of values and culture; personal enrichment; continuing education; academic scholarship; and research to discover, expand, or apply knowledge.”

We can assume that WSU will develop several core themes related to student learning, such as “undergraduate preparation for advanced study and professional careers” and “graduate and professional preparation for scholarship and research.”

NWCCU’s calendar would appear to require the University to provide a Year 1 report in 2011 that answers these points:

Section II: Core Themes

For each Core Theme: [Maximum of three (3) pages per theme]

a. Descriptive Title
b. Goals or Intended Outcomes for the Core Theme
c. Indicators of Achievement of the Core Theme’s Goals or Intended Outcomes
d. Rationale as to Why the Indicators are Assessable and Meaningful Measures of Achievement of the Core Theme’s Goals or Intended Outcomes

For the WSU core themes tied to student learning outcomes, the university will need assessments conducted with respect to the goals of its themes – implemented in ways that can be “rolled up” from program to college to university levels. WSU may elect to use something like its Six Learning Goals as the intended outcomes for its core themes that deal with student learning.

The Problem Statement

The challenge is to help programs move toward having and using indicators of achievement of WSU’s chosen outcomes, and to do so in a way that helps programs and colleges use the data to improve, rather than just performing a compliance activity (that is, developing a transformative approach to their assessment). The further challenge is to accomplish this in a resource-constrained environment.

Theron and I have been looking for a place that all ~100 undergraduate programs at WSU could start working toward meeting these requirements in time for a 2011 delivery date of a Year 1 report.

What we describe here is based around WSU’s goals, but the concept will likely work equally well for another campus with a different set of institutional learning goals

The Strategy

Figure 1 is our whiteboard of the concept. It starts with the idea of collecting sample assignments from each program (along with some metadata about each assignment) in order to provide feedback about the assignments to instructors and the program from their community of “critical friends.”  This process is intended to provide us baseline information. The baseline is in two forms: an assessment practice that programs can build on and data about the university’s teaching practices, such as the types of assignments used in programs and the kind of feedback (beyond grades) that the assignments can provide to learners, along with some demographics about the assignments. (If programs want to do something else, or something more, they can, see below.)

Figure 1. A brainstormed diagram of the kinds of metadata that would need to be collected with each assignment and the graphical analysis that could be done with the data.

Figure 1. A brainstormed diagram of the kinds of metadata that would need to be collected with each assignment and the graphical analysis that could be done with the data.

Here are some benefits that we see:

  • A simple message to communicate.

“Give us 3 assignments, we’ll help you get feedback to improve the assignments AND to meet NWCCU requirements”. (Provost to deans, deans to chairs, chairs to faculty, CTLT staff to WSU community. ) A simple message is less likely to get confused and become a “telephone game” nightmare.

  • A manageable and understandable process.

Every program has assignments. They are easily collected and can be assessed online using Harvesting techniques. Its not everything that might be included in program-level outcomes assessment, but it’s a starting point

  • Feedback from communities.

Specifically, communities important to faculty and instructors.

  • A common reference point.

With most programs doing the same thing, WSU can develop opportunities for shared models, resources and interdisciplinary partnerships.

  • Feedback from a broad range of stakeholders.

Assignments are artifacts that impact many people (downstream faculty, community, industry).  Those stakeholders can answer questions such as, “Does this assignment prepare students for your course? For your workplace? For life outside of the university? How would you improve it to meet your context?”

WSU’s Center for Teaching Learning and Technology has previously done work to map assignments to WSU’s Six Learning Goals using this proposed form (developed with the WSU Honors College): and in this case study with a academic program.

Or the mapping to WSU’s goals could be implemented more indirectly by scoring the assignment with WSU’s Critical and Integrating Thinking Rubric and then mapping the rubric to WSU’s six goals. If the program already has a rubric that is has been using, that rubric could be incorporated into the process and mapped to the WSU goals, see figure 2.

Figure 2.  Diagram of the mapping process from the Food Science rubric to the WSU Six goals and the the University of Idaho (UI) five goals. If the University changes its goals, the mapping can be readily changed, as illustrated by this joint WSU-UI program that is mapping its rubric to WSU's 6 goals and UI's 5 goals simultaneously.

Figure 2. Diagram of the mapping process from the Food Science rubric to the WSU Six goals and the the University of Idaho (UI) five goals. If the University changes its goals, the mapping can be readily changed, as illustrated by this joint WSU-UI program that is mapping its rubric to WSU's 6 goals and UI's 5 goals simultaneously.

The rating itself could be managed in a manner similar to the one we demonstrated to harvest feedback on an assignment, figure 3. Unlike that demonstration, in the pilot year of this proposed assessment plan, programs might only be asked  to provide sample assignments and not the associated student work. In subsequent years, programs may elect to use the full harvesting model, or may elect to use other assessments to provide triangulation to this approach.

Figure 3. Harvesting Feedback process gives results to the instructor (about the assignment) and to the program (about the assignments in aggregate and about the utility of the rubric used).

Figure 3. Harvesting Feedback process gives results to the instructor (about the assignment) and to the program (about the assignments in aggregate and about the utility of the rubric used).

Summary of the Process

1. Assessment for improving, not just proving:

  1. Academic programs deposit 3 sample assignments and fill in a form describing the courses where the assignments are given.
  2. The assignments are scored with a rubric (based on the WSU’s Goals, or the Critical Thinking Rubric, or a rubric that the program already uses to assess student work. The program can choose the rubric.) The program can help recruit the assessors from communities that have interest in the program’s success.
  3. The rubric is mapped to WSU’s chosen goals so the results can be rolled up from Program to College to University levels.
  4. Programs examine the feedback from the assessments, engage in conversations about it and choose action plans for the coming year; instructors can get specific feedback about their assignment to engage in SOTL or other process improvement.
  5. The whole process is assessed with a transformative assessment rubric to judge the quality of the assessment activity and suggest refinements in the assessment practice.

2. Or, if a program already has an assessment procedure (perhaps required by another accrediting body):

  1. The results of that assessment can be submitted, along with a mapping of the relevant data to WSU’s Goals.
  2. The assessment process is assessed, as above, to judge the quality of the assessment activity and to benchmark the program against all other WSU programs.

By using the Harvesting process to implement both the assessment of the assignment and the assessment of the assessments, it should be fairly simple to gather input about the program from a community that is relevant to, and interested in, the academic program’s success and to document the success of the assessment efforts.

Once programs have begun to engage in the discussions that we think these activities will trigger, they may elect to proceed in many directions that include: broadening the scope their assessment activities, refining the assignments or the criteria for assessing them, expanding the communities giving feedback, or assessing student work along with the assignments to gage how effective the assignments really are in impacting student learning.


We have proposed programs get started toward the new accreditation standards by harvesting assessments on their assignments, but this is not the only place to put a toe into the water. Syllabi or student work would be another place to start. Ultimately, we think programs should have direct measures of both student learning and faculty learning, and be able to talk about action plans related to improving that learning and/ or improving the assessment of that learning.