Developing WSU’s Guide to Effective Student Learning Outcomes Assessment Rubric

The table below charts the steps that Washington State University’s Office of Assessment and Innovation (OAI) and its stakeholders went through to develop the Guide to Effective Program Assessment rubric used in the WSU System of Student Learning Outcomes Assessment.

Description Generalized Model Specific example of application at WSU’s Office of Assessment and Innovation.
Determine the scope of the program assessment initiative. Solicit stakeholder input about the purpose of the assessment. Determine which aspects of program assessment will be most useful for the institution to assess Gary met with provosts and assistant deans to frame the general parameters.
Develop a framework for assessing assessment at the meta-level. Research existing literature (and tools) to begin delineating rubric dimensions. Developed a framework for the rubric based on the WASC Evidence Guide, writings from Peter Ewell and others, and the Transformative Assessment Rubric, an EDUCAUSE Learning Initiative project to evaluate the responsiveness of assessment plans.
Flesh out criteria for each dimension. October 2009 Begin drafting rubric criteria, periodically reviewing with stakeholders and revising as needed. Flesh out rubric dimensions with specific performance criteria share with the Assessment Liaison Council and external reviewers, getting feedback about what was clear, what was not, and how much detail was useful. (Guide to Assessment Rubric (Oct 2009))
Test a draft rubric November 2009 Solicit program assessment plans for low-stakes review. Solicit a program’s report written for WSU’s 2009 NWCCU accreditation self-study and test the rubric with OAI and external reviewers.
Pilot an initial assessment cycle Dec 2009 (ratings done Dec 2009-Feb 2010) Solicit program assessment plans for formative review. Norm raters and launch and initial review. Return rubric-based feedback and scores to programs and report program scores to the institution. Via program liaisons, all WSU undergraduate programs were required to submit a first draft of an assessment self-study by Dec 18, 2009. Programs were given a template with areas for each rubric dimension. In the first cycle, only three of the four dimensions were required. Reviewers participated in a norming session but in the initial phase all scores were reconciled if they were more than a point apart or if there was a split at the “competency” level. Occasionally, a 3rd rater was required. Assessment plans were scored with the rubric but it was emphasized that this was an initial and provisional rating. Guide to Assessment (Dec 2009)
Revise rubric/ assessment process. February-March 2010 Solicit feedback from programs as well as reviewers about what worked and what didn’t. Revise rubric and assessment process to be more useful and efficient. The rubric was revised based on feedback captured from notes reviewers made as they were using the rubric as well as feedback from programs via an online survey. Informal observations from programs to their OAI contacts were also included. The 4 dimensions remained essentially the same but number of levels, level names and wording of rubric criteria changed considerably. All wording was changed in a positive format (what was happening versus what was missing) and a definition of terms was added as a cover page. The 6 point scale remained the same.
Test rubric April May 2010 Solicit feedback from stakeholders in the process of using the rubric on a sample report The revised rubric was tested internally by rating a report written for the December 2009 cycle. The rubric and report were also used by an audience at a regional assessment conference. Despite not norming together the internal and external reviewers agreed fairly closely.
Launch a second assessment cycle. May-August 2010 Solicit program assessment plans. Norm raters on revised rubric and begin review process. Return rubric-based feedback and scores to programs and report program scores to the institution. Programs were required to submit a second draft of an assessment self-study by May 17, 2010 (with an option to delay to August 20). They used a similar template with areas for each rubric dimension. In the second cycle, all four dimensions were required. Reviewers participated in an extensive norming session over several days and the rubric was tweaked slightly.Guide to Effective Program Assessment Rubric (May 2010) and the slightly revised version Guide to Effective Program Learning Outcomes Assessment Rubric (August 2010)
October – December 2010 Review quantitative and qualitative evidence of the review process Studies of interrater agreement were conducted along with collection of observations from using the rubric (The scoring tool had a place for reviewers to make comments about the rubric in the context of a specific rating effort.). These were used to begin framing the next revision of the rubric and template. 

OAI Interrater analysis (version 1) excel datasheet

Draft next revision of rubric and report template (Halted Dec 2010) Review literature and data from previous uses of rubric, look for patterns of rater disagreement. OAI staff began examining their experiences and the kinds of inter-rater disagreements, reviewed literature for key performance criteria, examined notes left by reviewers as they used the rubric May-Sept. The resulting notes were intended as input to the next revision.Guide to Effective Program Learning Outcomes Assessment Rubric (Dec 2010 notes) In addition, the template programs used to complete the report was revised to better prompt the writing. Program Learning Outcomes Assessment Template Revision Dec 2010

Rubric revision process

With the NWCCU report filed we turned to some assessment of our inter-rater reliability, viewing that as an opportunity to understand what might be differences in our shared understandings. These may be rubric elements or interpretations of rubric language.

This google doc is the space where we are looking at that rating data and recording results of a read-aloud norming protocol. https://docs.google.com/document/d/159bJfDiwtIHEc4ZE2a5pE1-Y9TE4Klgkpu7VL54hbXE/mobilebasic?id=159bJfDiwtIHEc4ZE2a5pE1-Y9TE4Klgkpu7VL54hbXE

How a rubric can communicate

OAI has been finishing up its 2009-10 cycle of reviews of program-level assessment, see the University’s portfolio for details about the process and the results.

One of the responses to a program regarding communication with stakeholders had a summary of the utility of a rubric as a communication tool:

Under “Communication” the report states: “Program Objectives and Outcomes will be more extensively discussed with the students in classes to encourage more participation in the assessment and improvement process.”

A programmatic assessment rubric could be a very useful tool to encourage students, and other stakeholders, to participate in the assessment and improvement process. For example a rubric:

  • Provides a reference point for students to consult repeatedly as they monitor their own learning and develop the skills of self-assessment. Students are supported in becoming better judges of quality in their own and others’ work.
  • Supports the development of a sense of shared expectations among students, faculty, staff, and external stakeholders.
  • Provides evaluators and those whose work is being evaluated with rich and detailed descriptions of what is being learned and what is not by facilitating a breaking down of outcomes into dimensions and of dimensions into criteria.
  • Provides criteria to shape and guide students’ engagement with one another and with course content.
  • Promotes a shared understanding among faculty, students, and stakeholders of the program outcomes.

Analysis of Inter-rater agreement 2009-10

Lee,

Thanks for telling me that you completed rating Honors also.

Our average ratings for that program were 5; 5.5; 4.5; 5  so we are a little lower than you, but in the same category “Integrating” in all but one.

You can see all our results here: https://universityportfolio.wsu.edu/2009-2010/Pages/default.aspx

We are exploring two measures of inter-rater reliability, within 1 point and within the same category.

In terms of scores, see the graph, which we think is good. 83% of our scores are within 1 point of each other

Regarding being in the same category, we are not doing as well, it seems that we often come close, but straddle the lines.

What is valuable about you rating 2 programs (one high and one low) is that we can begin to get a sense that you see our measure in the same way that we do.  Another kind of test we need to do is see if outsiders agree with us in the messy middle.

We have more work like this to do with external stakeholders to see how well our tool plays in the wider arenas

Nils

On 10/13/10 4:40 PM, “Lee” wrote:

> Hi Nils,
>
> I sent in my review of Honors.  I gave them all top marks.  Was I right?  They
> struck me as being the Bar we’re all trying to reach!  It’s almost like you
> wrote the review rubric to FIT what they’ve done!?
>
> Lee
> ________________________________
> From: Nils Peterson [nils_peterson@wsu.edu]
> Sent: Tuesday, September 28, 2010 3:47 PM
> To: Lee
> Subject: Another WSU program to review
>
> Lee,
>
> Rather than Business, I’m giving you our Honors program. This is a program
> that has worked with our unit for several years and we know them.
>
> I think you will find it contrasts from Biology’s report in ways that may help
> you exercise more of the rubric’s scale.
>
> Thanks for your interest and help

Examining the quality of our assessment system

With most of the 59 programs rated for this round, we are beginning an analysis of our system of assessment.

To re-cap, we drafted a rubric about a year ago and tested it with Honors college self-study and a made-up Dept of Rocket Science self-study. We revised the rubric with discussions among staff and some external input. In December we used the rubric to rate reports on 3 of 4 dimensions (leaving off action plan in the first round). Based on observations in the December round, the rubric was revised in mid-spring 2010.

We tested the new rubric at a state-wide assessment conference workshop in late April, using a program’s report from December. The group’s ratings agreed pretty well with our staff’s (data previously blogged).

The May-August versions of the rubric are nearly identical, with only some nuance changes based on the May experiences.

The figure below is a study of the ratings of OAI staff on each of 4 rubric dimensions. It reports the absolute value of difference of the ratings for each pair of raters — a measure of the inter-rater agreement. We conclude that our ratings are in high agreement [a 54% are 0.5 point or closer agreement (85/156); 83% are 1.0 point or closer]. We also observe that the character of the distribution of agreement is similar across all four of the rubric dimensions.

Norming on the Guide to Assessment Rubric

Results of Norming on the Guide to Assessment rubric This image captures the data from OAI’s norming work on the first 2 dimensions of the May 17 Guide to Assessment rubric. The sample being assessed is the Sociology self-study from December 2009.

The red data are from Gary & Nils’ presentation to the Assessment, Teaching and Learning conference (“Compared to what?”) the end of April in Vancouver. ATL is a statewide conference hosted by the State Board of Community and Technical Colleges. We shared the rubric and the Sociology self study, divided the audience of 40 into 4 groups and had each group rate (un normed) the self-study using the rubric. We captured their data and displayed it in real time as a radar plot. From that effort we recruited several audience members to serve as external raters of the May 17 self studies.

Institutional Self-Assessment Rubric

This post is in support of a TLT webinar, the series is titled “Power of Rubrics
[Archive of session]
Gary Brown, Theron DesRosier, Jayme Jacobson & Nils Peterson, Washington State University

Introduction and Background on the Problem

Washington State University is in the process of responding to changes in by its accrediting body, NWCCU. The response includes the transformation of the former Center for Teaching Learning and Technology (CTLT) to the Office of Assessment and Innovation (OAI).

The University is in the process of developing its response to NWCCU’s changed standards and OAI is helping move the institutional thinking toward one that embeds assessment in ways that help faculty to think about student learning outcomes, and about the processes that programs are using to assess their work on improving outcomes.

This work builds on work of the former CTLT known to as “Harvesting Gradebook.” Previous reports will provide context on using the Harvesting Gradebook with students: AAC&U report Jan 2009 Update Fall 2009. This report links to a webinar archive that paints a picture of how to roll harvesting up, From Student Work to University Accreditation.
Using Harvesting Feedback with Academic Programs

In the previous webinar (From Student Work to University Accreditation) we described a vision for how harvesting could be used to move data from the level of an individual piece of student work up through levels of assessment and reflection to a university-level accreditation report. Presently OAI is engaged in deploying a middle level piece of this vision, the assessment of program-level self studies with an “Assessment of Assessment” rubric. The most current version of the rubric and other materials for the process are linked from grey portion of OAI website banner.

Figure 1. The process involves the academic program collecting evidence, writing a self study, and having the self study assessed with the University’s rubric (Called Guide to Assessment on the OAI website, formerly called Assessment of Assessment rubric). This image shows the process from data sources (upper left, to self study, to rubric-based assessment, to radar graph of results. This diagram represents work on the level of an academic program, a “middle tier” in the vision presented in From Student Work to University Accreditation.

Readers interested in trying the process are invited to do so at the WSU University Portfolio site for 2009-10. The Department of Rocket Science (on the site as of 12/7/09) was created as a sample. Other programs appearing on the site (beginning in January 2010) are actual WSU programs seeking formative feedback. (strikeout March 2010) Contact us if you want.

A Prezi visual of the WSU assessment calendar, provides an interactive picture of the assessment cycle and calendar and will serve as a “Dashboard” for monitoring progress.

Guide to Assessment – Rubric
Because of the wide diversity of programs in the WSU 4-campus system, a one size approach to learning outcomes assessment will not fit all. Consequently, WSU is developing a rubric to assess the self-study plans (short form and long form). Like the AAC&U VALUE project, the WSU rubric assumes that “to achieve a high-quality education for all students, valid assessment data are needed to guide planning, teaching, and improvement.”

The Guide to Assessment is the tool OAI is creating to help programs assess the quality of their student learning outcomes assessment activities. Using the Harvesting mechanism, programs will be able to gather evidence from stakeholders outside the university — a requirement of the accreditor — as well as gathering self-, peer- and OAI reviews.

Short form of the Rubric