Examining the quality of our assessment system

With most of the 59 programs rated for this round, we are beginning an analysis of our system of assessment.

To re-cap, we drafted a rubric about a year ago and tested it with Honors college self-study and a made-up Dept of Rocket Science self-study. We revised the rubric with discussions among staff and some external input. In December we used the rubric to rate reports on 3 of 4 dimensions (leaving off action plan in the first round). Based on observations in the December round, the rubric was revised in mid-spring 2010.

We tested the new rubric at a state-wide assessment conference workshop in late April, using a program’s report from December. The group’s ratings agreed pretty well with our staff’s (data previously blogged).

The May-August versions of the rubric are nearly identical, with only some nuance changes based on the May experiences.

The figure below is a study of the ratings of OAI staff on each of 4 rubric dimensions. It reports the absolute value of difference of the ratings for each pair of raters — a measure of the inter-rater agreement. We conclude that our ratings are in high agreement [a 54% are 0.5 point or closer agreement (85/156); 83% are 1.0 point or closer]. We also observe that the character of the distribution of agreement is similar across all four of the rubric dimensions.

Draft timeline for Assessment Self-study process

Draft timeline for Assessment Self-study process Showing how the self study data feeds in from programs to the template and flows back out from the template to various reporting needs.

This diagram needed to explain system to NWCCU and also to share with Liaisons and Program Points to help them understand that the goal is focus and labor saving.

[We never finished this diagram or communicated it out. Ed.]

Bottlenecks to improving program outcomes assessment

Bottlenecks to improving program outcomes assessment.
Today we had a discussion about bottlenecks (and mis-perceptions) programs seem to have about assessment and an assessment system.  The conversation was based on what we’ve observed in our review of the Dec. 18 self studies.
It serves as input to the review of the near final draft of the version of the rubric being prepared for May 17.

This image is a record of the discussion

As promised, reflection on the retreat, finally

Re: draft note to staff
I was thinking last night that the persistent concern expressed in our still recent retreat was less about job security and more about qualifications in our protean institution.  If I no longer should be doing what I’ve done in the past, what skills do I need to develop to do what I am expected to do?  And how long, one anonymous note asked, before we change yet again?

In my own thinking, I am coming to understand that it is more about the affect than the skill set, though they are not fully separable.  As the CTLT our work was like teaching an elective course in the major.  Most of interactions were with faculty who wanted to work with us.  And now we fear our new charge is not only requiring us to work with those who share fewer of our concerns and values, but we are drifting into work increasingly with those who clearly do not.  We are now like those who teach required courses.  And the required course we teach is not one that many find rewarding.

Our challenge as a CTLT was encouraging participation in our efforts commensurate with the institutions’ investment in us.  As the OAI, our challenge is, with some irony, helping the institution achieve measurable outcomes.  In CTLT we worked with colleagues who were mostly or who became friends.  As an OAI, we must strive to work collegially.

The challenge is ultimately less about our skills.  We are amply qualified.  It is a challenge to our dispositions.  To a person this is not the work we signed on for.  It has a “feel” about it like a garment not cut quite right.  It is frequently somehow oddly irritating.

“Do we still work with the individual faculty we have worked with?”   No, is the answer.  If the work cannot be parlayed to the program level, then the job requires us to extricate ourselves from that work …and, at least in the work context, away from that friendly working relationship.

“Will we change our organization again?” Probably.  The garment may not fit quite right, but the unit will grow into it.

As the CTLT, however, we were unique and recognized internationally for our transformative focus.  Some aspects of that focus were occasionally a source of some contention, but it was a shared understanding we had of our mission.  We were guided by the belief that not only can we but we must do a better job in our institutions of learning at improving the student learning experience and student learning…outcomes.

So here we are.  Be careful what you wish for, the saying goes. Our new vision, rubric, and the harvesting of community input are being followed nationally. We have a tremendous opportunity to promote real, deep, and meaningful change at WSU.

If we can change.

Dr. Gary R. Brown, Director
The Office of Assessment and Innovation
Washington State University
509 335-1352
509 335-1362 (fax)

Wiki Page for the Self-Study Review Process

From: Joshua Yeidel
Date: Thu, 25 Feb 2010 11:40:52 -0800
To: OAI messages
Subject: Self-Study Review Process Wiki Page

OAI folks,
In the course of working through the December Self-Study cycle, we have
developed a lot of informal knowledge about the process of handling
Self-Study Reviews.  Now it is time to capture and preserve that knowledge,
so that we can apply it to the May cycle.

I have started a wiki page with an outline of the process (the steps
identified in the Process Actions list “Next Actions” field).  Please add
what you know about the process steps to this page.


link reproduced here:  Assessment of Assessment Self-Study Review Process – ctltwiki

— Joshua

Planning program highlights for sharing

Planning program highlights for sharing This was a discussion of creating short highlights/ summaries (aka Baseball Cards) for programs with well developed features in their assessment plan, based on the Dec 18 self-study

Re-design of the workflow and web store following Dec 09 cycle

Re-design of the workflow and web store following Dec 09 cycle As the work from assessment and feedback on the Dec 09 self studies comes to a close, the team met to review the (admittedly) ad hoc tools assembled in January. These boards reflect that discussion. Implementation is intended before the May 17 deadline.