Examining the quality of our assessment system

With most of the 59 programs rated for this round, we are beginning an analysis of our system of assessment.

To re-cap, we drafted a rubric about a year ago and tested it with Honors college self-study and a made-up Dept of Rocket Science self-study. We revised the rubric with discussions among staff and some external input. In December we used the rubric to rate reports on 3 of 4 dimensions (leaving off action plan in the first round). Based on observations in the December round, the rubric was revised in mid-spring 2010.

We tested the new rubric at a state-wide assessment conference workshop in late April, using a program’s report from December. The group’s ratings agreed pretty well with our staff’s (data previously blogged).

The May-August versions of the rubric are nearly identical, with only some nuance changes based on the May experiences.

The figure below is a study of the ratings of OAI staff on each of 4 rubric dimensions. It reports the absolute value of difference of the ratings for each pair of raters — a measure of the inter-rater agreement. We conclude that our ratings are in high agreement [a 54% are 0.5 point or closer agreement (85/156); 83% are 1.0 point or closer]. We also observe that the character of the distribution of agreement is similar across all four of the rubric dimensions.

Calendar for the last month before NWCCU Report

This image shows more details of our understanding of the last 30 days before the NWCCU report on 10.15.2010. A year ago we “guessed” the date at 10.10.10, which still shows in the figure.

Late reports have come in around the 9.17 deadline, driven by President/Provost goal of getting to 100% reporting. The report (or at least the exec summary) is going thru Provost to Regents, hence the 9.17 deadline before the Sept Regents meeting.  Another post in this chronicle gives more details of the web reporting that needs to be accomplished in the remaining weeks.

Planning for the Assessment Report 2009-2010 Web Presence

Planning for the Assessment Report 2009-2010 Web Presence Activity underway to modify Accreditation.wsu, OAI website, and University Portfolio to reflect our understandings of the needs for the online portions of the NWCCU reporting and public face.

At Accreditation we seek a link. OAI serves as our “news” page. Its contents evolve over time. UP home needs to be pretty durable over time, an introduction to the concept that there is a growing portfolio of evidence.  2009-10 tab in UP will soon become locked, a historic record of the year past. A 2010-11 tab will open up for the dynamic materials.

The image shows a variety of elements that are needed to populate 2009-10.

—— Forwarded Message
From: Joshua Yeidel
Date: Fri, 24 Sep 2010 18:49:15 -0700
To: Nils Peterson , Jayme Jacobson , Corinna Lo , Gary Brown
Subject: Plan for Assessment Report 2009-2010 Web Presence

I have created a Google Doc with prose to accompany the photo Nils kindly made of the whiteboard from Thursday’s meeting.  The URL was sent to your Gmail account.  Please review and correct/update.  I tried to record the assignments as I understood them (in bold)j — since the time frame is so short (preview is due for review by higher-ups in 10 working days), it is important that if you are NOT the right assignee, you let everyone know.

— Joshua

attached PDF Planfor2009-2010AssessmentReportWeb

—— End of Forwarded Message

Comment at end of original post

Atttachments

Edit

Note that a photo of the whiteboard from the meeting and the “prose” plan are attached above.

at 9/27/2010 1:10 PM

P2PU Assessment Workshop

Philipp Schmidt of P2PU was attending the HASTAC P3 meeting last week. He’s hosting a meeting (P2PU Assessment Workshop) next week at Carnegie Foundation for the Advancement of Teaching, Palo Alto. CA.  The meeting’s agenda is evolving in the wiki in the link above.  Theron DesRosier will be attending with me.

I have one key question about this meeting, regarding the assumptions and goals of the organizers: Are we trying to enable learning and learners or are we trying to build a university? (Theron and I are hoping for the former.)

Which makes me think of Terry Wassell interesting thought in response to a discussion in Dave Cormier’s blog post “An emerging model for open courses” :

@Scott Leslie. Thanks for your comment on the language of ‘courses’, or in my case ‘modules’. It has helped me realise that my approach to open education post my looming retirement may be trapped in the wrong mindset. I have been trying to think of how I can convert a module I teach at Leeds Uni that dies when I retire to an OE resource ‘in the wild’. I have been thinking about how it can be packaged as an OE module that a community of network of open learners can engage with and exploit/re-purpose according to individual and collective needs. I assumed that I and others would somehow organically become mentors (open tutors?) and flexibly help out as required. Perhaps I should be trying to develop links with existing communities engages in discussions and project around the discipline of my module and try and contribute there somehow. I think your comment illustrates the difficult transition in moving between open education as content (based on a formal education model) and open education as process that engages disparate audiences with varied agendas and objectives. [emphasis added]

Another perspective related to this discussion can be found in Stephen Downes’ 2007 post “Open Source Assessment.”   To Stephen’s thinking, I would add the thought that the assessment needs to be conducted in public (in contrast to a typical university process where student assessment is private between student and teacher). Think instead of the example of a master class in violin, where multiple learners are observing the dialog between teacher and student.

The important aspect of conducting the assessment in public, using public criteria, is that the community can learn from the experience as ‘legitimate peripheral participants.’

So, returning to Terry’s question, I’m wondering if Terry can create an open assessment of the learning in his modules. Terry might then post his content and the assessment where it can be accessed by the community.