Analysis of Inter-rater agreement 2009-10

Lee,

Thanks for telling me that you completed rating Honors also.

Our average ratings for that program were 5; 5.5; 4.5; 5  so we are a little lower than you, but in the same category “Integrating” in all but one.

You can see all our results here: https://universityportfolio.wsu.edu/2009-2010/Pages/default.aspx

We are exploring two measures of inter-rater reliability, within 1 point and within the same category.

In terms of scores, see the graph, which we think is good. 83% of our scores are within 1 point of each other

Regarding being in the same category, we are not doing as well, it seems that we often come close, but straddle the lines.

What is valuable about you rating 2 programs (one high and one low) is that we can begin to get a sense that you see our measure in the same way that we do.  Another kind of test we need to do is see if outsiders agree with us in the messy middle.

We have more work like this to do with external stakeholders to see how well our tool plays in the wider arenas

Nils

On 10/13/10 4:40 PM, “Lee” wrote:

> Hi Nils,
>
> I sent in my review of Honors.  I gave them all top marks.  Was I right?  They
> struck me as being the Bar we’re all trying to reach!  It’s almost like you
> wrote the review rubric to FIT what they’ve done!?
>
> Lee
> ________________________________
> From: Nils Peterson [nils_peterson@wsu.edu]
> Sent: Tuesday, September 28, 2010 3:47 PM
> To: Lee
> Subject: Another WSU program to review
>
> Lee,
>
> Rather than Business, I’m giving you our Honors program. This is a program
> that has worked with our unit for several years and we know them.
>
> I think you will find it contrasts from Biology’s report in ways that may help
> you exercise more of the rubric’s scale.
>
> Thanks for your interest and help