We have done a process to review language and scores on individual responses. A couple weeks ago, Ashley led an activity to compare (in each dimension) across 4 programs. That found a couple anomalies.
This is an experiment to compare across the 22 programs that provided a report May 17. (Actually a few are not finished in the scoring process)
The activity uses a spreadsheet of the OAI contact, the Reviewer, the program and its scores. The activity is for each OAI Contact to filter the list on themselves as contact, and then sort the programs by score on one dimension at a time — and see if the programs are in a reasonable ordinal order.
The activity continues for each reviewer, to filter on the programs they reviewed and then sort the programs by score on one dimension at a time — and see if the programs are in a reasonable ordinal order.
The result is a variety of transects across the data, with the opportunity to spot something that is out of order
Different ideas were offered about what to look for:
- programs who’s scores were more than 1/2 point out of wack
- programs that were not in the right “bin” relative to others in that bin
The spreadsheet looks like this