Sanity Checking the May 17 scores

We have done a process to review language and scores on individual responses.  A couple weeks ago, Ashley led an activity to compare (in each dimension) across 4 programs. That found a couple anomalies.

This is an experiment to compare across the 22 programs that provided a report May 17. (Actually a few are not finished in the scoring process)

The activity uses a spreadsheet of the OAI contact, the Reviewer, the program and its scores.  The activity is for each OAI Contact to filter the list on themselves as contact, and then sort the programs by score on one dimension at a time — and see if the programs are in a reasonable ordinal order.

The activity continues for each reviewer, to filter on the programs they reviewed and then sort the programs by score on one dimension at a time —  and see if the programs are in a reasonable ordinal order.

The result is a variety of transects across the data, with the opportunity to spot something that is out of order

Different ideas were offered about what to look for:

  1. programs who’s scores were more than 1/2 point out of wack
  2. programs that were not in the right “bin” relative to others in that bin

The spreadsheet looks like this

Agenda June 30, 2010 Review of Recent Feedback to Programs Part II

Agenda June 30, 2010 Review of Recent Feedback to Programs Part II
Goal 1: Assess accuracy and alignment between a program’s score and feedback.
·      We have made progress on this goal through last week’s review of Anthropology, Music, Geology and Communication feedback reports.
·      Each OAI contact should carry out a similar exercise with each of the contact’s program feedback reports to ensure alignment. When there is misalignment, a discussion with the reviewers and reconciliation of scores and discrepant language needs to occur.  This should currently be happening with Geology, Music and Communication.
·      A next important exercise is to ensure consistency of look and overall message across feedback reports.
1)      Standardized Language: each feedback report must have the following standard language at the top of the report (let’s read it now and offer feedback so we can make any necessary revisions asap):
https://assessment.wsu.edu/sites/OAITeam/2009-2010/Documents/Forms/AllItems.aspx

Please find below the feedback to your program’s 5/17/09 assessment report, the second in the WSU Learning Outcomes Assessment System cycle.  As in the December round, your program’s report was read independently by two reviewers from the Office of Assessment and Innovation (OAI). Your OAI contact was not one of the reviewers (simulating the accreditation review process we anticipate in fall 2010). Additional comments and suggestions by your OAI contact are located at the end of the document.

Each OAI feedback report includes observations, commendations and recommendations, using language from the WSU Guide to Effective Program Assessment Rubric, which are intended to help programs shape their next steps for assessment as well as future reporting to OAI and accreditors. Recommendations are in italics. This feedback is intended to guide programs in their next steps for assessment as well as future reporting to OAI and accreditors. Your OAI contact is happy to discuss any of the comments, questions or recommendations.  We encourage you to proceed with action plans that will make your assessment efforts more fruitful; the OAI is available to help in those efforts.

OAI hopes that it has effectively conveyed the goal that the report you have shared is to be understood as a living document that will be used to respond to those interested accreditors and stakeholders in WSU’s mission– The Northwest Commission for Colleges and Universities (NWCC&U), The Washington state Higher Education Coordinating (HEC) Board, the Office of Fiscal Management (OFM), WSU’s Annual Program Review (APR) process as well as others as needed.

The next program assessment report is due Friday, AUGUST 20, 2010.  This should include revisions to address OAI feedback. The report submitted in August will be used as evidence for WSU’s assessment report to NWCC&U, which will be submitted October 15, 2010. Please download a fresh template from the OAI website at http://www.oai.wsu.edu This site also provides a number of resources that you may find helpful during the revision and drafting process.
2)      Any personalized comments must be put into an email and not be included in the official feedback report.  A copy of this email must be placed somewhere for OAI future reference (Location TBA).  There is also standard language that must be included in the email to program points. It is similar to the feedback report language and can be accessed at this link:https://assessment.wsu.edu/sites/OAITeam/2009-2010/Documents/Forms/AllItems.aspx

3)      All radar charts must be de-wonkified. Currently they are all looking different. We need to decide whether or not the ratings themselves should be noted on the chart and/or on the headers for each section of the feedback report.

4)      Reviewers’ comments must seem as ‘one voice’ and be edited by the OAI contact for consistency/alignment/clarity.  No Reviewer 1, Reviewer 2. If there is concern regarding over-editing, the OAI contact should discuss with the reviewers.  There has been a suggestion to organize the feedback by dimension category  – what do you think about that?

5)      Be sure to review the feedback given to the program from round one prior to sending the feedback back to the program point.  In the case that the programs are receiving lower scores, then that will need to be part of the conversation with the program point.  We are currently working on crafting some language regarding changes made to the rubric that could be used to explain lower scores.  (Nils will show his draft and talk about it).

6)      There has been a suggestion to add a scoping/contextualizing statement at the top of the August Template. (Kimberly will show her draft and talk about it).

7)      We need to do another round of looking across programs to ensure consistency of scores/categories in order to move forward on Goal 2: Assess consistency of feedback and scores across programs — does the same score/category represent similar levels of quality?
The 14 programs who’s data is in process actions today are tabulated below.

8)      Other?

Norming and workflow for May 17 self study reviews

Norming and workflow for May 17 self study reviews This is another post linked to the notes for OAI Team in this note  https://communitylearning.wordpress.com/2010/06/02/process-for-creating-may-17-responses-to-programs/

There are three images
Results of norming on the Sociology Self-Study from Dec using the May 17 rubric. This chart compares OAI results with the results Gary & Nils brought back from the Assessment, Teaching and Learning conference the end of April. In that setting about 40 conference attendees rated Sociology, with about 10 people rating each dimension. No person rated all 4 dimensions. This session was done without norming, as part of sharing the WSU system.

A key agreement in the norming process was how to treat the self-studies.
1) the whole document is being read to inform each dimension, even if the particular section of the report does not describe the features sought in the rubric.
2) no “benefit of the doubt” or “inferring;” if the program did not describe a feature of its process, then it is to be treated as missing — an attempt to simulate the practice of outside accreditors. It was hoped that the feedback would be used by raters to communicate to programs in constructive ways when the score did not reflect what was known to exist.

Work flow for rating reports walks through the raters and OAI Contact roles and the flow of the documents and data. This process is somewhat changed from the December process, with more focus on how the raters reconcile and provide feedback to the OAI contact– with more emphasis on how to provide feedback to the program.

Plan for first week is a sketch of the timeline and scheduling to help OAI think about how to coordinate among various raters and reviewers.

Self-Study Process Alerts

Self-Study Process Alerts OAI folks,

Don’t forget that you can set an “alert” on the Sharepoint “Spring Self-Studies Process” list, so you get emails that keep you up to date on changes to the list.  Here is a sample (it may appear slightly shrunk due to the way it was captured):

I have my alert set so it is delivered once a day with all the day’s changes;  you can have immediate alerts on each change if you wish.

To set an alert,
go to the current OAITeam site:  https://assessment.wsu.edu/sites/OAITeamat the top page near the right edge, click on “Welcome ” to drop down a menu.Select “My Settings” from the Menu.  You will see your profile page.From the blue toolbar about 1 inch below the top of the page, click on “My Alerts”.  You will see the “My Alerts on this Site” page. From the blue toolbar, click on “Add Alert”.  You will see the “New Alert” page.From the libraries and lists, select “Spring Self-Studies Process”.  Click “next” to see the alert settings page.Choose your settings and click “OK”.

Note that because this list is based on the “Task” list template, you will get an email when a task is assigned to you in any case.

— Joshua

Process for creating May 17 responses to programs

I am still finalizing a few details, but I know several of you are in position to begin writing your response to programs, so here is a start.

Documents and templates you need are in the left menu “Site Hierarchy/Documents library” of the Assessment.wsu.edu site where we are working https://assessment.wsu.edu/sites/OAITeam/2009-2010

As part of Process Actions Steps 7 & 8, the OAI Contact should:

More on what I think the remaining steps are:

  • Josh and I are still working on how you will record the  reconciled rater scores
  • Post the completed Response to Program in the AWE folder along side the program’s self study
  • Set the status of the Process Actions to alert Ashley to read the draft.
  • When Gary finishes his review, use the OAI Email template to send comments back to programs

Assessment Report Card for Provost’s Council (May 2010)

We might anticipate response to this report to be presented to the Provost’s council by the Vice Provost tomorrow.

attached file Assessment Report Card 5-10-10

Flexing the May Report deadline

Hi [Program Point],
We discussed the update on your recent assessment meeting, which was a great start.  Your OAI contact also shared that you may be feeling a bit stressed by the May timeline, so I want to say we’d rather keep working together to get a pithy, focused, and clear document that reflects the real* work you are doing.  So if you need more time and if you are open to the ongoing collaboration, when the results start coming in we’re happy to help pull together a report ready outline, find some time to sit down with you, and help get the report into shape.  If that goes into June or so, that’s fine on this end.
Let me know.

Gary

*”real” is the operative word that makes flexibility possible.
Dr. Gary R. Brown, Director
The Office of Assessment and Innovation
Washington State University
509 335-1352
509 335-1362 (fax)
browng@wsu.edu
https://mysite.wsu.edu/personal/browng/GRBWorld/