Updated OAI web presence

The OAI website is updated along with new landing page for UniversityPortfolio/2009-2010 (the ‘dirdash’ page) that provides summary data from the Dec 18 self-study effort. In addition, preparations are in place for the ‘May Folder’ with materials for writing the May self-study and a more general ‘Resources folder’ for Outcomes Assessment resources not related to writing a particular self-study. Finally, conversations are open with the Provost’s office for revisions to the University’s Accreditation page. Still missing is opening up public access to the root of UniversityPortfolio, with a timeline history of the assessment activities.

Status of reports for 2009

Status of reports for 2009

Hi Larry,
Here is where we are at.

1.      “Ready” means we have it and are reviewing it now (or have reviewed and rated).
2.     “Not ready” means we are either talking with or working with programs to get their planning documents (and plans) in shape.
3.      “Not received” means just that, though the overall response is complicated by the limbo status of professional programs.  We will review the few we have received and work with their interested people.

At this point, we will “score” them not using the full scale of the rubric, but categorically into three bins which will be sorted on Monday.  All documents that have been shared with us will receive written feedback.  The three bins:

1.      On Target and going well (high)
2.      On Target with work
3.      Doubtful that adequate progress will be made this spring.

We will press to meet zero week with category #2.  I suspect category #3 represents the meetings with deans that you may want to initiate.

I’m taking my hardworking crew (those that have lasted through this adventure) to Riccos for some Nogg of the Egg at 2:00.  If you are free, please consider joining us.

And have a great holiday!

Gary
Attached spreadsheet with details by program status of 2009 self-study reports

Recognition from VP Quality, Curriculum, and Assessment at AAC&U

From a 12/15/2009 webcast, Terry Rhodes,Vice President for Quality, Curriculum, and Assessment at AAC&U:
https://admin.na6.acrobat.com/_a738382050/p17163215/

Questioner: “How is VALUE & Power of Rubrics to assess learning playing in the VSA [Volunteer System of Accountability] Sphere?”

Rhodes: [VSA is ] very concerned about comparability among institutions, but they have indicated they would love campuses to use rubrics and to report on them, but they want to have some way that they can provide comparability. I think again the work that is going on at Washington State begins to provide a way to do that. It’s not necessarily a score but is a wonderful rich way to provide the multiplicity and multiple dimensions of learning in a graphic way that is easily represented and easily communicated.

Questioner: “Are there any accreditor responses to the use of rubrics (vs VSA test scores) share?”

Rhodes: “All of the accrediting workshops at SACS at Middle States–are very heavy into that. Northwest is one area that has lagged a little behind on this, but I think with Washington State pushing them they are going to get more enthusiastic. All of the accreditors have actually viewed rubrics, and the use of them, and the reporting of learning using rubrics as much more useful for campuses than a single test score.

Suggested Script for Cold Calls to Program Points

DRAFT DRAFT DRAFT
Cold Calls to WSU Programs

Who you are
·        This is a beginning of the process of actively contacting all programs to solicit their participation in the Dec 18 self-study reporting. Prior to this point, contact was indirect, via the College-level program liaisons.

My name is _______________ with the Office of Assessment and Innovation.  As your College Assessment Liaison probably told you, accreditation and accountability expectations for higher education have been changing, and the accreditation process for us will now require updates every two years with a report on learning outcomes progress due next fall, 2010.

What your role is
·       I am the designated OAI contact selected to work with you and your program.  As you may know, the Office of A and I was established to meet new assessment requirements and to help do so in ways that keep the focus on improving the student learning experience.  Our charge is to help WSU programs develop, if they are not already in place, assessment strategies that meet new assessment requirements in ways that leverage faculty expertise and that are meaningful and useful for WSU faculty.

What you understand their role to be
·       As you may know, in order to meet the fall report deadline, some level of assessment needs to be conducted next spring so that the results can be used to guide improvements.  It is using evidence to ‘close the loop’ that is the common denominator of assessment requirements coming from all of our stakeholders, including the NWCC&U, professional accreditors, OFM, the legislature, or the HEC Board (all now pressing us for accountability).  I’m calling because I understand you are the point person for your program and I want to make sure you are aware that there is a December 18th deadline for sharing your plans.

The Process Overview
·       After you have shared your planning document in the template provided to you by your college assessment liaison, available from me, or accessible online at (https://universityportfolio.wsu.edu/2009-2010/Pages/default.aspx ), your plan will be reviewed and feedback provided. That review and feedback will be based on criteria available in the ‘Guide to Assessing Assessments [A of A].’  The Guide is being developed to help clarify the principles of assessment that our accreditors expect, and of course it might be a useful resource as you prepare your plan.
·       The same systematic process, our charge, will be conducted as WSU’s and your program’s assessment continues, and ideally you and your team will join us in refining the criteria and process to help us make the activity as useful as we can for improving the WSU student experience.
·       To be clear, this process at this point is intended to be formative assessment that we will use so that we can do more than comply with accreditation–we want to help shape it while the opportunity for us and for our accreditors is still possible.  We hope you may also help us identify colleagues in your field who might find this exercise useful and to ensure that the review of your program is conducted by experts you know and trust.
What we can do to help
·       It may be that what this activity entails is overwhelming or confusing right now.  The OAI was established to help you navigate this challenge and to do it in a way that is effectively integrated into your everyday teaching practice.  We have developed a number of strategies over the last several years that you may find useful, and we have developed and identified a number of resources that can help.  Let me know if you want to meet and talk about this, and of course I welcome the opportunity to meet with any of your assessment team who are available as well.

Comments added to original post

questions for an initial mtg, F2F or phone or whatever

Here’s what I try to run through to start the discussion of prog assessment:

Initial conversation about program assessment (phone or F2F)

Tell me about your program.
I’m not very familiar with XXXXXXXXX; could you tell me a bit about your program, students, faculty, and capstone to help us plan an appropriate pilot assessment.

What do you think is working well in your program?  What’s an issue that you all struggle with?  (See if they can identify a question.)

Does the program already have student learning goals?

What direct measures to target

Capstone:
•Does your program have a capstone project?
(Basic info re mode, size, how many students / projects each semester/year,  All seniors?  Mostly seniors?  Timing  )
•Which student work / class is appropriate for capstone assessment?  Individual or group work?
(If it’s a huge project, could students write a short piece about the project,  3-4 pages providing an overview of their thinking about the project, and reflection – like something you might include in a portfolio or prep for a job interview —  and the project is an “appendix” to that, for our assessment??)
•Logistical issues to address?  Format of project or other.  Can we collect digital copies? Need clean, unmarked.
•Assignment prompt and course syllabus
•Logistics of collecting student work this semester.

Lower division core class
•Does your program have a lower division core class or classes?
(Basic info re mode, size, how many students / projects each semester/year,  All first year students?  Mostly?  Timing  )
•Which student work / class is appropriate for assessment?  Individual or group work?
•Logistical issues to address?  Format of project or other.  Can we collect digital copies? Need clean, unmarked.
•Assignment prompt and course syllabus
•Logistics of collecting student work this semester.

Other stuff
•Do a program inventory.
•Who will participate in assessment?  Faculty, instructors, TAs, others
•Look at the timeline for getting started. General timeline this semester and next (attach our sample timeline)
•Student learning goals – into a rubric
•How do your faculty share ideas about teaching, or learn new teaching practices?  What kind of teaching resources (articles, workshops, conferences, brownbags, etc) are used by instructors?  How are they shared, informally or formally?

Green, Kimberly at 12/18/2009 12:33 PM

December Agenda Shared with Liaisons

Suggestion for you to send to Liaisons
In particular, the note was shared:

The new OAI website is up http://oai.wsu.edu and we are assembling resources there that will be helpful to you and program contacts in creating the plans that are due Dec 18. Especially note the three links on the right end in the banner. These are the template for writing a plan, the guide for assessing a plan and a folder of additional resources.

I want to remind you that OAI staff are available to talk with your program contacts about these materials and to consult in the development of plans. Please call 335-1355 to arrange an appointment

agenda for liaisons, december

A of A Rubric Draft December (revised)

document attached to email with no body  a of a rubric expanded revised 12_1

Dr. Gary R. Brown, Director
The Office of Assessment and Innovation
Washington State University
509 335-1352
509 335-1362 (fax)
browng@wsu.edu
https://mysite.wsu.edu/personal/browng/GRBWorld/

Assessment Liaison Updates and Tasks

Folks,

Please share with your Program Assessment Point people:

1.      OAI contacts have been assigned for each program.
You can see the list:  https://universityportfolio.wsu.edu/20082009/Lists/Liaisons/Liaison%20Council.aspx
If you click on the names, you will find more contact information, but if that fails, all OAI contacts can be reached at 5-1355.

No doubt this list will be changing.  Please make changes or send them to me or Judy (judyrumph@wsu.edu ) as they occur on your end.

2.      Attached is the 3rd version of the rubric for assessing assessment and the corresponding template.  The rubric has been rendered in three versions (not perfectly aligned in this draft just yet).  There is the over view, the ‘digest’ version (by popular demand), and the expanded version, which we have found tends to produce better results though perhaps requires a greater initial investment.)  We anticipate one more round of deep revision of the rubric and template following the activities associated with this release, but the principles will remain the same.  We are also close to releasing a timeline and checklist, but anticipate that document merits discussion at our next meeting.

3.      We strongly recommend you forward the blue below and encourage each of your teams to review the assessment process using the rubric to evaluate the mock report (Rocket Science) before submitting their spring assessment plans.  We also recommend they consider their previous self-study reports in light of these criteria.

We also suggest that the assessment of assessment process be conducted synchronously in collaboration between your program assessment teams and their OAI contacts.

And we always welcome feedback but particularly at this juncture as we gear up for spring assessment.

Here is the link to the online Mock/Model Template-based report–Rocket Science.  The process is the same as the one we did with Honors.  If you log into this site, you will find the directions to guide you through the Rocket Science assessment process:

https://universityportfolio.wsu.edu/2009-2010/Pages/default.aspx [since retired]

Again, the process:

1.      Read over the Assessment Criteria (you can download the revised rubric at the link above or read it online).

2.      Read the Rocket Science self Study

3.      Go through the online rubric and assign Rocket Science scores on each of the four dimensions of the rubric.

Meanwhile, we are also sharing this process statewide and with selective professional groups and have received very positive feedback.  More on that when next we meet, but the upshot is transparency.

4.      Finally, don’t forget the deadline for spring assessment plans.  We need plans for spring assessment activity from all programs before December 18th, 2009.  We will review (with the appropriate criteria of the rubric) and provide feedback for each plan as soon as possible but no later than early January. (The sooner we receive them, the sooner we can provide feedback.)

Don’t hesitate to contact me with any questions or concerns. And don’t forget the next Liaison Council meeting:  December 4th at 1:00 in Lighty 403.

Gary

PS, some program points have asked for more models.  A few from 2008 we can point to are here: https://teamsite.oue.wsu.edu/progeval/default.aspx
(follow ‘Assessment Highlights’ and the ‘case reports’ below the highlights)

Dr. Gary R. Brown, Director
The Office of Assessment and Innovation
Washington State University
509 335-1352
509 335-1362 (fax)
browng@wsu.edu
https://mysite.wsu.edu/personal/browng/GRBWorld/

attachment with original email  a of a set beta (3)

the email also references a mythical program “rocket science” as a vehicle to be used testing the rubric. Those files are included here for completeness: