Assessment Report Card for Provost’s Council (May 2010)

We might anticipate response to this report to be presented to the Provost’s council by the Vice Provost tomorrow.

attached file Assessment Report Card 5-10-10

Draft note to Deans/Chairs to Come From Provost

Draft note to Deans/Chairs

Larry,

Here’s our first draft.  With your modifications, I hope we can entice a few hold outs to join in the fun.

Deans and Chairs

WSU is preparing its response to NWCCU’s Final Report. As part of that effort, the Office of Assessment and Innovation is working with all undergraduate programs to collect self-studies on program-level assessment activities. OAI has provided feedback to programs who submitted draft self-studies in December, and my office also used that preliminary information as part of a report to the HEC Board in March.

I am reminding you of OAI’s deadline of May 17 for a revised self-study, including the Action Plan and Evidence section which was not required last December. I am asking you to ensure that your program provides a self-study in May, using guidelines found on the OAI.wsu.edu website, or that your program is in contact with OAI to make other arrangements.

Timely delivery of self-studies will help OAI by allowing the time necessary to review and compile data from all programs for the report that WSU must provide to NWCCU in the Fall of 2010. My office is also working with OAI to develop a mechanism where the information you provide in these self-studies can be used to also meet many of the requirements of the HEC Board’s Annual Program Review. OAI, Institutional Research and my office are working to streamline as much of this reporting as possible.

College Liaisons, Program Points and OAI Contacts can all be found here https://universityportfolio.wsu.edu/2009-2010/Lists/Programs/All.aspx

NWCCU’s Final Report can be found on the Accreditation.wsu.edu website.

Thanks

Re-thinking Course Evals for Prog Eval purposes / Turmoil in Colleges

Re-thinking Course Evals for Prog Eval purposes / Turmoil in Colleges A conversation on the bus home from work with a College Liaison led to this exchange

On 4/6/10 9:18 AM, “Nils Peterson” wrote:
Further to our conversation on the bus. Collecting actionable data, and acting on it, are the things that we believe are likely to engage students and therefore to increase response rates on course evals. How about we get together and talk about some ways to re-think the course eval process. I’m looking for a partner that is in a position to take a bit of an adventurous view of the topic.

(I want to see if this Liaison would venture into something like paired assessment where faculty declare their goals for the course and students say what they perceived happened. We’d get curricular maps, which could then be used by the program to help students navigate course selection.)

Liaison replies:
Nils,
I will be interested in doing this.  However, let’s put off the conversation for several weeks.  Amongst other things, I still do not know where I am going to land beginning July 1.  That is to say, I do not know what my position will be and in what college.

A Leader Asks, What is the Rain King Chronicle

It’s our blog of the story, or history of this assessment initiative we’ve launched.  We summarize and don’t name names (except our own).  Bits of notes go in this chronicle, for instance the liaison who told us “it is inappropriate for us to share our rubrics.”

It’s not just about keeping a history, which we are wishing we had done on a variety of projects in the past, but, we are also in this work reminded of the Bellow novel.  In the story, the lead character by happenstance becomes the Rain King of a remote African tribe only to learn that the tribal elders are plotting to have him beheaded….

As promised, reflection on the retreat, finally

Re: draft note to staff
I was thinking last night that the persistent concern expressed in our still recent retreat was less about job security and more about qualifications in our protean institution.  If I no longer should be doing what I’ve done in the past, what skills do I need to develop to do what I am expected to do?  And how long, one anonymous note asked, before we change yet again?

In my own thinking, I am coming to understand that it is more about the affect than the skill set, though they are not fully separable.  As the CTLT our work was like teaching an elective course in the major.  Most of interactions were with faculty who wanted to work with us.  And now we fear our new charge is not only requiring us to work with those who share fewer of our concerns and values, but we are drifting into work increasingly with those who clearly do not.  We are now like those who teach required courses.  And the required course we teach is not one that many find rewarding.

Our challenge as a CTLT was encouraging participation in our efforts commensurate with the institutions’ investment in us.  As the OAI, our challenge is, with some irony, helping the institution achieve measurable outcomes.  In CTLT we worked with colleagues who were mostly or who became friends.  As an OAI, we must strive to work collegially.

The challenge is ultimately less about our skills.  We are amply qualified.  It is a challenge to our dispositions.  To a person this is not the work we signed on for.  It has a “feel” about it like a garment not cut quite right.  It is frequently somehow oddly irritating.

“Do we still work with the individual faculty we have worked with?”   No, is the answer.  If the work cannot be parlayed to the program level, then the job requires us to extricate ourselves from that work …and, at least in the work context, away from that friendly working relationship.

“Will we change our organization again?” Probably.  The garment may not fit quite right, but the unit will grow into it.

As the CTLT, however, we were unique and recognized internationally for our transformative focus.  Some aspects of that focus were occasionally a source of some contention, but it was a shared understanding we had of our mission.  We were guided by the belief that not only can we but we must do a better job in our institutions of learning at improving the student learning experience and student learning…outcomes.

So here we are.  Be careful what you wish for, the saying goes. Our new vision, rubric, and the harvesting of community input are being followed nationally. We have a tremendous opportunity to promote real, deep, and meaningful change at WSU.

If we can change.

Gary
Dr. Gary R. Brown, Director
The Office of Assessment and Innovation
Washington State University
509 335-1352
509 335-1362 (fax)
browng@wsu.edu
https://mysite.wsu.edu/personal/browng/GRBWorld/

HEC Board Report (Feb 25, 2010)

Sent and accepted by WSU administration….
attached report  HEC Board Outcomes Report (March 2010 FINAL)

A program’s response to next steps after receiving feedback

A program’s response to next steps after receiving feedback On Jan 28, 2010 the program representative replied

Nils,

After our meeting on Tuesday, I began recruiting reviewers for the norming session.  Everyone is busy and no one volunteered.  So, I brought this up with the chair who is unwilling to commit any resources, including faculty time, to reviewing the theses.   Therefore, the rating of the theses is not an option at this time.

However, I am willing to continue to work on preparing the self study to be submitted by May 17th.  Once I receive your propose changes, I will review them and distribute to the faculty for their comments.

Thanks,

On Tue, Jan 26, 2010 at 9:52 AM, Nils Peterson wrote:

Thanks for meeting this AM. In the future, I’m happy to reciprocate and go to your office, just share that with Judy.

Here are my notes summarizing what I believe we agreed. Please feel free to clarify or correct anything below.

Regarding the rating of thesis documents:
You have the 6 thesis documents, each about 20 pages. You are working on identifying raters, at present yourself and 2-3 others are likely.

I am going to organize OAI resources for a norming session to take place in about 10 days — perhaps the week of Feb 8. I will need your help arranging a 90 minutes that will work for the raters you are identifying. Please confirm if that week or the week following are preferable.

The process will use paper copies of the thesis, with names removed and code numbers applied. I will get you a guide sheet for doing this document preparation.

I will create an online survey using the rubric in your program’s self study document, and make you an owner of that survey so you can take over the process moving forward.

Following the norming session, raters will have until the end of spring break, March 21, to complete their rating tasks.

OAI will help you get reports of the data and think about next steps after the rating.

Regarding the December draft self-study
You have been sharing the OAI feedback with the chair and are continuing to think about the recommendations and which are politically and practically feasible

I will use Track Changes to propose a new draft of your self-study by the week of Feb 8.

You will accept/modify/reject my suggestions based on the context and we will discuss further revisions.

We did not discuss, but I would hope we have a revised version to send through the OAI rating process by March 8.

Regarding the May 17 self-study
Beginning after March 21, I will help you think about the results of ratings and the feedback on the revised shelf study in the process of creating the self-study due May 17

Thanks again for your attention to all this

Nils

Working towards 100% participation by faculty

I chatted with Gary after someone in a program asked me what actual threat existed that he could use to coerce his (more senior) colleagues into participating in program assessment.

I mentioned the carrot approach — all the potential benefits — but I figure that this question will continue to come up, and I sought out Gary for a more official answer regarding the Stick.

This is what I came up with after that conversation with Gary, which afterwards got his ok.

Regarding broad participation of faculty in assessment:

One thing the accreditors cited WSU for last year (2009) is the low level of participation in assessment found in many programs … a few people doing most or all of the work of assessment.

Getting more folks to participate:

Carrots:
·       What does your faculty want to know about its program, about student learning?  Program assessment can and should be designed to answer a question you all care about.

·       A significant number of faculty must be involved in assessment, including senior faculty, for these efforts to systematically impact a program.

·       Quality assessment supports ongoing program improvement, contributing to your program’s reputation.

The stick:
Programs whose assessment doesn’t have broad faculty involvement —  i.e., not close to full participation — may  jeopardize WSU’s institutional re-accreditation, which is risky particularly  in the current era of funding cuts at the state level and of increased competition (including online education) in general.

Status of reports for 2009

Status of reports for 2009

Hi Larry,
Here is where we are at.

1.      “Ready” means we have it and are reviewing it now (or have reviewed and rated).
2.     “Not ready” means we are either talking with or working with programs to get their planning documents (and plans) in shape.
3.      “Not received” means just that, though the overall response is complicated by the limbo status of professional programs.  We will review the few we have received and work with their interested people.

At this point, we will “score” them not using the full scale of the rubric, but categorically into three bins which will be sorted on Monday.  All documents that have been shared with us will receive written feedback.  The three bins:

1.      On Target and going well (high)
2.      On Target with work
3.      Doubtful that adequate progress will be made this spring.

We will press to meet zero week with category #2.  I suspect category #3 represents the meetings with deans that you may want to initiate.

I’m taking my hardworking crew (those that have lasted through this adventure) to Riccos for some Nogg of the Egg at 2:00.  If you are free, please consider joining us.

And have a great holiday!

Gary
Attached spreadsheet with details by program status of 2009 self-study reports

Suggested Script for Cold Calls to Program Points

DRAFT DRAFT DRAFT
Cold Calls to WSU Programs

Who you are
·        This is a beginning of the process of actively contacting all programs to solicit their participation in the Dec 18 self-study reporting. Prior to this point, contact was indirect, via the College-level program liaisons.

My name is _______________ with the Office of Assessment and Innovation.  As your College Assessment Liaison probably told you, accreditation and accountability expectations for higher education have been changing, and the accreditation process for us will now require updates every two years with a report on learning outcomes progress due next fall, 2010.

What your role is
·       I am the designated OAI contact selected to work with you and your program.  As you may know, the Office of A and I was established to meet new assessment requirements and to help do so in ways that keep the focus on improving the student learning experience.  Our charge is to help WSU programs develop, if they are not already in place, assessment strategies that meet new assessment requirements in ways that leverage faculty expertise and that are meaningful and useful for WSU faculty.

What you understand their role to be
·       As you may know, in order to meet the fall report deadline, some level of assessment needs to be conducted next spring so that the results can be used to guide improvements.  It is using evidence to ‘close the loop’ that is the common denominator of assessment requirements coming from all of our stakeholders, including the NWCC&U, professional accreditors, OFM, the legislature, or the HEC Board (all now pressing us for accountability).  I’m calling because I understand you are the point person for your program and I want to make sure you are aware that there is a December 18th deadline for sharing your plans.

The Process Overview
·       After you have shared your planning document in the template provided to you by your college assessment liaison, available from me, or accessible online at (https://universityportfolio.wsu.edu/2009-2010/Pages/default.aspx ), your plan will be reviewed and feedback provided. That review and feedback will be based on criteria available in the ‘Guide to Assessing Assessments [A of A].’  The Guide is being developed to help clarify the principles of assessment that our accreditors expect, and of course it might be a useful resource as you prepare your plan.
·       The same systematic process, our charge, will be conducted as WSU’s and your program’s assessment continues, and ideally you and your team will join us in refining the criteria and process to help us make the activity as useful as we can for improving the WSU student experience.
·       To be clear, this process at this point is intended to be formative assessment that we will use so that we can do more than comply with accreditation–we want to help shape it while the opportunity for us and for our accreditors is still possible.  We hope you may also help us identify colleagues in your field who might find this exercise useful and to ensure that the review of your program is conducted by experts you know and trust.
What we can do to help
·       It may be that what this activity entails is overwhelming or confusing right now.  The OAI was established to help you navigate this challenge and to do it in a way that is effectively integrated into your everyday teaching practice.  We have developed a number of strategies over the last several years that you may find useful, and we have developed and identified a number of resources that can help.  Let me know if you want to meet and talk about this, and of course I welcome the opportunity to meet with any of your assessment team who are available as well.

Comments added to original post

questions for an initial mtg, F2F or phone or whatever

Here’s what I try to run through to start the discussion of prog assessment:

Initial conversation about program assessment (phone or F2F)

Tell me about your program.
I’m not very familiar with XXXXXXXXX; could you tell me a bit about your program, students, faculty, and capstone to help us plan an appropriate pilot assessment.

What do you think is working well in your program?  What’s an issue that you all struggle with?  (See if they can identify a question.)

Does the program already have student learning goals?

What direct measures to target

Capstone:
•Does your program have a capstone project?
(Basic info re mode, size, how many students / projects each semester/year,  All seniors?  Mostly seniors?  Timing  )
•Which student work / class is appropriate for capstone assessment?  Individual or group work?
(If it’s a huge project, could students write a short piece about the project,  3-4 pages providing an overview of their thinking about the project, and reflection – like something you might include in a portfolio or prep for a job interview —  and the project is an “appendix” to that, for our assessment??)
•Logistical issues to address?  Format of project or other.  Can we collect digital copies? Need clean, unmarked.
•Assignment prompt and course syllabus
•Logistics of collecting student work this semester.

Lower division core class
•Does your program have a lower division core class or classes?
(Basic info re mode, size, how many students / projects each semester/year,  All first year students?  Mostly?  Timing  )
•Which student work / class is appropriate for assessment?  Individual or group work?
•Logistical issues to address?  Format of project or other.  Can we collect digital copies? Need clean, unmarked.
•Assignment prompt and course syllabus
•Logistics of collecting student work this semester.

Other stuff
•Do a program inventory.
•Who will participate in assessment?  Faculty, instructors, TAs, others
•Look at the timeline for getting started. General timeline this semester and next (attach our sample timeline)
•Student learning goals – into a rubric
•How do your faculty share ideas about teaching, or learn new teaching practices?  What kind of teaching resources (articles, workshops, conferences, brownbags, etc) are used by instructors?  How are they shared, informally or formally?

Green, Kimberly at 12/18/2009 12:33 PM