Draft note to Deans/Chairs to Come From Provost

Draft note to Deans/Chairs

Larry,

Here’s our first draft.  With your modifications, I hope we can entice a few hold outs to join in the fun.

Deans and Chairs

WSU is preparing its response to NWCCU’s Final Report. As part of that effort, the Office of Assessment and Innovation is working with all undergraduate programs to collect self-studies on program-level assessment activities. OAI has provided feedback to programs who submitted draft self-studies in December, and my office also used that preliminary information as part of a report to the HEC Board in March.

I am reminding you of OAI’s deadline of May 17 for a revised self-study, including the Action Plan and Evidence section which was not required last December. I am asking you to ensure that your program provides a self-study in May, using guidelines found on the OAI.wsu.edu website, or that your program is in contact with OAI to make other arrangements.

Timely delivery of self-studies will help OAI by allowing the time necessary to review and compile data from all programs for the report that WSU must provide to NWCCU in the Fall of 2010. My office is also working with OAI to develop a mechanism where the information you provide in these self-studies can be used to also meet many of the requirements of the HEC Board’s Annual Program Review. OAI, Institutional Research and my office are working to streamline as much of this reporting as possible.

College Liaisons, Program Points and OAI Contacts can all be found here https://universityportfolio.wsu.edu/2009-2010/Lists/Programs/All.aspx

NWCCU’s Final Report can be found on the Accreditation.wsu.edu website.

Thanks

Why Public Reporting

One faculty member responds to OAI assessment plan feedback:

“Oh -kiss my #$$, I have been here long enough to know that THAT thing
will be flushed down the toilet.”

[SACS] Student Learning Outcomes – Business

On the question, does every major have to report outcomes:

From: SACS Commission on Colleges Discussion Forum [mailto:SACS-L@LISTSERV.UHD.EDU]
Sent: Wednesday, March 31, 2010 2:57 PM
To: SACS-L@LISTSERV.UHD.EDU
Subject: Re: [SACS] Student Learning Outcomes – Business

I can guarantee it is every major within a degree. Also you must show that every program has been assesed and that improvement strategies have been implemented; not plans to improve. Also at your reaffirmation every program must be finished with this process not just a portion.

Comment to the original post

SACS is not NWCCU

Edit
I believe they are somewhat farther down the assessment trail. 

Can we ask this same question directly of our accreditation agency?

Yeidel, JoshuaNo presence information at 3/31/2010 5:39 PM

Faculty Senate

At Faculty Senate, Thursday, February 18th, concerns were expressed about the assessment initiative.  The concerns largely reflected, we are informed, questions about who was being asked to do this.

On Monday, February 22nd, we have been invited to provide a report.  I have proposed we share the HEC Board report we are preparing this week.

Status Update: From Harvesting Gradebook to Learning Outcomes Assessment

The Cal State University system has been holding an internal series of webinars on ways to integrate and assess general education, including the use of ePortfolios, the VALUE rubrics, and themes like “sustainability.” By invitation we just presented a summary of the harvesting gradebook and beyond

Thursday, February 11, 2010
10:00-11:00 a.m.

Draft abstract

WSU does not have a centrally endorsed ePortfolio tool, and has been moving to view the web itself as an ePortfolio. You can read a summary of our (somewhat radical) thinking here: https://communitylearning.wordpress.com/2009/09/27/not-your-fathers-portfolio/

One of the challenges inherent in the strategy is how to manage assessment when student work is scattered across the Internet. To meet that challenge, we have been developing a tool called the “Harvesting Gradebook” that allows multiple reviewers, both inside and outside the institution, to give rubric-based feedback to learners wherever the learners’ work resides.

This “embedded assessment” approach has the advantage that it can be rolled up from program to college to university level for meaningful program review. WSU is piloting such a system for use in their NWCCU accreditation reporting. Their concept piece can be found here: https://communitylearning.wordpress.com/2009/09/21/from-student-feedback-to-university-accreditation/

In the effort to balance the tension between accountability and assessment, WSU is currently refining a rubric ( https://universityportfolio.wsu.edu/2009-2010/December%20Packet/Guide%20to%20Assessment%20(Expanded).pdf ) to provide formative feedback to academic programs about their assessment work.

WSU’s work is attempting to articulate, coordinate and be accountable for student learning outcomes across many scales within a diverse university setting.

Institutional Self-Assessment Rubric

This post is in support of a TLT webinar, the series is titled “Power of Rubrics
[Archive of session]
Gary Brown, Theron DesRosier, Jayme Jacobson & Nils Peterson, Washington State University

Introduction and Background on the Problem

Washington State University is in the process of responding to changes in by its accrediting body, NWCCU. The response includes the transformation of the former Center for Teaching Learning and Technology (CTLT) to the Office of Assessment and Innovation (OAI).

The University is in the process of developing its response to NWCCU’s changed standards and OAI is helping move the institutional thinking toward one that embeds assessment in ways that help faculty to think about student learning outcomes, and about the processes that programs are using to assess their work on improving outcomes.

This work builds on work of the former CTLT known to as “Harvesting Gradebook.” Previous reports will provide context on using the Harvesting Gradebook with students: AAC&U report Jan 2009 Update Fall 2009. This report links to a webinar archive that paints a picture of how to roll harvesting up, From Student Work to University Accreditation.
Using Harvesting Feedback with Academic Programs

In the previous webinar (From Student Work to University Accreditation) we described a vision for how harvesting could be used to move data from the level of an individual piece of student work up through levels of assessment and reflection to a university-level accreditation report. Presently OAI is engaged in deploying a middle level piece of this vision, the assessment of program-level self studies with an “Assessment of Assessment” rubric. The most current version of the rubric and other materials for the process are linked from grey portion of OAI website banner.

Figure 1. The process involves the academic program collecting evidence, writing a self study, and having the self study assessed with the University’s rubric (Called Guide to Assessment on the OAI website, formerly called Assessment of Assessment rubric). This image shows the process from data sources (upper left, to self study, to rubric-based assessment, to radar graph of results. This diagram represents work on the level of an academic program, a “middle tier” in the vision presented in From Student Work to University Accreditation.

Readers interested in trying the process are invited to do so at the WSU University Portfolio site for 2009-10. The Department of Rocket Science (on the site as of 12/7/09) was created as a sample. Other programs appearing on the site (beginning in January 2010) are actual WSU programs seeking formative feedback. (strikeout March 2010) Contact us if you want.

A Prezi visual of the WSU assessment calendar, provides an interactive picture of the assessment cycle and calendar and will serve as a “Dashboard” for monitoring progress.

Guide to Assessment – Rubric
Because of the wide diversity of programs in the WSU 4-campus system, a one size approach to learning outcomes assessment will not fit all. Consequently, WSU is developing a rubric to assess the self-study plans (short form and long form). Like the AAC&U VALUE project, the WSU rubric assumes that “to achieve a high-quality education for all students, valid assessment data are needed to guide planning, teaching, and improvement.”

The Guide to Assessment is the tool OAI is creating to help programs assess the quality of their student learning outcomes assessment activities. Using the Harvesting mechanism, programs will be able to gather evidence from stakeholders outside the university — a requirement of the accreditor — as well as gathering self-, peer- and OAI reviews.

Short form of the Rubric

Suggested Script for Cold Calls to Program Points

DRAFT DRAFT DRAFT
Cold Calls to WSU Programs

Who you are
·        This is a beginning of the process of actively contacting all programs to solicit their participation in the Dec 18 self-study reporting. Prior to this point, contact was indirect, via the College-level program liaisons.

My name is _______________ with the Office of Assessment and Innovation.  As your College Assessment Liaison probably told you, accreditation and accountability expectations for higher education have been changing, and the accreditation process for us will now require updates every two years with a report on learning outcomes progress due next fall, 2010.

What your role is
·       I am the designated OAI contact selected to work with you and your program.  As you may know, the Office of A and I was established to meet new assessment requirements and to help do so in ways that keep the focus on improving the student learning experience.  Our charge is to help WSU programs develop, if they are not already in place, assessment strategies that meet new assessment requirements in ways that leverage faculty expertise and that are meaningful and useful for WSU faculty.

What you understand their role to be
·       As you may know, in order to meet the fall report deadline, some level of assessment needs to be conducted next spring so that the results can be used to guide improvements.  It is using evidence to ‘close the loop’ that is the common denominator of assessment requirements coming from all of our stakeholders, including the NWCC&U, professional accreditors, OFM, the legislature, or the HEC Board (all now pressing us for accountability).  I’m calling because I understand you are the point person for your program and I want to make sure you are aware that there is a December 18th deadline for sharing your plans.

The Process Overview
·       After you have shared your planning document in the template provided to you by your college assessment liaison, available from me, or accessible online at (https://universityportfolio.wsu.edu/2009-2010/Pages/default.aspx ), your plan will be reviewed and feedback provided. That review and feedback will be based on criteria available in the ‘Guide to Assessing Assessments [A of A].’  The Guide is being developed to help clarify the principles of assessment that our accreditors expect, and of course it might be a useful resource as you prepare your plan.
·       The same systematic process, our charge, will be conducted as WSU’s and your program’s assessment continues, and ideally you and your team will join us in refining the criteria and process to help us make the activity as useful as we can for improving the WSU student experience.
·       To be clear, this process at this point is intended to be formative assessment that we will use so that we can do more than comply with accreditation–we want to help shape it while the opportunity for us and for our accreditors is still possible.  We hope you may also help us identify colleagues in your field who might find this exercise useful and to ensure that the review of your program is conducted by experts you know and trust.
What we can do to help
·       It may be that what this activity entails is overwhelming or confusing right now.  The OAI was established to help you navigate this challenge and to do it in a way that is effectively integrated into your everyday teaching practice.  We have developed a number of strategies over the last several years that you may find useful, and we have developed and identified a number of resources that can help.  Let me know if you want to meet and talk about this, and of course I welcome the opportunity to meet with any of your assessment team who are available as well.

Comments added to original post

questions for an initial mtg, F2F or phone or whatever

Here’s what I try to run through to start the discussion of prog assessment:

Initial conversation about program assessment (phone or F2F)

Tell me about your program.
I’m not very familiar with XXXXXXXXX; could you tell me a bit about your program, students, faculty, and capstone to help us plan an appropriate pilot assessment.

What do you think is working well in your program?  What’s an issue that you all struggle with?  (See if they can identify a question.)

Does the program already have student learning goals?

What direct measures to target

Capstone:
•Does your program have a capstone project?
(Basic info re mode, size, how many students / projects each semester/year,  All seniors?  Mostly seniors?  Timing  )
•Which student work / class is appropriate for capstone assessment?  Individual or group work?
(If it’s a huge project, could students write a short piece about the project,  3-4 pages providing an overview of their thinking about the project, and reflection – like something you might include in a portfolio or prep for a job interview —  and the project is an “appendix” to that, for our assessment??)
•Logistical issues to address?  Format of project or other.  Can we collect digital copies? Need clean, unmarked.
•Assignment prompt and course syllabus
•Logistics of collecting student work this semester.

Lower division core class
•Does your program have a lower division core class or classes?
(Basic info re mode, size, how many students / projects each semester/year,  All first year students?  Mostly?  Timing  )
•Which student work / class is appropriate for assessment?  Individual or group work?
•Logistical issues to address?  Format of project or other.  Can we collect digital copies? Need clean, unmarked.
•Assignment prompt and course syllabus
•Logistics of collecting student work this semester.

Other stuff
•Do a program inventory.
•Who will participate in assessment?  Faculty, instructors, TAs, others
•Look at the timeline for getting started. General timeline this semester and next (attach our sample timeline)
•Student learning goals – into a rubric
•How do your faculty share ideas about teaching, or learn new teaching practices?  What kind of teaching resources (articles, workshops, conferences, brownbags, etc) are used by instructors?  How are they shared, informally or formally?

Green, Kimberly at 12/18/2009 12:33 PM

Building a learning community online

I have been thinking about dissemination and adoption of knowledge as our organization (formerly WSU’s Center for Teaching Learning and Technology) is re-organized to become the Office of Assessment and Innovation (OAI). Our unit’s new challenge is to help the university develop a “system of learning outcomes assessment” in response to new requirements from our accrediting body, NWCCU.

We have captured a discussion about how our unit’s web presence might be changed in a series of notes and whiteboard shots attached to this blog post.

One part of our ideas for this “accreditation system” can be found in this presentation for the TLT Friday Live on harvesting feedback across multiple levels of the university.  We have a prototype of one of the middle tiers running now to test and refine the rubric.

Our system depends on OAI staff working with “Liaisons” for each College and their “Points” in each academic program, and developing skills among the Liaisons and Points so that they can provide useful feedback to programs on the assessment activities that the programs are undertaking. Because of the diversity of WSU, the specific learning outcomes assessment that programs undertake will need to vary by program. What the university seeks is “robust” assessment of student learning. Our method (links above) involves a meta-assessment of the assessment practices of the programs. For programs to understand and develop robust assessment strategies, I believe that the OAI’s, the Liaisons’, and the Contacts’ professional development needs to be a key component of the “system of assessment.” [That is, professional learning in the discipline.]

The challenge is to provide professional development in the context of a multi-campus university, with programs and learners at diverse places in their own learning.
We have advocated that learners find their community of practice and join it and work on their problem in the context of that community. However, that assumes the community of practice exists in an organized way that can be joined. Presently, the COP’s I’m aware of are loose knit collections of bloggers who have developed the skills of tracking one another. Novices would need to learn these skills as an entry requirement to their own participation. That barrier for entry/participation is probably too high for the WSU community we need to reach. I previously wrote a manifesto describing how our unit should change its web strategy. That proposal also included the concept of finding/ building the community of practice online — but it did not solve the problem of how to build that community.

In 2006, Dave Cormier proposed the idea of a “feedbook” of readings that was based on an RSS feed, rather than on traditional paper media. The book was more dynamic because it could be based on blogs or other contemporaneous sources.  In Dave’s later reflections he points to the interesting perspective that the feedbook is (can be) a collaborative effort among a community of learners.

“In addition to the freshness of the material, the multiplicity of voice and perspective and the fact that your textbook will never be out of date, one of the first things that would happen is a decentralization of the instructor. While the instructor would usually be responsible for the basic set of links…gone will be the rabbit out of a hat magic that comes from controlling the flow of knowledge. Students will actually be able to add to that flow of knowledge as their research brings up new sources of course material.”

Dave’s thoughts about multiplicity of voice and perspective seems to fit with Lave and Wenger’s Situated Learning: Legitimate Peripheral Participation Cambridge U Press, 1991

John Seely Brown summarizes several of Lave and Wenger’s writings in a piece written for Xerox PARC:

“This work unfolds a rich, complex picture of what a situated view of learning needs to account for and emphasizes, in particular the social, rather than merely physical nature of situatedness…

“Next, a few clarifications are probably helpful. First, as Lave (1991) herself notes, the situation is not simply another term for the immediate, physical context. If it is to carry any significant conceptual import, it has to be explored in social and historical terms. Two people together in a room are not inevitably identically situated, and the situated constraints on practice do not simply arise in and through such isolated interactions. The people and the constraints importantly have social and historical trajectories. These also need to be understood in any situated account.

“Second, community of practice denotes a locus for understanding coherent social practice. Thus it does not necessarily align with established communities or established ideas about what communities are. Community in Lave & Wenger’s view is not, a “warmly persuasive term for an existing set of relations” (Williams, 1977). Communities can be, and often are, diffuse, fragmented, and contentious. We suspect, however, that it may be this very connotation of warm persuasiveness that has made the concept so attractive to some.

“Third, legitimate peripheral participation (LPP) is not an academic synonym for apprenticeship. Apprenticeship can offer a useful metaphor for the way people learn. In the end, however, in part because of the way apprenticeship has historically been “operationalized,” the metaphor can be seriously misleading, as LPP has occasionally been located somewhere between indentured servitude and conscription.

“As Lave and Wenger put it:
‘Legitimate peripheral participation is not itself an educational form, much less a pedagogical strategy or a teaching technique. It is an analytic viewpoint on learning, a way of understanding learning. We hope to make it clear that learning through legitimate peripheral participation takes place no matter which educational form provides a context for learning, or whether there is any intentional educational form at all. Indeed, this viewpoint makes a fundamental distinction between learning and intentional instruction.’ [1991: 40]

This quote above is one that I’m still trying to fully absorb [np]

JSB continues:

“One of the powerful implications of this view is that the best way to support learning is from the demand side rather than the supply side. That is, rather than deciding ahead of time what a learner needs to know and making this explicitly available to the exclusion of everything else, designers and instructors need to make available as much as possible of the whole rich web of practice-explicit and implicit-allowing the learner to call upon aspects of practice, latent in the periphery, as they are needed.”

“… The workplace, where our work has been concentrated, is perhaps the easiest place to design [for legitimate peripheral participation] because, despite the inevitable contradictions and conflict, it is rich with inherently authentic practice-with a social periphery that, as Orr’s (1990) or Shaiken’s (1990) work shows, can even supersede attempts to impoverish understanding. Consequently, people often learn, complex work skills despite didactic practices that are deliberately designed to deskill. Workplace designers (and managers) should be developing technology to honor that learning ability, not to circumvent it.”

Applying these ideas to OAI/WSU

In the process of becoming the OAI we are re-vamping our website and proposing that it contain several elements:

  • a branded page that provides a basic OAI presence within the university
  • an archive of the former CTLT site, with its various linked resources (many of which retain some value and the URL has some reputation in search engines)
  • a university portfolio space (a showcase)  where we assist the university in mounting is publicly viewable and assessable evidence for accreditation (the system demonstrated above)
  • an assessment workspace for collaborations on assessment activities with academic units (these collaborations may require managed authorizations)
  • a “social” space for collaboration around professional development related to the problems we are working on

It is the last space that presents interesting design challenges, an opportunity to facilitate LPP, and is the cause of this reflection.

Given that the social space will be closely linked to the OAI’s main university presence, we have a requirement that it be “professional.” Dave’s comment above about multiplicity of voice and perspective is the opportunity/concern to manage. Its possible that in a community of learners, some members (perhaps more novice ones) will make contributions at a lower level of professionalism or with less insight. The community, and the visitor, should have ways of both hearing, and not over-valuing, these comments and the community should have ways of responding to these comments that can facilitate learning for multiple players in multiple ways.

In face to face environments, there are various protocols for making contributions, and cues that give an observer orientation to the hierarchy of expertise and authority within the community. An observer can use these cues to conclude which contributions carry the greatest authority in the group. Protocols for responding to contributions can also help organize the group dynamic. In online settings, these cues may be absent and a visitor may sense immaturity or cacophony (e.g., in a feedbook’s content) where in fact what is happening is that novices are exploring their (partial) understanding by sharing within the community.

The problems as I see them:

  • How can an open online community that embraces legitimate peripheral participation maintain a coherence that allows members and visitors to appreciate its professionalism/maturity while still making space for novices participation?
  • What mechanisms can be employed to create “emergent” authority in a web-based learning community without a central oligarchy?
  • Much of the literature on LPP was developed in the context of face-to-face communities. How can an Internet-based community exploit “incidental” (e.g., drop in) participation by experts, who would not have appeared in face-to-face settings? This question is a recognition that via a feedbook mechanism an item from an expert outside the community can be routed into the stream of the community’s readings (thus ‘incidental’).

Seeking NWCCU Feedback on Rain King Initiative

From: Karen Houmiel [mailto:khoumiel@nwccu.org]
Sent: Tuesday, October 20, 2009 2:04 PM
To: Brown, Gary
Subject: RE: Seeking feedback

Hi Gary,

I have forwarded your email to Dr. Baker who is the person with whom you will want to speak.  If you need anything else, please don’t hesitate to let us know!  Have a great day!

Thank you,
Karen

From: Brown, Gary [mailto:browng@wsu.edu]
Sent: Tuesday, October 20, 2009 1:52 PM
To: Karen Houmiel
Subject: Seeking feedback

Hi Karen,
I have recently been appointed to lead our new Office of Assessment and Innovation and Washington State University.  We are developing a process for helping our programs at WSU assess their assessment, and in that process we are developing criteria and a template, doing our best to align it with principles of good practice and the new NWCCU standards.  It would be a tremendous help to talk about the process and invite some early feedback on our efforts now, and perhaps, if we are successful, gaining an endorsement in principle.

I recognize that improved learning outcomes and reaccreditation is the endorsement that matters most, but I am seeking some guidance and support to help WSU programs help the institution (and our students!) in achieving those ends.  Is it possible to speak with somebody in advance of the gathering in Seattle in November?

Gary Brown

Dr. Gary R. Brown, Director
The Office of Assessment and Innovation
Washington State University
509 335-1352
509 335-1362 (fax)
browng@wsu.edu
https://mysite.wsu.edu/personal/browng/GRBWorld/

Specialized Accreditation–Gary Responds a college(purpose again)

Subject: RE: Follow up (1 of 2)

Thanks again for your explanation,  Gary – I look forward to participating!

Subject: RE: Follow up (1 of 2)

The good news is that almost all professional accrediting organizations are working with CHEA and the regional accreditors to align the work so that when a program satisfies one it satisfies all.  As Larry and I explained in our meetings this summer (and will when we meet with you and your team soon), a key goal of ours is to develop a system that is similarly efficient.  The not so good news is that the Northwest Commission and ACPE may or may not be there just yet (I don’t know), and we may have additional work to do in order to meet the fall 2010 goals we have assumed in the face of the challenge to implement an institutional assessment system.  Part of what we’re now exploring is an assessment strategy that will, among other things, help us find out.  (You may have seen the report in the Chronicle last week, by the way, about some of the relationships between professional accreditors and the challenges they now face.)

In short, though, your personal bias – that we build a sustainable process – is precisely what we are hoping to do–implement something that is systematic, meets multiple audiences, and, most importantly, is about teaching and learning improvements. If faculty don’t see the value of assessment in their daily work, then it will not be sustainable or, ultimately, useful.

Subject: RE: Follow up (1 of 2)

Hi Gary,
Thanks for the explanation – I will be learning as we go – from a large university perspective.  Recognizing that my college just completed a rigorous professional accreditation evaluation by my professional accrediting agency and received a full 6 year accreditation – how will our efforts and experience play into the big picture of university accreditation?  I imagine there are other professional programs at WSU who have similar professional academic standards they are required to meet?

My personal bias is that we should create an assessment tool which provides important data/information and includes a sustainable process for programmatic and curricular assessment.  This may be a premature opinion since I haven’t reviewed anything yet for this project, but for programs who have already demonstrated that they are on track with assessment (such as through professional accreditation), I am not in favor of having them re-generate or ‘re-do’ tasks for the sole purpose of assessment.  Does that make sense?

Thanks,