Agenda June 30, 2010 Review of Recent Feedback to Programs Part II

Agenda June 30, 2010 Review of Recent Feedback to Programs Part II
Goal 1: Assess accuracy and alignment between a program’s score and feedback.
·      We have made progress on this goal through last week’s review of Anthropology, Music, Geology and Communication feedback reports.
·      Each OAI contact should carry out a similar exercise with each of the contact’s program feedback reports to ensure alignment. When there is misalignment, a discussion with the reviewers and reconciliation of scores and discrepant language needs to occur.  This should currently be happening with Geology, Music and Communication.
·      A next important exercise is to ensure consistency of look and overall message across feedback reports.
1)      Standardized Language: each feedback report must have the following standard language at the top of the report (let’s read it now and offer feedback so we can make any necessary revisions asap):

Please find below the feedback to your program’s 5/17/09 assessment report, the second in the WSU Learning Outcomes Assessment System cycle.  As in the December round, your program’s report was read independently by two reviewers from the Office of Assessment and Innovation (OAI). Your OAI contact was not one of the reviewers (simulating the accreditation review process we anticipate in fall 2010). Additional comments and suggestions by your OAI contact are located at the end of the document.

Each OAI feedback report includes observations, commendations and recommendations, using language from the WSU Guide to Effective Program Assessment Rubric, which are intended to help programs shape their next steps for assessment as well as future reporting to OAI and accreditors. Recommendations are in italics. This feedback is intended to guide programs in their next steps for assessment as well as future reporting to OAI and accreditors. Your OAI contact is happy to discuss any of the comments, questions or recommendations.  We encourage you to proceed with action plans that will make your assessment efforts more fruitful; the OAI is available to help in those efforts.

OAI hopes that it has effectively conveyed the goal that the report you have shared is to be understood as a living document that will be used to respond to those interested accreditors and stakeholders in WSU’s mission– The Northwest Commission for Colleges and Universities (NWCC&U), The Washington state Higher Education Coordinating (HEC) Board, the Office of Fiscal Management (OFM), WSU’s Annual Program Review (APR) process as well as others as needed.

The next program assessment report is due Friday, AUGUST 20, 2010.  This should include revisions to address OAI feedback. The report submitted in August will be used as evidence for WSU’s assessment report to NWCC&U, which will be submitted October 15, 2010. Please download a fresh template from the OAI website at This site also provides a number of resources that you may find helpful during the revision and drafting process.
2)      Any personalized comments must be put into an email and not be included in the official feedback report.  A copy of this email must be placed somewhere for OAI future reference (Location TBA).  There is also standard language that must be included in the email to program points. It is similar to the feedback report language and can be accessed at this link:

3)      All radar charts must be de-wonkified. Currently they are all looking different. We need to decide whether or not the ratings themselves should be noted on the chart and/or on the headers for each section of the feedback report.

4)      Reviewers’ comments must seem as ‘one voice’ and be edited by the OAI contact for consistency/alignment/clarity.  No Reviewer 1, Reviewer 2. If there is concern regarding over-editing, the OAI contact should discuss with the reviewers.  There has been a suggestion to organize the feedback by dimension category  – what do you think about that?

5)      Be sure to review the feedback given to the program from round one prior to sending the feedback back to the program point.  In the case that the programs are receiving lower scores, then that will need to be part of the conversation with the program point.  We are currently working on crafting some language regarding changes made to the rubric that could be used to explain lower scores.  (Nils will show his draft and talk about it).

6)      There has been a suggestion to add a scoping/contextualizing statement at the top of the August Template. (Kimberly will show her draft and talk about it).

7)      We need to do another round of looking across programs to ensure consistency of scores/categories in order to move forward on Goal 2: Assess consistency of feedback and scores across programs — does the same score/category represent similar levels of quality?
The 14 programs who’s data is in process actions today are tabulated below.

8)      Other?

Shared Language-Jargon Dilemma

Our University-hosed wiki is about to be retired. This page, developed in Sept-Oct 2005 seemed work keeping. I’ve ported it here changing links to preserve the navigation. – np

Shared language

Most academic disciplines and other discourse communities develop context specific language. They use such specialized language to communicate complex ideas efficiently among members of the group. EE Kim uses the term SharedLangage to describe this concept of communication.


Unfortunately, what is effective and efficient for established members of disciplines and discourse communities presents extreme challenges for students, novices and outsiders. Since students have not yet mastered the shared language, they may experience the shared language as jargon impeding their entry into the discipline.

E.E. Kim: Wikis resolve the dilemma

Kim suggests that the discourse communities can resolve the tension between jargon and shared language by using wikis to make the shared language more explicit: [[1]] Kim argues that wikis support the development of shared language in ways that Content Mangement Systems and standard websites do not. He urges writers to use automatic WikiWord linking to identify shared language and explain it.

Using Wiki to Develop Communities

Our University-hosed wiki is about to be retired. This page, developed in Sept-Oct 2005 seemed work keeping. I’ve ported it here changing links to preserve the navigation. – np

The (perhaps not obvious) way to find community in MediaWiki installations (and some others) is to use the history page to see who is contributing to the article. The page provides a portfolio of the work of an ad hoc team collaborating around a topic area. Examination of the history page would show not only who contributed, but the character of their contribution. From this, one might glean ideas about how knowledgeable contributors were to the topic. By then exploring other pages edit by these people (Special:Contributions&target=user) one could learn more about the scope and character of these individuals.

In WSUWiki, there are additional, more explicit mechansims to enable users to find one another. The best developed of these is CTLT’s Morning Reading Group (MRG). Using a Category page, MRG is building pages to capture the group’s thinking about articles that are read bi-weekly. Rather than the process above where interest in a topic is inferred by edits shown on the History page, MRG members are encouraged to edit their user page to include a Category tag (for example see User:Nils peterson.

To facilitate these group-forming activities, two templates have been created, one for the Groups’ page, describing it and inviting others to join, and the other for a User’s page, declaring and describing reasons for interest in the group.

Categories in MediaWiki are essential to the operation of this system. As MRG members create or find pages of interest to the group, they tag the pages with the MRG category. This causes the pages to list in the index of the MRG page).

Student-Faculty Interaction

Another application of this strategy of users adding themselves to category pages can be seen in the work developing to faciliate more faculty-student interaction. The concept started out being called ‘Undergraduate Research,’ but came to be understood as something more general, presently its working name is Matchmaking.

The objective is to help faculty advertise ways in which they would like to interact with students in scholarship activities, and for students to find faculty (and perhaps groups of students and faculty) with shared interests and experiences. The only example implemented as of this writing is by Nils Peterson.

Other campuses have undertaken similar initiatives, using custom database solutions. WSU is exploring Wiki as its solution for two reasons:

  • Its more ad hoc, and can support data and structures that might be invented later more readily than a traditional database solution.
  • Wiki can support other group formation (in addition to Matchmaking) using the same mechanisms, allowing users to invent other purposes and groups without requiring new tools.

See also:
Using_Wikis_For_Learning and especially

Stubbing out as a teaching tool

Our University-hosed wiki is about to be retired. This page, developed in Sept-Oct 2005 seemed work keeping. I’ve ported it here changing links to preserve the navigation. – np

Stubbing out as a teaching tool is one way to support constructivist thinking.

Somewhere in a learning experience, a question arises about a topic. It may be confusion over wording, varying perspectives, lack of knowledge (and/or skill), or a whole host of other reasons. The point is that there is a “hole” in understanding (can be individual but it is more likely to be a “hole” because of a lack of shared understanding). One example of a hole in collective understanding is the concept of jargon/shared language.

Once we discover a hole in our understanding, someone can stub it out in the wiki with the topic/controversy. When I was learning about discourse communities I posted this stub in Wikipedia to see if others would fill in the gaps. Other Wikipedia contributors, with more knowledge about discourse communities than I had, filled in the details (see history of the page). In this case, the stub was a simple definition of discourse communities taken from the discourse community literature. But a stub could be as simple as a title for an article or as complex as a complete article. The idea is to get something out there that others can interact with.

Interaction with the growth of the knowledge is key to learning. As learners engage in the construction of the knowledge, they become part of that knowledge. Stubbing (and engaging in) a wiki topic meets all three principles for promoting contextual knowing as outlined by Baxter Magolda (1996). She argues that we must “validate students as knowers,” situtate “learning in the students’ own perspective,” and “define learning as mutually constructing meaning” (p. 286).

The underlying power of the wiki is that it allows learners (students and teachers alike) to become part of the knowledge construction process.

For more on stubbing in wiki’s, see Stub in Wikipedia

Other posts that relate using wiki’s in the classroom.

Using Wikis for Learning

Wikipedia in the Classroom
Baxter Magolda, M. B. (1996). Epistemological development in graduate and professional education. The Review of Higher Education, 19(3), 283-304.

Wiki as a Portfolio

Our University-hosed wiki is about to be retired. This page, developed in Sept-Oct 2005 seemed work keeping. I’ve ported it here changing links to preserve the navigation. – np

It generally takes time for different applications of a new technology to be realized — sometimes a long time — and wiki seems to be no different.

Regarding electronic portfolios (another new technology), Wikipedia Wikipedia:Electronic_Portfolio is currently focused on a specific approach to e-portfolios that uses large-scale purpose-built software.

However, at CTLT we understand e-portfolios to include that model as well as other tools — like wiki — which meet the criteria for ‘folio thinking.’ E-portfolio tools need to facilitate activities where learners a) create artifacts that evidence their engagment with core concepts, issues and skills, b) reflect on what those artifacts reveal about how their efforts line up with what they’re trying to accomplish, and c) present the package to an audience. Depending on the context, that audience could be a group of peers who are supporting one another through the learning process, or experts in the area (faculty, community partners, etc.), or even potential employers and/or other institutions and educational programmes to which the student is applying or considering applying.

Blogs are one such tool, as several have noted (Nils’ analysis [new thoughts since that link was originally posted]), and, as this illustrates, wikis are another. The essence of the process is that learners engage in authentic work in a wiki; see for example a course design being tried at Washington State University.

Key elements which make wiki useful for e-portfolios include the “My Contributions” feature, which lets a user easily locate the work they have done, the “diff” listings, which let the user document the change(s) that he/she made in the pre-existing page (and the changes that were made afterwards), and the User page, which offers a place to create a reflection about the diffs.
Locate other articles from Washington State University related to this topic here.

Students Contribute to Wikipedia

Our University-hosed wiki is about to be retired. This page, developed in Sept-Oct 2005 seemed work keeping. I’ve ported it here changing links to preserve the navigation. – np

CTLT Reflection on Experience

Andy Carvin has acknowledged “the hostility that many educators have towards the [Wikipedia] website, particularly their concerns that it can’t be considered a reliable source.”

At WSU’s Center for Teaching, Learning, and Technology, we are approaching classroom use of Wikipedia not from a concern about “reliable” reference sources. Instead, we view it as an active learning environment, and a collabortative one, where students can learn by engaging an authentic task. Furthermore, we want to challenge some of the criteria that have traditionally been used to determine the quality of learning. To get at broader criteria, it is helpful to ask ourselves what kinds of intellectual performance give evidence of deep understanding. As students write, revise, edit, substantiate, and collaborate, Wikipedia offers them opportunities to:

  1. Develop an intellectual voice by persuading an actual community of interested readers
  2. Justify and defend thought in a public setting
  3. Increase awareness of, and engagement with, the history of the writing — the controversies, analysis, perspectives, assumptions, and evidence
  4. Collaborate with others, beyond disciplinary silos
  5. Engage in an authentic task

Recognizing that students might be at varying stages of readiness to contribute to Wikipedia, we have added a scaffolding process where students can explore a topic area, using Wikipedia and other resources, identify key terms and concepts, identify gaps in Wikipedia (missing articles, articles with missing concepts or citations) and then develop and submit new entries to Wikipedia.

To facilitate this scaffolding work, WSU has launched its own instance of MediaWiki, where students and instructors can do things that would be unacceptable in Wikipedia (for example, create categories for specific courses, post syllabi, or other instruction), and generally work in a quiet area outside the rough and tumble of Wikipedia itself. By using MediaWiki for our wiki, students can practice the markup, software features, and protocols of Wikipedians while at the same time learning in a safer space.

See Collaborative Reflection on The Cultural Politics of Sport

See also Wiki as a Portfolio which provides an assessment mechanism for the work above.

Using Wikis for Learning

Our University-hosed wiki is about to be retired. This page, developed in Sept-Oct 2005 seemed worth keeping. I’ve ported it here changing links to preserve the navigation.

NOTE: Some links may not work because this item is quite old.

Andy Carvin and Jimmy Wales have started a public conversation (ca Sept 2005) to see what educators have to say about Wikipedia. They asked questions that focus on Wikipedia in the classroom:

“For example, is Wikipedia something you’d want your students using in the classroom? Do you consider it an appropriate teaching tool?”

These questions suggest a perspective that misses what Carvin suggested in an earlier post on this topic (see the example 5th grade class mid-way down). The broader, better question I think Carvin and Wales mean to ask is, “How can Wikipedia be used to meet education goals like critical thinking, research, and collaborative knowledge generation?” This question gets away from factual accuracy of Wikipedia and toward transforming educational instituions — and even defining how a Wiki-versity would be created.

Carvin’s prompt seems to be focused on K-12 education, Washington State University is able to provide perspective for Wikipedia in Higher Education, which we think may inform the K-12 conversation.

This is an introduction to a multi-voice response by staff in the Washington State University Center for Teaching Learning, and Technology. Some of what we have to say is not being implemented in/with Wikipedia, perhaps can’t be, or shouldn’t be. Some is not in the classroom, but outside. The responses below might suggest how Wikipedia could/should change, or confirm that there is another role for wikis in parallel to Wikipedia.

Discussion about using wikis for learning continues as a recent post on the POD listserve was asking for a good presenter about using wiki’s in the classroom. Our response was to point to this work and to say “if you like what you see, contact us.” One other response suggested Timothy W. Spannaus at Wayne State University who conducted a workshop on wiki use in the classroom at the Lilly National Conference at Miami University, Oxford, OH in 2005.

Students Contribute to Wikipedia

In Students_Contribute_to_Wikipedia we describe a scaffolded course design where students learn about a topic through a reflective process of preparing to make entries to Wikipedia. The process involves using Wikipedia as a text and analyzing its deficiencies, then doing other research to locate resources that would address those issues, and finally developing new articles, editions to existing articles, or bibliographies to enhance existing articles and posting them into Wikipedia.

[For another, more contemporary example, see the Murder, Madness, Mayhem project conducted by a course at UBC.]

Wiki as a Portfolio

Wiki_as_a_Portfolio describes another aspect of the course design above, illustrating how the “My Contributions” feature of Wikipedia can be viewed as a repository of artifacts that a student can use to compose a portfolio of their learning process. It describes how User pages in WSU’s MediaWiki serve as the place to develop reflections on specific “difs” the user is making, and on “difs” that others make to the user’s contributions.

The Shared Language-Jargon Dilemma

Shared Language-Jargon Dilemma addresses the issue of learners entering a community of discourse. Building off EEKim’s post on the Essence of Wiki. In this article, Steve Spaeth develops some more thinking on the Janus-like relationship between jargon and shared language. The process of novices entering a community of discourse can be view as one of moving from jargon to shared language.

Stubbing out as a Teaching Tool

Todd Vanek shows an application of Spaeth’s thinking in Stubbing_out_as_a_Teaching_Tool, where stubbing out pages is a strategy to focus students’ wiki contributions, and at the same time attend to the jargon/shared language issue.

Using Wiki to Develop Communities

Using Wiki to Develop Communities within a university context. In addition to the Wikipedia-supported mechanisms for discovering people with shared interests, WSU is exploring the use of Templates in this wiki to facilitate groups announcing themselves and their intersts, and for users to advertise their membership in groups. Seen in the context of DesRosier’s work, this begins to suggest how students learn about topics outside of the silos of courses.

Norming and workflow for May 17 self study reviews

Norming and workflow for May 17 self study reviews This is another post linked to the notes for OAI Team in this note

There are three images
Results of norming on the Sociology Self-Study from Dec using the May 17 rubric. This chart compares OAI results with the results Gary & Nils brought back from the Assessment, Teaching and Learning conference the end of April. In that setting about 40 conference attendees rated Sociology, with about 10 people rating each dimension. No person rated all 4 dimensions. This session was done without norming, as part of sharing the WSU system.

A key agreement in the norming process was how to treat the self-studies.
1) the whole document is being read to inform each dimension, even if the particular section of the report does not describe the features sought in the rubric.
2) no “benefit of the doubt” or “inferring;” if the program did not describe a feature of its process, then it is to be treated as missing — an attempt to simulate the practice of outside accreditors. It was hoped that the feedback would be used by raters to communicate to programs in constructive ways when the score did not reflect what was known to exist.

Work flow for rating reports walks through the raters and OAI Contact roles and the flow of the documents and data. This process is somewhat changed from the December process, with more focus on how the raters reconcile and provide feedback to the OAI contact– with more emphasis on how to provide feedback to the program.

Plan for first week is a sketch of the timeline and scheduling to help OAI think about how to coordinate among various raters and reviewers.

Self-Study Process Alerts

Self-Study Process Alerts OAI folks,

Don’t forget that you can set an “alert” on the Sharepoint “Spring Self-Studies Process” list, so you get emails that keep you up to date on changes to the list.  Here is a sample (it may appear slightly shrunk due to the way it was captured):

I have my alert set so it is delivered once a day with all the day’s changes;  you can have immediate alerts on each change if you wish.

To set an alert,
go to the current OAITeam site: the top page near the right edge, click on “Welcome ” to drop down a menu.Select “My Settings” from the Menu.  You will see your profile page.From the blue toolbar about 1 inch below the top of the page, click on “My Alerts”.  You will see the “My Alerts on this Site” page. From the blue toolbar, click on “Add Alert”.  You will see the “New Alert” page.From the libraries and lists, select “Spring Self-Studies Process”.  Click “next” to see the alert settings page.Choose your settings and click “OK”.

Note that because this list is based on the “Task” list template, you will get an email when a task is assigned to you in any case.

— Joshua

Process for creating May 17 responses to programs

I am still finalizing a few details, but I know several of you are in position to begin writing your response to programs, so here is a start.

Documents and templates you need are in the left menu “Site Hierarchy/Documents library” of the site where we are working

As part of Process Actions Steps 7 & 8, the OAI Contact should:

More on what I think the remaining steps are:

  • Josh and I are still working on how you will record the  reconciled rater scores
  • Post the completed Response to Program in the AWE folder along side the program’s self study
  • Set the status of the Process Actions to alert Ashley to read the draft.
  • When Gary finishes his review, use the OAI Email template to send comments back to programs