Beyond The University

“More than one-third of the world’s population is under 20. There are over 30 million people today qualified to enter a university who have no place to go. During the next decade, this 30 million will grow to 100 million. To meet this staggering demand, a major university needs to be created each week.” —Sir John Daniel, 1996 in John Seely Brown Minds on Fire: Open Education, the Long Tail, and Learning 2.0 (EDUCAUSE Review) | EDUCAUSE

In Can “The Least Of Us” Disrupt and Change Education for “The Rest Of Us?” Rob Jacobs writes: “Disruption will come when the poor of the world figure out ways to educate themselves and their neighbors via the Internet. Of course this education won’t match the focus, rigor, and quality of Western schools, but never the less, the drive and need to learn will create a youth movement in these developing countries for using the Internet as a tool to educate themselves and others.”

By ‘disruption’ I gather Jacobs means in the Innovator’s Dilemma sense of Clayton Christensen, where a new business model (or new technology) disrupts the existing businesses using the existing model.

John Brockman quotes Don Tapscott “Universities are finally losing their monopoly on higher learning… There is fundamental challenge to the foundational modus operandi of the University — the model of pedagogy. Specifically, there is a widening gap between the model of learning offered by many big universities and the natural way that young people who have grown up digital best learn.”

There seem to be two intertwined ideas here: technology access and pedagogy.

So, if a large unmet demand for education exists at a price point well below that of a university degree, and the Internet infrastructure is available to deliver both the content and the collaboration, will the higher education model in the developed world succumb to Jacob’s disruptive revolution from below? And if so, when? And which players will be the facilitators?

Lets take the time dimension out of the problem. Will the revolution be substantially underway by 2020 (10 years out)? I pick 10 years because it is about one cycle of accreditation review for most US universities and it would be at the outer edge of most corporate planning. Here is why I conclude the preconditions for Jacob’s bottom up revolution, the technical infrastructure, will exist within 10 years if they don’t already exist today. The question is, will learners develop a reflective practice within an optimistic, constructivist and collaborative pedagogy to realize the potential, and if they do, will this self-organizing revolution be sufficient to engulf the developed world’s educational systems.

The Technical Infrastructure Exists

First, a conversation between Charlie Rose and Eric Schmidt, CEO of Google points to the direction for the technology of continued exponential improvement of hardware (mobile devices, Google’s servers and the network bandwidth) following Moore’s law with video recording and GPS-aware applications standard on phones. As illustration, my cell phone in 2000 had a small B&W screen, could store phone numbers and make calls. My cell phone today is an iPhone. In another decade my cell phone will be equally unrecognizably different from my iPhone of today.

Second, Seb Paquet proposed the idea of making group-forming ridiculously easy and postulated an extension to Reed’s Law: “The value of a group-forming network increases exponentially with the number of people in the network, and in inverse proportion to the effort required to start a group.”

Mechanisms for finding, forming or joining groups can be expected to be enhanced. Amazon and Google demonstrate the impact of personalize recommendations based on data about you. Their capacity to recommend will grow, and those recommendations could help learners discover and expand niche communities.

Third, people younger than myself seem to think nothing of making video to communicate their ideas. The rate of production of user content on YouTube will increase beyond the current 1000 hrs of user created video uploaded per hour. Micheal Wesch describes how media are becoming environments that change our conversations.

Fourth, Internet World Stats provides information on the level of Internet access worldwide by region.  On a planetary basis, 24% of the world’s population has access today and the growth rate is 362% over the period 2000-2009. In another decade it seems credible that almost anyone wanting Internet access (e.g., wanting an education) will have it.

I contend that the technology is in place (MediaWiki alone is adequate to the task) and that there is a sufficiently large user base to start the revolution (over a billion people outside North America and Europe connected Internet (World Internet Stats, 9/2009)).

The Pedagogy Exists (and is being implemented outside of universities)

In her review (ca 2000) of George Hillocks Jr.’s book Ways of Thinking, Ways of Teaching. New York: Teachers College Press, 1999. Carol Rutz (Writing Program Administration 23.3, Summer 2000, 127-129) describes Hillocks’ analysis of his subjects’ teaching approaches:

“[Their] four ways of thinking are derived from two epistemological stances, the objectivist—knowledge is “out there” to be apprehended and understood—and the constructivist—knowledge is constructed actively by learners as they interact with the world. These positions are modified by the teacher’s attitude toward students. The pessimist views students as defective creatures unable to learn without close supervision, whereas the optimist views students as capable and eager to learn.”

Hillocks constructs a 2×2 matrix of these stances and uses them to analyze the behaviors of English composition teachers.

Near the end of her review, Rutz notes and laments, “Had Hillocks interviewed students as well as teachers, his case for the effects of reflective practice within an optimistic, constructivist pedagogy would likely become even stronger.”

Unspoken in Rutz’ review is that this is an analysis of classroom teachers, published in 1999 before the emergence of Web 2.0. What happens if one relaxes the assumption that teachers are meeting students in classrooms and explores the “effects of reflective practice within an optimistic, constructivist pedagogy” on learners in Internet communities? That is a key phrase ‘reflective practice within an optimistic, constructivist pedagogy’ and a key observation, it is not held by all teachers.

In a 2007 study, we have found similar spectrum of teacher’s beliefs about teaching, ranging from teacher-centered, to learning-centered, to learner-centered and as part of our Harvesting Gradebook work we found s spectrum of learning beliefs among students, ranging from teacher-authority to learner-as-agent.

John Seely Brown begins to give us an idea how widely this practice/pedagogy is held by learners (outside the university).

“The most profound impact of the Internet, an impact that has yet to be fully realized, is its ability to support and expand the various aspects of social learning. What do we mean by “social learning”? Perhaps the simplest way to explain this concept is to note that social learning is based on the premise that our understanding of content is socially constructed through conversations about that content and through grounded interactions, especially with others, around problems or actions. The focus is not so much on what we are learning but on how we are learning.”

“[I]nstead of starting from the Cartesian premise of “I think, therefore I am,” and from the assumption that knowledge is something that is transferred to the student via various pedagogical strategies, the social view of learning says, “We participate, therefore we are.” “

“A contemporary model that exemplifies the power of this type of social learning is provided by the distributed virtual communities of practice in which people work together voluntarily to develop and maintain open source software. The open source movement has produced software such as the Linux operating system and the Apache web server, which have offered surprisingly robust alternatives to commercial products. …

“Open source communities have developed a well-established path by which newcomers can “learn the ropes” and become trusted members of the community through a process of legitimate peripheral participation. [‘legitimate peripheral participation’ is another key phrase for this new learning model. np]  New members typically begin participating in an open source community by working on relatively simple, noncritical development projects … As they demonstrate their ability to make useful contributions and to work in the distinctive style and sensibilities/taste of that community, they are invited to take on more central projects. Those who become the most proficient may be asked to join the inner circle … Today, there are about one million people engaged in developing and refining open source products, and nearly all are improving their skills by participating in and contributing to these networked communities of practice.”

The “hole in the wall experiment” (below) is the low tech version of this idea.

The Long Tail in Learning (learning without the University)

Seely Brown continues…

“Chris Anderson, the editor of Wired, has shown that Internet-based e-commerce differs from commerce in the physical world. In the world of physical retailing, and particularly in areas of selling goods like books, music, and movies, sales are usually dominated by best-sellers. Typically, 20 percent of titles generate 80 percent of all sales, [b]ut Anderson notes that e-commerce sites such as, Netflix, and Rhapsody don’t follow this pattern. They are able to maintain inventories of products—books, movies, and music—that are many times greater than can be offered by any conventional store. The result is an economic equation very different from what has prevailed in the physical world … the bulk of their sales comes from their vast catalogs of less-popular titles, which collectively sell more than the most popular items. … From the customers’ standpoint, online enterprises offering unprecedented choice are able to cater much more efficiently to individual tastes and interests than any brick-and-mortar store.”

“As more of learning becomes Internet-based, a similar pattern seems to be occurring. Whereas traditional schools offer a finite number of courses of study, the “catalog” of subjects that can be learned online is almost unlimited. There are already several thousand sets of course materials and modules online, and more are being added regularly. Furthermore, for any topic that a student is passionate about, there is likely to be an online niche community of practice of others who share that passion. … The Faulkes Telescope Project and the Decameron Web are just two of scores of research and scholarly portals that provide access to both educational resources and a community of experts in a given domain. The web offers innumerable opportunities for students to find and join niche communities where they can benefit from the opportunities for distributed cognitive apprenticeship.”

Seely Brown gives an example of this change:

“A very different sort of initiative that is using technology to leverage social learning is Digital StudyHall (DSH), which is designed to improve education for students in schools in rural areas and urban slums in India. The project is described by its developers as “the educational equivalent of Netflix + YouTube + Kazaa.” Lectures from model teachers are recorded on video and are then physically distributed via DVD to schools that typically lack well-trained instructors (as well as Internet connections). While the lectures are being played on a monitor, a “mediator,” periodically pauses the video and encourages engagement among the students by asking questions or initiating discussions about the material they are watching. The recorded lectures provide the educational content, and the local mediators stimulate the interaction that actively engages the students and increases the likelihood that they will develop a real understanding of the lecture material through focused conversation.”

The biggest example of all may be Wikipedia, who’s goal is to get a free encyclopedia to everyone in the world. It averaged 379 Million page views/day (all languages) in Sept 2009. More importantly than being a top global website, Wikipedia demonstrates a how to create a self-organizing learning organization using volunteers.

While the Seely Brown and Wikipedia examples points to some sophisticated (and already educated) communities, Sugata Mitra gives an example of third-world children using similar strategies to teach themselves to use computers and the Internet.  In his Hole in the Wall project, young kids figured out how to use a PC on their own — and then taught other kids. He asks, “what else can children teach themselves?”


Extending Clay Shirkey’s analysis of the problems facing newspapers, we might conclude Universities may be denying the impending revolution.

“When reality is labeled unthinkable, it creates a kind of sickness in an industry. Leadership becomes faith-based, while employees who have the temerity to suggest that what seems to be happening is in fact happening are herded into Innovation Departments, where they can be ignored en masse. This shunting aside of the realists in favor of the fabulists has different effects on different industries at different times. One of the effects on the newspapers is that many of their most passionate defenders are unable, even now, to plan for a world in which the industry they knew is visibly going away.

“The curious thing about the various plans hatched in the ’90s is that they were, at base, all the same plan: “Here’s how we’re going to preserve the old forms of organization in a world of cheap perfect copies!” The details differed, but the core assumption behind all imagined outcomes (save the unthinkable one) was that the organizational form of the newspaper, as a general-purpose vehicle for publishing a variety of news and opinion, was basically sound, and only needed a digital facelift.

“With the old economics [of newspaper publishing] destroyed, organizational forms perfected for industrial production have to be replaced with structures optimized for digital data. It makes increasingly less sense even to talk about a publishing industry, because the core problem publishing solves — the incredible difficulty, complexity, and expense of making something available to the public — has stopped being a problem.”

I think there is evidence to believe a “reflective, self-organizing, constructivist and collaborative pedagogy” is already practiced by many people organized into communities that make use of “legitimate peripheral participation.” So, to paraphrase Shirky: It makes increasingly less sense even to talk about a University industry, because the core problem Universities solve — the incredible difficulty, complexity, and expense of building knowledge and making it available to the public — has stopped being a problem.

Dry tinder is in the box, all that is needed is a hot spark.

What will the spark be, and where would it most effectively land?

Not your father’s Portfolio

We were working with a writer on an article about ePortfolios to appear in Campus Technology (its here 11.2009). One of our examples to illustrate our thinking about ePortfolios was Margo Tamez’ El Calaboz Portfolio. Our writer got back with this:

“The editor for my article about eportfolios had a question about my coverage of Margo Tamez’s eportfolio usage. She had expressed concern that the eportfolio have a home beyond the duration of the court case. Does Washington State have any kind of official policy or practices specifically about the life of its student eportfolios? Is there any kind of guarantee that it will live on after the student has left the institution? Anything you can say about that?”

There is a short answer and a long answer to the question.

Short answer: WSU has no policy or procedure in place to delete a student’s SharePoint mySite (where Tamez portfolio is) after graduation, but after 12 months this site becomes read-only unless the graduate makes a specific request to have management access restored.

Long answer: The problem with the short answer is that it focuses on the technical survival of a specific thing at a specific URL. Thinking about a specific collection of artifacts in a specific system at the specific URL is too narrow a focus for our understanding of an ePortfolio.

At the risk of insulting the Campus Technology editor by paraphrasing a Oldsmobile ad, an ePortfolio ‘is not your father’s Portfolio,’ by which I mean that in our view an ePortfolio is not at all an electronic counterpart of the paper portfolio.

An electronic portfolio is both more durable and more tenuous than its paper predecessor. Its also more powerful. Its not a thing or a place, its a practice.

Googling Margo Tamez (she is lucky to have a fairly unique name)  illustrates that she built her electronic reputation in many places, that is, her ePortfolio is not in one place. Rather, it is the sum of artifacts imbedded in the contexts of the communities where she was working. This image is a Touchgraph (link to live Touchgraph) of a search for Margo. It shows her portfolio as a collection of the web 2.0 places she is working.

TouchGraph rendering of Google results for Margo Tamez

Due to the nature of the problem she was working on, Margo intentionally built her portfolio in a distributed fashion. Many of the key documents were emailed to a list of readers, where the body of the email served as description and context for the document. Her WSU ePortfolio was the recipient of a cc: of those emails. Other pieces of her work were created in wikis or as guest posts in blogs. She worked in her community, keeping the artifacts of her work (her ePortfolio) in the places that were best suited to them.

As part of our ePortfolio case study work, we interviewed Dennis Haarsager, now Senior Vice President for System Resources and Technology at National Public Radio, about blogging and building portfolios in public places. In our reflection we said:

“In our interview, Haarsager argued for the public lectures he gives on his chosen problem. The lecture is a showcase portfolio of Haarsager’s current, best thinking. The medium is mostly broadcast, but he feels it allows him to reach new audiences, and to get kinds of feedback about his ideas that he does not get in comments on his blog.

“Tamez is also creating showcase “mini-portfolios” in the form of printed fliers and media interviews. These productions may have some of the risk-related prestige that Tenner ascribes to printed books, while at the same time having the new audience-reaching and immediacy values that Haarsager associates with his lectures. In her learning portfolio, these mini-portfolios document where Tamez’ thinking was at points in her learning trajectory.”

Thus, our thinking is that ePortfolios are created as by-products of work, and are scattered across the venues and contexts in which the work is conducted. An ePortfolio is continually dissipating as systems storing the work go away, and continually growing as new work is added.

I have been struggling for awhile with the problem of describing a 21st century resume.  It too is not like its 20th century counterpart. In that 2007 post I did not yet fully recognize the obvious, which I’m coming to see here. My blog(s) and the other places I post online are my ePortfolio (and my resume).

Rather that focusing on the durability of an ePortfolio system or URL, the most important things we see about an ePortfolio (and ePortfolio as 21st century resume) are the abilities to:

  1. find your work when you need it for reflection or repurposing,
  2. establish that you are indeed the author (possibly under multiple identities) of the works you wish to claim, and
  3. leverage the Google Juice of your work so that it helps you be found by people who share your interests and can help you in your work.

The first of these requirements is most likely met with a hybrid of several Web 2.0 tools. It could be supplemented with a social bookmark service where you track yourself.

The second challenge, proving that the work is “yours” is probably done by making a claim to a corpus of works rather than to a single piece, and by making an appeal to a community and context in which the work was done. (Unlike Catherine Howell‘s thought (ca 2005)  that “universities have a role in ‘authenticating’ individuals [and endowing]… them with certain attributes,” we think an ePortfolio world that enables community-based learning and community-based credentials breaks those assumptions about the university, see a recent piece for AAC&U.)

The third requirement is met by working in public and working in venues where your community of practice will likely congregate and then linking from those contexts to works you created in other contexts that contribute to the conversation.

This third point can be illustrated if you Google me (Nils Peterson). You will discover that there are two people using that name with different career trajectories.  Nils Peterson the Poet is in the Bay Area and worked at San Jose State. His identify is authenticated by a variety of news stories (that is, a community of other writers know that he is who he claims and they are in agreement in their accounts of him).

I claim to be other Nils Peterson who is (currently) prominent in Google, the Nils Peterson who publishes in Campus Technology as well the author here, and the blogger at  I have made a consistent effort to create user identities using Nils+Peterson in many systems and to link from one system to another. This strengthens my claim to be the Nils Peterson who is saying all those things. I don’t depend on my employer or the universities that educated me to substantiate my claims, but I do depend on the corroboration of the communities in which I work.

But, the claim is circumstantial, like solving a jig saw puzzle by inferring which pieces fit together. Following the notions of Helen Barrett, and because I work online in public, my ePortfolio (and resume) is a life lifelong and life wide  web of the works Google associates with me, where ever they exist.

Crowd-sourcing feedback

David Eubanks commented on our recent Harvesting Feedback demo. I’ll save replying about inter-rater reliability to focus now on his suggestion of using Mechanical Turk and the very insightful comment about the end of “enclosed garden” portfolios.

I think David correctly infers that Mechanical Turk is a potential mechanism to crowd-source the Harvesting Feedback process we are demonstrating. Its an Amazon marketplace to broker human expertise. The tasks, “HITs” (Human Intelligence Tasks) are ones that are not well suited to machine intelligence, in fact the site bills itself as “artificial artificial intelligence.”

To explore Mechanical Turk, I joined as a “Worker” to discover that “Requesters” (sources of HITs) can pre-qualify Workers with competency exams. I’m now qualified as a ‘”Headshot” Image Qualifier’ a skill to identify images that meet certain specific criteria important to requester Fred Graver. I also learned that workers earn (or maintain) a HIT approval rate, which is a measure of how well the worker has performed on past tasks. One might think of this as how well the worker is normed with the criteria of the task (though the criteria in this case are not explicit (which is a weakness in our view)). Being qualified for a task might be analogous to initiation to a community of practice; but one would need to then practice “in community” which Mechanical Turk does not seem to support.

We’ve also been exploring a couple other crowd-source feedback sites that help flesh out the character of this landscape. Slashdot and Leaf Trombone (website and video). Slashdot is a technology-related news website that features user-submitted and editor-evaluated current affairs news with a “nerdy” slant. Leaf Trombone is a game for the iPhone that lets you use your iPhone to play a slide trombone to a world audience.

The three systems are summarized in this table:

Mechanical Turk Leaf Trombone Slashdot
Goal of site/ developer’s reason for using reputation in the site Distributed processing of non-computable tasks/ sort for suitable workers Selling an iPhone app/ use ego to encourage players Building a reliable source of information/ screen for editors who can take high level tasks
Type of reputation / Participant’s purpose for having a good reputation Private reputation/ to secure future employment; earn more income Public reputaiton/ status in the community as player and judge; ongoing participation Public reputation/ enhanced opportunity to contribute to the common good (as opposed to being seen as clever fellow
Type of Reward/ Motivation for participant Money/ Personal gain Personal access to perform on world stage/ learning & fun “Karma” to enable future roles in the community/ improve the information in the community
Performance Space/ durability of the performance Private space (enclosed garden)/ durability is unknown, access to the performance is only available to the Requestor Public stage & synchronous; a new playback feature makes performances durable, but private for the artist Public stage & asynchronous/ permanent performance visible to public audience
Kind of feedback to the participant/ durability of the performance Binary (yes/no) per piece of work completed; assessments are accumulated as a lifetime “approval rate” score Rating scale & text comment per performance/ assessment are stored for the performer Rating scale per posting/ assessments are durable and public for both individual items and are accumulated into the user’s “Karma” level
Assessment to improve the system This could be implemented by the individual “Requester” if they desire ? High “Karma” users engage in meta-assessments of the assessors
Kind of learning represented Traditional employer authority sets a task and is arbiter of success; the goal is to weed out unsuccessful workers Synchronous, collaborative individual learning – judge as learner; performer as learner Asynchronous collaborative community learning
Type of crowd-sourcing Industrial model applied to crowd of workers Ad hoc judges gathered as needed for a performance Content and expertise are openly shared

The three systems represent an interesting spectrum, and each might be applied to our challenge of crowd-sourcing feedback. But looking at the different models they would have very different impacts on the process. I believe that only Slashdot’s model could be sustained by a community over an extended period of time, because it is the only one that has the potential to inform the community and build capital for all the participants.

The table above got me to think about another table we made, diagraming 4 models for delivering higher education. At one side of the chart is the industrial, closed, traditional institution. It progresses through MIT’s open courseware and Western Governor’s University’s student collected content and credit for work experiences to the other end of the chart that we called Community-based Learning.

Three rows in our chart addressed the nature of access to expertise, the assessment criteria, and what happens to student work. The table above informs my thinking on those dimensions. As I’ve charted it, in the Slashdot model expertise is open, assessment is open. (while assessment criteria are obscure, the meta-assessment helps the community maintain a collective sense of the criteria) and the contributer’s (learner’s) work remains permanently as a contribution to the community. This is what I think David is referring to when he applauds the demise of the “enclosed garden” portfolio.

A reason to work in public is to take advantage of an open-source/ crowd-wisdom strategy. David illustrated the power of “We smarter than me”  when called our attention to Mechanical Turk.

Another reason is the low cost to implement the model. Recently the UN Global Alliance for Information and Communication Technology and Development (GAID) announced the newly formed University of the People, a non-profit institution offering higher education to the masses. In the press briefing, University of the People founder Shai Reshef said that “this University opened the gate to these [economically disenfranchised] people to continue their studies from home and at minimal cost by using open-source technology, open course materials, e-learning methods and peer-to-peer teaching.” [emphasis added]

We propose that to be successful the University of the People must implement its peer-to-peer teaching as community-based learning and include a community-centric, non-monetary mechanism to crowd-source both assessment and credentialing.

Harvesting feedback on a course assignment

This post demonstrates harvesting rubric-based feedback in a course, and how the feedback can be used by instructors and programs, as well as students. It is being prepared for a Webinar hosted by the TLT group. (Update 7/28: Webinar archive here. Minutes 16-36 are our portion. Minutes 24-31 are music while participants work on the online task. This is followed by Terry Rhodes of AAC&U with some kind comments about how the WSU work illustrates ideas in the AAC&U VALUE initiative. Min 52-54 of the session is Rhodes’ summary about VALUE and the goal of rolling up assessment from course to program level. This process demonstrates that capability.)

Webinar Activity (for session on July 28) Should work before and after session, see below.

  1. Visit this page (opens in a new window)
  2. On the new page, compete a rubric rating of either the student work or the assignment that prompted the work.

Pre/Post Webinar

If you found this page, but are not in the webinar, you can still participate.

  • Visit the page above and rate either the student work or assignment using the rubric. Data will be captured but not be updated for you in real time.
  • Explore the three tabs across the top of the page to see the data reported from previous  raters.
  • Links to review:

Discussion of the activity
The online session is constrained for time, so we invite you to discuss the ideas in the comment section below. There is also a TLT Group “Friday Live” session  being planned for on Friday Sept 25, 2009 where you can join in a discussion of these ideas.

In the event above, we demonstrated using an online rubric-based survey to assess an assignment and to assess the student work created in response to the assignment. The student work, the assignment, and the rubric were all used together in a course at WSU. Other courses we have worked with have assignments and student products that are longer and richer, we chose these abbreviated pieces for pragmatic reasons, to facilitate a rapid process of scoring and reporting data during a short webinar.

The process we are exploring allows feedback to be gathered from work in situ on the Internet (e.g., a learner’s ePortfolio), without requiring work be first collected into an institutional repository. Gary Brown coined the term “Harvesting Gradebook”  to describe the concept, but we have come to understand that the technique can “harvest” more than grades, so a better term might be “harvesting feedback.”


This harvesting idea allows a mechanism to support community-based learning (see Institutional-Community Learning Spectrum). As we have been piloting community-based learning activities from within a university context, we are coming to understand that it is important to assess  student work and assignments and the assessment instruments.

Importance of focusing assessments on Student Work

Gathering input on student projects provides the students with authentic experiences, maintains ways to engage students in  authentic communities, helps the community consider new hires, and gives employers the kind of interaction with students that the university can capitalize when asking for money. But, we also have come to understand that assessing student learning often yields little change in course design or learning outcomes, Figure 1. (See also )

graph of 5 years of outcomes data

Figure 1. In the period 2003-2008 the program assessed student papers using the rubric above. Scores for the rubric dimensions are averaged in this graph. The work represented in this figure is different than the work being scored in the activity above. The “4” level on the rubric was determined by the program to be competency for a student graduating from the program.

The data in Figure 1 come from the efforts of a program that has been collaborating with CTLT for five years. The project has been assessing student papers using a version of the Critical Thinking Rubric tailored for the program’s needs.

Those efforts, measuring student work alone, did not produce any demonstrable change in the quality of the student work (Figure 1). In the figure, note that:

  • Student performance does not improve with increasing course level, eg 200,300,400-level within a given year
  • Only one time were students judged to meet the competency level set by the program itself (2005 500-level)
  • Across the years studied, student performance within a course level did not improve, e.g., examine the 300-level course in 2003, 2006, 2007, 2008

Importance of focusing assessments on Assignments

Assignments are important places for the wider community to give input, because the effort the community spends assessing assignments can be leveraged across a large group of students. Additionally, if faculty lack mental models of alternative pedagogies, assignment assessment helps focus faculty attention on very concrete strategies they can actually use to help students improve.

The importance of assessing more than just student work can be seen in Figure 1. As these results unfolded, we suggested to the program that it focus attention on the assignment design. They just did not follow through as a program in reflecting on and revising the assignments, nor did they follow through with suggestions to improve communication of the rubric criteria with students.

Figure 2 shows the inter-rater reliability from the same program. Note that the inter-rater reliability is 70+% and is consistent year to year.

Graph of inter-rater reliability data

Graph of inter-rater reliability data

This inter-rater reliability is borderline and problematic because, when extrapolated to high stakes testing, or even grades, this marginal agreement speaks disconcertingly to the coherence (or lack there of) of the program.

Figure 3 comes from a different program. It shows faculty ratings (inter-rater reliability) on a 101-level assignment and provides a picture of the maze, or obstacle course, of faculty expectations that students must navigate. Higher inter-rater reliability would be indicative of greater program coherence and should lead to higher student success.

Interrater reliability detail

Importance of focusing assessments on Assessment Instruments

Our own work and Allen and Knight (table 4) have found that faculty and professionals place different emphasis on the importance of criteria used to assess student work. Assessing the instrument in a variety of communities offers the chance to have conversations about the criteria and address questions of the relevance of the program to the community.


The intention of the triangulated assessment demonstrated above (assignment, student work and assessment instrument) is to keep the conversation about all parts of the process open to develop and test action plans that have potential to enhance learning outcomes. We are moving from pilot experiments with this idea to strategies to use the information to inform program-wide learning outcomes and to feed that data into ongoing accreditation work.

Blackboard + Angel = reason for open learning

In response to the Wired Campus article about Blackboard’s acquisition of Angel Learning,  Scott Leslie commented about moving beyond the LMS to networked learning options. His comment led me to this 2005 post where he saw the social software light and a later post looking for help making the case for “fully open” content.

I think it would be useful to go beyond open content to other aspects of openness, such as Downes’ open assessment. Beyond open content and assessment lie open problems worked in community. One of the problems, if students are working in the cloud, is how to manage the assessment of the distributed student work. WSUCTLT has been exploring how to gather assessments from the cloud, with an idea called Harvesting Gradebook. This has led them to look at teaching/learning practices along a spectrum from institution-based to community based (PDF). That work began with perspectives shaped by institutional contexts, but is now branching out to examine open learning outside the univeristy‘s walls.

Scott, as you suggest, part of making the case for open, is the potential for greater scalability. When the institution locks up the content and locks up the expertise in a course, then its hard to scale. MIT’s open content takes one step. Western Governor’s University takes another step with its “bring your own content and experience” strategies. Community based learning unlocks the content, the experts and the assessment, making for the most scalable solution. (Diagram of WSUCTLT thinking on these four models (PDF).)