One of the reasons that online course discussions fail is the lack of an assessment system that not only accurately scores quality student participation but actually encourages high quality participation.
Some courses actually do not grade student discussions. I am not sure why, but I suspect it is because instructors similarly fail to grade face to face discussions and somehow see it as an inappropriate interference in the students’ free expression. It is possible that it can become exactly that, but that is only with poorly designed systems. If there is one thing that I have learned about the difference between online education and face to face instruction is that in online, if you aren’t grading it, they aren’t doing it. In face to face, your mere physical observation of their participation (or lack of it) is perceived to be a form of assessment that encourages participation. That physical observation is missing in online discussions.
The most common ways of grading discussions are the most destructive. If you really want to have a dull, meaningless discussion, make the number of posts the primary factor in assessment. In general, whenever you have a scoring criterion that places a primacy on quantity over quality, you will usually get quantity without quality. Ask for three posts per student, and you will get three posts per student, with the content of each being something like “post one,” “post two,” and “post three.” You will get many an “I agree” or “I disagree” post that adds nothing to the discussion. In almost all rubric creation, I do everything I can to avoid having actual numbers.
The best way to assess discussions, by far, is with a high quality rubric. The problem is that most instructors have little or no training in rubric creation, and they will usually create a poor rubric when given the chance. I believe that is because of an incorrect notion that they have to have objective measures of performance in the rubric. That is why you get numbers of posts as a criterion. If you look, in contrast, at the scoring rubrics used by the College Board to assess Advanced Placement essays, you won’t see anything like that. These rubrics describe different levels of quality in the real terms you would use to describe a quality performance, without any need to identify objective criteria. They use terms like “well written,” for example. Somehow they are able to score these essays with an inter-rater reliability close to 90%.
Start by identifying the characteristics of high quality participation. Such people post early in the discussion so that others have the benefit of their initial thinking. They make thoughtful comments, using supporting information related to the topic of the discussion as appropriate. They stay on topic. They are respectful toward others. When they agree with others, they make their reason for agreement clear, often adding additional, complementary thoughts. When they disagree, they do so respectfully, giving clear reasons for their disagreement. When they are confronted with new arguments in opposition to theirs, they consider them carefully and even change their minds if appropriate.
Next, think of the characteristics of poor participation. Such posters post late, often submitting a flurry of posts at the end of the discussion period to make it seem as if they have been actively participating throughout. They can be rude or insensitive. They are often off topic. They write empty posts that simply agree or disagree without explanation. They offer no new insights or information. When they are confronted with new arguments in opposition to theirs, they cling blindly to their original position, even when that position becomes untenable.
Notice that the descriptors above do not require the poster to agree with a specific line of thought, so they do not stifle student free thought. (A question that demands that students come to a specific line of thought is rarely a well designed question, but that is a topic for another article.)
Now, take these various descriptors and arrange them in blocks ranging from a superior performance to a failing performance, and you have a quality rubric that will not only accurately measure student performance against real world criteria, it will actually teach them how to participate effectively. Most importantly, it will help create that fine level of discussion you always wanted. Some instructors will grade each discussion individually, and some instructors will give an overall grade to a group of discussions. Neither approach is necessarily right or wrong. I think it depends upon the number and type of discussions in the course.
If your discussions have been failing to meet your objectives, try playing with your assessment system. You may be pleasantly surprised by the results.
Filed under: Uncategorized |
John Mark Walker: "If educational communities can continue to push platform integration and content portability, in the future, students may be able to design their own personalized degrees from smaller, modular chunks that cross institutional barriers" (
Richard Koubek
Judith McDaniel
Tim Fraser-Bumatay: "Although the format leaves us far-removed physically, the online forum has its own sense of intimacy" (Judith McDaniel, "
Ryan Kelly: "For me to be able to work with people clear across the country for an extended period of time opened me up to new things" (Judith McDaniel, "
Daniel Herrera: "As a Mexican American, I know that words of identity are powerful; so to discuss white privilege with my professor and classmates in a face-to-face class would have been terrifying and impossible" (Judith McDaniel, "
Cathy Gunn: "Traditional methods for effecting change at my institution aren’t getting us even to a trickle yet, let alone to thinking about or planning for a wave!" (
Educational Technology and Change Journal 
Hi John,
The first online course I took was Development and the Internet by the Harvard Berkman Center, on their H20 platform, in 2003.
There was no assessment in that case, but the H20 platform has a nifty feature for discussions called Rotisserie (see the rotisserie discussions for that Development and the Internet course).
Everybody got a question to answer within a given deadline when all answers got published, then everybody got assigned by the software to comment on someone else’s answer within a new deadline, etc. As explained in the part about the Rotisserie in About H20:
As H20 is free software, maybe it could be tweaked to add an assessment feature to the rotisserie? But anyway, the point of discussion assessment is to make people better at discussing, not vice-versa, surely? And even without a built-in assessment feature, it would be easier to assess the quality of each participant’s contribution if they all have to start each stage of the discussion on a par, I think.
John, you bring up a number of good points and offer excellent suggestions. However, my sense is that assessing e-discussions is a complex issue with a number of variables that dictate a teacher’s options.
For example, in my online classes, I have approximately 80 students posting comments in multiple forums and threads every week. Practically speaking, there’s no way I could assess all of their posts.
This is an area where e-teaching assistants would be a godsend. But we’re not there — yet.
Thus, I have to be selective in where I place my assessment efforts. For many discussions, a simple self-report is all I require, i.e., an “X” before the activity in a learning log.
However, many of the discussions have built-in incentives, e.g., students are often required to quote classmates’ opinions and arguments in their papers, and being quoted is considered an honor by most students. Thus, they put effort into their posts, and being quoted is a thumbs up as far as quality is concerned.
As a writing teacher, I assess posts that are directly related to the writing process. For example, students review their classmates’ drafts and post their comments in the writer’s blog. The criteria for these peer reviews are spelled out in the assignment and in the general guidelines for all drafts.
I score the reviews, and the scores impact their grade for a particular assignment.
This evaluation of student reviews is labor-intensive. Again, this is an area where online teaching assistants would be extremely helpful.
In the classroom, teachers are often forced to make compromises based on practical considerations. With the right kind of support, we could reduce or even eliminate some of these compromises. This is one of the reasons why I’m critical of funds earmarked for online learning being diverted to developing and maintaining bureaucracies to oversee online teachers. The limited dollars that go into administering ought to be going into teaching. -Jim S
You don’t have to assess every post. You just have to get a good overall sense of the kind of participation people are getting.
In the forum I mentioned in reference to an earlier article, I mentioned a large forum, with thousands of participants. One person started a thread on this topic: who are the best participants in these forums? There were hundreds of posts that followed, with different people getting nominations and explanations for those nominations.(I am happy to say I received my share of nominations.) Out of all those hundreds of posts, there were not all that many nominations, when you compare it to the potential total. The top names were mentioned over and over again.
If you are reading the posts, then you know who is participating at a top level and who is not. The key is to avoid getting too anal about the grading. If they are doing what you expect, give them the top score and move on.
As Claude suggested, the real point of the rubric is to give them guidance on what is expected of them. If you see that they are responding to that guidance as you read, then your grading is done.
it is the ones who are not responding appropriately who really stick out.
Thanks for the clarification, John. Yes, the process you describe can definitely be implemented. I’ve tried a similar rating system in the past. I posted criteria for excellent comments as well as examples from past discussions. Selected students emailed their ratings to me; each student was required to serve as a rater at some point during the semester. I announced the top commenters at the end of the rating period. This worked to some extent. I’ve tried different variations, too. The problem was time. This, too, required time I didn’t have. -Jim S
I agree, John, that playing with your assessment strategy is important, but I think we need to start at the other end of our course design–how we create the prompts or ask the questions that are going to get the discussion going.
Here are a few pointers I use to remind myself when i build a new course or a new discussion forum:
•Be specific. If I want a substantive response, I need to tell folks what that means. Respond to all of the articles? Some of them? Attempt this assignment and tell me what you discovered?
•Be inclusive. If I want them to finish this assignment and then move on to another, let them know that now. Too much information is always better (if it is clearly stated) than too little, and I can always repeat it later if they’ve forgotten.
•Be exclusive. Leave out the things that don’t matter. Just because I can work in my favorite quote or picture—that doesn’t make it pertinent or useful.
•Be relevant. It has to matter to folks, not just because they signed up for this discussion, but because it is relevant to or applicable to their lives. That’s when I have the absolutely best discussions. “Women between 20 and 55 use more healthcare than men. Are insurance companies therefore justified in charging them more for health insurance?” Now THAT was a discussion!
Judith, we cover a lot of ground when we combine your tips with John’s! Your “be relevant” is especially helpful for me. Now that you mention it, the times when I’ve done that, discussions have picked up. Now I’ll have to remember to apply this rule consistently. -Jim S