By Jim Shimabukuro
George Siemens, in an interview with Audrey Watters, says, “In terms of evaluation of learners, assessment should be in-process, not at the conclusion of a course in the form of an exam or a test” (“How Data and Analytics Can Improve Education,” O’Reilly Radar, 25 July 2011). In the context of online learning, he’s underscoring the data mining tools built into learning management systems to do just that – provide on demand information on student log-ins, participation, completion of activities, etc. that can be used to formatively monitor progress. He also mentions the capacity to mine more complex data such as the quality of a student’s performance, but this area is still relatively unexplored.
Still, teachers are discovering that online classes provide mountains of qualitative digital data for each student. In essence, everything that’s done by everyone in an online class is automatically recorded and archived. For example, I teach completely online writing courses and have access to a mind-boggling amount of performance information. For each student, I can access all email exchanges that we’ve had, all discussion forum and chat posts, all confirmations of tasks completed, all evaluations written for classmates’ drafts, and all drafts written and comments received from peers and from me. Because this data is in digital form, it’s also searchable, fluid, and portable.
This means that the student’s writing process is visible to a degree that we could only dream about in face-to-face hardcopy classrooms. Using traditional on-ground methods, much of the learning process was beyond our reach. For example, we didn’t make and keep hardcopies of every draft that had been reviewed by students and teachers as well as every comment that was made. We didn’t record all discussions. This is not to say that we couldn’t. We could, but the labor would have been so intense that it would have been all but impossible. Add to this the cabinets and space needed to store these files as well as the effort required to search, by hand, documents and files and we begin to understand the enormity of this undertaking.
The unprecedented mother lode of formative qualitative information, though, introduces new problems. Because we can do something online that we can’t on-ground doesn’t mean we ought to. The slowest part of the virtual instructional process, the bottleneck, is that which requires human processing. A human being still has to dig through, select, and interpret the data to use it. And this requires the one commodity that’s always in short supply for teachers – time. Thus, even though the information is only a few clicks away, teachers may not be able to use much of it.
In the interest of time, I use an academic triage system to determine which students I’m going to mine. I routinely monitor the reviews that students write for one another’s drafts and generate scores based on their knowledge of assignment criteria and writing guidelines and their ability to apply them in their assessments. When students fall below a certain score, I dig into their process. Thus, when I review their current drafts, I also review past drafts and comments they’ve received. This approach reveals patterns, or more precisely, patterns of failure. I can share these patterns with students and pinpoint corrective actions.
Ultimately, though, triage systems like this are stopgaps. It’s just a matter of time before we invent efficient and effective methods to mine qualitative data. In the digital world, this means “smarter” programs that will allow teachers to set parameters for an application that will, in turn, hunt for, process, and instantly generate the kinds of reports that we need to assess learning as it’s occurring and in a context that sheds light on the entire process. With a click, we’ll literally be able to “see” how the student is learning and intervene at any point to guide performance.
Filed under: Uncategorized |