Educational Technology and Change Journal

By Jim Shimabukuro
Editor

We gathered a panel of editors, writers, and professors to generate a comprehensive list of questions for Dr. Rory McCorkle. The panel members were: Stefanie Panke, Bert Kimura, Judith McDaniel, Leigh Dooley, and Harry Keller. Colleges around the world are developing online programs, and an invariable topic is remote test proctoring as an adjunct or alternative to on-campus testing. We hope that our questions and Dr. McCorkle’s responses will shed more light on the issues involved in reviewing and selecting a proctoring service.

Perhaps the most meaningful takeaway for us is the realization that the ultimate quality of the testing process is determined to a large extent by the participating institutions. In short, GIGO. Colleges should determine their needs prior to shopping for a proctoring service. The general services and features in the PSI Bridge™ platform tend to be standard among major providers, and it’s in the details that the critical differences begin to emerge.

Dr. Rory McCorkle

Dr. Rory McCorkle is the Senior Vice President of Certification and Education Services for PSI. He helps PSI clients meet their testing, educational and strategic goals through the suite of services offered by PSI, while leading a team of business development, account management, and consulting professionals. Dr. McCorkle has worked with over 750 testing organizations, including well-known universities and colleges, licensing bodies, and renowned certification programs.

The questions and answers below are roughly grouped into four areas: students, proctors, teachers, and PSI services.

I. STUDENTS

Bert Kimura: How do the services and data collected conform with FERPA and other privacy issues?

Rory McCorkle: PSI Services LLC (PSI) holds privacy and security as a highest priority. We require very limited personally identifiable information (PII) for students and limit access to PII to only those who require access for provision of the services. In practice, we take care to not only be secure, but non-invasive as well. Only a single login is required for users to take their tests, and PSI proctors do not require access of a user’s machine to verify test compliance. All security measures are integrated and automated to ensure proper compliance and client privacy, while simultaneously minimizing security risks and providing a smooth user experience. In addition, PSI offers a proprietary, customizable lock-down browser and self-serve check-in and authentication. PSI complies with major regulations such as FERPA and GDPR. We are also PCI compliant. Our remote proctors receive extensive training on how privacy and privacy regulations are central to their roles, which is coordinated by PSI’s Chief Compliance Officer.  

Jim Shimabukuro: If each student’s test-taking is recorded, where is the recording stored and for how long? Who has access to the recordings? For example, do teachers have access? How are these archives secured?

RM: The recording is stored on PSI’s cloud-based server, which complies with best-in-class security measures. The only individuals with access to the recordings are the institution’s administrator, the assigned faculty member, and PSI remote proctoring supervisors or auditors.

JS: What are the chances of a student being accused or suspected of cheating when s/he is actually innocent? Is there a review or appeal process?

RM: All decisions on the final actions to take with a student suspected of cheating rest with the institution. PSI’s role is solely to identify and catch incidents, as defined by the business rules established with the institution. Since this decision is made by a human proctor, and such decisions are regularly reviewed by remote proctoring supervisors and audited by our remote proctoring auditing function, the chances of this scenario occurring are low. The recording, with incidents logged, is also available for the faculty member or administrators to make a final decision.

JS: Does the proctoring environment negatively impact certain students? In other words, do some students perform worse than expected in this environment?

RM: While limited research has been conducted on the differences between student anxiety between an in-person and remote proctoring delivered exam, disparate trends have been observed by PSI and other remote proctoring providers. Some students are less anxious due to being in a familiar environment, without the stressors of traveling to an in-person test center or classroom and being in a stressful environment. Others are more stressed due to the unfamiliarity of the remote proctoring environment; however, this tends to decrease after more exposure to the method. In a study conducted in 2014-2015, PSI did find that there was no impact on pass rates nor test taker satisfaction, regardless of test method.

Leigh Dooley: Do you provide special support services for students with disabilities?

RM: Yes, we provide American Disabilities Act (ADA) accommodations, where the student is approved by the institution for an accommodation.

JS: Do you provide proctors with second language skills for ESL students?

RM: We currently have proctors who are fluent in Japanese, Mandarin Chinese, and Spanish; however, these proctors are typically not used for ESL students.

JS: Do you provide student orientations or workshops on how to take your proctored exams? Are staff available 24/7 to answer student questions about the testing process?

RM: We provide videos that walk students through the process. We also provide FAQs and other resources to answer questions, frequently customized to the individual institution. PSI staff are available to assist with scheduling or technical issues 24/7.

JS: What are the requirements from a student’s perspective? What are her/his hardware and software needs? For example, will she need a special type of computer, operating system, browser, or camera to record her performance?

RM: No special type of computer or browser is required. The student is required to have a Mac or PC operating system and webcam. There are certain requirements with regard to minimum bandwidth and hardware requirements (e.g., RAM, processor speed), but PSI requires very low connectivity speeds. We recommend 750 kbps in regard to bandwidth but require 500 kbps. No special browser is required as PSI uses a proprietary secure browser to ensure that students cannot access other internet browsers, programs, and other resources not permitted by the institution.

JS: From a student’s perspective, what are the dress or grooming requirements, if any? Why?

RM: There are no particular dress or grooming requirements, with the exception of any bulky clothing that could be used to obscure electronic devices. For example, typically “hoodies” are not permitted due to the ability to hide recording devices.

JS: Are students allowed bathroom breaks during the testing? Are they allowed to snack while testing?

RM: Typically, neither breaks nor snacks are allowed; however, this is determined by the business rules established by the institution. Some of the more creative uses to access cheat sheets have included putting notes on soda bottle labels and candy wrappers, which is why we typically do not allow snacks.

JS: From a student’s perspective, what are the location requirements for participating in this proctoring environment? For example, does she need access to an enclosed or secure room? Or can she take the proctored test from anywhere?

RM: The location must be private (e.g., no one else in the room during the duration of the exam). The room cannot have notes or other “cheating” resources when scanned before the examination.

JS:  Is this proctoring service available 24/7 to accommodate working students, care providers, etc.?

RM: Yes, provided that 24/7 testing is permitted by the institution.

JS: If a student is uncomfortable with your testing process, are you able to provide alternatives that will ensure her comfort?

RM: This is governed by the allowable testing methods and business rules established by the institution.

Judith McDaniel: Is your test environment culturally neutral, i.e., does it work for or against students from certain socioeconomic or cultural backgrounds?

RM: Yes, it is culturally neutral. There is nothing in the tool or service that would disadvantage any student. PSI also trains our proctors to make students feel comfortable through both customer service and cultural awareness training.

II. PROCTORS

BK: Who serves as proctors (if real people)? What are their qualifications? How are they vetted? How are they trained to proctor?

RM: Proctors are real people and are all PSI employees. They are selected for their customer service aptitude, attention to detail, ability to communicate effectively via chat with test takers around the globe, and consistency in applying security standards. They are vetted with an interview process, pre-employment testing, and a background check. They undergo a multi-week training process that includes self-guided training through our security standards, a certification test to ensure they understand requirements, shadowing an experienced proctor, and finally, proctoring under the observation of an experienced proctor or supervisor.

JS: Will a live proctor be standing by for the duration of the test to observe and respond to any questions from a student?

RM: If the examination is delivered through our live proctoring method, yes. If it is a record-and-review examination, then a resource is available for technical issues throughout the exam.

JS: Do your proctors rely on a cheater profile to alert them to specific student behaviors? What are some of these behaviors and are they highly correlated with cheating?

RM: PSI proctors use a variety of different flags to alert us to potential cheating. These flags include biometric flags in addition to traditional observation techniques, but we cannot disclose further information.

JS: Do proctors warn students about suspicious behaviors or environmental features that might indicate cheating? Are students allowed to eliminate these to proceed with the testing?

RM: This depends on the business rules set by the institution, as institutions can decide to warn students about incidents. Incidents of a certain severity can result in immediate termination of the examination.

III. TEACHERS

Stefanie Panke: How does the system work from the teachers’ perspective? Could you walk ETC readers through the process of setting up an exam, for example in a Biology 101 class?

RM: Setting up an exam from the instructor’s perspective is as simple as logging into the institution’s LMS and navigating from the dashboard.

Through the LMS integration, data transfer is automatic and exam structures do not require additional set-up by faculty.

Teachers can set up customized rules to meet their class’s or exam’s specifications. This includes whether the exam should be live or record-and-review proctored (assuming their institution allows both), as well as the proctoring business rules for the exam.

JS: Do teachers have the power to turn certain features or requirements of the testing process on or off?

RM: Like other testing elements, this is governed by the institution. Faculty members can typically select certain options, such as allowable resources, if any.

SP: What type of test assignments and assessments does the system support? Can you give some examples?

RM: Basically, any assessment that is supported within the institution’s LMS can be supported within the PSI Bridge™ platform. While most institutions use the system for either midterms or end of course assessments such as finals, some teachers use the platform to administer anything from weekly quizzes through multi-hour capstone exams.

JS: Do you proctor different types of exams, e.g., essay, open-book?

RM: Yes, we proctor assessments of all different types, ranging from multiple choice, to essays, to performance tests (in application testing).

SP: Is there a test environment that allows interested educators to take a test run and see what the look and feel is from the students’ perspective?

RM: We provide videos walking the faculty through the process so they understand what the student sees.

IV. SERVICES

JS: If an instructor encourages it, does your system allow students to open multiple windows for web searches during exams?

RM: If allowed by the institution, yes; however, this would be a rare allowance.

JS: What happens if a test is somehow interrupted by a technical problem, making it difficult or impossible for a student or students to complete it?

RM: Typically, when this occurs, it is due to an issue with the student’s internet service. In this case, the examination can be rescheduled for another time to allow them to complete. Regardless, technical issues are first troubleshooted with PSI’s Technical Support. If issues cannot be resolved, which are rare, then the student is offered a free retest at an appropriate time, typically using another computer or from another location.

JS: Are test results delivered with a guarantee that no cheating occurred? Is this guarantee qualified with an error margin?

RM: We do not offer such a guarantee, as there are not any proctoring services that are perfect. However, we can offer SLAs for incidents on an institution by institution basis.

JS: What makes your service better than the competitors’ (e.g., ProctorU)?

RM: PSI’s on-staff experts have pioneered remote proctoring technologies and the company itself has more than 15 years of experience in this delivery method. Our multimodal remote proctoring system, PSI Bridge™, brings live remote proctoring and record-and-review proctoring together in a single system that can be customized to an institution’s needs and easily connected to any LMS. We provide this cloud-hosted software as a service for higher education with LMS integration via the PSI Bridge™ LTI app that includes single sign-on and automatic data transfer, saving faculty time by eliminating their need to set up tests.

Within the LMS, faculty can define rules, configure assessments on an exam-by-exam basis, and adjust exam policy as needed to accommodate custom resources. Faculty can also see proctored exam results in the LMS and view specific students’ exams alongside any flagged violations and annotated video. Additionally, through the “rule implementer” feature, proctors will not need to manage rules manually, enabling higher accuracy and faster intervention.

We will also soon be rolling out facial biometrics to further secure testing and verify student identity.

PSI selects and trains proctors for not just their customer service aptitude and attention to detail, but their technological savvy, as well. Our proctor selection process includes pre-employment testing and background checks, and we implement extensive training process, including certification of all remote proctors.

LD: How do you verify student identity?

RM: Student identity is first verified by the integration with the institution’s LMS. The student signs in through the LMS, which is the first identity verification step, then transfers over to PSI Bridge™ from this system. Then the student must take a picture of themselves, along with a photo of their identification. This is reviewed by the proctor, along with the required room scan. The proctor may ask the student to show their identification again, manipulate it (such as moving it to show security features), or other steps to ensure the match between the student and their identification.

Later this year, PSI Bridge™ will also incorporate an automated tool powered by facial recognition software. This will automatically compare the photo and identification, as well as ensure the legitimacy of the identification.

LD: Do you offer different levels (e.g., live, record and review, automated) of proctoring?

We offer live and record-and-review proctoring.

LD: Do you use an internal auditing system?

RM: Yes, we have a separate auditing team that watches a percentage of examinations delivered every day. Issues found are discussed with the proctors by their supervisors. Significant issues identified by audits can result in proctor de-certification, retraining, and/or dismissal.

LD: How does your service integrate with LMSs?

RM: Most commonly, our service is fully integrated with institution LMSs through LTI and other such integrations.

LD: Do you have safeguards against cut-and-paste cheating strategies to keep students from sharing test questions with others?

RM: The primary step is our proprietary secure browser, which prevents students from using copy/paste, print screen, or a number of other hotkeys. In addition, by sharing their desktop, proctors can monitor the student’s test to ensure that they have not switched to an application not permitted by the institution.

LD: What are the costs for startup and per test? How do they compare with other proctoring services?

RM: Pricing is scaled based on the length of the test, proctoring method chosen, and yearly volume.

Harry Keller: No proctoring service is 100% effective against cheating. No matter what safeguards you come up with, someone somewhere will find a means to defeat it. What are some novel schemes that you’ve discovered and dealt with? What are some proactive steps that you’re taking?

RM: We divide incidents into two types of issues: Cheating and attempting to steal content. For those trying to cheat, the methods used are fairly traditional: notes hidden on someone’s laptop, written on an arm, attempts to access a mobile phone during a test, etc. Our procedures are quite successful at catching such attempts. More creative uses to access cheat sheets have included putting notes on soda bottle labels, candy wrappers, etc. Again, this is one of the reasons why students are typically not allowed access to food or drink during the exam. We have also caught students having individuals hold up answers or notes outside of their window when testing. Again, this can be caught through careful proctor attention and watching where the student is looking. Most test takers are concentrating on the examination and do very little during the test – cheaters are often identified by frequent glances to an area not on their screen.

Stealing content is a very difficult issue to battle. Technology has enabled just about anyone to become James Bond when it comes to stealing content – two companies now have patents for cameras embedded into a contact lens. With the increase in availability of spyware (a quick Amazon search of “spy camera” yields more than 8,000 items), proctors are not only trained in the latest and greatest in spyware, they’re also trained to recognize behavioral patterns that are indicative of stealing content. This includes behavioral patterns such as people moving their bodies in strange ways or angling their bodies in a peculiar angle from their computer screen.

While we train proctors on the latest trends, we also work with our clients to ensure they understand that no proctoring service will catch everything and how to use test development techniques to avoid overexposure of examination content.

Filed under: Uncategorized |