Milestone admissions: using AI against cheating

March 27, 2020 | Orsolya Lányi

The impact of coronavirus may create opportunities to develop in new areas. This is certainly true for the admissions process of hopeful Milestone candidates, as the March round of written tests had to be delivered via a secure online platform.
With Tünde Szabó, the Institute’s Head of Academic Operations, we talked about the admissions process, new challenges and the opportunities provided by artificial intelligence in education.

It is the admissions period at Milestone, but this year the new type of coronavirus has brought an unexpected change to the life of the Institute. At what point did you start thinking about how the admissions process could be changed?

Having followed the international events closely, we did expect the COVID-19 outbreak to reach Hungary – the only question was when this would happen. Therefore, by the time the first case was confirmed in Hungary on 4th March, we had already been thinking about how this would affect our operation. The three main areas in which we started research and planning together were transitioning our module teaching and mentoring activities, our Student Life and the admissions process to online platforms. In terms of the first two areas, we already had an existing framework to build on, as our Academic Team has been working on the development of various innovative teaching methods, including online teaching, for a while: in fact, this functionality was an important factor in choosing Canvas, the advanced Learning Management System (LMS) used by the Institute for several years. Thanks to this, the transitioning of our educational and Student Life activities, where in addition to Canvas our Student Societies and Leaders had been actively using social media platforms as well, could be done quickly and with relative ease – of course, this was only possible with the openness and cooperation of our student community and faculty. For conducting our admissions process online, we had no well-trodden paths, so we saw this as an exciting challenge.  

What has the admissions process been like so far and what changes has the online transition brought about?

The admissions process consists of three stages. First, students have to submit their online application, through which we aim to find out as much as possible about students’ interests and motivation in both academic and other areas. Based on the evaluation of these applications, candidates are invited to a written test (in English), which has three main parts and tests candidates’ logical thinking skills, problem-solving abilities, reasoning and English language skills. Successful applicants are invited to the interview round, which involves a 20-minute discussion with two interviewers, also in English. Our Institute traditionally organises the written tests in two rounds. By early March, the first session had already been delivered, so it was organising the second sitting and the interviews that we had to find a new solution for. For the speaking part, we had a readily available option: interviews will be held using video communication channels, with the format, apart from the absence of personal meetings, left unchanged. In terms of the written tests, the main goal with the online transition, in addition to practical considerations and providing equal opportunities to our candidates, was maintaining the validity and reliability of the entrance exam: that is, we had to ensure that the test measures what it is intended to measure, and reliably, in an online format as well. These were the main criteria we applied when looking for an online testing platform: we needed an option that would guarantee the integrity of the examination process and that we could switch to quickly, without our candidates feeling that this was an emergency measure.

The programme identifies cases of potential malpractice during the test using Artificial Intelligence (AI). Can you tell us a bit more about how it works?

It was an essential goal that the results of the first round of written tests, which were delivered under strict live invigilation, are comparable to those of the second, online round, and that we can deliver the whole admissions process as per the initially planned timeline – this is where AI helped. It is important to underline here that the built-in AI functions of the platform merely assist us in our work, all admissions decisions are still made by our Admissions Committee. Therefore, it cannot happen that an applicant is excluded solely due to the AI, possibly erroneously, identifying a certain exam event as suspicious. What this means in practice is that, with the candidate’s permission, both the test-taker and their device’s screen are recorded while the test is being taken, and these recordings are then stored by the system together with candidates’ responses. Reviewing such a huge amount of footage, however, is rather a lot of work – this is where we were primarily relying on the help of AI.

The AI tool of the software flags all cases in both the webcam and the screen recordings that are potentially suspicious.

 Occurrences that are picked up include, for example, a candidate navigating away from the test window, if they speak or if their faces are only partially visible when there is no valid reason for this, or if an object (such as a mobile phone) or another person appears in the view of the camera. The Academic Team have reviewed all of the footages and as we were doing so, we mainly (but not exclusively) focused on the sections flagged by the AI tool. In cases where the first review made this necessary, a second check was performed by members of the Admissions Team. The results of these second checks were then discussed in detail by the Admissions Committee, which made their decisions in light of these investigations.

What has the overall experience with online testing been like?  

It is generally true that students of secondary school age are quite confident when it comes to digital devices, and they are quick to master the use of newer and newer online platforms. Still, as in the case of our admissions tests the online transitioning was an unexpected change introduced at a fairly short notice, we felt it particularly important that we provide our applicants and their parents with as much advance information as possible and that we can offer satisfactory responses to any issues or questions that may arise. To achieve this aim, we prepared detailed guidelines about the online testing process and the preparation needed, as well as about the data handling aspects. Candidates were also given an opportunity to check the suitability of their equipment in advance, and, in order to familiarise themselves with the platform, they could also take a brief sample test. Finally, during the test weekend, my colleagues and I were also readily available to handle all technical or other issues that may occur, ensuring that all our candidates can perform to the best of their abilities. Thanks to all the preparation, the second round has also been conducted successfully, but once the admissions period has closed, we are of course planning to get more detailed feedback from all our test-takers as well.

What are the prospects of progressive, online education using artificial intelligence? Is it possible that AI will be a useful tool for public education as well?

Artificial intelligence is already used in several areas in education, and this trend will continue. For example, examinations have been available for years where not only ‘closed’ type of questions (such as multiple choice) but also essay-type tasks and speaking exam performances are evaluated by AI, though in the latter cases this is usually done in a so-called hybrid model, with human assessors also involved to a varying extent. There are adaptive tests, which set test questions in a dynamic way, based on the answers submitted by the test-taker, and there are also programmes that use the results of a certain type of assessment (for example, a weekly subject test or homework) to build personalised study plans for learners.

It is clear that the use of AI provides opportunities to support the learning and teaching process that go beyond assessment.

 Just to mention a few more examples: speech recognition software can aid students who struggle with writing or have limited mobility, virtual learning assistants can guide students in processing educational content individually, and augmented reality (AR) software can make the learning experience more enjoyable. In my opinion, it is still adaptive learning technologies that will have the greatest impact, as these make it possible to produce individualised syllabi tailored to the learner’s existing knowledge, skills and learning habits both in quantity and content. In an ideal scenario, this can mean that teachers spend less time performing administrative and other tasks that could be done by AI and focus their energies on imparting knowledge and developing skills where AI does not present an alternative such as creative and critical thinking or collaborative skills.

How do you envision Milestone admissions tests in the future? Can there be a change further to this new system having been used?

Based on the positive experience so far, we are indeed considering whether written tests could be delivered online in the future, especially as we often have applicants who are studying abroad at the time of admissions. In order to make an informed decision on this, we will have to conduct further analyses once the admissions process has closed, and getting feedback from this year’s applicants will form an important part of this work. When it comes to the interviews, however, we will always be aiming to conduct these face-to-face, as we feel it important to form a personal relationship with applicants. 

The impact of coronavirus may create opportunities to develop in new areas. This is certainly true for the admissions process of hopeful Milestone candidates, as the March round of written tests had to be delivered via a secure online platform.
With Tünde Szabó, the Institute’s Head of Academic Operations, we talked about the admissions process, new challenges and the opportunities provided by artificial intelligence in education.

It is the admissions period at Milestone, but this year the new type of coronavirus has brought an unexpected change to the life of the Institute. At what point did you start thinking about how the admissions process could be changed?

Having followed the international events closely, we did expect the COVID-19 outbreak to reach Hungary – the only question was when this would happen. Therefore, by the time the first case was confirmed in Hungary on 4th March, we had already been thinking about how this would affect our operation. The three main areas in which we started research and planning together were transitioning our module teaching and mentoring activities, our Student Life and the admissions process to online platforms. In terms of the first two areas, we already had an existing framework to build on, as our Academic Team has been working on the development of various innovative teaching methods, including online teaching, for a while: in fact, this functionality was an important factor in choosing Canvas, the advanced Learning Management System (LMS) used by the Institute for several years. Thanks to this, the transitioning of our educational and Student Life activities, where in addition to Canvas our Student Societies and Leaders had been actively using social media platforms as well, could be done quickly and with relative ease – of course, this was only possible with the openness and cooperation of our student community and faculty. For conducting our admissions process online, we had no well-trodden paths, so we saw this as an exciting challenge.  

What has the admissions process been like so far and what changes has the online transition brought about?

The admissions process consists of three stages. First, students have to submit their online application, through which we aim to find out as much as possible about students’ interests and motivation in both academic and other areas. Based on the evaluation of these applications, candidates are invited to a written test (in English), which has three main parts and tests candidates’ logical thinking skills, problem-solving abilities, reasoning and English language skills. Successful applicants are invited to the interview round, which involves a 20-minute discussion with two interviewers, also in English. Our Institute traditionally organises the written tests in two rounds. By early March, the first session had already been delivered, so it was organising the second sitting and the interviews that we had to find a new solution for. For the speaking part, we had a readily available option: interviews will be held using video communication channels, with the format, apart from the absence of personal meetings, left unchanged. In terms of the written tests, the main goal with the online transition, in addition to practical considerations and providing equal opportunities to our candidates, was maintaining the validity and reliability of the entrance exam: that is, we had to ensure that the test measures what it is intended to measure, and reliably, in an online format as well. These were the main criteria we applied when looking for an online testing platform: we needed an option that would guarantee the integrity of the examination process and that we could switch to quickly, without our candidates feeling that this was an emergency measure.

The programme identifies cases of potential malpractice during the test using Artificial Intelligence (AI). Can you tell us a bit more about how it works?

It was an essential goal that the results of the first round of written tests, which were delivered under strict live invigilation, are comparable to those of the second, online round, and that we can deliver the whole admissions process as per the initially planned timeline – this is where AI helped. It is important to underline here that the built-in AI functions of the platform merely assist us in our work, all admissions decisions are still made by our Admissions Committee. Therefore, it cannot happen that an applicant is excluded solely due to the AI, possibly erroneously, identifying a certain exam event as suspicious. What this means in practice is that, with the candidate’s permission, both the test-taker and their device’s screen are recorded while the test is being taken, and these recordings are then stored by the system together with candidates’ responses. Reviewing such a huge amount of footage, however, is rather a lot of work – this is where we were primarily relying on the help of AI.

The AI tool of the software flags all cases in both the webcam and the screen recordings that are potentially suspicious.

 Occurrences that are picked up include, for example, a candidate navigating away from the test window, if they speak or if their faces are only partially visible when there is no valid reason for this, or if an object (such as a mobile phone) or another person appears in the view of the camera. The Academic Team have reviewed all of the footages and as we were doing so, we mainly (but not exclusively) focused on the sections flagged by the AI tool. In cases where the first review made this necessary, a second check was performed by members of the Admissions Team. The results of these second checks were then discussed in detail by the Admissions Committee, which made their decisions in light of these investigations.

What has the overall experience with online testing been like?  

It is generally true that students of secondary school age are quite confident when it comes to digital devices, and they are quick to master the use of newer and newer online platforms. Still, as in the case of our admissions tests the online transitioning was an unexpected change introduced at a fairly short notice, we felt it particularly important that we provide our applicants and their parents with as much advance information as possible and that we can offer satisfactory responses to any issues or questions that may arise. To achieve this aim, we prepared detailed guidelines about the online testing process and the preparation needed, as well as about the data handling aspects. Candidates were also given an opportunity to check the suitability of their equipment in advance, and, in order to familiarise themselves with the platform, they could also take a brief sample test. Finally, during the test weekend, my colleagues and I were also readily available to handle all technical or other issues that may occur, ensuring that all our candidates can perform to the best of their abilities. Thanks to all the preparation, the second round has also been conducted successfully, but once the admissions period has closed, we are of course planning to get more detailed feedback from all our test-takers as well.

What are the prospects of progressive, online education using artificial intelligence? Is it possible that AI will be a useful tool for public education as well?

Artificial intelligence is already used in several areas in education, and this trend will continue. For example, examinations have been available for years where not only ‘closed’ type of questions (such as multiple choice) but also essay-type tasks and speaking exam performances are evaluated by AI, though in the latter cases this is usually done in a so-called hybrid model, with human assessors also involved to a varying extent. There are adaptive tests, which set test questions in a dynamic way, based on the answers submitted by the test-taker, and there are also programmes that use the results of a certain type of assessment (for example, a weekly subject test or homework) to build personalised study plans for learners.

It is clear that the use of AI provides opportunities to support the learning and teaching process that go beyond assessment.

 Just to mention a few more examples: speech recognition software can aid students who struggle with writing or have limited mobility, virtual learning assistants can guide students in processing educational content individually, and augmented reality (AR) software can make the learning experience more enjoyable. In my opinion, it is still adaptive learning technologies that will have the greatest impact, as these make it possible to produce individualised syllabi tailored to the learner’s existing knowledge, skills and learning habits both in quantity and content. In an ideal scenario, this can mean that teachers spend less time performing administrative and other tasks that could be done by AI and focus their energies on imparting knowledge and developing skills where AI does not present an alternative such as creative and critical thinking or collaborative skills.

How do you envision Milestone admissions tests in the future? Can there be a change further to this new system having been used?

Based on the positive experience so far, we are indeed considering whether written tests could be delivered online in the future, especially as we often have applicants who are studying abroad at the time of admissions. In order to make an informed decision on this, we will have to conduct further analyses once the admissions process has closed, and getting feedback from this year’s applicants will form an important part of this work. When it comes to the interviews, however, we will always be aiming to conduct these face-to-face, as we feel it important to form a personal relationship with applicants.