Artificial Intelligence: Transforming the Future of Feedback in Education PDF

Journal of Applied Testing Technology, Vol 23(Special Issue 1), 95-116, 2022 Artificial Intelligence: Transforming the Future of Feedback in Education Tarid Wongvorachan1*, Ka Wing Lai1, Okan Bulut2, Yi-Shan Tsai3 and Guanliang Chen3...

Journal of Applied Testing Technology, Vol 23(Special Issue 1), 95-116, 2022 Artificial Intelligence: Transforming the Future of Feedback in Education Tarid Wongvorachan1*, Ka Wing Lai1, Okan Bulut2, Yi-Shan Tsai3 and Guanliang Chen3 Department of Educational Psychology, University of Alberta, Canada; [email protected], 1 [email protected] 2 Centre for Research in Applied Measurement and Evaluation, University of Alberta, Canada; [email protected] 3 Department of Human Centred Computing, Monash University, Australia; [email protected], [email protected] Abstract Feedback is a crucial component of student learning. As advancements in technology have enabled the adoption of digital learning environments with assessment capabilities, the frequency, delivery format, and timeliness of feedback derived from educational assessments have also changed progressively. Advanced technologies powered by Artificial Intelligence (AI) enable teachers to generate different types of feedback supporting student learning. Despite the rapid uptake of digital technologies in education, previous studies on educational feedback primarily focused on the theoretical underpinnings of feedback practices, which are limited in terms of their coverage of AI-based technologies. This paper aims to inform both researchers and practitioners about the present and future of AI applications in feedback practices, identify and organize potential areas for the use of AI for feedback purposes, and establish venues for AI research and practice in educational feedback. Furthermore, the role of the three branches of AI (i.e., natural language processing, educational data mining, and learning analytics) in feedback practices and potential areas for their future development are discussed. Keywords: Artificial Intelligence, Educational Data Mining, Educational Feedback, Learning Analytics, Natural Language Processing 1. Introduction be differently equipped to access, understand, and use their feedback, both feedback information and communication Feedback—a process where learners make sense of the strategy need to be considered to maximize the benefit provided information to reduce the gap between their of feedback to students and provide a lasting change; current and desired performance—is a crucial component hence, the importance of personalized feedback (Hattie & of student learning (Carless & Boud, 2018; Watling & Timperley, 2007; Kochmar et al., 2020). Ginsburg, 2019). Rather than being a piece of static Teachers have their role to effectively formulate and information, this paper conceptualizes feedback as the communicate their feedback while students use the instructional process that encompasses the component information to update their knowledge and change the of information and communication strategy to enhance corresponding behavior (e.g., learning strategy, approach students’ understanding of their learning (Gamlem & to the task, and the use of learning resources) to achieve Smith, 2013; Hattie & Timperley, 2007). For example, the desired outcomes(s) (Boud & Molloy, 2013; Forsythe feedback to improve students’ math problems can be & Johnson, 2017). Feedback is used for both formative conveyed by verbally explaining, solution demonstrating, and summative purposes during the learning process. or both. As each student has their own condition and may *Author for correspondence  Artificial Intelligence: Transforming the Future of Feedback in Education Teachers use formative feedback to provide students (Rodway-Dyer et al., 2011), or video-based feedback with opportunities for continual improvements while (Ketchum et al., 2020). This article focuses on technology using summative feedback to inform students about their that enhances the provision of verbal feedback, both performance in the course (Barana et al., 2019; Marriott manually written by teachers as informed by results from AI & Teoh, 2012). (i.e., semi-automated process) or generated by computers As advancements in technology have enabled new (i.e., fully automated process) and non-verbal feedback ways of learning and changed the dynamics of education such as computer-generated graphs. Some cutting-edge (e.g., the transition from traditional paper-and-pencil examples of AI applications for feedback include the assessments to digital online assessments), the frequency, usage of Machine Learning (ML) and Natural Language delivery format, and timeliness of feedback derived from Processing (NLP) to evaluate student performance in educational assessments have also changed progressively real-time and produce personalized feedback for students to meet the needs of students (Jurs & Špehte, 2021). For who are at risk of low performance (Jimenez & Boser, example, digital score reporting has enabled students 2021). Educators can also utilize learning analytics (LA), to receive immediate and personalized feedback from which involves using AI and relevant techniques to computerized assessments to best inform students in provide real-time personalized feedback to all students their learning (Bulut et al., 2019; Zenisky & Hambleton, and thereby enhance their learning experience (de Laat 2012). Similarly, intelligent tutoring systems (ITSs) can et al., 2020; Tsai et al., 2021). Such technologies can provide granular and specific feedback to students as empower students to use feedback through the speed and they complete learning tasks personalized based on their efficiency of AI applications (Zhang et al., 2019). unique interests and proficiency levels (Ai, 2017; Kulik & In sum, using AI allows educators to provide Fletcher, 2016). The examples (i.e., digital score reporting feedback to a large number of students in a short time and ITS) can be further developed through AI. frame in contexts such as Massive Open Online Courses Specifically, in the field of education, AI itself refers to (MOOCs), minimal disruptions from time and space the development of computer systems that can perform barriers, and an ability to process large-scale educational education-related tasks that require human intelligence, data such as Trends in International Mathematics and such as grading students’ exams, personalizing learning Science Study (TIMSS) (e.g., Bethany et al., 2021) and materials, or providing recommendations for assignment Programme for International Student Assessment (PISA) tasks based on real-time data analysis (L. Chen et al., 2020). (e.g., Organisation for Economic Co-Operation and In other words, the usage of AI includes all activities that Development [OECD], 2019) with educational data involve applying computer systems to extract information, mining (EDM) and ML methods (Gardner et al., 2021; solve problems, and answer questions, from a simple rule- Witten et al., 2017). based decision to a complex process of image or voice Given the increasing use of advanced technologies recognition (L. Chen et al., 2020; Loyola-Gonzalez, 2019). and AI in education worldwide, it is important to inform The definition also expands to the involvement of AI in both researchers and practitioners about the present human decisions, such as the usage of an unsupervised and future of AI applications in feedback practice learning approach to assign students into groups (e.g., (United Nations Educational, Scientific and Cultural adaptive learners, deep learners) to inform teachers in Organization [UNESCO], 2019). Previous studies their feedback communication strategy (Tempelaar, on educational feedback primarily focused on the 2020). Therefore, any action that involves the assistance theoretical underpinnings of feedback practices, such of computer systems to simulate human intelligence in as the development of feedback literacy (e.g., Carless & information extraction and decision making, whether in Boud, 2018; Carless & Winstone, 2020), learner-centered part or in full, can constitute the usage of AI. feedback (e.g., Molloy et al., 2021), and the advocacy for In the education field, AI also plays a role in supporting formative feedback practices in education (e.g., Boud, feedback practices by providing fully automated or semi- 2020). However, these studies are generally limited in automated feedback in various forms such as written their coverage of AI-based technologies for generating feedback (Zhang et al., 2019), audio-based feedback and delivering student feedback. Additionally, most 96 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen studies on the application of AI for feedback purposes by discussing the directions where AI can be harnessed only focus on a single application of each technology, to transform the future of feedback practices. Our goal such as the use of AI to develop ITS (Ubani & Nielsen, is to form an anchoring concept before revolving into 2022) or personalized feedback (Chan & Zary, 2019). This the specific application of AI in each research field to situation warrants a systematic review of the application show the current landscape of the area. Note that the of AI for feedback provision to provide an introductory organization of this paper takes on a specific pattern by overview of the area for current and new researchers in discussing NLP, EDM, and LA in this order throughout the field. the paper for ease of understanding. This theoretical paper aims to inform researchers and practitioners about the current and future landscape of AI applications in feedback, identify and organize potential 2. The Four Levels of Feedback areas for the use of AI for feedback (e.g., AutoTutor or Framework QuizBot chatbots with NLP, interactive feedback with Feedback is an essential element in student learning as it online data visualizations, and intelligent recommender helps facilitate student development by stimulating their systems), and establish venues for AI research and learning process and optimizing their understanding practice in educational feedback (G. Chen et al., 2020; of class materials for improved performance in the task Yildirim-Erbasli & Bulut, 2021). Educators could also use (Hounsell, 2007). In a feedback spiral, students reflect results from this paper to guide their implementation of on feedback from their instructors to update their task- AI in their feedback practice (e.g., LA-driven feedback or related knowledge and behavior in response to the NLP for automatic feedback generation). received feedback; for example, students who receive The paper begins by discussing the extent to which feedback from their mid-term exam can use it to adjust feedback can affect students with the four levels of their learning strategy such as investing more time to feedback framework as one of our supporting frameworks study the course content they did not do well to prepare in the paper. Then, we introduce the involvement of AI for the final exam (Carless, 2019). Characteristics of high- in educational feedback to lay out the groundwork for quality feedback include thorough coverage, appropriate the contribution of AI to feedback practices. We discuss tone, straightforward language, and transparency in its the application of AI technologies in the fields of NLP, guidance (Hounsell, 2007). Feedback should also be EDM, and LA for feedback purposes by defining them timely and relevant to both the course itself and student and their scope and then introducing the feedback circumstances to maximize its actionability, especially in dimension framework and weaving it together with the the distance learning context where communication is four levels of feedback framework. We conclude the paper Table 1. The four levels of feedback Feedback Level The Effect on Students’ Level of Change This level concerns how the tasks are performed (e.g., correctly vs. Task Level incorrectly). This level concerns the thought process needed to perform the task and its Process Level related variant. This level concerns how students monitor, direct, and regulate their Self-Regulated Learning Level actions toward learning goals. This level concerns personal aspects of the students themselves (e.g., well Self Level done). Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 97  Artificial Intelligence: Transforming the Future of Feedback in Education impeded by the lack of physical proximity (Bulut et al., instruction allows students to learn more efficiently while 2020; Hounsell, 2007; Jurs & Špehte, 2021). at the same time allowing instructors to address issues that Feedback provided by teachers could affect students at can only be identified through results from a fine-grained four levels: 1) task level, 2) process level, 3) Self-Regulated level data analysis (Barana et al., 2019; Jimenez & Boser, Learning (SRL) level, and 4) self level, as suggested by 2021). For example, AI can be used to grade a large Hattie and Timperley (2007)’s model of feedback level; number of exams and at the same time identify patterns of these four levels are defined and compared in Table 1 student performance with data mining to inform teachers on how they differently affect students’ level of change in their feedback provision, such as providing more in their task performance. An effective feedback process feedback detail in the content areas that the student cohort can influence students beyond task-level to process- or did not do well (L. Chen et al., 2020; Jimenez & Boser, SRL-level via feedback-informed action; conversely, 2021). feedback disconnected from student context tends to be Additionally, a number of testing organizations such disregarded instead (Bulut et al., 2020; Carless, 2019). as Educational Testing Service (ETS) and Pearson have An example of actionable feedback could be “you could implemented an automatic essay scoring system (AES) improve your performance in domain X by reviewing to assess written essays from test takers for a more lecture Y on topic Z”; this way, the feedback will be able to efficient workflow; the system evaluates the essay based guide students in their actions. Instructors could also add on elements such as grammatical error, writing style, statements that encourage students to reach out to them and discourse structure not only to ease the scoring to maintain interaction between student and instructor process but also to provide relevant feedback to the test for potential follow-up. takers for their improvements such as grammar usage, vocabulary diversity, or essay organization (Gardner et al., 2021). Another instance of AI-related assessment is 3. The Involvement of AI in computerized adaptive testing (CAT), which is usually Educational Feedback implemented in high-stakes testing such as the Graduate The application of AI in feedback practices and education Management Admission Test (GMAT) or the Graduate is increasing, and this trend is likely to continue as more Record Examination (GRE) via internet delivery than 50% of human proficiency levels such as literacy, (Gardner et al., 2021). CAT automatically tailors item numeracy, and problem-solving can be covered by AI. selection by matching the test taker’s estimated ability Specifically, the current capability of AI can fully cover to items to be administered with a rule-based computer level 2 proficiency that 53% of OECD adults can achieve system based on Item Response Theory, so that the test and can increasingly cover level 3 proficiency that 36% can maximize the gained information by delivering of OECD adults can achieve (Elliott, 2017; Holmes et items with appropriate parameters (e.g., difficulty) to al., 2019). The mentioned level 2 and level 3 proficiency examinees (Magis et al., 2017). Test providers can then include acting on less explicit mathematical information provide personalized feedback from the information and ideas, comprehending lengthy and non-continuous gained from the assistance of CAT as each examinee texts, and solving problems requiring multiple steps and received a different set of test items (Economides, constant monitoring (National Center for Educational 2005). Aside from the educational assessment area, the Statistics [NCES], 2022). intelligent tutoring system (ITS) has been used to provide The early application of AI for feedback purposes is corrective feedback and suggestions to student errors dated back to the 1950s, during which AI was used for and human tutors as an enhanced teaching practice adaptive learning (the self-adaptive keyboard instructor) (Ai, 2017). A meta-analysis of 50 controlled evaluations or computerized assessment (Holmes et al., 2019; of ITS found that students who receive assistance from Pask, 1982). AI will likely continue to play important ITS exhibited greater performance than students from roles in education in the future due to its benefits. The conventional human-only classes (Kulik & Fletcher, combination of AI technology and high-quality human 2016). 98 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen 4. Primary Applications of AI (Roberts, 2019). Textual data used in NLP can come in any shape ranging from simple words or sentences such Technologies in Educational as student-authored course reviews to complex essays Feedback with various structures and writing styles from GRE/ GMAT examinees (Moreno & Redondo, 2016; Roberts, 4.1  Definition and Scope of AI Technology 2019). Algorithms used in NLP can automatically convert in the Three Fields the data into understandable formats and extract non- AI is an umbrella term covering a wide area of machine trivial information from it by analyzing elements such capability, from basic problem solving such as rule-based as syntax, semantics, morphology, or even the basic message delivery to advanced decision-making such as frequency of words (Goddard, 2021; Moreno & Redondo, multi-class machine learning-based classification. The 2016). Some applications of NLP in education include most relevant AI technologies for feedback are situated text summarization to extract essential elements from in NLP, EDM, and LA (Gardner et al., 2021; Lemay et al., unstructured documents (e.g., theses, essays, or reports), 2021; Zhang et al., 2019). Table 2 presents the juxtaposition machine translation to bypass or mitigate language between the application of AI technologies in these three barriers, and sentiment analysis to gain insights into research fields in terms of their definition and capability. public opinion (Goddard, 2021). In terms of data requirement, the application of AI in The application of AI in EDM focuses on using the three research fields can process any kind of data as ML techniques such as clustering or classification of they work together; for example, ML techniques used in educational databases for knowledge discovery (Guo et EDM can either process numerical data by themselves al., 2015; Hussain et al., 2018; Qazdar et al., 2019). Data or process textual data with the help of NLP to support used in EDM can be both numerical and textual in nature feedback practices; thus, it is impossible to attribute the and can take various forms, such as student performance application of AI in the three fields to any specific types as indicated by their GPA, their history of grade repetition of data as the three fields share overlapping space in the as indicated by self-reported binary indicator (i.e., yes vs. actual practice. Note that despite being applicable to the no), or even the educational level of their parents; such educational field, NLP’s capability also spans to non- data can come from large-scale sources such as student educational settings as well. records or educational surveys (e.g., PISA, TIMSS) The application of AI in NLP focuses on manipulating (Bethany et al., 2021; Hussain et al., 2018; OECD, 2019). unstructured textual data for understanding, interpreting, Some applications of EDM include the prediction of and potentially generating relevant textual output student performance from school databases to identify Table 2. Focus and capability of AI technologies in the three fields Focus of the Application of AI Research Field Capability Technology The understanding and Convert textual data for translation NLP manipulation of textual data or pattern extraction Automatically extracting information Knowledge extraction from EDM using machine learning to discover educational databases insights Process learning activity data The leveraging of student activity LA to support human judgment in data for classroom optimization classrooms Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 99  Artificial Intelligence: Transforming the Future of Feedback in Education potential low achieving students or the development of adaptive learning systems with student log data to provide personalized lessons (Elatia et al., 2016; Qazdar et al., 2019). Such insights can support stakeholders in the educational context, such as principals, teachers, or even parents, to make informed decisions on matters such as curriculum design or school development (Elatia et al., 2016). The application of AI in LA focuses on collecting and leveraging accumulated data on student learning processes and activities for classroom optimization (Larusson & White, 2014; Sipes, 2017). Similar to EDM, the research focus of LA lies in the translation of data-driven insights Figure 1. The scope of learning analytics, educational data into practical recommendations to guide the process of mining, and natural language processing1. planning, decision making, and intervention (Larusson & White, 2014; Siemens, 2012). Like EDM, LA can also process numerical and textual data (Lemay et al., 2021). teaching effectiveness with technology-enhanced learning However, the difference between the two fields is that environments or to develop an early warning system to LA operates from a holistic framework that considers assist with academic advising (Larusson & White, 2014; student data as a whole while performing descriptive and Sipes, 2017; Tempelaar, 2020). Rather than being mutually diagnostic analyses with an emphasis on the process of exclusive, the applications of AI in the three fields share teaching and learning. On the other hand, EDM focuses overlapping space in their usage of educational data as more on knowledge discovery from various analysis they are often utilized together for innovative applications techniques (Lemay et al., 2021). of AI in education. Figure 1 shows a visual representation EDM researchers focus more on the automated of the intersection among NLP, EDM, and LA in solving discovery aspect, such as predictive or descriptive analytics educational problems. and machine learning models. In contrast, LA researchers One example of the combination of EDM and LA focus on leveraging data to inform human judgment, such (EDM+LA) is the usage of ML techniques such as as investigating data patterns, resource allocation, and cluster analysis to find potential groups of students (e.g., their effect on learning and teaching practice (G. Chen et low achievement vs. high achievement) and predictive al., 2020; Siemens, 2012). In other words, we could say that model to predict students’ learning outcomes to extract LA is more end-users-oriented while EDM is automation- insights from educational databases before displaying oriented. Some applications of LA is the leveraging of a them along with other information such as learning data-rich environment by collecting digital footprints process data (e.g., students’ test-taking time) on a LA of learning activity from learning management systems dashboard to inform students in their learning strategy (LMS) data such as records of course material access, (Larusson & White, 2014; Lemay et al., 2021; Qazdar student demographics, or course history to maximize et al., 2019); this combination leverages the capability 1 Note. LA stands for learning analytics, EDM stands for educational data mining, and NLP stands for natural language processing. The LA part is adapted from “Using learning analytics in SoTL,” by S. Sipes, 2017, Center for Innovative Teaching and Learning @ IUB, (https://blogs.iu.edu/citl/2017/12/13/using-learning-analytics-in-sotl). Copyright 2021 by Indiana University Bloomington. The EDM part is adapted from “A machine learning algorithm framework for predicting students performance: A case study of baccalaureate students in Morocco,” by A. Qazdar, B. Er-Raha, C. Cherkaoui, and D. Driss, 2019, Education and Information Technologies, 24(6), p. 3579 (10.1007/s10639-019-09946-8). Copyright 2021 by Education and Information Technologies. The NLP part is adapted from “5 Natural language processing examples: How NLP is used,” by T. Roberts, 2019, Bloomreach, (https://www.bloomreach.com/en/blog/2019/09/natural-language-processing). Copyright 2021 by Bloomreach. 100 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen of EDM by applying ML models to educational data instructors in formulating feedback information (i.e., the and leverages the power of LA with its dashboard for what) and the communication of feedback (i.e., the how) student improvements. Researchers can also combine the in various ways to enhance students’ learning experience capacity of EDM and NLP (EDM+NLP) by extracting and outcomes. both textual data (e.g., writing quality of students’ written assignment) with NLP and numerical data (e.g., course 4.2 The Feedback Dimension Framework grade) with EDM. The combined information can predict Applying AI technology to feedback requires more students’ class completion to aid teachers in their course specificity within the educational context to ensure that planning, including feedback provision (Crossley et al., the result contributes to feedback delivery for teachers 2015; Moreno & Redondo, 2016). For the combination and feedback understanding for students. In this paper, of NLP and LA (NLP+LA), instructors could extract we use Gamlem and Smith (2013)’s framework to organize students’ learning process data with LA and their textual aspects of feedback practice into two dimensions, namely, data with NLP to automatically generate verbal feedback feedback strategy and feedback content. Each category for students (Piotrkowicz et al., 2017). The three fields consists of various sub-characteristics of feedback, such (LA+NLP+EDM) can also work together by extracting as focus, clarity, or honesty. We have summarized and students’ learning process data from LA and combining it defined the sub-characteristics that could be applied to with results from machine learning analyses of students’ the application of AI in the three fields in Table 3. The data from EDM. This information can inform the definition of feedback characteristics from Gamlem and generation of data-supported feedback in writing with Smith (2013)’s framework. NLP and non-verbal format (e.g., graphs, numbers) via From the framework summarized in Table 3, the a LA dashboard (Li & Xing, 2021; Pardo et al., 2017). feedback content dimension covers the description of The mentioned combinations show that AI can support Table 3. The definition of feedback characteristics from Gamlem and Smith (2013)’s framework Dimension of Sub-Characteristics of Definition Feedback Feedback The targeted level of change (e.g., task level vs. Focus thought process level). The data to which students’ performance was Comparison compared. Feedback Content The purpose of the feedback itself (i.e., descriptive Function vs. judging). The tone of the feedback content (i.e., positive vs. Valence negative) The format in which the feedback is delivered (i.e., Mode verbal vs. non-verbal). The frequency and timeliness in which the feedback Timing is delivered. Feedback Strategy The utility of feedback related to the given Use timeframe for students to use the feedback. The manageability of the feedback by students and Management teachers. Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 101  Artificial Intelligence: Transforming the Future of Feedback in Education Table 4. The Summary of AI Applications in Feedback Dimension Natural Language Educational Data Learning Analytics of Feedback Processing Mining Content Focus Provide task-level Personalized process-level Task-level feedback via score feedback for domain- feedback as informed reporting and point-of-error specific tasks (e.g., by student behavior and identification. academic writing, learning style from data physics) mining techniques (e.g., Personalized process-level clustering). feedback as informed by Provide process-level student behavior, profile, and feedback such as verbal Provide SRL-level other manually imported hints through natural feedback via findings learning analytics data. language generation from EDM models using student behavior and Provide SRL-level feedback Provide SRL-level survey-based SRL-related through the monitoring feedback by generating construct as features. of student activity both context-relevant feedback individually and as a group. to aid in self-reflection Assess manually written feedback in terms of feedback level (task-level vs process-level vs self- level) Comparison Self-referenced feedback Primarily self-referenced Self-referenced from based on previous work. from students’ previous students’ previous profiles. profiles. Norm-referenced Criterion-referenced as feedback based on writing indicated by the domain of work in databases such as the task. Expertiza or SWoRD Norm-referenced Criterion-referenced as indicated by class feedback based on performance. domain-specific standards such as grammatical rules or mathematics. Function Judging feedback based Data-driven feedback can Primarily descriptive due to on aspects such as be either descriptive or its formative nature. content type, coverage, judgemental as intended and plagiarism. Outputs by the instructor. are in the form of scores/ grades. Descriptive feedback by flagging errors, their locations, and suggestions for improvements. 102 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen Valence Implied neutral as Instructor dependent. Instructor dependent. generated by the system. Strategy Mode Automatically generated Verbal feedback via Verbal feedback via manually verbal feedback on natural language input dialogues from the the quality of work generation or manual instructor. and location of errors. input from the instructor. Automatically generated Non-verbal feedback with verbal prompts on Non-verbal feedback with data visualization delivered suggestions for revision. data visualization. via an interactive dashboard. Non-verbal feedback in scores or metrics assessing the quality of elements as indicated by the rubric. Both feedback modes can be delivered via chatbots. Timing Immediate feedback Immediate output Semi-automatic output delivery via natural delivery with automatic by allowing instructors to language generation. feedback formulation. co-author feedback with algorithms. Immediate output delivery with automatic feedback formulation. Longitudinal feedback (i.e., across time) is also available. Use High usability as NLP- EDM-informed feedback High usability due to its based feedback is usually is frequently used for formative nature. formulated for formative formative purposes, purposes, implying which implies the opportunities for students opportunity to use the to put feedback into received information. action. Management User-friendly interface Feedback personalization Feedback personalization at with self-paced feedback and user-friendly both individual- and group navigation. interface make the level makes the feedback feedback easily easily manageable and Error localization makes manageable and relatable. relatable. the feedback more manageable. An interactive dashboard for step-by-step guidance also increases manageability. Interface localization (e.g., Chinese version) makes the feedback more accessible. Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 103  Artificial Intelligence: Transforming the Future of Feedback in Education feedback itself (i.e., the what) with four sub-characteristics inform students to adjust their learning strategy before that include: 1) the focus of feedback, 2) the comparison of the final exam; therefore, it has high usability compared feedback, 3) the function of feedback, and 4) the valence to summative feedback given after a course is over. Lastly, of feedback. The focus aspect of feedback synergizes the management of feedback concerns the manageability well with Hattie and Timperley (2007)’s framework of of the feedback by students and teachers; for example, feedback level (see Table 1) as they both concern levels personalized feedback could be easier to grasp than of change in students from receiving feedback. Therefore, generic feedback and, therefore, more manageable to we describe the focus of feedback with the four levels students (Pardo et al., 2019). of change (i.e., task-, process-, SRL-, and self-level) as Both feedback aspects are equally important to suggested in Hattie and Timperley (2007)’s framework. convey information about students’ performance for their The comparison of feedback concerns how students’ improvement. The content part ensures that the message performance was compared. For example, in criterion- is meaningful and actionable for the students, while the referenced comparison, students’ scores are compared strategy part ensures that the feedback is appropriately with specific criteria (e.g., 95-100% = A, 90-95% = A-), delivered to students at the right time, with the right whereas in norm-referenced comparison, students’ scores amount, and on the right channel (Brookhart, 2008). We are compared against each other (e.g., the top 10% of applied the literature on the application of AI to the three students receive an A, the next 30% gets a B). Also, it research fields to Gamlem and Smith (2013)’s framework is possible to apply self-referenced comparisons where as the anchoring point and discussed how AI could benefit students’ scores are compared against their previous the formulation and delivery of feedback at a fine-grained scores (e.g., comparing scores from the first and second level. See Table 4 for the summary of AI applications in midterms). The function of feedback concerns the purpose feedback. This Table could serve as the current landscape of the feedback itself, such as a descriptive function that of what AI can do to benefit feedback practices Table 4. describes students’ performance as is (e.g., you got 80/100) The summary of AI applications in feedback or a judging function that forms conclusions about students’ performance (e.g., you did well with 80/100). 4.3 The Application of NLP, EDM, and LA The “you did well” part implies that students’ performance to Educational Feedback is judged positively, whereas the descriptive feedback only As summarized in Table 4, the primary benefit of NLP in provided students’ scores. Lastly, the valence of feedback feedback practices is the additional capability to generate concerns the tone of the feedback itself (i.e., positive, feedback on students’ written performance based on negative, neutral). linguistic components such as writing quality, syntactic The second dimension of feedback strategy covers complexity, and grammatical errors before providing how the feedback is delivered from instructors to verbal feedback or numerical scores to students. The students (i.e., the how) with four sub-characteristics benefit of EDM to feedback practices relies on the results that include: 1) the mode of feedback, 2) the timing of of machine learning techniques to establish and deliver feedback, 3) the use of feedback, and 4) the management data-supported feedback via data visualization such as of feedback. The feedback mode concerns the format normative curves or bar charts. EDM can also be used in which the feedback is delivered; for example, verbal to provide verbal feedback by using NLP-based systems feedback could be manually written feedback as informed or relying on manual input from the instructors as by results from AI (i.e., semi-automatic process) or an informed by results from the algorithm. The benefit of automatically generated sentence (i.e., fully automated LA for feedback practices relies on monitoring students’ process). The timing of feedback concerns the frequency activity data such as time use or interaction history with and timeliness in which the feedback is delivered in real course material to provide personalized feedback via time (i.e., immediately available) or not in real time (i.e., an interactive dashboard. Like EDM, LA feedback can taking time to inspect the feedback before releasing it to be either semi-automatic or fully automatic, depending the student). The use of feedback concerns the utility of on the implemented system. The application of AI can feedback related to the given timeframe for students to provide immediate feedback with high usability and use the feedback. For example, formative feedback can 104 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen Figure 2. Feedback practice in NLP. manageability due to its formative nature and user- provide feedback that could stimulate self-reflection as friendly interface. well as encourage learners with motivational prompts NLP. The application of NLP to feedback primarily focuses (e.g., “you are on the right track”, “keep going”) at the same on text processing for constructed response tasks such as time to support student self-regulated learning with SRL- essay writing through the usage of NLP-based models for level feedback (Desai & Chin, 2020; Pengel et al., 2021). In text-based feature extraction, language recognition for the case of manually written feedback, NLP can be used feedback selection, or natural language generation for to develop a content classifier model to assess whether the automatic feedback generation (Li & Xing, 2021; Zhang et written feedback falls into task-level, process-level, SRL- al., 2019). Figure 2 visualizes how NLP technologies can level, or self-level feedback to allow instructors to provide enhance feedback practices in both feedback content and personalized feedback to students and target the intended feedback strategy dimensions. level of learning (Cavalcanti et al., 2020). For its application to feedback content, task-level In terms of the comparison aspect, feedback can be feedback is generated to assess the quality of domain- provided based on the previous work of the students specific tasks such as essay structure via the automatic or a normative database via a web-based writing and essay scoring system or constructed response tasks in revising platform such as the eRevise, the Expertiza, and Physics (Dzikovska et al., 2014; Zhang et al., 2019). the Scaffolded Writing and Rewriting in the Discipline NLP-based feedback software programs can also (SWoRD) project where students’ work is compared with flag the location of the error, assess the clarity of the instances of other students (Ramachandran et al., 2017; content, and provide feedback to assist students in their Zhang et al., 2019). The system can also provide feedback improvement (Lan et al., 2015; Xiong et al., 2012). For based on grading standards (or criteria) of domain- example, AcaWriter is a web-based writing assistance tool specific tasks such as mathematics, grammatical rules, or that assesses students’ analytical and reflective writing physics formulas (Dzikovska et al., 2014; Kochmar et al., and provides real-time feedback on academic writing 2020; Lan et al., 2015; Perikos et al., 2017). characteristics such as clarity, conciseness, and rhetorical NLP-based feedback systems can provide both connotation (Knight et al., 2020). Process-level feedback judging and descriptive feedback. The system can is usually provided via Intelligent Tutoring System (ITS) evaluate students’ writing to provide judging feedback by generating step-by-step hints on working toward the such as scores/grades on content type, coverage, tone, correct answer. The ITS can also use students’ meta- volume, and plagiarism or even categories of the cognitive data from the behavioral log to enhance its performance itself (high vs. low quality) (Ötleş et al., procedural feedback (Kochmar et al., 2020; Perikos et al., 2021; Ramachandran et al., 2017; Zhang et al., 2019). 2017). In addition, NLP-based conversational agents can The system can also provide descriptive feedback to flag also generate context-relevant feedback by comparing and localize the detected elements and assess their clarity students’ level of knowledge to the course material and with keyword detection (Dzikovska et al., 2014; Lan et al., Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 105  Artificial Intelligence: Transforming the Future of Feedback in Education 2015; Perikos et al., 2017; Xiong et al., 2012). Lastly, the from”), error localization, and clarity assessment are also valence of the feedback provided by NLP is implied to be available on the feedback provision platform such as LMS neutral as generated by the system (Kochmar et al., 2020). for students to better understand their feedback as well By applying NLP to the feedback strategy, the system (Dzikovska et al., 2014; Xiong et al., 2012; Zhang et al., can provide verbal prompts and suggestions for students 2019). to improve their work through a user interface (Xiong et al., 2012; Zhang et al., 2019). Descriptors of work EDM. EDM primarily contributes to feedback via quality (e.g., effective-mediocre-ineffective-other) can be insights extracted from data mining techniques such as delivered via a user interface in natural language sentences association rule mining, clustering, and classification, (Ötleş et al., 2021; Solano et al., 2021). The developers can most of which fall under machine learning (Pechenizkiy combine the mentioned features with a pop-up chatbot to et al., 2008; Ray & Saeed, 2018). Additionally, EDM can automate the feedback process and increase interactions also process both numerical data and textual data with with the students (Kochmar et al., 2020). Non-verbal the help of NLP, thus, enabling it to leverage data of feedback is also available on the same interface as scores many forms to support feedback in education. Figure 3 or metrics of each element (e.g., relevance or plagiarism) visualizes how EDM technologies can enhance feedback or as locations of the error, such as “paragraph X” or “page practice in both feedback content and feedback strategy Y” (Lan et al., 2015). dimensions. NLP-based feedback software programs can provide Feedback content-wise, EDM can inform teachers to immediate feedback with its automation in terms of provide feedback at either task-, process-, or SRL-level. feedback timing. Feedback provided by NLP-based Personalized process-level feedback can be formulated software programs through their user interface is usually based on data on student behavior, learning style, and used for formative purposes; that is, the feedback is web usage behavior as logged by the LMS (Anjewierden supposed to be used to improve students’ work before et al., 2007; Romero & Ventura, 2010). Instructors can their next submission or post-test in subjects such as also adjust their feedback to task level to fit their needs mathematics or academic writing (Kochmar et al., 2020; (Merceron & Yacef, 2005). Feedback from EDM is Lan et al., 2015; Zhang et al., 2019). For some software primarily self-referenced as students can reflect on data programs, such as Natural Language to First-order Logic of their previous lessons, but instructors can provide both (NLtoFOL), students can also request formative feedback norm- and criterion-referenced feedback by tailoring before their submission (Perikos et al., 2017; Xiong et al., their feedback to fit the needs of the class (Merceron & 2012; Zhang et al., 2019). For the feedback management Yacef, 2005; Romero & Ventura, 2010). For SRL-level aspect, feedback is delivered via a user-friendly interface feedback, predictive models (e.g., decision tree) or that students can navigate at their own pace (Perikos et clustering models (e.g., k-means clustering) from EDM al., 2017; Zhang et al., 2019). Additional prompts (e.g., can identify and operationalize student behavior (e.g., “provide more detail from…” or “use more evidence system gaming, careless response) to provide feedback Figure 3. Feedback practice in EDM. 106 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen that is relevant to their motivation to inform students in & Ventura, 2010). The feedback can be both formative developing their self-regulation (Winne & Baker, 2013). and summative, so students may use feedback and the In addition, instructors can include surveys that measure provided resource to inform their subsequent activities, SRL-related constructs such as the Online Self-Regulated such as second submission or post-test (Merceron & Learning Questionnaire (OSLQ) and the Motivated Yacef, 2005; Ray & Saeed, 2018). Lastly, EDM-informed Strategies for Learning Questionnaire (MSLQ) in the feedback is usually provided via an online learning LMS to extract features that can inform EDM models in environment, which can be designed in a user-friendly providing SRL-level feedback as well (Araka et al., 2019). format for better manageability (Merceron & Yacef, 2005; EDM-driven feedback can also be either judging or Romero & Ventura, 2010). Feedback personalization descriptive. Descriptive feedback is provided in the form based on student profiles also makes the message more of recommendations such as “from your performance, you relatable to students (Pechenizkiy et al., 2008; Ray & may want to check out course material X” (Pechenizkiy et Saeed, 2018). al., 2008; Romero & Ventura, 2010). However, instructors may also set up the dialogue to be judging, such as “you LA. Like EDM, LA contributes to feedback by providing have a satisfactory performance” (Merceron & Yacef, 2005; insights extracted from student profiles with differences in Romero & Ventura, 2010). In terms of feedback valence, its primary focus to inform human decisions in learning feedback provided by EDM can be either positive, and teaching instead of model generalization (Gardner negative, or neutral, as instructors who give the feedback et al., 2021; Lemay et al., 2021). Despite being similar can adjust the tone to align with their teaching style. to EDM-driven feedback, LA-driven feedback sets itself For EDM application to feedback strategy, EDM- apart based on its capability to monitor student activity informed feedback can provide both verbal and and provide feedback in both cross-sectional formats non-verbal responses. Predetermined dialogues and (e.g., one time) or longitudinal format (e.g., across the natural language generation can provide written answers semester). Also, LA-driven feedback is delivered via from teachers, while the system can give graphical an interactive dashboard as opposed to a static format, information with data visualization (Anjewierden et al., which is the characteristic that emphasizes the practical 2007; Ray & Saeed, 2018; Romero & Ventura, 2010). aspect of the data more when compared to EDM-driven Depending on the system, EDM-informed feedback can feedback (G. Chen et al., 2020). Figure 4 visualizes how take time if the feedback is human-processed (Merceron LA technologies can enhance feedback practice in both & Yacef, 2005). Some systems allow automatic feedback feedback content and feedback strategy dimensions. formulation to provide immediate non-verbal feedback Content-wise, LA can provide task-level feedback to student users via an online learning platform or LMS by displaying students’ performance in each task in the (Pechenizkiy et al., 2008; Ray & Saeed, 2018; Romero format of scores or error localization (i.e., where they Figure 4. Feedback practice in LA. Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 107  Artificial Intelligence: Transforming the Future of Feedback in Education did wrong) (Sedrakyan et al., 2020). LA can also provide where the program processes student data to inform the process-level feedback by using additional information instructors in their feedback write-up. Also, it can be from learners’ behavior data such as their approach to the fully automatic in the case of AcaWriter where feedback task, course history, academic proficiency profile, or even is automatically and immediately generated based on the surface-level data such as clickstream in addition to the document that students provide to the software (Knight et task performance data (Gardner et al., 2021; Tempelaar al., 2020; Tsai et al., 2021). Feedback can also be presented et al., 2015). Instructors can also manually import their across time with a longitudinal design to inform students data into the system to inform their feedback formulation of their improvements (Sedrakyan et al., 2020). For the (Tsai et al., 2021). For SRL-level feedback, LA can track feedback use aspect, LA-based feedback is intended for SRL-related features to inform its feedback formulation, formative purposes, so students usually have time to such as a competency tracking dashboard that monitors process and apply it to their learning strategy (Sedrakyan students’ domain knowledge and motivational factor et al., 2020). LA-based feedback has high manageability that could affect them in their self-regulation, learning as it is personalized to match the profile of each student, design tracking that monitors students’ activity to inform as well as presented via an interactive dashboard with instructors of course design, and even teamwork tracking step-by-step guidance to inform students about their that monitor students’ activity in relation to their roles in thinking (Barana et al., 2019; Sedrakyan et al., 2020; Tsai the group through social network analysis; information et al., 2021). With customization, students can choose can then be processed and displayed in a dashboard as whether they want to view step-by-step feedback hints, SRL-level feedback to inform students of their progress consider an example of the task, or send a message to the and create awareness in their learning strategies (Matcha instructor (Tempelaar et al., 2015). Instructors can also et al., 2020; Viberg et al., 2020). localize the output dashboard with the local language LA-based feedback can be either self-referenced (e.g., Chinese) to make the feedback and its software even as informed by previous profiles of the students more accessible for non-English speakers (Wang & Han, for personalized suggestions, criterion-referenced as 2021). Lastly, instructors can choose to deliver feedback informed by domain-specific criteria (e.g., mathematical to groups of students or tailor it to one student, making solutions), or even norm-referenced should the the feedback process more manageable for both students instructors wish to give feedback in the form of peer- and instructors (Pardo et al., 2019). comparison (e.g., class average) (Barana et al., 2019; Pardo et al., 2019; Wang & Han, 2021). For the function aspect, LA-based feedback is usually descriptive to fit its 5. Transforming the Future of formative purposes (Knight et al., 2020; Sedrakyan et al., Feedback in Education 2020). Content of the feedback could be suggestions for The previous section discussed the current state of the improvements, descriptions of the current performance, application of AI to the fields of NLP, EDM, and LA for and prescriptive feedback for correction (Pardo et al., educational feedback. However, as technology evolves, 2019; Wang & Han, 2021). Lastly, in terms of feedback the AI applications could be improved to maximize valence, instructors can frame their feedback as either the benefit that students receive from their feedback as positive, negative, or neutral to fit their teaching style. technology continues to advance. This section discusses For the application of LA to feedback strategy, LA-based feedback can be both verbal and non-verbal. future directions where AI can be enhanced to transform Verbal feedback such as suggestions is usually prepared the future of feedback practice. by the instructors as informed by results from LA, Table 5 summarizes the actions, characteristics, and while non-verbal feedback takes the form of a graphical functionalities we can acquire to improve the application display of numerical information (e.g., line chart) via of AI technologies in the three research fields to further interactive dashboards (Barana et al., 2019; Gardner et al., the capability of educational feedback based on the 2021; Wang & Han, 2021). LA-based feedback software reviewed literature. The suggestion for improvement programs can be semi-automatic in the case of OnTask is based on Gamlem and Smith (2013)’s Framework of 108 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen Table 5. The future direction of AI applications to improve feedback in education Natural Language Dimension of Feedback Educational Data Mining Learning Analytics Processing More fine-grained profiling More data for coverage in for deeper personalization terms of corpus structure, Contextualization with (e.g., student preference, topics, and examples. more variable features for interest, bias, and Increase accessibility with deeper personalization. biofeedback) localization and machine Utilize appropriate Shared open-source databases Content translation measures to extract across institutions for more Contextualization with domain knowledge domain content coverage. domains-related factors for enhanced Consider informal social for accurate content contextualization. learning and personal interpretation. learning environments for more profile coverage. Interoperability with AI in Utilizing longitudinal records other research fields (e.g., across the course and school LA) for innovative modes of year for future-oriented delivery (e.g., dashboard). Fully integrated EDM feedback information. Eliminate redundant feedback Strategy with LMS for enhanced Multidirectional cycle for efficiency. automation capability. communication between Contextualization with learners, the system, and different domains and student instructors for appropriate profiles for appropriate feedback strategy. feedback delivery strategies. feedback dimension. To further improve the application Future studies applying NLP techniques to enhance of AI to benefit feedback content, more data is needed feedback practices could consider expanding the for AI algorithms to expand the scope of vocabulary and coverage of the system with more data on domain-related contextual-relevant patterns for the feedback system to topics to improve relevance and practicality (Ötleş et recognize and inform its results (Bengfort et al., 2018; al., 2021; Solano et al., 2021; Zhang et al., 2019). Output Elatia et al., 2016). To improve the application of AI in relevance in both feedback content and feedback delivery feedback practices, researchers could consider applying could also be improved by considering contextual AI technologies across the three fields together. Also, factors such as student profile, knowledge gap, domain- they can utilize the capability of different technologies for specific convention (e.g., mathematical expression), enhanced effectiveness, such as using machine translation writing complexity, differences between first and second with an LA dashboard to expand the accessibility of a language learners, and the interaction of item difficulty learning platform to international students or integrating and feedback effectiveness to make the feedback more EDM into a learning environment at a deeper level context-specific; some of these data can be obtained from (i.e., using the computer assistance more) for enhanced LA (Goddard, 2021; Kochmar et al., 2020; Lan et al., 2015; automation in the feedback process. Perikos et al., 2017; Xiong et al., 2012). Further, improving error localization and clarity assessment of students’ work Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 109  Artificial Intelligence: Transforming the Future of Feedback in Education with higher precision by expanding the keyword database to its relevant learning outcome in future subjects to and optimized algorithm could reduce redundancy in examine how past feedback informs future performance the feedback cycle, resulting in a quick turnaround and (e.g., introduction to calculus to intermediate calculus) more efficiency in the feedback process (Dzikovska et al., (Ryan et al., 2019). The multidirectional channel between 2014; Ramachandran et al., 2017). Researchers could also students, instructors, and the system could also be increase the accessibility of the output with localization established to enable the instructors to reflect on their and machine translation, which could be combined instructional design via learners’ results to formulate with an interactive dashboard for enhanced feedback appropriate feedback strategies (Sedrakyan et al., understanding (Barana et al., 2019; Moreno & Redondo, 2020). 2016; Wang & Han, 2021). Future research in the application of EDM to 6. Discussion feedback practice could consider extracting domain- specific knowledge and contextual factors (e.g., students’ This theoretical paper aims to identify the current performance in related courses) with appropriate applications of AI technology to feedback in education procedures to add more variable features into the and to identify venues for future research in technology- predictive model for deeper feedback personalization driven feedback practice. The application of AI in each (Lan et al., 2015; Merceron & Yacef, 2005; Pechenizkiy of the three fields (i.e., NLP, EDM, and LA) has its et al., 2008). Researchers could also consider developing unique contribution to the feedback of different natures, a fully-integrated LMS with a built-in EDM component some examples are the usage of AES from NLP, process for a higher automation capability that can handle a large mining from EDM, and an analytics dashboard from amount of data with minimal human processing (i.e., LA. With adequate expertise, researchers could develop manually data importing) (Ray & Saeed, 2018; Romero a feedback system that is situated across the three fields & Ventura, 2010). such as a predictive model that can process textual data For the application of LA to feedback, researchers by combining the capability of NLP with EDM (Lan et could collect student profiles at a deeper level, such as al., 2015). learner differences in preference, interest, and bias via An AI-driven feedback system could leverage the survey, more performance data via stealth assessment, capability of online educational platforms by increasing the and biological data via wearables to expand coverage quality of provided feedback and enhancing the efficiency across student characteristics, disciplines, and grades of the feedback process. A real-time conversation-based (i.e., K-12 vs. higher education); however, ethical issues system embedded in online assessments could stimulate on privacy should be considered as well (Gardner et information exchange between students and the digital al., 2021; Karaoglan Yilmaz & Yilmaz, 2021; Pardo tutor and thus improve their learning engagement and et al., 2019; Sedrakyan et al., 2020; Tsai et al., 2021). motivation (Yildirim-Erbasli & Bulut, 2021). Further, Researchers could also expand data coverage by sharing displaying verbal and non-verbal feedback together via the database for writing patterns across institutions, or a LA dashboard could synergize the capability of both they could collect data on informal social learning and feedback formats, help students understand patterns of personal learning environments (e.g., forum discussion, their learning progress with data visualization, and receive homework, in-class activities) into account as they are personalized messages as informed by predictive models primary predictors of academic performance (Knight et (Qazdar et al., 2019; Wang & Han, 2021). Thus, using AI al., 2020; Sedrakyan et al., 2020; Tempelaar et al., 2015). technologies in feedback could constitute the best practice Future research could also explore the use of future- of effective feedback communication in terms of design, oriented feedback by utilizing a longitudinal record to contents, and ancillary materials (Zenisky & Hambleton, identify the impact of feedback across the course duration. 2012). This would allow instructors to link feedback information 110 Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology  Tarid Wongvorachan, Ka Wing Lai, Okan Bulut, Yi-Shan Tsai and Guanliang Chen Automated or semi-automated feedback systems generation with features relevant to their fields, such could also enhance the efficiency of the feedback process as students’ programming background in the STEM by reducing the time and resources required from the field (Mbunge et al., 2021). On the student side, future instructors to provide formative feedback (Knight et al., research could explore topics about students’ data 2020). Specifically, automated feedback systems enable literacy, which influences their interpretation of data, students to learn and receive feedback asynchronously, and their experience with non-verbal feedback displayed allowing the instructors to focus on addressing concerns via a LA dashboard (e.g., data literacy skills for students or providing a one-on-one feedback session to students to meaningfully interpret graphs provided in a LA who need extra guidance (Perikos et al., 2017). The use dashboard) to help them make the most of their feedback of EDM also allows instructors to discover patterns of (Wasson et al., 2016). student performance in large-scale learning environments (e.g., MOOCs) with relative ease to provide feedback 7. References to students as a group (Romero & Ventura, 2010). Finally, by integrating a venue for students to respond Ai, H. (2017). Providing graduated corrective feedback in an to instructors about the quality of feedback, instructors intelligent computer-assisted language learning environ- can use such data to inform their development of ment. ReCALL, 29(3), 313–334. https://doi.org/10.1017/ feedback models (Flodén, 2017; Zenisky & Hambleton, S095834401700012X Anjewierden, A., Kolloffel, B., & Hulshof, C. (2007). Towards 2012). educational data mining: Using data mining methods As technology rapidly evolves, this paper could for automated chat analysis to understand and support serve as a starting point in the field of AI in educational inquiry learning processes. International Workshop on feedback by introducing the current landscape and Applying Data Mining in E-Learning (ADML 2007), future possibilities of the research area. However, 27–36. there are limitations to what AI can currently do. First, Araka, E., Maina, E., Gitonga, R., & Oboko, R. (2019). A predictive results about students’ learning outcomes conceptual model for measuring and supporting self- can be computed based on immutable variables such regulated learning using educational data mining on as gender, age, or socio-economic status that could learning management systems. 2019 IST-Africa Week hardly be used to form actionable recommendations Conference (IST-Africa), 1–11. https://doi.org/10.23919/ (Ramaswami & Bhaskaran, 2010). Instead of immutable ISTAFRICA.2019.8764852 predictors, instructors are encouraged to harness Barana, A., Conte, A., Fissore, C., Marchisio, M., & Rabellino, malleable learning predictors based on learning theory, S. (2019). Learning analytics to improve formative assess- ment strategies. Journal of E-Learning and Knowledge such as their formative assessment activities (Ramaswami Society, 15(3), 75–88. https://doi.org/10.20368/1971- & Bhaskaran, 2010; Tempelaar, 2020). Second, the 8829/1135057 automatically-generated feedback could overlook minor Bengfort, B., Bilbro, R., & Ojeda, T. (2018). Applied text but meaningful background information such as students’ analysis with Python: Enabling language-aware data relevant medical conditions (e.g., learning disorder products with machine learning (First edition). O’Reilly history). If possible, instructors should constantly check Media, Inc. the record of automatically generated feedback to ensure Bethany, F., Foy, P., & Yin, L. (2021). TIMSS 2019 user guide its relevance and improve the system whenever it is for the international database (2nd ed.). TIMSS & PIRLS appropriate. International Study Center. Future studies are needed to further explore and Boud, D. (2020). Challenges for reforming assessment: investigate the growing capabilities of technologically- The next decade. International Virtual Meeting: enhanced feedback practices for the betterment of Teaching, Learning & Assessment in Higher Education, education. On the system side, instructors could develop Universidad del Desarrollo. https://educacionsupe- rior2020.udd.cl/ EDM predictive models or NLP models for feedback Vol 23(Special Issue 1) | 2022 | http://jattjournal.net/ Journal of Applied Testing Technology 111  Artificial Intelligence: Transforming the Future of Feedback in Education Boud, D., & Molloy, E. (2013). Rethinking models of feed- Crossley, S., McNamara, D. S., Baker, R., Wang, Y., Paquette, back for learning: The challenge of design. Assessment & L., Barnes, T., & Bergner, Y. (2015, June 26). Language Evaluation in Higher Education, 38(6), 698–712. https:// to completion: Success in an educational data min- doi.org/10.1080/02602938.2012.691462 ing massive open online class. International Conference Brookhart, S. M. (2008). How to give effective feedback to on Educational Data Mining (EDM). International your students (1st ed.). Association for Supervision and Conference on Educational Data Mining (EDM), Spain. Curriculum Development. de Laat, M., Joksimovic, S., & Ifenthaler, D. (2020). Artificial Bulut, O., Cutumisu, M., Aquilina, A. M., & Singh, D. (2019). intelligence, real-time feedback and workplace learning Effects of digital score reporting and feedback on students’ analytics to support in situ complex problem-solving: A learning in higher education. Frontiers in Education, commentary. The International Journal of Information 4(65). https://doi.org/10.3389/feduc.2019.00065 and Learning Technology, 37(5), 267–277. https://doi. Bulut, O., Cutumisu, M., Singh, D., & Aquilina, A. M. (2020). org/10.1108/IJILT-03-2020-0026 Guidelines for generating effective feedback from Desai, S., & Chin, J. (2020). An explorative analysis of the e-assessments. Hacettepe University Journal of Education, feasibility of implementing metacognitive strategies in 1–13. https://doi.org/10.16986/HUJE.2020063705 self-regulated learning with the conversational agents. Carless, D. (2019). Feedback loops and the longer-term: Proceedings of the Human Factors and Ergonomics Towards feedback spirals. Assessment & Evaluation in Society Annual Meeting, 64(1), 495–499. https://doi. Higher Education, 44(5), 705–714. https://doi.org/10.10 org/10.1177/1071181320641112 80/02602938.2018.1531108 Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., & Carless, D., & Boud, D. (2018). The development of stu- Campbell, G. (2014). BEETLE II: Deep natural language dent feedback literacy: Enabling uptake of feedback. understanding and automatic feedback generation for Assessment & Evaluation in Higher Education, 43(8), intelligent tutoring in basic electricity and electron- 1315–1325. https://doi.org/10.1080/02602938.2018.146 ics. International Journal of Artificial Intelligence in 3354 Education, 24(3), 284–332. https://doi.org/10.1007/ Carless, D., & Winstone, N. (2020). Teacher feedback literacy s40593-014-0017-9 and its interplay with student feedback literacy. Teaching Economides, A. A. (2005). Personalized feedback in CAT. in Higher Education, 1–14. https://doi.org/10.1080/1356 WSEAS Transactions on Advances in Engineering 2517.2020.1782372 Education, 2(3), 174–181. Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Elatia, S., Ippercieli, D., & Zaiane, O. (Eds.). (2016). Data Nascimento, A., Freitas, F., & Gašević, D. (2020). How mining and learning analytics: Applications in educa- good is my feedback?: A content analysis of written feed- tional research. John Wiley & Sons. back. Proceedings of the Tenth International Conference Elliott, S. W. (2017). Computers and the future of skill demand. on Learning Analytics & Knowledge, 428–437. https:// OECD. https://doi.org/10.1787/9789264284395-en doi.org/10.1145/3375462.3375477 Flodén, J. (2017). The impact of student feedback on teach- Chan, K. S., & Zary, N. (2019). Applications and challenges ing in higher education. Assessment & Evaluation in of implementing artificial intelligence in medical educa- Higher Education, 42(7), 1054–1068. https://doi.org/10. tion: Integrative review. JMIR Medical Education, 5(1), 1080/02602938.2016.1224997 e13930. https://doi.org/10.2196/13930