Understanding Error Patterns in Students' Solutions to Linear Functions PDF

Learning and Instruction 92 (2024) 101895 Contents lists available at ScienceDirect Learning and Instruction...

Learning and Instruction 92 (2024) 101895 Contents lists available at ScienceDirect Learning and Instruction journal homepage: www.elsevier.com/locate/learninstruc Understanding error patterns in students’ solutions to linear function problems to design learning interventions Noor Elagha *, James W. Pellegrino Department of Psychology, University of Illinois at Chicago, USA A R T I C L E I N F O A B S T R A C T Keywords: Background: Reasoning with and about linear functions (LF) is considered essential knowledge for college Mathematical knowledge readiness, but evidence shows that students experience difficulty in this topic. There were two overarching aims Error patterns of the reported studies. One was to assess students’ understanding of LF and discern the cognitive underpinnings Problem solving of common errors they make in these types of problems. The second was to explore designs of a learning Worked examples Learning from errors intervention that can ameliorate these misunderstandings. Math learning and instruction Methods: In Study 1, analyses of performance on Verbal Description, Table, and Graph LF problem types showed substantial and interpretable errors on the latter two. Errors reflected systematic conceptual and procedural misunderstandings associated with interpretation of structural features of Table and Graph LF problems. The results for LF Table and Graph problems suggested that students could benefit from error focused interventions. In Study 2, information about common errors from Study 1 was used to inform the design of instructional in­ terventions using three worked example conditions: (1) error detection and correction, (2) error correction only, and (3) no error (control). Results & conclusions: Results of quantitative and qualitative analyses showed enhanced performance in all intervention conditions and across both problem types and were particularly impactful for both problem types when error correction was required in worked examples. Findings from both studies are discussed in terms of implications for learning environments, including initial instruction and diagnostic assessment. 1. Introduction (Mielicki & Wiley, 2016; Mielicki, Martinez, DiBello, Hassan, & Pelle­ grino, 2019). Reasoning with and about linear functions (LF) is considered As described in the sections that follow, there is a substantial liter­ essential knowledge for college readiness (e.g., Conley, Drummond, de ature in both the mathematics education and cognitive psychological Gonzalez, Rooseboom, & Stout, 2011). Simple LF problems are ubiqui­ literatures on the types of knowledge required for solving LF problems tous in multiple large-scale mathematics achievement tests intended for and the specific cognitive processes underlying solution of different high school and college bound students, including the U.S. National types of LF problems. However, the literature lacks a robust comparative Assessment of Educational Progress and international assessments such analysis of how the same individuals perform across different forms of as TIMSS and PISA (see e.g., Wijaya et a., 2014). The literature suggests simple LF problems and the nature and consistency of the errors they that mastering linear functions in a formal instructional environment make. A study was therefore conducted to evaluate the contents of brings with it conceptual and cognitive difficulties (Dorier & Sierpinska, student responses to typical LF problems which vary in task format and 2001). Understanding essentials of linear functions requires cognitive the conceptual and procedural knowledge and processing strategies flexibility, such as moving between tabular, algebraic, graphic, and se­ needed to solve them. This was done to inform the design and imple­ mantic representations, as well as moving between abstract, algebraic, mentation of a second study exploring the efficacy of instructional in­ and geometric languages, sometimes all in one task (Alves-Dias & terventions for LF problems that could lead to improved performance, Artigue, 1995; Dorier, 2000). Evidence suggests that college students including learning from worked examples with and without error experience considerable difficulty correctly solving even simple LF detection and error correction. problems and those difficulties vary depending on the problem format * Corresponding author. E-mail address: [email protected] (N. Elagha). https://doi.org/10.1016/j.learninstruc.2024.101895 Received 26 January 2023; Received in revised form 9 January 2024; Accepted 18 February 2024 Available online 16 March 2024 0959-4752/Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 1.1. Mastering linear functions ability to translate between different modes of mathematical represen­ tations in order to abstract and deeply understand mathematical con­ Mastering algebra, which includes understanding and solving linear cepts (Lesh & Doerr, 2003). From a cognitive perspective, to construct a functions problems, constitutes one of the most important proficiencies robust and sustainable schema of any mathematical concept and to be that students need for success in a variety of college majors and potential able to reason about the concepts, a student must learn to interpret careers (Conley et al., 2011). Such knowledge is typically acquired various modes of representation within the concept. It is important to before or during high school instruction. Nevertheless, students struggle expose students to different modes of representation because it eluci­ with LF problem solving for multiple reasons (e.g., Britton & Henderson, dates different essential features of mathematical concepts (Duval, 2009; Mielicki et al., 2019). Mielicki et al. (2019) found that when 2006). Knowing and understanding such features contributes to schema testing the mathematics ability of college students on problems development of the domain in that the relations between the aspects of requiring understanding and knowledge of linear equations, variables the concept are established and/or reinforced. Consequently, the ability and patterns, and linear functions, students’ performance was poorest to translate between representations is a predictor of problem solving for linear functions problems. A variety of explanations have been skills and the acquisition of conceptual and procedural knowledge offered for students lack of mastery of algebraic knowledge and LF (Wakhata, Balimuttajjo, Mutarutinya, & Rwamagana, 2023). Thus, function problem solving in particular. mastering and translating between multiple representations can pro­ One such explanation is the obstacle of formalism. Dorier, Robert, mote problem solving, reasoning, and procedural fluency within a Robinet, and Rogalski (2000) explain that because linear functions is domain (e.g., Mainali, 2021; Wakhata et al., 2023). Representational highly theoretical and abstract in nature, it causes what students report fluency can also aid in specificity in the constraints of interpretations as feeling a “fog” (Carlson, 1993). Formalism coupled with the many and foster a deeper understanding of the topic (Ainsworth et al., 2006). new definitions and concepts that only tangentially relate to mathe­ According to Sierpinska (2000), while problem solution requires matical skills and concepts they already know, contribute to confusion, conceptual thinking, students tend to overly rely on the use of proce­ difficulty, and error in solving functions problems. This stems from the dural knowledge. Conceptual thinking refers to consciously reflecting on lack of a foundation of Cartesian geometry, which can help students to the different representations of information. It specifically concerns intuitively solve linear functions problems (Dorier, 1998). Linear func­ systems of concepts (as opposed to collections of independent infor­ tions uses abstract, algebraic, and geometric language and students must mation and ideas), as well as engaging in reasoning by making con­ be able to understand each of these systems and move between them. nections within a system. Procedural knowledge is guided by practical According to Alves-Dias, understanding linear functions requires thinking which guides physical, goal-oriented actions that support the cognitive flexibility, which entails moving between tabular, algebraic, actions of conceptual thinking (Sierpinska, 2000). These different graphic, and semantic representations, as well as moving between ab­ methods of thinking are both important to understanding functions, but stract, algebraic, and geometric languages, sometimes all in one task the tendency for students to exclusively apply procedural knowledge is (Alves-Dias & Artigue, 1995; Dorier, 2000). Students must also intui­ what causes erroneous reasoning in functions problems (Sierpinska, tively understand each language, as taking these languages literally will 2000). Specifically, this tendency refers to difficulty in thinking contribute to the difficulty (Hillel, 2000). Additionally, during the analytically. That is, when students solve such problems, they do not knowledge acquisition process students are prone to effectively encode think beyond typical examples and representations of linear functions, misinterpreted conceptual and/or procedural knowledge because of and thus only apply procedural knowledge to the task at hand. This imprecision in the learning process (Ben-Zeev, 1998). This often results tendency could be attributed to the overwhelming amount of new in students applying erroneous rules to complete a problem without concepts and abstractions presented during the algebra learning process experiencing an impasse in the solution process. (Dorier & Sierpinska, 2001), which is an exemplification of why students The process of operating in one representation and providing an experience conceptual difficulties and how cognitive flexibility is answer in another representation has been noted in explaining a pertinent. disconnect between the understanding of equations and their applica­ tion in a graphical context (Leinhardt, Zaslavsky, & Stein, 1990; 1.2. Problem variation and problem solution Schoenfeld, Smith, & Arcavi, 1993). Examples of this difficulty include not being able to determine the y-intercept from a given graph, the In the discussion that follows we focus on the nature of the problem valence of coordinates from a given point on a graph, or the valence of solving process for three distinct types of LF problems known respec­ the slope of a given line on a graph (e.g., Kaput, 2018; Moschkovich, tively as description problems, graph problems, and table problems. Schoenfeld, & Arcavi, 2012). Additionally, according to Janvier (1987), Description problems are the ubiquitous and often studied word prob­ the processes of moving between the different representations are not of lems where students must produce a linear equation that corresponds to equal difficulty. For example, the process of moving from an equation to a specific verbal problem description. Graph problems require students a graph is not as difficult as moving from graphs to their corresponding to produce a linear equation, or component of same, that corresponds to equations. This is because moving from a graph involves pattern a specific illustrated graph. Table problems require students to produce detection, which requires conceptual reasoning, while moving from an a linear equation that corresponds to a tabled set of specific values for equation requires a series of steps that are comparatively more covarying elements. Further details and examples of all three problem straightforward and may require only procedural knowledge (Leinhardt types are presented after first considering the general nature of the et al., 1990). Several specific forms of knowledge associated with con­ problem solving actions needed for successfully addressing all three ceptual and procedural modes of thinking are important for solving LF types of LF problems. problems. The nature of such knowledge has been explicated in detail in Regardless of problem type, there are two principal types of actions the Common Core Standards for Mathematics (Common Core State needed to solve linear functions problems which can be classified as Standards Initiative, 2010) (2010). construction actions and interpretation actions. Construction actions The importance of mastering and translating between different rep­ comprise generation of something new, while interpretation actions resentations can be explained in light of several perspectives and bodies comprise making sense of a functional equation, graph, verbal descrip­ of research. Lesh’s theory on multiple representations states that stu­ tion, or table of covarying values. These actions are not mutually dents should be exposed to multiple embodiments of mathematical ideas exclusive, and can vary in their local or global nature, as well as whether and be able to comprehend, link, and transform between and within a they are quantitative or qualitative (Leinhardt et al., 1990). On a concept to create meaning out of them (Lesh, 1979). Based on this cognitive level, solving LF problems involves recognizing and following theory, Lesh’s translation model emphasizes the importance of the relevant identification and extraction rules based on the interpretation 2 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 of given information, then using the extracted parts in the process of theory suggests that one must be able to build a proper representation of formulating a problem model (Adu-Gyamfi & Bosse, 2014). the situation presented in the problem (interpretation) which requires To address a construction action, students produce something new the execution of cognitive processes that call upon relevant domain (Leinhardt et al., 1990; Simon, 1978). Construction actions involve knowledge (e.g., Kintsch & Greeno, 1985). If one lacks relevant con­ building an algebraic function (including calculating and determining ceptual knowledge, or if it is incorrect, the situation representation the elements of a function, such as slope and intercept), plotting points (interpretation) will demonstrate a lack of understanding that will then from a table, or building a graph. In construction actions, the generation carry forward into the problem solution (construction). Analyzing the process is challenging because students must generate an answer from a traces of solutions to linear functions problems can provide insight into part that is not given, as opposed to interpretation in which they are students’ competency levels in a domain, specifically, the understanding asked to react to given information. For example, it can be difficult to they have and their ability to map that to a representation of the prob­ construct an equation from a given graph because it is unclear what they lem. It is also important to understand how this knowledge gets called need to implicitly know to generate it (Markovits, Eylon, & Bruck­ upon in executing each action. This can be explored by looking into the heimer, 1986; Stein & Leinhardt, 1989; Yerushalmy, 1991). cognitive underpinnings of the solution process which is conveyed by To address an interpretation action, students must first understand the students’ schematic knowledge for representation of the given what the given information represents (Leinhardt et al., 1990). This problem. Being able to correctly recognize and classify problem types information, which can be in the form of a description, graph, table, or allows for information given in the problem to be represented correctly equation, can represent a functional relationship or a situation in which and efficiently which then allows for the solution procedure to be car­ a student must address a question that asks them to make meaning out of ried out correctly. the given information. For example, interpretation actions involve interpolation (continuing a graph or table), discerning a pattern, or 1.3.1. Description problems understanding the valence of slope in tabular and graphical form. In Items in which students are asked to provide an equation that rep­ interpretation actions, a common struggle is a bias toward pointwise resents a verbal description of a linear function, often called algebra interpretations. For example, students tend to think of tables and graphs story or word problems, are the most frequently studied form of LF as just a compilation of individual points, instead of understanding that problems in the educational and cognitive psychology literature (e.g., those points make up a conceptual entity expressing a functional rela­ Christou, Reid, & Vosniadou, 2019; Hinsley, Hayes, & Simon, 1977; tionship between and among the individual values (Kaput, 2018; Koedinger, Alibali, & Nathan, 2008; Koedinger & Nathan, 2004; Mayer, Moschkovich et al., 2012; Oehrtman, Carlson, & Thompson, 2008). 1981). Such problems require the construction of a cognitive represen­ Linear functions questions can involve either or both interpretation tation of the information stated verbally. Kintsch and Greeno (1985) and construction actions. However, it is common for construction ac­ presented a processing model that explicated the procedures one en­ tions to build on interpretation, but not all interpretation actions gages in during this process. The model comprises two main compo­ necessitate construction (Leinhardt et al., 1990). Fig. 1 shows an nents, (1) a propositional textbase, which is a propositional structure example of a functions problem that involves both types of actions. We and situation model of the information presented, and (2) a problem can learn more about students’ difficulty with solving these problems by model, which represents the calculation strategies required to solve the discerning the type of action for which mistakes most often occur. problem. These components require the knowledge of propositional framing (translating sentences to propositions), properties and relations of sets, and arithmetic operations. Essentially, for every relevant text­ 1.3. Cognitive basis of task processing and problem solution base in the problem, a propositional frame is created which reflects perception of the components of the textbase from which an individual To successfully solve a linear functions problem one must have begins to form parts of a problem model. This is a task-specific structure specific forms of relevant linear functions knowledge and understanding that represents the conceptual understanding of given information which are in turn required to execute key aspects of a cognitive problem- (Kintsch & Greeno, 1985). solving process for specific types of LF problems. For example, cognitive To solve the problem, conceptual and procedural knowledge is required to form the cognitive representation accurately and correctly produce an expression that represents the information to carry out necessary interpretation and construction actions. Responses to these problems that do not demonstrate the correct execution of these actions can be further analyzed to identify discrepancies in the inductive pro­ cess. Errors in either or both of these procedures suggest a misunder­ standing in the respective knowledge base. On the other hand, errors in description problems could suggest miscomprehension of the text as opposed to a broad conceptual and/or procedural misunderstanding of the domain (Cummins, Kintsch, Reusser & Weimer, 1988). Examining error patterns across these types of problems can indicate whether errors are caused by systematic discrepancies or other extenuating conditions. 1.3.2. Graph problems Perceptual and cognitive models are constructed while solving problems that present information in the form of a graph. Specifically, interpretive processes are executed to discern meaning from the graph, and integrative processes use semantic cues to guide information extraction from the graph (Carpenter & Shah, 1998). These processes occur in tandem as student’s typically engage in the following steps, according to Lohse (1991). The initial step in graph comprehension is encoding spatial features, such as general pattern recognition, then Fig. 1. Example solution of functions problem with stepwise indications of the creating a structural description representing the graph in short term type of corresponding action. memory. This prompts the graph schema, which allows one to access 3 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 conceptual and procedural knowledge needed to identify and extract shows evidence of learning benefits when students are presented with relevant information to ultimately interpret a graph (Lohse, 1991). Once both correct and incorrect information (Groβe & Renkl, 2007; Schnotz, information is extracted, relevant conceptual and procedural knowledge 2001; Schnotz, Vosniadou, & Carretero, 1999) in such a way that it of functions operations are used to complete the problem. triggers impasse-driven episodes (VanLehn, 1999). It is imperative that students have robust conceptualizations and According to VanLehn (1999), learning can be promoted through adaptive procedural knowledge on how to solve graph problems beyond experiencing impasses, which prompt positive learning episodes. These prototypical examples. A review of the literature on graph comprehen­ impasses are characterized by noticing and reflecting on errors and are sion by Shah and Hoeffner (2002) cites three primary processes required critical to accurate knowledge acquisition. Error detection and error for graph comprehension: processing perceptual features of the graph, correction tasks could trigger what VanLehn (1999) refers to as usage of knowledge about graphs, and the processing the content of the impasse-driven episodes to facilitate learning and performance on sub­ data. Specifically, students’ schemas of graphs comprising their prior sequent problems. Learning from examples with indicated errors and/or knowledge about slope and how it maps to the features of a graph affects error detection and correction can ameliorate the difficulty that students how they encode a given graph and the corresponding problem situa­ experience with typical LF problems because it guides them to appro­ tion. In many graph problems, similar to those used in this study, not all priately implement procedural steps to solve the problems while also relevant information is explicitly presented, requiring students to apply producing a deeper understanding of why these procedures work (Gott a deeper conceptual understanding in order to extract relevant infor­ et al., 1993), which is crucial to be able to flexibly apply pertinent as­ mation. Especially for a domain in which the obstacle of formalism is pects of learned procedures while solving new problems (Catrambone, pervasive, a robust foundational understanding of the topic, beyond rote 1996, 1998; Gott, Parker, Pokorny, Dibble, & Glaser, 1993). Thus, it memorization of solution procedures and general formulaic under­ helps students understand the problem features more deeply and allows standing, is imperative (Schoenfeld, 1985; 1988; 1992). For example, as them to replace incorrect knowledge or build on deficient knowledge seen in Fig. 1 shown earlier, a line could be presented on a graph with no (Booth, Lange, Koedinger, & Newton, 2013). Further discussion of the explicitly given coordinates of collinear points which are needed for a implementation of conditions for use of worked examples with LF comprehensive understanding of the graph to extract other information problems will be presented following the results of Study 1. from the representation. 1.5. Errors in the context of conceptual change 1.3.3. Table problems Problems that display discrete information in tables contain data The underlying causes and potential solution to remediating errors in points which require the extraction of relevant information and the LF problem solving can be explored in light of conceptual change theory. construction of a representation of the relationship between two vari­ Conceptual change is a knowledge building process that underlies ables (Vessey, 1991). Contrary to graphs, tables do not contain sub­ meaningful learning through revising prior knowledge. This process stantial spatial information or require symbolic or semantic mapping of occurs when learners are faced with a situation in which they must information (Lohse, 1993; Vessey, 1991). Inductive reasoning is the reconcile their prior knowledge with competing information that is new process of evaluating specific information to formulate a rule that rep­ and/or at odds with what they already know (Chi, 2000; Mayer 1981; resents given premises (Johnson-Laird & Byrne, 1993; Klauer, 2001). Vamvakoussi et al., 2007; Vosniadou, 2007), resulting in internally This analytic process is required to extract and use information from inconsistent mental models. To resolve the conflict, a learner must tables in order to detect patterns and make inferences about given in­ augment the existing knowledge structure by assimilating and reor­ formation (Vessey, 1991; Neubert & Binko, 1992). In the context of LF ganizing the new information with their existing knowledge (Vosniadou, table problems, the solution process requires specific inductive 2013). Based on conceptual change theory, this can be addressed reasoning processes: pattern induction and pattern extrapolation which through exposure to instruction such as the use of worked examples with comprise analytic and systematic evaluation of given values then error detection/correction. Such instructional interventions can help inferring the pattern of relations of the values relative to each other, elucidate inconsistencies between the correct conceptualizations of the respectively. These are the processes required in the interpretation and topic and a learner’s flawed understanding (Vosniadou, 2013). Study 2 construction actions that make up the solution process to LF table explores instructional designs using worked examples that may achieve problems. Responses to these problems that do not demonstrate the this drawing upon the error analyses that are at the core of Study 1. correct execution of these actions can be further analyzed to identify specific discrepancies in the inductive process. Analyses of student so­ 1.6. Significance of current research lution processes for table problems are relatively rare in the literature in comparison to the cumulative body of work on the solution of descrip­ As noted earlier, the existing research on students’ ability to solve LF tion or graph problems. problems studies each problem type in isolation from one another. The current work addresses the question of whether a student who is able or 1.4. Improving LF performance unable to solve one type of problem should or should not be able to solve the other types as well and the reasons for consistency or variability in The aforementioned research suggests that students could benefit performance across problem types. Existing data regarding performance from instructional interventions that attend to the specific deficiencies on the different LF problem types as discussed above is not representa­ that students have in their knowledge of LF with an emphasis on their tive of a student’s overall understanding of LF and only indicates a comparative ability to move between problem types that make partic­ student’s ability to carry out the problem solving steps relevant only to a ular demands on aspects of LF knowledge and processing. One specific given task type. While simple description, graph and table problem types form of intervention that has proven successful in other areas of math­ are all in fact “equivalent” in the LF information they represent, the ematics problem solving involves learning from worked examples with literature suggests that the prior knowledge and reasoning skills and without error detection and correction. There is considerable required to solve each of them do indeed differ. Study 1 aims to evaluate research showing that learning from worked examples for various students’ ability to engage in solving multiple types of LF problems mathematics problem types promotes deep understanding during the instead of just one type with the goal of revealing patterns in perfor­ learning process because it maximizes germane cognitive load (Groβe & mance associated with student (mis)understandings of LF that may Renkl, 2007). In particular, students can allot more cognitive resources contribute to variable and low performance in this domain of mathe­ to the learning of important content and less to performance demands matics problem solving and that could be ameliorated through an (Renkl, Gruber, Weber, Lerche, & Schweizer, 2003). Research also intervention such as the use of worked examples. 4 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 2. Study 1 representing the graphed data. In Appendix C, the last two items were not included in the Mielicki et al. (2019) study. These tasks were chosen The goals of this study were to: (1) evaluate college students per­ to be similar in requirements to the other graph related tasks and were formance on LF problems that systematically vary in how information is drawn from NAEP and a practice test for the College Board’s SAT exam. presented, (2) observe and identify specific errors that students tend to All the tasks in Appendices A, B, and Crequire interpretation and con­ make for each type of problem so that they can be mapped to the hy­ struction actions as specified in the Common Core Math Standards pothesized knowledge and cognitive processes needed to solve each (Common Core State Standards Initiative, 2010). problem type, and (3) determine specific errors that are most prevalent All tasks were presented in the open response format shown in and that could be addressed in an instructional intervention using Appendices A, B, and C allowing students to explicitly show how they worked examples. solved these problems. Similar to what was presented earlier in Fig. 1, a Data were collected from 187 total subjects who were enrolled in an majority of the steps in solving these problems require interpretation introductory psychology class at an R1 public university in the U.S. with and construction actions. For each item type, mistakes in responses were a diverse student population comprising 36% Latina/o/x/e, 23% White, categorized based on the action(s) that corresponds with each step in 21% Asian, 8% Black, 8% International, 3% Multi-Race, 2% Other. The solving the problems within each general class. This was determined by sample contained 61.5% (n = 115) females and 38.5% (n = 72) males, whether the mistake indicated a lack of proper understanding of one or and the average age was 19.2 (SD = 0.87). Because sample size cannot more of the relevant math standards. be definitively determined a priori for qualitative studies (Sim, Saun­ ders, Waterfield, & Kingstone, 2018), it was determined by other similar 2.1.2. Design and procedure studies that model student responses in math learning (Baker, Cooley, Ethical approval was obtained from the university’s Internal Review Trigueros, & Trigueros, 2000; Koedinger & Nathan, 2004). Board (IRB protocol number 2020-0186). Prior to participating in the Students were presented a series of linear functions problems that experiment, subjects signed an informed consent. required them to demonstrate the actions they take in generating a so­ Participants were presented with the set of 21 math problems using lution. Those actions presumably call upon students’ understanding and an online survey platform, Qualtrics, in an open response format. All application of relevant aspects of the mathematics knowledge described items were divided into 3 blocks that were counterbalanced such that in reference documents such as math standards (CCSI, 2010). Responses there were 3 unique block orders across subjects. Additionally, each were analyzed for overall accuracy, and inaccurate responses were block included items representing each of the 3 major item types in further analyzed for mistakes of construction and/or interpretation as random order. Students were asked to solve each of the problems and to well as general misunderstandings. Identifying sources of difficulty in explicitly show how they arrived at their answer. Answers were coded solving the varying problem types can inform subsequent research for accuracy, and when an answer was inaccurate, the written work was focused on interventions such as worked examples with and without subsequently analyzed for mistakes in a construction action and/or in an error detection and error correction to remedy common mistakes that interpretation action, or general misunderstanding. students make while solving different types of LF problems. 2.1.3. Scoring and analysis 2.1. Methods Each item was scored for overall correctness using a dichotomous 0–1 scoring procedure. For those cases where errors occurred, the nature 2.1.1. Materials of the error was categorized using a qualitative classification system that The items used in this study were derived from the item pool used by is described in the section that follows. The distribution of error types Mielicki et al. (2019). The latter materials were drawn from multiple was evaluated for each item and cluster of similar items. The open sources, including practice tests for the ACT exam and the College response nature of these tasks can yield ambiguous responses that are Board’s SAT exam, and items publicly provided from the National difficult to classify. For cases in which errors occurred but the type of Assessment of Educational Progress (NAEP), the Third International error could not be clearly identified, responses were coded as “unclear”. Mathematics and Science Study (TIMSS), and the Partnership for Readiness for College and Careers (PARCC). The Mielicki et al. (2019) 2.2. Results and discussion study included 136 items that varied in terms of content and construct representation relative to Common Core mathematics standards for In the sections that follow, various analyses of performance are middle-school and high-school levels, 44 of which were LF questions. presented and the main results of each analysis are briefly discussed. The The items in Mielicki et al. (2019) were presented in their original presentation begins with analyses of accuracy patterns on the various five-alternative multiple-choice format using a computer-based assess­ item types and then turns to analysis of specific types of errors for each ment administration platform. In the present study, items were chosen problem type. The error classification analyses are followed by analyses from the set of LF items used by Mielicki et al. (2019) based on their of the consistency of error types within and across the LF problem types. mapping to Common Core math standards for algebra and were pre­ sented in an open response format. 2.2.1. Analyses of item performance Appendices A, B, and C display the total set of 21 items included in Scatterplots and frequency distributions of Subject performance on the item pool. The items shown vary in the information presented and each of the three classes of items showed that some subjects had uni­ the nature of the problem students are asked to solve. Information is formly low performance on all item types. A subsequent examination of provided in either a graphical, tabular, or descriptive format, and the their actual responses indicated that these 36 subjects were not items require students to engage in interpretation and construction attempting the tasks in any serious manner, including the easier processes that ostensibly call upon understanding of relevant linear Description items. What constituted an unserious attempt was one that functions math knowledge. The items in Appendix A present verbal contained unintelligible or generally irrelevant responses that did not descriptive information and ask students to produce an expression rep­ give any indication that the subject acknowledged the nature of the task resenting the situation verbally described. The items in Appendix B including the requirement to provide an equation representing the present information in a tabular format and ask students to produce an problem situation. Thus, their data were eliminated from the dataset expression that represents the relationship underlying the tabled values. submitted to further quantitative and qualitative analysis. The excluded The items in Appendix C present information in a graphical format and sample did not differ from the final analytic sample on the basis of age or ask students to produce an expression representing the graphed data or gender. A t-test showed no difference between the age of the final ana­ calculate and provide the slope or intercept of the expression lytic sample (151) and the excluded sample (36), t (185) = 0.03965, p = 5 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 0.968. A chi-squared test of independence showed no differences in Table 2 gender between the two groups, X2 (1) = 0.71, p = 0.398. The final Correlations between performance on the three item types. analytic sample contained 60.3% (n = 91) females and 39.7% (n = 60) Description Graph Table males, and the average age was 19.4 (SD = 1.95). Description 1 Overall accuracy scores were obtained for each subject on each Graph 0.49* 1 problem type. Table 1 shows the means and variances of subject per­ Table 0.44* 0.60* 1 formance on each of the three item types. Correlations were then Note: *p < 0.001. computed between each of the three sets of items using overall subject performance on each item type. The correlations were modest and in onto an equation in proper form the error is not one of construction but expected directions given differences in the means and variances of interpretation. This classification of what constitutes a construction student performance for each item type (Table 1). As shown in Table 2, error is consistent across all three item types given that they all ulti­ the highest correlation was between performance on Table and Graph mately require students to produce a simple linear equation or some items, r (149) = 0.60, p < 0.001). transform of one. To do this they must produce a “syntactically” correct Fig. 2 displays the bivariate distribution of individual students’ linear equation, and use it to solve for the relevant unknown which re­ performance across the two most difficult problem types as indicated quires procedural knowledge for execution. above. Individuals whose performance lies along the diagonal per­ Below are actual student responses which serve as examples of formed equally well or equally poorly on both item types. If performance erroneous solutions to description, table, and graph problems using the is below and away from the diagonal they performed better on table general interpretation and construction coding scheme and which have items than graph items. If performance is above and away from the di­ been further detailed into subcategories of each error type to precisely agonal they performed better at graph items than table items. A majority identify the specific knowledge discrepancy in interpretation and con­ of individuals fall into a cluster just below the line and around the center struction actions. The subcategories will be described for each corre­ of the x-axis, which is consistent with results indicating that average sponding item type in the sections that follow. performance on graph items was lower than performance on table items. Individuals whose performance lies toward the left extremity of the di­ agonal generally performed somewhat better on the graph items than 2.2.2.1. Description problem errors. As shown in Table 1, description the table items. In contrast, individuals whose performances lies toward items showed higher accuracy than the graph and table items, resulting the right extremity of the diagonal generally performed better on the in comparatively fewer erroneous responses to classify and analyze. The table items than the graph items. A least-squares regression line was fit majority of these were interpretation errors such that there is a to the data and is plotted in Fig. 2. The regression line is represented by misrepresentation of a relationship stated verbally in constructing the the equation, y = 0.46x + 1.60. As noted above, at the lower end of the symbolic expression using the canonical y = mx + b form for a simple performance scale students performed better on Graph than Table items, linear equation. Some erroneous solutions to description problems are thus the non-zero intercept, but as performance on Table items improves shown below. performance on Graph items improves at roughly half the rate. The table The Student Response to Description Item 1 (Fig. 3) represents one of for this regression line is presented in Appendix E. the few common errors in description problems. The work suggests that their propositional framing of the given information “twice as many 2.2.2. Analysis of errors adults’’ prompts multiplication knowledge leading to an interpretation A coding system was used to classify the type(s) of errors observed of this as the variable multiplied by 2. It also suggests that their prop­ for each problem type. Errors can be classified as interpretation errors, ositional framing of the given information “4 more than” also prompts construction errors, both, neither, or unclear. In the process of classi­ multiplication knowledge leading to an interpretation of this as 2c fying erroneous responses, student work was analyzed to identify where multiplied by 4. It suggests that there is no propositional distinction and why the error occurred in the problem-solving process. When between “twice a value” and “a value more than”, and that both text­ classifying an erroneous response as an interpretation error, the bases prompt the same knowledge of properties and relations of sets. The response must depict a lapse in the student’s grasp of the conceptual discrepancy in the framing yielded an erroneous interpretation of the underpinnings for interacting with the given information and/or a verbal description which carried over to the constructed equation. discrepancy in their cognitive representation of the given information. Essentially, this response demonstrated a discrepancy in their concep­ This is manifest in different ways based on item type because of the tual understanding of the description of the different values and the different conceptual and procedural knowledge required for solving relation of those values to the critical elements of the linear equation. each item type. When classifying an erroneous response as a construc­ Similarly, the Student Response to Description Item 2 (Fig. 4) shows tion error, the response must depict a lapse in the conceptual and/or that their interpretation of the given information “$48 for each hour procedural knowledge necessary to construct a simple linear equation worked” was interpreted as “where the plumber starts to charge” in using the canonical y = mx + b form. Construction errors focus on a which they represented $48 as the additional cost; and their interpre­ students’ failure to apply relevant knowledge in using the information tation of the given information “plus an additional $9 for travel” as “the extracted from the problem to produce a final solution expressed as a $9 can change depending on the location of the individual” in which linear function. Thus, if a subject incorrectly represents one of the key they represented $9 as the value that is contingent on the number of elements in the problem situation, such as the slope or intercept, in their hours worked. It suggests a discrepancy in the framing of the proposition interpretation of the problem but otherwise maps those interpretations “$48 for each hour worked” and the proposition “plus an additional $9 for travel” in their cognitive representation of the problem and the mapping then to the components of the symbolic expression. Addition­ Table 1 ally, these responses would not be considered construction errors Subject-level performance data for each item type. because the students attempt to represent the relationship between the Item Type Mean SD given values as a linear equation in y = mx + b form, showing that they are familiar with the essential elements of a simple linear equation and Description 5.46 1.34 Graph 3.08 2.40 how to represent the given or extracted information. Table 3.85 2.66 Because description items showed higher accuracy than the graph and table items, resulting in comparatively fewer erroneous responses to Note. Indicated are the mean and standard deviation of items solved correctly out of 7 total items. classify and analyze, they were excluded from further error analysis. 6 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 Fig. 2. Scatterplot Depicting Distribution of Individual Students’ Performance Across Graph and Table Problems. Note. Performance is represented by the number of problems solved correctly out of 7 total problems in each item set. Fig. 3. Example student response to a description item. Fig. 4. Example student response to a description item. Instead, the bulk of the error analysis effort was devoted to the Table and corresponding x and y values. These solutions suggest erroneous in­ Graph items as described below. terpretations of the data pattern in the given table and a failure to demonstrate the necessary inductive reasoning process (pattern induc­ 2.2.2.2. Table problem coding scheme. The following are examples of some tion and extrapolation), which is germane to solving table problems. common erroneous solutions to a table problem and their coding using the These responses can also be considered construction errors because they aforementioned classification scheme. Student Response 1 (Fig. 5) is do not show evidence of attempting to represent the relationship be­ representative of responses in which students appear to evaluate the tween the x and y values as a linear equation in y = mx + b form. More changes in the x values and the y values separately and this is but one generally, they show a lack of knowledge of the relationship between the example of this pattern of problem solution. Student Response 2 (Fig. 5) pattern in the table and a mapping of the pattern to the canonical form of is representative of responses in which students provide a set of separate a linear function. equations that only focus on describing the relationship between 7 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 Fig. 5. Example student responses to a table item. 2.2.2.3. Graph problem coding scheme. Some erroneous solutions to knowledge of graphs to use the given information in the graph to Graph problems are shown below. These solutions suggest the erroneous ascertain collinear points in order to solve the problem. Thus, both re­ interpretation of the specific graph and weakness in understanding key sponses suggest erroneous interpretations of the graph shown due to aspects of graphs and how they map to the elements of a linear equation. knowledge issues associated with interpretation and representation of Students appear to be relying on procedural knowledge or prototypical graphs. knowledge of graphs. These responses are representative of different Additionally, the work in Student Response 1 demonstrated knowl­ solution variations for common errors. edge of the formation of a linear equation, showing that they are familiar The Student Response to Graph Item 1 (Fig. 6) represents common with all of the aspects of the equation and where to plug in the given or demonstrations of a discrepancy in subjects’ conceptual knowledge of extracted (albeit incorrect) information, so this response would not be graphs and linear functions (aside from the incomplete solution of this considered a construction error. However, this is also a demonstration of particular response). This response shows a discrepancy in the under­ how relying on procedural knowledge of linear functions is not sufficient standing of how the slope of a line is calculated and, more holistically, for correctly solving these types of problems. Because the work in Stu­ how the slope describes the properties of the line, suggesting an inter­ dent Response 2 does not demonstrate familiarity with the correct pretation error. Here they used the given information to apply their procedural knowledge required to solve the problem, it would also be broad knowledge of the slope equation – identifying the corresponding x considered a type of construction error. and y intercepts as the “rise” and the “run” - to find the slope of the line, There are many variations of these responses which ultimately as well as not considering the valence of the line and how that would demonstrate interpretation aligned with elements of the appropriate contribute to the slope. This would also be considered a type of con­ knowledge and cognitive processes for particular types of items. The struction error (aside from the fact that they did not complete the so­ responses shown are representations of many common solutions that lution) because they identified slope as “rise over run” suggesting that suggest erroneous interpretations rooted in weak or flawed knowledge they possess procedural knowledge to solve for slope, but did so incor­ of graphs and tables as representations of linear functions and their re­ rectly. This is a demonstration of how relying on procedural knowledge lationships to the process of generating a proper linear equation and thus of linear functions is not sufficient for correctly solving these types of adequate solution of linear functions problems of this general class. This problems. is in contrast to erroneous constructions of responses to these items. In a Graph Item 2 Student Response 1 (Fig. 7) suggests that the student is few cases, erroneous responses to these problems were unclear in estimating the point, based on the given information, at which the line conveying their interpretation of the given information and how the crosses the y-axis which defines the y-intercept. It suggests that their student arrived at their final answer. These responses were coded as understanding of graphs is not robust enough to discern the inability to “unclear” along the interpretation dimension, but respectively coded visually identify the y-intercept in this graph similar to the way that it is along the construction dimension. typically presented. Similarly, Graph Item 2 Student Response 2 repre­ Based on the error coding described above, Table 3 provides a sents common errors in using noncollinear points to solve graph prob­ summary of all the error subtypes identified for Table and Graph items. lems. This is common in problems in which collinear points are not explicitly presented and furthermore require the use of conceptual Fig. 6. Example student response to a graph item. 8 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 Fig. 7. Example student responses to a graph item. resolved all discrepancies, which revealed coding errors that were pre­ Table 3 dominantly non-systematic misclassifications by the raters, but ulti­ Error Types and Subtypes Across Table and Graph items. mately did not result in any changes to the established scoring scheme. Error Type Table Items Graph Items The raters then coded the remaining responses of the subset which Interpretation Describes change in the x- Visually estimates and/or resulted in near perfect agreement between both scorers for all items (M Errors values and/or y-values provides response using the = 0.98, SD = 0.02). The few resulting discrepancies in the remaining separately, either by providing given information at face- subset were resolved, and again, did not require any changes to the an expression of the change in value (G.I.E1). only x-values, only y-values, or scoring scheme. These results suggest that the scoring scheme for error an amalgamation of both (T.I. categorization is a reliable classification tool for the types of errors E1). observed in LF problems presented as tables and graphs and can be used Describes change in each pair Identifies a given non- to infer specific conceptual lapses in students’ LF knowledge. of corresponding x-values and collinear point from the graph y-values (T.I.E2). as relevant information to solve problem (G.I.E2). 2.2.4. Prevalence of sub-errors by item type Response is incorrect and does Response is incorrect and does Analyzing the frequencies of the subtypes of each error allows for a not clearly show how they not clearly show how they more precise examination of the specific mis-understandings students arrived at the final answer arrived at the final answer have in these actions. To discern the distribution and prevalence of er­ (source of error cannot be (source of error cannot be identified) (T.Unclear). identified) (G.Unclear). rors across both items and subjects, the frequencies of the error subtypes Construction Demonstrates familiarity with Demonstrates familiarity with in each item type group were tabulated. Responses were coded as correct Errors the formation of a linear the formation of a linear when there was no error and responses with errors were classified based equation, but incorrectly equation, but incorrectly on the specific error observed in the interpretation action and con­ carries out the steps to carries out the steps to struction action. Table 3 lays out the descriptions of the specific errors in construct the final solution (T. construct the final solution (G. C.E1). C.E1). both actions for graph and table problems. Two types of interpretation Response does not show Response does not show errors in responses to graph problems were (1) visually estimating and/ evidence of attempting to evidence of attempting to or using the given information at face-value (G.I.E1) and (2) identifying represent the relationship represent the relationship a given non-collinear point from the graph as relevant information to between the x and y values in between the x and y values in the canonical form of a linear the canonical form of a linear solve problem (G.I.E2). When the response is incorrect but does not function (y = m x + b) (T.C. function (y = m x + b) (G.C. clearly show how they arrived at the final answer and the source of the E2). E2). error cannot be identified, it was coded as unclear (G.Unclear). Two Note. The denotation in each category coincides with the code representation of types of construction errors in graph problems were (1) demonstrating the respective error subtype for further reference. familiarity with the formation of a linear equation, but incorrectly car­ rying out the steps to construct the final solution (G.C.E1) and (2) not showing evidence of attempting to represent the relationship between 2.2.3. Scoring reliability the x and y values in the canonical form of a linear function (y = m x + b) To evaluate the reliability of the scoring scheme to support the claim (G.C.E2). that these classifications represent common errors among these re­ It is possible for responses to demonstrate different combinations of sponses, two raters scored the responses for 30% of the respondents (n = the types of interpretation and construction errors. It is also possible for 50) with the goal of high agreement between both raters on all di­ responses to demonstrate an error in one action but not in another (e.g., mensions of error categorization. Initially, the responses of three re­ an interpretation error but not a construction error; a construction error spondents were discussed among the raters as a training session. Then, but not an interpretation error). The error combinations comprise the both raters independently coded 10% of the subset and Cohen’s Kappa specific subtype of interpretation and/or construction errors (and/or no values1 were derived for each item which resulted in high reliability (M errors) within each item type. = 0.89, SD = 0.05) (Viera & Garrett, 2005). Raters then discussed and Table 4 displays the frequency of each sub-error combination that was observed in the 982 total responses to the graph problems that were 1 scored. The overall accuracy was 44% with 75 missing responses coded The reported mean and standard deviation of kappa values are based on the kappa values derived for each item. This represents the average of the agree­ as NA As shown in Table 4, most responses showed no error in the ment across items. construction actions in (N = 746–76%). In contrast, almost half of all 9 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 Table 4 Frequency counts of error types on graph problems. Graph Interpretation Error 1 Graph Interpretation Error 2 Graph No Interpretation Error Totals G.I.E1 G.I.E2 Error Unclear G.Unclear Graph Construction Error 1 55 39 0 37 131 G.C.E1 Graph Construction Error 2 35 40 22 8 105 G.C.E2 No Construction Error 202* 88 0 456 746 Totals 292 167 22 501 982 Note. The asterisk (*) denotes the single most frequent error pattern. 982 responses contained errors in the interpretation actions (N = demonstrated the T.I.E1 type interpretation error (225/351–64.1%). If 459–46.7%), and almost two thirds of those interpretation errors an interpretation error occurred (351/978–35.9%), it was almost always demonstrated the G.I.E1 type interpretation error (63.6%–292/459). associated with a construction error (345 of 351 cases – 98.3%). Few Overall, the single most prevalent error pattern in graph problems was construction errors occurred without an associated interpretation error an interpretation error and no construction error with this pattern ac­ (26 of 392–6.6%). counting for 55.1% of all incorrect responses – 290 of 526. Specifically, As noted above, the most prevalent error combination in table when solving graph problems students tend to make errors in the problems was both an interpretation error and construction error. Spe­ interpretation action in which they visually estimate relevant elements cifically, when solving table problems students tend to make errors in of the linear function from the given information and/or use the given the interpretation action in which they describe the change in the x- information at face-value to solve the problem of generating elements of values and/or y-values separately, as well as errors in the construction the equation, and they then correctly carry out the construction actions action in which they don’t attempt to represent the relationship between based on the flawed interpretation. An example of this error is repre­ the x and y values in the canonical form. An example of this combination sented in Student Response 1 in Fig. 7. Interpretation and construction of errors is represented in the Student Responses shown in Fig. 5. errors co-occurred in 32.1% of all incorrect responses 169 of 526. Across all table and graph problems only a few responses were Construction errors in the absence of an interpretation error were very considered unclear (Graph n = 22; Table n = 21) (Tables 4 and 5). Re­ infrequent representing 8.6% of all incorrect responses – 45 of 526. sponses that were considered unclear along the interpretation dimen­ Table 3 also lays out the descriptions of the specific errors in inter­ sion aligned with the G/T.C.E2 classification, in which there was no pretation and construction actions for table problems. Two types of evidence of attempting to represent the given information in the ca­ interpretation errors in responses to table problems were (1) describing nonical form of a linear function. This means that responses in which the change in the x-values and/or y-values separately, either by providing interpretation of the given information was unclear also did not show an expression of the change in only x-values, only y-values, or an familiarity with the elements of a LF equation. Missing responses were amalgamation of both (T.I.E1), and (2) describing change in each pair of also tabulated for each item (Tables 4 and 5) to determine if an item was corresponding x-values and y-values (T.I.E2). When the response is more likely to be skipped. Across all responses to all table and graph incorrect but does not clearly show how they arrived at the final answer problems not many were left blank (Graph n = 79; Table n = 75). and the source of the error cannot be identified, it was coded as unclear (T.Unclear). 2.2.5. Sub-error consistency across subjects and items Two types of construction errors in table problems were (1) The sub-error patterns were analyzed for consistency to determine demonstrating familiarity with the formation of a linear equation, but two things: (1) whether the same types of interpretation and construc­ incorrectly carrying out the steps to construct the final solution (T.C.E1), tion errors are exhibited across students for the items within a given and (2) not showing evidence of attempting to represent the relationship problem type, and (2) whether students are internally consistent in the between the x and y values in the canonical form of a linear function (y types of interpretation and construction errors they make across items = m x + b) (T.C.E2). It is also possible for responses to demonstrate within each given problem type. different combinations of the types of interpretation and construction errors. 2.2.5.1. Consistency across items. Intraclass correlation coefficients Table 5 displays the frequency of each sub-error combination that (ICC) were used as a reliability measure of sub-error pattern consistency was observed in the 978 total responses to the table problems that were across items within each item type. This measure conveys the extent to scored. The overall accuracy was 55% with 79 missing responses coded which students’ responses to these items exhibited the same error pat­ as NA. As shown in Table 5, there were 398 total errors and almost all of terns across each item type. Two ICC tests were conducted, a test of those contained errors in the construction actions (N = 392–98.4%), and consistency and a test of absolute agreement. The test of consistency most of those errors demonstrated the T.C.E2 type construction error determines whether the error patterns are consistent across items, and (304/392–77.6%). Almost all of the total errors also contained errors in the test of absolute agreement determines whether the variation in the the interpretation actions (351–88.2%), and most of those errors individual error subtypes are the same across items. It is useful to obtain Table 5 Frequency counts of error types on table problems. Table Interpretation Error 1 (T.I.E1) Table Interpretation Error 2 (T.I.E2) Table No Interpretation Error Totals Error Unclear T.Unclear Table Construction Error 1 (T.C.E1) 36 31 0 21 88 Table Construction Error 2 (T.C.E2) 183* 95 21 5 304 No Construction Error 6 0 0 580 586 Totals 225 126 21 606 978 Note. The asterisk (*) denotes the single most frequent error pattern. 10 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 both measurements because, although items could be highly correlated Table 7 in their observed error patterns, it is important that the error subtypes ICC values and confidence intervals for the test of consistency and the test of themselves exhibit the same patterns across items. This ensures that all absolute agreement for graph items. items are designed in such a way that students will solve them in a ICC 95% CI similar manner, verifying that the item set is a reliable tool to specif­ Consistency 0.85 0.62–0.96 ically explore students’ solution processes, and ultimately, their un­ Agreement 0.84 0.59–0.96 derstanding of the LF domain as manifest by that item type. To do this, the counts of the error subtypes for each item were converted to per­ centages of the sum of the total errors and entered into the analysis. All Overall, these error analyses suggest that students are consistent in were included except for those in the “unclear” category their item-level performance within and across each LF problem type, ICC values indicate the level of reliability observed: a value of 0.5 including the specific mistake they make in the interpretation and suggests poor reliability, a value between 0.5 and 0.75 suggests mod­ construction actions of the solution process. In other words, when stu­ erate reliability, a value between 0.75 and 0.9 suggests good reliability, dents make a mistake on one item of a given type, they are likely to make and values greater than 0.90 suggest excellent reliability (Koo & Li, mistakes on many other items of that same item type, and the mistakes 2016). The consistency and absolute agreement tests both resulted in they do make are consistent across most or all items in the subset. moderate to excellent ICC values for the table items (Table 6) and graph items (Table 7). This means that the error patterns varied consistently 2.2.7. Discussion across items, and that the individual subtype patterns did not vary much Quantitative and qualitative analyses of student responses to three from item to item. Thus, all items are consistent in their ability to un­ different types of LF items reveal college students’ ability to solve such cover how students carry out interpretation and construction actions in problems and, more specifically, the types of errors they make and the these types of LF problems and that they are all calling upon the same extent of their struggles with materials representing this important knowledge. subdomain of mathematics. Overall results showed that students per­ formed better on Description items compared to Table and Graph items, 2.2.5.2. Consistency within subjects. It is useful to analyze the error consistent with prior data (MielickiMartinezDiBello et al., 2019). Rela­ consistency on the subject level because it will ensure that individual tively high accuracy on Description items suggests that, generally, stu­ students are consistently calling upon the same knowledge to solve each dents possess the knowledge and ability to carry out interpretation and of these types of problems. This will indicate whether the knowledge construction actions necessary to solve problems in which the infor­ that is called upon is a reflection of the students’ firm understanding, or mation is verbally presented and a simple linear equation is the desired misunderstanding, of the domain as manifest within that problem type. product. This is generally consistent with prior research on students’ To test whether subjects are internally consistent in the types of errors solution of simple algebra “story problems” (e.g., Cook, 2006; Koedinger they make, a subset of subjects were evaluated on the basis of the con­ & Nathan, 2004). An instructional intervention that focuses on sistency of their error patterns across items. A binomial probability improving the knowledge and relevant cognitive processes associated calculation was done to identify the subset as well as the criteria for with solving Description items is not necessarily warranted. This is in determining the reliability of students’ error patterns. An explanation of contrast to the relatively low accuracy observed in solutions to the criteria is included in the Supplementary Materials section. Results Table and Graph items and the respective misunderstandings identified of this calculation suggests that a large portion of subjects who made from error analysis for both problem types. The results suggest consis­ errors on more than half of all items in each item type were consistent in tent erroneous interpretations of the given information in such problems their error patterns. This supports the conclusion that as students solve and related erroneous constructions of the responses to these items. Such each of these types of problems, they are consistently calling upon the errors are rooted in weak or flawed conceptual knowledge of graphs and same, albeit incorrect, knowledge and verifies that their responses are a tables as representations of linear functions and the use of such true reflection of their understanding, or misunderstanding, of the knowledge in the process of solving linear functions problems. The error domain as manifest within that problem type. analyses suggest a need to address the specific knowledge discrepancies, conceptual and procedural, that have been identified for both problem 2.2.6. Sub-error patterns in low performers types. It is useful to examine the action-level error patterns in these prob­ The data revealed some differences between the three problem types. lems for low performers to highlight the discrepancies in the schemas of While these problems are in fact similar in the information they present, those who are incorrectly solving these types of problems. To do this, the the knowledge required to interpret each problem type is different. The lowest performing 27% (n = 39) of subjects (Kelley, 1939) were data also revealed that students do not have an equal level of prior extracted on the basis of their error patterns on table problems and on knowledge required to engage with and reason about these problems. graph problems independently. Analyses of these students’ sub-error They seem to have a more robust and developed knowledge base and patterns are included in the Supplementary Materials section. Error skill set to solve the Description problems than the Table and Graph patterns observed in the low performance groups are consistent with the problems. As such, this study has illuminated the discrepancy that exists error patterns observed across all subjects as a whole in both graph and in students understanding of the LF domain. It has also provided an table items. These findings reveal that schemas in those who are weakest explanation for why students’ performance on these problems may be at solving these problems exhibit discrepancies in aspects of the low compared to other essential topics in algebra. problem-solving process that may reflect fundamental conceptual mis­ understandings in their LF knowledge, similar to what was found with 2.2.7.1. Cognitive implications of rational errors. There is evidence that the larger group (across all subjects). suggests that the errors observed are not random and illogical and do not stem from general unfamiliarity with the LF domain as a whole. For example, most of the college students tested did well on the description Table 6 items even though many then performed poorly on the table and/or ICC values and confidence intervals for the test of consistency and the test of graph problems. Thus, they seem to know how a linear function is absolute agreement for table items. expressed as an algebraic equation in the canonical form of y = mx + b. ICC 95% CI High consistency in the error patterns for table and/or graph problems Consistency 0.86 0.64–0.96 suggests that students are overwhelmingly and consistently making the Agreement 0.82 0.54–0.95 same errors across items of a given type. 11 N. Elagha and J.W. Pellegrino Learning and Instruction 92 (2024) 101895 One error is that students recognized the relevant approach to the stem from the misspecification of constraints. The solution to this graph problems by applying their knowledge of LF, evidenced by problem type requires systematic evaluation of all given values and showing familiarity with the elements of the canonical form, but they inferring the pattern of relations of the values relative to each other. did not recognize this relevance for table problems. Students appear to However, students appear to have a biased conceptual understanding of make surface-level interpretations about the given information based on linear functions in the context of discerning and expressing given value the structure in which it is presented. This means that the knowledge patterns in which they show a lack of consideration of the constraints regarding the procedural approach for linear functions problems was during the process of formulating an equation that represents the pattern effectively called upon for graph problems but not for table problems. of the given values. Specifically, responses showed that the patterns of For example, while solving table problems, students do not call upon the the x and y values were evaluated separately, disregarding the rule relevant linear function knowledge, even if they do possess it. This is which requires the evaluation of multiple corresponding x and y values because, regardless of other cues in the problem, when seeing a problem to execute the appropriate pattern induction process required to solve in which information is structured in a table format with discrete values, these problems. Consequently, this is reflected in their construction it is not classified as a LF problem, and therefore their LF knowledge is actions such that their conceptual understanding of pattern induction not called upon. This suggests that when these students are solving this does not align with the subsequent procedure which comprises the type of problem, the knowledge that is called upon is not relevant to this process of solving for the elements of the LF equation then constructing problem type which leads to a flawed representation of the situation. it. Instead, responses were constructed in a way that aligns with their Their approach to solving the problems, however, was logically biased understanding of pattern induction and extrapolation. The errors consistent and systematic, but was not the correct approach given the observed in both graph and table problems demonstrate the common nature of the problem. This shows that the errors are rational errors and tendency for students to think of tables and graphs as just a compilation are not random and/or due to a fundamental lack of knowledge of LF. of individual points, instead of understanding that those points make up Students are in fact familiar with the general LF domain but seem to a conceptual entity (Schoenfeld et al., 1993; Stein, Baxter, & Leinhardt, make systematic errors in the interpretation and construction actions 1990; Yerushalmy, 1991). Thus, it seems that the surface-structural that suggest that they lack the ability to properly build representations characteristics of these problems contribute to the erroneous solutions of these problems. of these types of LF problems. The combination of errors that are most prevalent in table and graph The implications of the errors discovered in Study 1 indicate that problems can be presumed to be indicative of the cognitive un­ these errors may have been due to a deficiency during earlier instruction derpinnings that lead to the apparent misunderstandings. A taxonomy in which 1) the specifications of how tables and graphs relate to LF were for rational errors developed by Ben-Zeev (1998) instantiates the not assimilated into prior knowledge, or 2) prior knowledge regarding mathematical reasoning processes which reveal underlying cognitive the conceptualizations of graphs and tables was inaccurate and in turn mechanisms. The basis of this framework is that, as observed in the subsequent LF instruction was built on faulty knowledge structures that results of this study, errors in problem solving are often systematic and resulted in incorrect mental models. Based on conceptual change theory, logically consistent rather than random and unsubstantiated (Ben-Zeev, this can be addressed through exposure to instruction that elucidates 1995; 1996). This rational error framework maps out what is referred to inconsistencies between the correct conceptualizations of the topic and a as inductive failures that seem to arise when one overgeneralizes rele­ learner’s flawed understanding (Vosniadou, 2013). Study 2 explores vant conceptual and procedural knowledge from prototypical examples instructional designs using worked examples that may help achieve this. which result in erroneous solutions to math problems. One type of inductive failur