How AI Detects Knowledge Gaps

Introduction

In 1982, three psychologists at the University of Wisconsin published a paper with a title that should have alarmed every teacher on earth. They called it "The Illusion of Knowing" [1]. Arthur Glenberg, Alex Wilkinson, and William Epstein gave college students passages containing blatant contradictions. One sentence said a fact. A sentence nearby said the opposite. The students read the passages and rated their comprehension as high. They did not notice the contradictions. They believed they understood what they had just read. They were wrong.

Seventeen years later, Justin Kruger and David Dunning at Cornell quantified the problem even more sharply. People scoring in the bottom quartile on tests of logic, grammar, and humor estimated their performance at the 62nd percentile [2]. Their actual score placed them at the 12th percentile. The gap between what they thought they knew and what they actually knew was fifty percentile points wide. Humans do not just fail to detect their own knowledge gaps. They systematically overestimate what they know. The worse they perform, the larger the overestimation.

This is the problem that six decades of computer science, psychometrics, and cognitive science have been trying to solve. How do you build a machine that knows what a learner does not know, especially when the learner cannot tell you? The answer turns out to involve hidden Markov models, recurrent neural networks, a Danish mathematician, a forgotten geography tutor from 1970, and a scheduling algorithm trained on 700 million flashcard reviews. This is the story.

The Brain That Lies to Itself

Before exploring how machines detect knowledge gaps, it helps to understand why the problem exists at all. The answer is biological.

The human brain does not have direct access to its own memory storage. When asked "do you know this?" the brain does not look up the answer in some internal database. Instead, it relies on cues. Asher Koriat at the University of Haifa spent two decades mapping these cues and in 1997 published the definitive account [3]. He showed that when people judge whether they have learned something, they rely on fluency (how easily the material comes to mind), familiarity (whether the topic feels recognizable), and recency (how recently they encountered it). None of these cues reliably measure actual retention.

This is why rereading feels productive. The second time through a chapter, every sentence feels familiar. The brain interprets that familiarity as understanding. But familiarity and understanding are different things. Rereading builds recognition. It does not build recall. And when the exam requires recall, the student discovers the gap too late.

John Dunlosky and Janet Metcalfe documented this machinery in their 2009 textbook on metacognition [4]. Metacognition, they explained, consists of two parts: monitoring (judging what you know) and control (deciding what to study next). If monitoring is inaccurate, control fails. You study the wrong material. You skip the gaps. You arrive at the exam feeling confident and leave feeling confused.

The Glenberg illusion, the Dunning-Kruger effect, and Koriat's cue-utilization research all point to the same conclusion. Humans are structurally bad at detecting their own knowledge gaps. Not occasionally. Not because of laziness. But because the monitoring system they rely on uses the wrong signals. This is not a bug that education can fix with motivation or willpower. It is a design limitation of the human cognitive system.

Machines do not share this limitation. A machine can track every question a student answered, every mistake, every hesitation, every pattern of forgetting. It does not rely on feelings of familiarity. It works with data. The question is how.

A Psychometrician in Copenhagen

The scientific foundation for measuring what someone knows was laid not by computer scientists but by a mathematician. Georg Rasch was a Danish statistician who worked at the Danish National Institute for Educational Research. In 1960, he published a short book with a long shadow: *Probabilistic Models for Some Intelligence and Attainment Tests* [5].

Rasch's idea was simple but powerful. Each student has an ability level (θ). Each test item has a difficulty level (β). The probability that a student answers correctly depends on the difference between ability and difficulty. If a student's ability exceeds the item's difficulty, the probability is above 50 percent. If the item is harder than the student, the probability drops below 50 percent. The formula is a logistic function, and it places students and items on the same scale.

This was a departure from classical test theory, which treated the total score as the unit of measurement. Rasch's model made it possible to say something specific about each individual item and each individual student. It was the difference between a bathroom scale (one number) and an MRI (a detailed picture).

Frederic Lord and Melvin Novick at the Educational Testing Service in Princeton formalized the broader framework in 1968. Their book *Statistical Theories of Mental Test Scores* [6], with contributed chapters by Allan Birnbaum, introduced what became known as Item Response Theory. Birnbaum added two more parameters: a discrimination parameter (how sharply an item separates high-ability from low-ability students) and a guessing parameter (the probability of getting a multiple-choice item right by chance). These became the two-parameter and three-parameter logistic models, known as 2PL and 3PL.

But the real breakthrough for gap detection came from Lord himself. In 1971, he proposed the idea of adaptive testing [7]. Instead of giving every student the same fixed test, the computer would choose each question based on what it had learned from the previous answers. If a student answered a hard question correctly, the next question would be harder. If they got it wrong, the next would be easier. Each question was selected to maximize information about the student's ability level. This was computerized adaptive testing, and it turned a static exam into a dynamic conversation between machine and learner. The underlying principle connects to what researchers call desirable difficulties: productive struggle that strengthens learning.

The principle is identical to how a skilled tutor probes a student's understanding by asking progressively harder or easier questions. But a computer can do it with mathematical precision, selecting the single most informative question from a bank of thousands.

The First Machine That Tried to Teach

The year was 1970. A computer scientist named Jaime Carbonell at Bolt Beranek and Newman published a paper in *IEEE Transactions on Man-Machine Systems* describing a program called SCHOLAR [8]. SCHOLAR tutored students on the geography of South America. It could ask questions, answer questions, and give feedback, all in natural language. Its knowledge was stored as a semantic network of facts and concepts. If a student got something wrong, SCHOLAR could identify the specific concept that was missing.

Nobody called it an intelligent tutoring system at the time. The term did not exist yet. But SCHOLAR is widely regarded as the first ITS, and its architecture anticipated ideas that would take decades to mature [9]. The key insight was separating domain knowledge (what needs to be taught) from student modeling (what the student already knows). That separation is still the backbone of every adaptive learning system built since.

The 1970s and 1980s produced a remarkable generation of experimental tutors. SOPHIE taught electronics troubleshooting. DEBUGGY diagnosed subtraction bugs in children's arithmetic. GUIDON layered teaching strategies on top of the medical diagnosis system MYCIN. Each pushed the boundary of what machines could infer about a learner's state.

Then in 1984, Benjamin Bloom published a two-page paper that gave the entire field a reason to exist [10]. Bloom reported that students who received one-on-one tutoring with mastery learning performed two standard deviations above students in conventional classrooms. The average tutored student outperformed 98 percent of conventionally taught students. Bloom called this the "2-sigma problem." The question was clear: could we find a method of group instruction as effective as one-on-one tutoring?

The answer, or at least a partial answer, came from John Anderson at Carnegie Mellon University. Anderson's ACT-R cognitive architecture [11] modeled human cognition as a system of production rules. Each rule described a step in solving a problem. Anderson and his students, particularly Kenneth Koedinger, built the Cognitive Tutor for algebra. The system tracked which production rules a student had mastered and which they had not. It selected problems to target the weakest rules.

Field studies were promising. A 1997 evaluation in Pittsburgh showed that Cognitive Tutor students scored 15 to 25 percent higher on basic skills and 50 to 100 percent higher on problem solving compared to control groups [12]. The later RAND Corporation evaluation at scale found a more modest but still positive effect of about 0.20 standard deviations in the second year [13].

Four Parameters and a Hidden Markov Chain

The mathematical foundation for modern AI gap detection was laid in 1995 when Albert Corbett and John Anderson at Carnegie Mellon published "Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge" [14]. Their approach was built on a specific statistical framework: a two-node hidden Markov model.

The idea is straightforward. At any moment, a student either knows a skill or does not. This is the hidden state. It cannot be observed directly. What can be observed is whether the student answers questions correctly or incorrectly. The model connects the hidden state to the observable behavior through four parameters.

The first parameter, p(L‚ÇÄ), is the probability the student already knew the skill before encountering any instruction. The second, p(T), is the transition probability, the chance of learning the skill after each practice opportunity. The third, p(S), is the slip rate, the probability of answering incorrectly despite knowing the skill. And the fourth, p(G), is the guess rate, the probability of answering correctly without actually knowing the skill.

With these four numbers, the model updates its estimate of the student's knowledge after every interaction. If a student answers correctly, the probability of knowing increases (unless the guess rate is high). If they answer incorrectly, it decreases (unless the slip rate is high). The estimate converges toward a reliable picture of what the student knows and does not know.

Corbett and Anderson tested this in the ACT Programming Tutor, which taught Lisp programming to university students. The system tracked each production rule independently, maintaining a separate probability estimate for each skill. When a skill's estimated probability of mastery crossed a threshold (typically 0.95), the system stopped drilling that skill and moved on.

The model was simple. Four parameters per skill. Binary hidden states. First-order Markov dynamics (only the current state matters, not the history). These simplifications made it practical. They also made it wrong in interesting ways. Real learning is not binary. Skills interact. Forgetting happens. But for its time, BKT was remarkably effective. Michael Yudelson, Koedinger, and Geoffrey Gordon at Carnegie Mellon showed in 2013 that individualizing the parameters per student, particularly the learning rate, produced measurable improvements [15].

Philip Pavlik, Hao Cen, and Koedinger proposed an alternative in 2009 called Performance Factors Analysis [16]. Instead of a hidden Markov model, PFA used logistic regression over the count of prior successes and failures per skill. It was simpler, more interpretable, and handled multi-skill items more naturally. But both approaches shared a fundamental limitation: they required human experts to define the skills and tag every question with the relevant skill labels. As course content grew, this became a bottleneck.

When Neural Networks Learned to Read Exam Papers

In 2015, a Stanford graduate student named Chris Piech did something that had not been tried before. He fed sequences of student interactions (which question was attempted, whether the answer was correct) into a Long Short-Term Memory network and asked it to predict the next response [17].

The results were startling. On the Khan Academy dataset, the LSTM achieved an AUC (area under the receiver operating characteristic curve) of 0.85. Standard BKT on the same data scored 0.68. On the ASSISTments dataset, the gap was even wider: 0.86 versus 0.69. Piech and his co-authors called this a "25 percent gain over the previous best reported result" [18].

What made DKT different was not just accuracy. It was the absence of human-defined skill labels. BKT needed experts to tag every question with the knowledge component it tested. DKT learned the structure directly from data. Each student interaction was encoded as a one-hot vector of length 2M (where M is the number of distinct questions, doubled for correct and incorrect responses). The LSTM's hidden state served as a continuous, high-dimensional representation of the student's knowledge. No binary "knows" or "doesn't know." Instead, a dense vector that captured nuances, interactions between skills, and gradual changes.

The paper was presented at NeurIPS 2015 and has since been cited over 1,200 times. It opened a floodgate. Within five years, more than a dozen new architectures appeared, each addressing a limitation of the original DKT.

Jiani Zhang and colleagues at the Hong Kong University of Science and Technology introduced Dynamic Key-Value Memory Networks (DKVMN) in 2017 [19]. The model separated knowledge storage into a static key matrix (representing concepts) and a dynamic value matrix (tracking mastery per concept). This made the model more interpretable than a raw LSTM because you could read off a student's estimated mastery for each concept directly from the value matrix.

The Cho et al. systematic review published in 2024 catalogued the full range of post-DKT models [20]. The field had moved from recurrent networks to attention mechanisms to full transformer architectures, each generation squeezing a few more AUC points from the data while adding new capabilities like forgetting awareness, exercise-level embeddings, and temporal lag features.

Attention, Memory, and the Machines That Forget

The most important post-DKT model is arguably AKT, Context-Aware Attentive Knowledge Tracing, published at KDD 2020 by Aritra Ghosh, Neil Heffernan, and Andrew Lan [21].

AKT solved three problems simultaneously. First, it introduced monotonic attention. Standard attention mechanisms let a model attend to any past interaction equally. But in learning, more recent interactions should matter more because of forgetting. AKT's attention weights include an exponential decay term that explicitly models the passage of time. Interactions from yesterday count less than interactions from five minutes ago.

Second, AKT brought psychometrics back into deep learning. Its question embeddings use a Rasch-style decomposition: each question's embedding is the sum of its concept embedding plus a difficulty deviation. This is the IRT model from 1960, but now embedded inside an attention layer. It is a clever marriage of classical measurement theory and modern deep learning.

Third, AKT used context-aware response embeddings that encode not just whether an answer was correct but how the student interacted with the material. Across multiple benchmark datasets (ASSISTments 2009, ASSISTments 2017, Statics 2011), AKT consistently outperformed DKT, DKVMN, and SAKT.

Shalini Pandey and George Karypis had introduced SAKT (Self-Attentive Knowledge Tracing) in 2019 [22], bringing transformer-style self-attention to the field. Youngduck Choi and colleagues followed with SAINT [23], a full encoder-decoder transformer that separated the exercise stream (encoder) from the response stream (decoder). SAINT+ added elapsed time and lag time features [24].

What does all this mean in practical terms? A student sits down to study. With every answer, the system updates a detailed probabilistic picture of what the student knows and does not know across dozens or hundreds of micro-skills. The picture accounts for forgetting. It accounts for the difficulty of each specific question. It accounts for how long the student took. And it does all this without the student ever filling out a self-assessment questionnaire. The machine infers the gaps from behavior, not from self-report.

This is the structural advantage over human metacognition. The machine does not rely on feelings of fluency or familiarity. It uses the statistical structure of right and wrong answers over time. And when it says "you probably do not know this," it is right about 86 percent of the time.

The Map of Everything You Know

While the machine-learning community was building neural models, a parallel tradition in mathematical psychology was approaching the problem from a completely different angle.

In 1985, Jean-Paul Doignon and Jean-Claude Falmagne published a paper that introduced Knowledge Space Theory [25]. Their idea started from a simple observation: knowledge has structure. If a student knows how to solve quadratic equations, they almost certainly know how to solve linear equations. The reverse is not true. This means the set of all possible knowledge states is not random. It forms a structured space, and only certain combinations of skills are psychologically plausible.

A knowledge space is a mathematical object: a collection of subsets (called knowledge states) of a domain of items. The collection is closed under union, meaning if two states are plausible, so is their combination. This constraint dramatically reduces the number of possible states the system needs to consider, making assessment efficient even for large domains.

Falmagne spent the next two decades turning this theory into a working system. The result was ALEKS (Assessment and LEarning in Knowledge Spaces), which became one of the most widely used adaptive learning platforms in K-12 mathematics. Eric Cosyn, Hasan Uzun, Christopher Doble, and Jeff Matayoshi described its architecture in a 2021 paper [26]. ALEKS assesses a student by asking questions chosen to be maximally informative about which knowledge state the student occupies. Each question has a predicted probability of success near 0.5 given the current state estimate, the same principle as adaptive testing in IRT. But instead of estimating a single ability parameter, ALEKS estimates a position in a combinatorial space of possible knowledge configurations.

The result is a detailed map. Not a score. Not a percentile. A map showing exactly which concepts the student has mastered, which are within reach (meaning all prerequisites are met), and which are far away. The student sees a pie chart. The teacher sees a heatmap. The algorithm sees a probability distribution over the knowledge space.

The Missing Link: Forgetting and Scheduling

Gap detection answers one question: what does the learner not know right now? But a second question is equally important: when will the learner forget what they currently know?

Hermann Ebbinghaus answered the second question in 1885 with his forgetting curve. Memory decays exponentially after learning, with roughly half the material gone within an hour and about 70 percent gone within a day [27]. But each successful retrieval flattens the curve. Space the reviews right and the memory stabilizes.

Modern spaced repetition algorithms merge gap detection with forgetting prediction. The most advanced current system is FSRS (Free Spaced Repetition Scheduler), created by Jarrett Ye. FSRS models each memory using three variables: Difficulty (how inherently hard the material is), Stability (how long before the memory decays to a threshold), and Retrievability (the probability of successful recall right now) [28]. When Retrievability drops below a target (usually 90 percent), the system schedules a review.

FSRS was integrated into the popular open-source flashcard platform in version 23.10 and became the default scheduler for new users in version 23.12. An analysis of roughly 700 million anonymized reviews showed that FSRS users needed approximately 20 to 30 percent fewer reviews than users of the older SM-2 algorithm to maintain the same retention rate.

AKT's monotonic attention mechanism works on the same principle, from the opposite direction. Instead of predicting when a specific card will be forgotten, it adjusts the weight of past interactions based on elapsed time. The longer ago a student answered a question, the less that answer counts toward the current knowledge estimate. Gap detection and spacing are two sides of the same coin: one identifies what is missing, the other determines when to act.

The frontier is systems that combine both. A 2025 preprint called LECTOR (LLM-Enhanced Concept-based Test-Oriented Repetition) fuses semantic similarity derived from large language models with FSRS-style scheduling [29]. Simulated learners using LECTOR achieved a 90.2 percent success rate compared to 88.4 percent for the best previous baseline.

Large Language Models Enter the Classroom

The arrival of large language models in 2022 and 2023 reshaped knowledge gap detection in a way that is still unfolding.

Traditional knowledge tracing requires structured data: a sequence of question-answer pairs tagged to specific skills. LLMs can detect gaps through conversation. A student types a confused explanation. The model notices the confusion, identifies the missing concept, and probes deeper. This is qualitatively different from multiple-choice pattern matching.

The most rigorous published study of this approach is KG2M (Knowledge Gaps to Mastery), published in *Computers and Education: Artificial Intelligence* in 2025 [30]. The system combines discussion forum data with course-specific content using retrieval-augmented generation (RAG). It was deployed across three computer science courses with 1,355 students and 2,878 posts. Instructors evaluated the results through semi-structured interviews and found the tool effective for surfacing class-level knowledge gaps and generating targeted learning activities.

A more radical approach appeared in a 2025 preprint called RPKT (Recursive Prerequisite Knowledge Tracing) [31]. RPKT frames the problem explicitly as the "unknown unknowns" challenge. The system recursively traces prerequisite concepts during a tutoring conversation until it reaches the student's actual knowledge boundary, the point where the student's responses shift from confident to confused. This is the conversational equivalent of adaptive testing, but conducted through natural language rather than multiple-choice items.

Several groups have explored integrating LLMs directly with traditional knowledge tracing. Lee et al. showed in 2024 that language models can perform knowledge tracing by encoding student interaction histories as text prompts [32]. Kim et al. proposed token-efficient approaches that weight textual options differently based on their informativeness [33]. Wang et al. introduced LLM-KT, a plug-and-play instruction layer that aligns language model outputs with knowledge tracing objectives [34].

Khan Academy's GPT-4 powered tutor, launched in 2023, scaled from 68,000 users to over 700,000 in one academic year, expanding from 45 to more than 380 school district partners. But peer-reviewed evidence for its learning effectiveness remains thin. A 2024 study at the University of Windsor with 69 undergraduates found significant within-condition learning gains but no statistically significant between-condition differences compared to Google search or paper-only study [35]. A large randomized trial is registered with J-PAL but results are not yet published.

The honest assessment is this: LLMs bring unprecedented flexibility to gap detection. They can work with free-text responses, handle novel domains without pre-tagged skill structures, and adapt to individual conversational patterns. But they lack the calibrated, persistent learner models that decades of BKT/DKT research have built. The most promising direction is hybrid systems that pair an LLM's conversational intelligence with a traditional knowledge tracing backend.

What the Evidence Actually Shows

The effectiveness of AI-powered tutoring systems that detect knowledge gaps has been studied in multiple meta-analyses. The numbers are encouraging but should be cited carefully.

Wenting Ma and colleagues published the most cited meta-analysis in 2014 in the *Journal of Educational Psychology* [9]. Across 107 effect sizes and 14,321 participants, intelligent tutoring systems produced an overall effect of g = 0.43. That means the average ITS student performed about 0.43 standard deviations above the average control student. Broken down by comparison: ITS versus teacher-led large-group instruction showed g = 0.42. ITS versus non-ITS computer-based instruction showed g = 0.57. ITS versus textbooks showed g = 0.35. ITS versus one-on-one human tutoring showed g = ‚àí0.11 (not statistically significant).

The takeaway: intelligent tutoring systems outperform textbooks and classroom instruction by a meaningful margin. They roughly match one-on-one human tutoring. They do not surpass it.

James Kulik and John Fletcher conducted a separate meta-analysis in 2016 for the *Review of Educational Research* [36]. They found a larger overall effect, g ≈ 0.62, with step-based tutors (like Cognitive Tutor) showing d = 0.76 and answer-based tutors showing d = 0.31. The type of tutoring matters. Systems that monitor each step of problem solving detect gaps at finer grain and produce larger effects than systems that only check final answers.

Steenbergen-Hu and Cooper focused on college students and found a smaller but positive range of g = 0.32 to 0.37 [37].

Arthur Graesser and colleagues developed AutoTutor, a natural-language dialogue tutoring system that models student knowledge by comparing student responses to expected answers and anticipated misconceptions [38]. Learning gains ranged from 0.3 to 0.8 standard deviations depending on the comparison condition [39].

The ASSISTments platform, developed by Neil and Cristina Heffernan at Worcester Polytechnic Institute, is both a tutoring tool and a research platform. A 2016 study published in *AERA Open* demonstrated a WWC-approved effect of approximately 0.22 standard deviations for online mathematics homework [40].

The Skeptics and the Limits

No honest account of AI gap detection can ignore its limitations. The field has real problems.

The first is the cold-start problem. Every knowledge tracing model needs data before it can make predictions. A brand new student with zero interaction history is invisible to the system. BKT uses a prior probability, but that prior is generic. DKT needs at least a few dozen interactions before its hidden state becomes informative. Solutions exist (transfer learning, demographic priors, prerequisite graphs), but none eliminates the problem entirely.

The second is interpretability. DKT can predict with 86 percent accuracy whether a student will answer the next question correctly. But it cannot always explain why. The hidden state of an LSTM is a 200-dimensional vector. What does dimension 137 mean? Nobody knows. DKVMN and AKT improved interpretability by forcing explicit concept representations, but the field has not fully solved the tension between accuracy and transparency.

The third is Bloom's 2-sigma problem itself. The 1984 paper that motivated the field reported that one-on-one tutoring produced a two standard deviation improvement. But the underlying evidence was two small dissertations (Anania 1982 and Burke 1984) on narrow topics. The claim has been questioned [36]. VanLehn's 2011 analysis found that human tutoring effects are closer to 0.79 sigma, not 2.0 [41]. The goal of matching human tutoring may be more achievable than Bloom suggested, but also less dramatic.

The fourth is equity. Knowledge tracing models are trained on data. If that data comes predominantly from one population, the model's predictions may be less accurate for underrepresented groups. Slip and guess parameters calibrated on students in affluent suburban schools may not transfer to students in underfunded rural schools. Bias in educational data is an active area of concern [42].

The fifth and most fundamental: these systems measure performance, not understanding. A student who answers five algebra questions correctly may have memorized a procedure without understanding why it works. Transfer, creativity, and deep conceptual understanding remain difficult to assess through interaction data alone.

Where the Story Goes Next

The arc from Rasch in 1960 to RPKT in 2025 covers 65 years of steady progress on the same fundamental question: how do you measure what someone does not know?

Each generation answered with the tools it had. Rasch answered with logistic models and paper tests. Corbett answered with hidden Markov models and desktop software. Piech answered with LSTMs trained on millions of interactions. Ghosh answered with attention mechanisms that model forgetting. And the current generation is answering with large language models that detect gaps through conversation.

The market reflects this trajectory. AI-in-education market estimates for 2025 range from roughly five to nineteen billion dollars depending on the research firm, with projections reaching thirty to over seventy billion by 2030 [43], [44].

But the numbers are less interesting than the principle. Humans overestimate their knowledge. Always have. Probably always will. The Glenberg illusion is not a character flaw. It is a structural feature of the metacognitive system that Koriat mapped in 1997. AI gap detection does not fix this flaw. It works around it, by inferring from behavior what self-assessment cannot reliably provide.

The next frontier is closing the loop between detection and action. Knowing that a student is weak on negative numbers means nothing if the system cannot generate an appropriate remediation path. This is where LLM-powered tutors may finally fulfill the promise Bloom articulated in 1984. Not by replacing the teacher, but by giving every student something close to what only the tutored students had: a system that knows exactly what they do not know, and responds accordingly.

The sixty-year arc bends toward a simple truth. The best teacher is the one who understands what you have not yet understood. The best machine is the one that can figure it out without asking.