Artificial Intelligence in Education: Impact of AI-Based Systems on Mathematics Achievement

Use of artificial intelligence in education (AIED) is growing rapidly and significantly
influencing the teaching and learning processes. AI-based systems can provide immediate
feedback and adapt instructions in real time, creating opportunities for interaction
and engagement in the learning process that are often difficult to achieve
in traditional classroom settings, where teachers cannot always offer individualized
support to struggling learners. Several research studies have shown promising results
when AI-based systems are used as a supplemental tool in K-12 classrooms.

AI-based systems such as ITS and ALS have the potential to deliver adaptive and
personalized learning to students, particularly in STEM subjects like mathematics,
which many students find challenging. This is more evident among disadvantaged
students from rural areas who frequently fall behind grade-level expectations due
to limited resources, a lack of highly qualified teachers, and the effects of poverty.

The learning loss caused by COVID -19 pandemic has significantly impacted students’
mathematics performance, further widening existing achievement gaps. To
address this issue and narrow the achievement gap, several schools across the
United States have implemented AI-based tools to support mathematics learning.
A rural high school in the southern United States piloted ALEKS, an ITS, during
the 2021-2022 academic year to support struggling eighth-grade students. In the
2022–2023 academic year, the school adopted Edmentum Exact Path, an ALS, to
continue providing support.

This thesis consists of four chapters: Chapter 1 discusses a study conducted with
ALEKS; Chapter 2 extends the first study for journal publication; Chapter 3 examines
the use of Edmentum Exact Path; and Chapter 4 presents a systematic literature
review on AI-based systems and their impact on mathematics achievement.

Chapter 1 focuses on the effect of intelligent tutoring systems on the mathematics
achievement of underachieving students. This quasi-experimental study used
ALEKS to evaluate its impact on 158 eighth-grade students, 62–68% of whom were
economically disadvantaged, with only 12% proficient in mathematics. This study
aimed to 1) Compare the effectiveness of ALEKS versus traditional instruction in improving
students’ mathematics achievement. 2) Assess students’ progress on gradelevel
math standards over one academic year using ALEKS. 3) Analyze differences
in achievement across class periods with ALEKS implementation.

We compared the results of pretest and posttest from teacher-led instructions and
ALEKS -led instructions across two consecutive years. In the first year, only McGraw
math curriculum Reveal was used. In the second year, ALEKS was implemented as a
supplemental tool in a math support class for 50 minutes every other day, alongside
the Reveal Math curriculum. We also analyzed five years of End-of-Grade (EOG)
state assessment data (without ALEKS) and compared it to EOG data from the year
ALEKS was implemented.

Findings showed that students receiving teacher-led instructions showed greater
mathematics achievement than those using ALEKS-led instructions. This outcome
may be attributed to ALEKS being in its pilot stage, with teachers still learning how
to use it effectively. Many students were working on prerequisite skills as they were
below grade level. The COVID-19 pandemic likely amplified this effect, as students
were promoted to eighth grade without taking state or school exams after missing
much of their seventh-grade instructions due to school closures.

This study also found that ALEKS contributed to improvement across all eleven
mathematics standards within five math domains. However, most students were unable
to complete all standards because ALEKS is mastery-based, requiring students
to achieve 80% accuracy on prerequisite skills before progressing further. Limited
access to ALEKS (only on alternate days) also restricted completion rates. Another
finding indicated that high-achieving students with strong work ethics performed
better compared to mixed-ability groups, which included students with disabilities.

Chapter 2 was extended for journal publication by incorporating statistical analysis,
including paired t-tests and ANOVA, to evaluate the efficacy of ALEKS on students'
mathematics achievement. A literature review and null hypothesis were also
added. The results indicated that teacher-led instruction was more effective, showing
higher test scores and lower variance compared to ALEKS. The study had two
main objectives: 1) to examine whether the use of ALEKS show a statistically significant
improvement in students' mathematics achievement compared to traditional
teacher-led instruction, and 2) to determine whether the use of ALEKS show statistically
significant improvement across grade-level mathematics standards over one
academic year. The analysis found that both ALEKS-led and teacher-led instructions
were statistically significant, with teacher-led instruction being more effective.
A similar pattern was observed when comparing five years of End of Grade (EOG)
data with and without ALEKS. While the use of ALEKS significantly improved all
mathematics standards, the gains varied, likely due to its mastery-based learning,
which requires 80% mastery before progressing to the next topic. Since students
only used ALEKS on alternate days, they were unable to complete all eleven standards.

Overall, the findings from this study provide valuable insights into the use of ITSs
in K-12 classrooms. Mathematics ITSs like ALEKS offer adaptive and personalized
learning opportunities and can significantly enhance achievement among underperforming
students.

In Chapter 3, we examined the effectiveness of Edmentum Exact Path, an AI-based
instructional system, in enhancing mathematics achievement and engagement (affective
and cognitive) among 8th-grade students in the Southern United States.
This quasi-experimental study included 78 students from socioeconomically disadvantaged
backgrounds. We compared an experimental group that received both
traditional teacher-led instruction and Edmentum Exact Path-led instructions to a
control group that received only traditional teacher-led instruction.

The three objectives for this study were 1) To compare the efficacy of Edmentum
Exact Path and traditional teacher-led instruction on students’ mathematics achievement.
2)To examine To investigate differences in students’ affective engagement
between Edmentum Exact Path instruction and traditional teacher-led instruction.
3) To examine differences in students’ cognitive engagement between Edmentum
Exact Path instruction and traditional teacher-led instruction.

This intervention lasted five weeks, with a daily session of 50 minutes each. Both
groups used the McGraw-Hill math curriculum Reveal, 8th-grade math curriculum.
and incorporated Edmentum Exact Path as a supplemental tool for the experimental
group. The experimental group also attended math support classes, where
they worked on individualized learning pathways in Edmentum Exact Path, created
based on diagnostic assessments administered at the beginning of the school
year. Mathematics achievement was measured using pretests and posttests, while
student engagement was measured using a 35-item, 5-point Likert-scale Student
Engagement Instrument (SEI), administered following the posttest to assess affective
and cognitive engagement. A significant limitation of this study is the absence
of a pre-intervention SEI survey, which restricts the ability to measure changes in
engagement over time.

Data were analyzed using t-tests and ANOVA. The result showed that both the experimental
and control groups showed statistically significant improvements in mathematics
achievement. However, the control group showed greater gains in affective
engagement, whereas no statistically significant differences were observed in cognitive
engagement between the two groups.

These results suggest that integrating AI-based systems like Edmentum Exact Path
may enhance mathematics achievement and cognitive engagement by addressing
individual learning needs. However, such tools may be less effective in increasing
affective engagement, possibly due to a lack of emotional responsiveness. Further
research is needed to better understand the role of AI in promoting student engagement,
particularly among the underserved population in rural areas.

In Chapter 4, we conducted a systematic literature review to investigate the impact
of AI-based systems on mathematics achievement in K-12 classrooms. The review
was guided by the following objectives: 1) To identify what types of AI-based systems
are used in mathematics education, and the educational level at which they
are implemented. 2) To identify the impact of AI-based systems on students' mathematics
performance in K-12 classrooms. 3) To explore whether AI-based systems
help reduce the mathematics. achievement gap among students from low socioeconomic
backgrounds, and which system features contribute to this effect.

We followed the PRISMA guidelines and searched six major databases: ACM Digital
Library, ERIC (EBSCO), JSTOR, Wiley, ScienceDirect (Elsevier), and SpringerLink
to locate peer-reviewed articles published between 2008 and 2023. An initial pool
of 1,945 studies was identified based on predefined inclusion and exclusion criteria.
After screening, 42 articles were selected for in-depth analysis. Data was organized
and analyzed using spreadsheets.

The findings indicate that AI-based systems are widely used in K-12 classrooms
across various countries to provide personalized and adaptive learning experiences
to support students’ mathematics learning. Most studies were conducted in the
United States. Both ITS and ALS are used at the elementary, middle, and high
school levels, with more frequent implementation at the middle school level and less
at the high school level. Several studies also reported the use of adaptive learning
games, such as Lynnette, DragonBox, Woot Math Adaptive Learning (WMAL), and
Math Whizz. The most commonly used AI-based systems in the U.S. include ALEKS,
CTA1, ASSISTments, HALF, Math IVLE, MathSpring, and Decimal Point. Studies
from other countries reported the use of AI-based systems such as MIT, dialoguebased
tutors, ACALS, Adaptive CER-based mathematics games, PEDALE, PAT2Math,
ZPDES, RiaRiT, AmritaITS, UZWEBMAT, APPEAL, and HINTS.

Overall, the findings suggest a moderately positive effect of AI-based systems on students’
mathematics achievement. Most studies reported moderate to significant improvements
in student performance, engagement, and retention. Several AI-based
systems, such as ALEKS, MathSpring, and eFit were associated with improved outcomes
among students from low socioeconomic backgrounds, highlighting their potential
to support educational equity.

The discussion presents the findings of each study of this doctoral thesis, along with
the contributions and the limitations of this research. Grounded in the integration
of AI-based instructional systems and personalized learning frameworks, the studies
demonstrate how AI-based systems can enhance student learning outcomes in
mathematics. These AI-based systems show potential in enhancing students’ cognitive
engagement and academic performance. However, the findings also reveal limitations
in promoting affective engagement, highlighting the challenges AI-based
systems face in replicating the emotional connection of human teaching. The study
shows limitations in affective engagement, showing the challenges of AI-based systems
to replicate the emotional and relational aspects of human instruction. This
dissertation adds to the broader dis-course within the AIED communities, providing
empirical evidence on the pedagogical impact of AI-based instructions on students
mathematics achievement and engagement in underserved population within rural
educational contexts.

Future research should investigate the impact of various AI-based systems by comparing
their effectiveness with ITS and ALS, such as ALEKS, and Edmentum Exact Path. It should consider factors like instructional design, student characteristics,
and specific learning outcomes. By leveraging the strengths of these AI-based
systems, educators and policymakers make informed decisions regarding their integration,
ultimately enhancing student achievement in mathematics and related
disciplines.