Houghton Mifflin Harcourt’sMATH 180program targets students struggling with early mathematics achievement. Developed for students in Grade 5 and above who are two or more years behind in math achievement, and in line with recommendations for a successful math intervention from the American Institutes for Research and the What Works Clearinghouse (WWC),MATH 180provides explicit and systematic instructional methods, motivational strategies, and data-based decision making for teachers, among other innovative components. Using a combination of whole-class instruction, computer adaptive instruction and practice, and teacher-led small-group instruction,MATH 180aims to create a learning community where students learn to persevere through mistakes, experience success, and develop a growth mindset to gain confidence and overcome previously experienced struggles with math. Teachers usingMATH 180can become more effective instructors with the use of real-time data, differentiated instruction, and embedded discussion questions that identify student thinking and misconceptions, and through continuously available support, professional learning, and coaching resources.

Prior research suggests thatMATH 180may have a positive impact on student math achievement. For example, an early outcomes study showed greater growth in math achievement for students usingMATH 180compared to other interventions, and a large majority of teachers reported thatMATH 180raised achievement in the classroom and improved their teaching (Houghton Mifflin Harcourt, 2014). However, this study did not involve random assignment, making causal conclusions problematic, and on average, students completed less than 50% of the program over the school year.此外,最近生产WWC指导如何best to help students struggling with mathematics reports that, in general, little evidence is available on the impact of interventions for low-performing students (Gersten et al., 2009). Thus, a more rigorous efficacy study is needed to further evaluate the impact of this program on student math achievement.

In order to expand on the research base behind theMATH 180program, JEM & Rconducted a quasi-experimental study in Cobb County, GA.

The overarching purpose of the study was to evaluate the effectiveness ofMATH 180in helping middle school intervention students attain critical mathematics skills and closing the achievement gap. Specifically, a one-year study was conducted and designed to address the following questions:

• What are the effects ofMATH 180on student mathematics achievement? Specifically, how do changes in mathematics test scores achieved byMATH 180students compare to changes achieved by similar students in the matched comparison group?
• How doesMATH 180differentially affect subgroups of students? Specifically, how do changes in mathematics test scores achieved by specific subgroups ofMATH 180students (male and female students; Black, Hispanic, and White students; English learners [ELs]; and students with disabilities) compare to changes achieved by subgroups of comparison group students?

In order to address these research questions, a quasi-experimental study (QES) was conducted during the 2017–2018 school year in Cobb County, Georgia. The study included fiveMATH 180teachers and their 128 students in Grades 6–8.

TheMATH 180研究开始于2017年10月,在May 2018. Two middle schools were selected as study schools within Cobb County Public Schools in Georgia. Students within these schools who were identified as needing additional math support and usedMATH 180over the school year participated in the study (n= 128). In total, seven math support classes and five teachers served as treatment (i.e.,MATH 180) classrooms.During the study, students usedMATH 180as part ofasecond math class period. As a result,MATH 180was usedin addition to acore math course.

In order to allow for rigorous comparisons to be made between groups, each of the participating schools was matched to a comparison school that was not implementingMATH 180during the 2017–2018 school year but possessed similar schoolwide demographic and prior performance statistics. While this helped to promote baseline equivalence, school level data with such a small sample cannot ensure baseline equivalence, which is a necessary precondition for making strong causal conclusions. Therefore, to ensure that there were no significant differences between students on important variables, math intervention students who met specific criteria (e.g., completing at least 10 sessions ofMATH 180) were included in the final sample, and each student was matched to a student from a non-MATH 180school.

The comparison sample was determined via propensity scoring methodology. Basically, a logistic regression model was used to model the propensity to be aMATH 180学生。从逻辑学的预测概率ic regression served as a measure of the propensity of being aMATH 180student and was also used as a distance measure to implement the matching procedure (nearest neighbor algorithm). This predicted probability served to reduce the multidimensional student-level characteristics into a single number that was used to matchMATH 180and non–MATH 180students. Control students were selected based on the distance measure so that a comparison sample of the most closely matched students was created. The final propensity score was based on the following variables:

• Grade
• Ethnicity
• Gender
• EL Status
• Special Education Status
• Percent Present 2016–2017 School Year
• Math InventoryData from Fall 2017
• Georgia State Assessment Data from Spring 2016

Trainings were designed to provide teachers with the necessary background and practical experiences to begin implementing the program with fidelity during Fall 2017. The focus of these trainings was on the instructional model of theMATH 180program, the use of the materials and implementation of the key components, and how the program could best be used to effectively help students build strong math foundational skills.

A Houghton Mifflin Harcourt^®(HMH^®) professional trainer provided three days of trainingas well as coaching visits every 4–6 weeks. In addition to initially providing the details of the program pedagogy and components (when and how to use), follow-up trainings tended to be customized according to the needs of each site and teacher, and teachers could ask for help with specific components they wanted to become more proficient at incorporating. In addition, together with the teacher, goals were set following coaching visits to help teachers keep on track with their implementation of the program. Throughout the course of the study, participants had email access to the professional trainer so they could communicate on a real-time basis any questions or issues regarding training or implementation that arose.

Math Assessment

TheMath Inventorywas utilized as an outcome measure in this study. Key features of this assessment include:

• Computer adaptive assessment that measures math abilities and longitudinal progress from kindergarten through Algebra II
• Measures of the National Council of Teachers of Mathematics (NCTM^®) five content standards: Number and Operations Geometry, Algebra/Patterns and Functions, Data Analysis and Probability, and Measurement
• Mathematics growth is measured on the Quantile^®Framework for Mathematics—a scientific taxonomy of over 500 math concepts and skills—placing student readiness and difficulty of math tasks on the same scale.
• 40-minute group administration, which can be offered 3–5 times per year

TheMath Inventorywas administered in Fall 2017 and Spring 2018 by Cobb County. Data from the prior school year (2016–2017) was also provided in order to 1) check for baseline equivalence, and 2) measure growth from last year. Tests were scored by each respective system, and individual level resultswere shared with researchers.

School Characteristics

The study sites consisted of six schools located in the suburbs of Atlanta and within Cobb County Public Schools. All study sample students received math intervention in addition to their core math instruction.

Demographic and prior performance information is presented in Table 1 for each of the study schools. As shown, the population of students atMATH 180School A and their matched comparison school (E) also tended have similar demographics.MATH 180School B and comparison schools C, D, and F tended to have similar demographics, including higher percentages of minority students, students receiving a free/reduced-price lunch, and EL students, and a lower percentage of historical math proficiency, especially compared to a statewide average of 42%.

Student Characteristics

In total, the study sample consisted of 307 sixth- through eighth-grade students at six middle schools. Specifically, the full analytical sample consisted of 128 intervention students who used theMATH 180program during the 2017–18 school year and 179 comparison students not using the program.

Demographic characteristics for both types of students in the study sample are shown in Table 2. The sample consisted of an ethnically diverse population, with over 80% minority students. As shown, the propensity matching procedure resulted inMATH 180and control groups that were fairly comparable across all demographic categories. Indeed, there were no significant differences between the two groups with regard to any measured demographic factors,ps > .05.

Importantly, groups did not differ with regard to the average number of days students were enrolled or the percentage of days students were absent (see Table 3). There were also no differences in baseline math proficiency across groups as measured by math scale scores on the previous year’s end-of-grade test, andMath InventoryQuantile^®measures from the beginning of the school year, ps > .05. Still, in order to ensure baseline equivalence and enhance the sensitivity of analyses, baseline math performance was taken into account as a covariate in analyses comparingMATH 180and non–MATH 180students.

MATH 180Program Description

MATH 180focuses on reasoning and using visual models to make sense of math.

Utilizing a blended approach that incorporates both digital and print materials, the instructional model forMATH 180is designed to help students master skills and advance at an accelerated pace. Instruction begins with a whole-class “Do Now” to help students warm up. Taking approximately 5–10 minutes, these classroom management routines encourage thinking, inspire mathematical habits of mind, and make connections to prior topics. Following this whole-class time, students break into two groups and rotate between group instruction and the personalizedMATH 180software. While one group of students utilizes the customized software component of the program for support and practice, the teacher works with the other half on building conceptual understanding, developing reasoning and communication skills, and interpreting student thinking. Following 20–25 minutes, students rotate again so that they experience both groups before the class period ends.

MATH 180consists of two courses. Course I is designed for students who need to build numerical understanding and reasoning skills—the foundational concepts that students need to think algebraically. Concepts covered include multiplication, division, decimals, and fractions. Course II transitions students to pre-algebra with an emphasis on building proportional reasoning with rates, ratios and linear relationships, and functions.

For teachers, the Teaching Guide provides daily step-by-step instruction to develop understanding through discussion and problem solving. In addition to the ongoing formative assessments, the mSkills tests assess students for mastery of key, standards-aligned skills and concepts taught during whole-class and group instruction.

A consumable student book,mSpace, is designed to promote students’ active participation by providing a place to record and share their work and understandings. The companion software’s game-like environment engages students with games, choices, and an opportunity to earn badges as rewards. In the Learn Zone, students gradually progress from guided to independent practice with interactive visual models, a metacognitive coach, and corrective feedback. In the Success Zone, students apply what they’ve learned to more difficult tasks.

What are the effects ofMATH 180on student mathematics achievement? How do changes in mathematics test scores achieved byMATH 180students compare to changes achieved by similar students in the matched comparison group?

In order to examine the effects ofMATH 180on math performance, students who usedMATH 180were compared to an equivalent sample of non–MATH 180users via ANCOVAs. As previously noted, groups were matched based on important demographic variables. Nevertheless, to account for any baseline differences between the two groups, analyses included pretest scores as a covariate. Therefore, a significant effect for the group would indicate that after controlling for baseline performance, there were significant differences in the posttest scores ofMATH 180and comparison students.

Results showed that in the overall sample, there was a significant group difference in posttest performance on theMath Inventory,p< .05. As shown in Figure 1, when adjusting for baseline (Fall 2017)Math Inventoryperformance, the gains inMath InventoryQuantile measures were greater forMATH 180students than comparison students (49 Quantile difference),p< .05.

Findings demonstrating greater learning gains amongMATH 180students were further supported by the effect sizes obtained in comparative analyses. Effect size is a commonly used measure of the importance of an observed difference. The effect size of the observed difference between groups on the Spring 2018Math Inventory(d= 0.29 for differences in posttestMath InventoryQuantile measures,d= 0.33 forMath Inventorygrowth during the school year) can be classified as small-moderate and educationally meaningful.

Growth among Student Subpopulations

Analyses were also conducted to assess whether there were differences in math performance betweenMATH 180and comparison students belonging to several subpopulations of interest. Specifically, sufficient data was available to conduct analyses by gender, ethnicity, EL status, and disability status (see Figures 2–5). As with the overall analyses, an ANCOVA was run to determine whether there was a significant interaction between subgroup categories (e.g., males and females) and intervention group (i.e.,MATH 180vs. comparison students). A significant interaction would suggest that the effect ofMATH 180differs for different categories within the subgroup. Because group differences may also exist within an individual subgroup category, separate analyses were also conducted to investigate potential group differences in specific subgroup categories. Analyses included baseline performance as a covariate in order to minimize the potential effect of any pre-existing group differences.

Results for analyses ofMath Inventorygrowth revealed one significant interaction between subgroup categories and intervention group. When controlling for Fall 2017 scores, EL students in theMATH 180program outperformed controls on the Spring 2018 test,p< .05. For all other subpopulations, interactions between subgroup category and intervention group were not statistically significant,ps > .05. However, analyses of individual subgroup categories yielded several group differences. In addition to differences among EL students, there were significant group differences among females, Hispanic students, and students without disabilities,ps < .05. There was also a marginally significant group difference inMath Inventorygrowth among Black students,p< .10. In all cases, learning gains were greater among students inMATH 180than those observed in their counterparts in the comparison sample.

结果the outcome measures indicate thatMATH 180is associated with positive math learning gains among struggling math students. In comparison to students who did not use theMATH 180program, significant differences in learning gains were observed. Overall,MATH 180学生的表现比较学生的Math Inventory.MATH 180students demonstrated higher math performance as compared to non–MATH 180students on theMath Inventory, which suggests thatMATH 180is designed to improve students’ foundational understanding of math concepts.

When assessing performance in different subpopulations, several significant differences emerged regarding improvements inMath Inventoryperformance amongMATH 180students. Specifically, there were significant group differences in the post-test scores among females, Black and Hispanic students, EL students, and students without disabilities. In all cases, differences reflected greater math learning gains among students usingMATH 180compared to the matched control sample. Such findings are particularly noteworthy given the small sample sizes involved.

In summary, theMATH 180program was associated with positive learning gains among struggling math learners. Moreover, comparisons made with non–MATH 180students suggest that this program is effective in improving math skills. However, more rigorous research with a larger generalizable sample and randomized groups is warranted in order to determine the efficacy of this program.