White Paper – Math Whisperer

Bernice German, Math Whisperer
John Tanner, Test Sense

Introduction

The Math Whisperer approach to mathematics offers an intervention program for students who struggle with math and/or are behind their grade level peers in mathematical achievement. Based on over twenty years of hands-on research, the approach has been refined into its present state. We call this present state our Research Edition and given its efficacy in classrooms chose to make it available to schools in a blended learning (books and online lessons) package.

Math Whisperer is based upon a developmental approach to mathematics that prioritizes specific mathematical concepts over all others. Over time and after much research as well as trial and error, approximately eighteen Critical Building Bocks (CBBs) were identified for their capacity to serve as a mathematical foundation. The hypothesis that these are in fact true CBBs has been proven repeatedly through research that showed the dramatic effect on a student’s learning when one or more of the CBBs shown to be missing from a students’ understanding of math was successfully inserted. Gains on standardized test scores for such students was often the equivalent of two to three grade levels after just a six month program that supplemented a student’s normal math sequence (German, Bruckhart, and Kiplinger, (2008), (Toker, Green, Vanchu-Orosco, 2014).

Math Whisperer is entirely unique in its approach, and also entirely unique in the results produced to date.

Approach

A number of countries have accomplished significant mathematics gains in recent years through a focus on a few concepts with depth of understanding as the goal, with the country of Singapore as a notable example. In the U.S. the paradigm for the past twenty years, including the recently developed Common Core, is coverage.  Evidence that the coverage approach is not optimal comes from twenty years of consistently flat scores on all measures.

Furthermore, our work with the concepts that eventually became the CBBs led us to realize that alignment with state standards posed an interesting scenario: the strongest match between each CBB and a state’s standards was often years prior to the student’s current grade level.  The CBBs and state standards can be considered as aligned, but significantly, the CBBs operate at a depth not generally indicated in state standards even in the years when they are taught. This means that teachers are prevented from teaching the CBBs at the depth required to generate understanding both at the grade level when a form of the CBB first appears, and in the current year of instruction where the CBB is presumed to be the responsibility of a prior teacher. This is clearly a frustrating predicament to those who teach that on the surface doesn’t offer a way out because this is a system problem.

The Math Whisperer work performed to date sidestepped the conversation about coverage and focused instead on the differences between students who possessed the CBBs and those who did not. We accomplished this using a simple survey that was found to have a high correlation with the presence or absence of the CBB, followed by an intervention methodology that supplemented a student’s normal instruction. During the pilot phase this sometimes occurred as extra time within the school day devoted to the CBBs, and sometimes as extra-curricular work assigned by the teacher. In a few instances the CBBs became the sole source for the curriculum.

The conclusion we came to as a result of the overwhelming evidence is that achievement in math can be accomplished at a greater rate with less effort than has been seen in the U.S. in over thirty years. It showed that students could be brought to grade level at a rate of two to three grade levels a year, even for those who are severely behind. But it also showed that even though we presented Math Whisperer as a remedial tool, its efficacy among students identified as high performing led to increases even for them. Thus a misidentification results in a positive benefit for the student, making it unique in the remedial landscape.

In short, the approach showed a very promising answer to a problem that has persisted now for a great many years.

Outcomes

Measurements of the outcomes of the Math Whisperer approach have occurred in a variety of ways, from ad hoc analysis in early pilots to rigorous statistical evaluations of various efforts. We offer a brief summary of these efforts below.

Many early pilots and many of the implementations since simply compared pre and post data and performed an ad hoc comparison to the overall population. In virtually every instance the results appeared to be overwhelmingly positive, and since Math Whisperer was the single intervention it was reasonable to infer that it was the Math Whisperer approach producing the effect. While caution must be exercised in interpreting such results given the lack of rigor in many of these analyses, the success of students in the programs when compared to what would otherwise have been expected of them was very encouraging.

More advanced studies such as one conducted at Pattonville School District in Pattonville, MO in the 2006-2007 school year approached a randomized control study in that middle school students were selected into various conditions by classroom/teacher that included no intervention and full teacher support. The Pattonville study and a number since show that students receiving the intervention achieved gains on standardized tests scores following a five and a half month intervention that were more than double the control group. The only difference between the two groups was the addition of the Math Whisper approach.

The 2006-2007 school year was also the first when we conducted studies that attempted to understand the effect of the Math Whisperer approach on state test scores. A study at Lincoln high School in Denver showed a dramatic increase on state test scores of students who participated in Math Whisperer. All students in the pilot had previously scored “low unsatisfactory” on the Colorado state assessment administered at the time (CSAP). The average scale score growth on CSAP for students in the pilot was 45 points compared with the average state growth that was 14 points at these lover levels.

In 2007-2008 70% of the high school students in a pilot again at Pattonville School District using the Math Whisperer approach showed significant growth on their Math MAP (Missouri Assessment of Progress) index scores compared to only 30% of students in the control group. As in virtually all the other plots the intervention lasted for five months and supplemented the regular instructional program.

In 2014 doctoral students at the University of Denver conducted a meta-analysis of some of the earlier pilot data to subject them to a more rigorous analytical approach. The idea was to determine if the gains seen by students were being properly ascribed to the intervention. Their conclusions were indeed that students who are behind grade level experience on average two to three years of growth in a five and half month intervention of the Math Whisperer approach, and that Math Whisperer provided a benefit even for students who were deemed to be on grade level (Toker, et. al.).

One additional pilot is also worthy of note. During the 2008-2009 school year Envision High Schools in the San Francisco Bay area asked us to focus on algebra-preparedness for students who were significantly behind their peers. At the time the 9th grade state exam was entirely algebra based. The pilot occurred at grade nine across four schools, two in San Francisco, one in Oakland, and one in Hayward.  Students received fast-paced lessons on CBBs from earliest grades, and then the focus changed to algebra only.

The result was that students grew one grade level, on average, between late August and February, as represented by NWEA MAP scores, which represented a significant improvement over the growth during the entire previous year. The only change in the math program was the inclusion of several Math Whisperer lessons for part of the school day on as regular a basis as the schools’ schedule allowed. The improvement among the students on the state test was from 5% of students scoring “basic” to 17% of students achieving the basic score far exceeded state averages for improvement.

Additional pilots and efforts since have shown the pattern regarding state test scores repeating itself in Missouri, California, and Colorado.  In all these instances—as well as in terms of the more recent Common Core and additional additions of standards, the common theme between the CBBs and the standards is their tie to the National Council of Teaching of Mathematics (NCTM) standards, which served as the foundation for each. Thus we feel confident stating that although we have not conducted formal studies in all fifty states, states that have standards that align with NCTM are highly likely to achieve similar gains. Our less formal work in classrooms when the approach was used as part of a normal course of study suggests this is indeed the case.

In summary, the Math Whisper approach with its focus on CBBs ,and the remedies for each, qualifies it at the time of this writing as a promising program from a formal statistical perspective.

Identifying the CBBs

Identifying the concepts that now qualify as a CBB represented an iterative process. The methodology involved finding correlations between simple open-ended items and candidates for CBB status (Kiplinger, 2009). Once the correlations were found the perceived gap was remedied with materials specific to that CBB and once remedied the student was again tested to see what sorts of overall gains resulted. Identification of each CBB came from selecting those with the highest correlation coefficients with state tests and the grade level NWEA Maps test.

In the end approximately eighteen concepts achieved CBB status. We say “approximately” given that our methodology never sought to “count” or “name” them. For example, what does one name the “thing” that exists when a student solves the problem, “what is 4 – 5 = ?” and answers: “1”? Is it that they lack an understanding of negative numbers? Or is the issue that they are missing a property of subtraction? We drew upon our teaching experience and the additional research opportunities we had to formulate the next step – lessons that would fill that specific gap.  For example, asking 4^th graders in a high performing school “4 – 5 = ?” helped formulate the strategy for this CBB.  Those students had no knowledge of negative numbers, but did know that subtraction was not commutative.  So a typical response from these students was:  That’s impossible.  We compared these responses to those from high school students from low performing schools, who typically answered “1.”

To determine which items would be selected as representing the CBBs we started by selecting short constructed response items and then correlating scores with those of state tests and the off-the-shelf tests for which we had data. Note that we found data from adaptive tests, most notably the NWEA Maps test, to be the most useful in this regard given its ability to provide reasonably precise estimates of achievement regardless of how far above or below grade a student was. We began with the assumption that if we could identify the underlying constructs represented in the tests we would have identified the various possible CBBs.

Our first attempts at creating the assessment showed correlation coefficients (Pearson’s r) that were at or about 0.6. While these were statistically significant they were not adequate for our needs. Subsequent improvements and modifications to the assessment led to correlation coefficients in the 0.9 range with 18 questions that took students five to ten minutes to answer (2009).

Our final assessment—a version of which we use to this day—showed a strong link between the  five to ten minute assessment with 18 questions and nationally recognized commercial tests as well as state tests in middle and high school that took four hours of student time for math alone. This link is of particular importance, not only so that we can declare alignment with state efforts, but also because it shows that prioritizing the resulting CBBs will at some point have a positive impact on state tests scores, which absent a major policy change remain prevalent source of concern for schools, low performing schools in particular. We say “at some point” for the simple reason that while the Math Whisperer approach regularly produces the equivalent of 2-3 years of academic growth, some students are so far behind that it will take several years of interventions to achieve grade level.

Some have been critical that a ten-minute survey be used to identify the absence of a CBB. However, the nature of the items selected is such that additional items add little to no value to the precision in the estimates until the number of items approaches that of traditional standardized tests, and yet the students are still directed towards certain CBBs in an almost identical manner. The increase in the statistical precision of an estimate does not lead to a meaningful difference as to which CBBs a student will be directed towards. We choose not to burden the student or the classroom any further than is absolutely necessary in order to obtain the data we need to target specific CBBs.

An interesting but unintended outcome of how the assessment was created is that if we chose to we can predict student scores on state tests, even from a test that takes just five to ten minutes. We do not formally report the predicted scores as doing so detracts from our overall mission and as is generally the case, scores with a predictive value in today’s world are often over or misinterpreted. Nevertheless, it speaks to the technical quality of the assessment.

Filling Gaps in the CBBs

The Math Whisperer program is designed to be supplemental to regular classroom activities. The approach recognizes the practical difficulties of substituting material in a current year’s standards with material that appears for all intents and purposes to have come from a prior year. Teachers using the Math Whisperer program have observed that at some time along the school year, as students master the missing CBBs  their belief in themselves as capable math students who expect to understand math even at grade level emerges.

To supply a student a missing CBB the Math Whisperer team created unique lesson sets for each CBB designed to develop a deep conceptual understanding of the CBB within the student. Each lesson set follows Piaget’s model of true understanding being built through the process of:

                                                                                          Concrete objects  →  Pictures  →  Abstract notation

Each CBB is introduced to the student with concrete objects, also known as manipulatives in math education parlance.  Students eventually move on to pictographic representations, using some that are provided for them but also learning to draw their own. In the end, students come to understand the meaning behind the abstract notation, or symbols as they are more commonly called, since it is at that level that most academic math is performed and students will be expected to excel at the notational level.

Practice that is designed to deepen the student’s understanding accompanies each activity.  All activities and practices are designed for the student to do, without teacher or parent modeling.  This leads to independent thinking and confidence on the students’ parts.  The activities and practices are specifically designed to be accessible to the student. Teachers can and should be active supporters, but for students who are behind their peers the more the student comes to see himself or herself catching up, the more likely they are to continue down that path.

Validation for each of the lessons was accomplished through a variety of analyses.  Pre and posttest results from parallel versions of the diagnostic showed evidence for when a gap had or had not been filled. Validation efforts included a brief analysis that showed that the results on the posttest were not random, i.e., it was important to note that when a student had not received instruction on a missing CBB that the student still shows a gap regarding the CBB. This would provide validity evidence that the absence or lack of the lessons being applied was a significant contributor to the result.

Perhaps one of the greatest arguments for the validity of the lessons as being capable of promoting the desired learning is in the evidence presented by students who were identified as being at grade level and still were steered into one or more of the lesson sets for a variety of reasons. Just like their underperforming peers, these students also experienced growth as represented in both off the shelf and state test scores.

Finally, we would again be remiss to suggest the Math Whisperer approach can remedy all CBB deficiencies in all students in a single year. The data suggest that for a majority of students performing below grade level it is reasonable to expect 1-3 years worth of growth on a standardized test, with a meaningful impact on state test scores as the student approaches grade level understanding. For some students who are severely behind, return to grade level will require multiple years of effort.

Summary

The identification of the CBBs and the process of filling them as described in the Math Whisperer approach offer a proven methodology for supporting teachers in the improvement of academic outcomes. Empirical proof that CBBs actually exist and that their absence results in serious learning deficiencies for students is an exciting development since we can now identify such deficiencies and then immediately begin the processing of teaching them to a student.

The Math Whisperer program offers a practical approach to remediating deficiencies in mathematical understanding.  The dramatic student growth of students who participated in the numerous pilots and studies to date is evidence that the approach is highly effective.

Bibliography

German, B., Bruckhart, G., and Kiplinger, V.L. (2008). Preliminary Evaluation of the Effectiveness of Peak Achievement’s Math Intervention System at Lincoln High School, Denver Public Schools, Peak Research Internal Research Report.

Kiplinger, V., (2009). Properties of Peak Achievement’s Diagnostic Assessment of Mathematics Gaps, Peak Achievement Research, Boulder, CO.

Toker, T., Green, K., Vanchu-Orosco, M., (2014). Evaluation Issues in Assessment: The Effects of a Math Achievement Program Based on Critical Mathematics Building Bocks, retrieved at https://turkertoker.wordpress.com/2014/10/06/effects-of-a-math-achievement-program-based-on-critical-mathematics-building-blocks-on-mathematics-test-scores-in-three-u-s-schools.

Toker, T., Green, K., Vanchu-Orosco, M., German, B., (2014). Evaluation Issues in Assessment: The Effects of a Math Achievement Program Based on Critical Mathematics Building Bocks, Poster, Monroe College of Education Research Methods and Statistics. AEA Conference, Denver, CO.

Vanchu-Orosco, M. (2014). Development and Validation of a Measure of Mathematics Deficiencies: The Critical Building Blocks Assessment. Research Proposal. Math Whisperer Archives.