MultiLit is committed to the promotion of scientific evidence-based best practice. The development of our programs is predicated on the best available scientific research evidence and we endeavour to provide evidence for the specific efficacy of our programs as a result of our own research.
Do you have research evidence to support the efficacy of MacqLit?
MacqLit was developed over a period of more than 10 years. Initially, a rough translation of the MultiLit Reading Tutor Program was developed for group instruction purposes and employed in our tutorial centres. Efficacy data were collected over this period. Over time, the new group program evolved and again we collected efficacy data. Over the years 2010-2015, what has come to be known as MacqLit was developed and trialled and, again, performance data on its efficacy were collected.
Do you have evidence from large scale studies?
We have evidence of efficacy based on large samples of students participating in our tutorial centres.
Students attending the tutorial centres we ran for the Exodus Foundation in Sydney and Darwin over the period 2009-2011 were assessed at the start and conclusion of two-term periods of intensive literacy instruction that included an early version of the program. It should be noted that these students received three hours of instruction daily and that while instruction was centred on the MultiLit group instruction program, other programs were also used, including, for example, Spelling Mastery. Students also participated in individual Reinforced Reading sessions with volunteers at the centres. Instruction was delivered by highly trained tutors employed by MultiLit. The literacy assessments were carried out by testers employed by MultiLit, but who were independent of the instructional delivery of the program.
Three hundred and sixty-two students attended the programs at the various tutorial centres between 2009 and 2011 for whom data from pre- and post-testing were available. At program commencement, the average age of the students was 10 years, 5 months (125 months). Assessments carried out before the start of the program and students were tested again at the end of their second term in the program, after about 19 weeks of instruction, on average.
At the start of the program, the average crude reading age for this sample for reading accuracy was 89 months (7 years, 5 months) as measured by the Neale Analysis of Reading Ability and 85 months (7 years, 1 month) for reading comprehension, Students were three years below chronological age for reading accuracy and almost three-and-a-half years below chronological age for reading comprehension.
Highly statistically significant gains (p<.0005), based on analyses of raw scores, were made on all of the six literacy measures used: reading accuracy (Neale Analysis), reading comprehension (Neale Analysis), single word reading (Burt Word Reading Test), word spelling (South Australian Spelling Test), text reading fluency (Wheldall Assessment of Reading Passages (WARP)), and phonological recoding (Martin and Pratt Nonword Reading Test). The effect sizes were large (≥0.8) for all measures.
On average, these students made crude gains of:
- 13 months in reading accuracy
- 6 months in reading comprehension
- 14 months in single word reading
- 20 months in spelling
- 20 months in phonological recoding, and
- could now read 35 (50%) more words correctly per minute.
It should be emphasised that the results gained in this model were based on a three-hour intensive program that incorporated more reading and spelling practice than that offered in MacqLit. Although the program was very successful, it was acknowledged that schools would not be able to operate such a time-intensive model and for this reason adaptations were made to the program.
In 2014, a new model of program delivery was implemented in the tutorial centres. To provide a more cost-effective model and to reach more students in need of assistance, the number of students receiving instruction in the program was increased and program delivery was divided into two daily two-hour sessions, as opposed to only one previously. This was implemented with 20 students attending each session, one held each morning and one each afternoon for five days a week. Students received group instruction in decoding, sight words, spelling, vocabulary and comprehension, along with group reading and individual Reinforced Reading with volunteers at the centres. The number of hours of instruction was reduced from three to two hours per group. While most of the instruction involved an early version of MacqLit, additional programs such as Spelling Mastery were, again, still used at this point.
Full data were available for 74 students who received the modified group instruction program for 19 weeks during Semester 2, 2014. At program commencement, the average age of the students was 10 years, 4 months (124 months). The average (crude) reading age at the start of the program for single-word reading as measured by the Burt Word Reading Test was 93 months (7 years, 9 months); more than two-and-a-half years below chronological age.
Again highly statistically significant gains (p<.0005), based on analyses of raw scores, were made on all measures: single word reading (Burt), spelling (South Australian Spelling Test), phonological recoding (Martin and Pratt), reading comprehension (Test of Everyday Reading Comprehension;) and text reading fluency (WARP). The effect sizes were large (>0.8) for gains on all measures, ranging from 1.32 to 2.59. On average, these students subsequently made crude gains of:
- 12 months in single-word reading
- 14 months in spelling
- 18 months in phonological recoding, and
- could now read 22 (34%) more words correctly per minute.
These gains achieved by students on all measures provided sound evidence of the efficacy of the two-hour model of the group instruction program.
Do you have evidence from randomised control trials?
The evidence reported above, as noted, is based on uncontrolled trials of early versions of what became MacqLit, carried out for longer periods and employing additional instructional materials. We have also completed, with our former doctoral student, Dr Jennifer Buckingham, a modest but rigorous randomised control trial (RCT) of an early version of MacqLit. In this RCT, instruction was delivered daily for one hour comprising exclusively MacqLit group program prototype materials: no other instructional materials or programs were used in the trial.
The trial was carried out in a school in a socially disadvantaged area of New South Wales. Students deemed eligible for the program (being in the bottom quartile for reading performance) were randomly allocated to experimental (‘treatment’) and control conditions. The students in the experimental treatment condition received the program in small groups for an hour each day for three school terms, while the control group continued to experience their usual classroom literacy activities. At the end of three terms, the conditions were reversed: the original experimental group now received regular classroom literacy instruction while the original control group became experimental group 2 and received the program in small groups. Instruction then continued for another three school terms.
Students were tested initially on a battery of tests of reading and related skills, and then retested again after three terms, and then again after three more terms. Participant attrition occurred in both trials as a result of students no longer being of an appropriate age for the program or leaving the school to attend high school or for other reasons, a not uncommon occurrence with this demographic.
The detailed findings from the two phases of the crossover design study are reported fully in two articles by Buckingham, Beaman and Wheldall (Buckingham, Beaman & Wheldall, 2012; Buckingham, Beaman-Wheldall & Wheldall, 2014) and will not be reported in detail here. This RCT initially comprised 30 students with 26 students remaining for the second phase of the trial, equally distributed across the two conditions.
In the first phase of the study (three terms of instruction), the original treatment group (MacqLit) made greater gains than the control group and a large and statistically significant mean difference in gain between the groups was evident for the measure of phonological recoding (reading nonwords), with a large effect size (0.94). Minimal effect sizes were apparent for the other literacy measures employed and they were not statistically significant.
In phase two of the study, however, the original experimental group did not make further significant gains on any measure but the former control group, now experimental group 2, made very large (and significant) gains on all measures (with large effect sizes), clearly demonstrating the efficacy of the program (Buckingham, Beaman-Wheldall and Wheldall, 2014) (see figures 1 to 5 for details). Note that the teachers in this study were new to the program and initially inexperienced. We attribute the greater gains on all measures in the second phase of the study to their increased confidence and competence in the delivery of the program.
Are the research studies published in refereed academic research journals?
Yes some of them are and they are listed, and references provided, at the end of this document.
Do you have multiple independent studies testifying to the program’s efficacy?
At this point, the studies reported above constitute all of the efficacy data we have available, together with the field trial data reported below.
Do you have efficacy data from trials conducted by researchers independent of MultiLit?
At this point, there have been no studies carried out in MacqLit completed by other researchers. Following its publication, we would welcome other researchers to conduct efficacy studies on the program and will endeavour to facilitate their research.
Do you have other evidence of the specific efficacy of the program?
Immediately, prior to the program’s publication we were able to complete a field trial of the final (published) version of MacqLit carried out in a primary school in Sydney. Two experienced MultiLit tutors provided an hour’s instruction, using exclusively the MacqLit materials, to four groups of about four older low-progress readers, daily for two terms. All students were assessed on a battery of standardised and curriculum-based literacy measures before and after the approximately 20 weeks of instruction. The literacy assessments were carried out by testers who were employed and trained by MultiLit but who were independent of the instructional delivery of the program.
Complete sets of pre- and post-test data were available for 13 of the original group of 17 students, 5 boys and 8 girls. Six students were in Year 3, 4 students were in Year 4, 2 students were in Year 5 and one student was in Year 6. At program commencement, the average age of the students was 9 years, 1 month (109 months; ranging from 93 to 129 months).
At program commencement, the average (crude) reading age for reading accuracy was 87 months (7 years, 3 months) and the average (crude) reading age for reading comprehension was 79 months (6 years, 7 months). This is almost 2 years below chronological age for reading accuracy and two and a half years below chronological age for reading comprehension.
Following two terms (twenty weeks) of instruction, this group of 13 students had made crude average gains of:
- 8 months in reading accuracy
- 8 months in reading comprehension
- 12 months in single-word reading
- 14 months in spelling
- 32 months in phonological recoding, and
- could now read 30 (45%) more words correctly per minute.
Statistically significant gains (p<.001), based on analyses of raw scores, were made on all six measures: reading accuracy (Neale), reading comprehension (Neale), single word reading (Burt), spelling (South Australian Spelling Test), phonological recoding (Martin and Pratt) and text reading fluency (WARP). The effect sizes were large (>0.8) for gains on all measures ranging from 2.06 to 2.97.
Data on reading fluency was also collected fortnightly using the WARP Progress Monitoring Passages. The graph below (Figure 6) shows the average WARP score for the group at fortnightly intervals over the two terms. Performance increases steadily over the two terms, showing the cumulative effect of MacqLit instruction.
Summary statement regarding efficacy
As may be seen, the MacqLit small group instruction program has been shown consistently to deliver significant and substantial gains in literacy performance for older low-progress readers.
References to published efficacy studies
Buckingham, J., Beaman, R., & Wheldall, K. (2012). A randomized control trial of a MultiLit small group intervention for older low-progress readers. Effective Education,4, 1-26. doi: 10.1080/19415532.2012.722038 (Published 2013).
Buckingham, J., Beaman-Wheldall, R., & Wheldall, K. (2014). Evaluation of a two-phase experimental study of a small group (‘MultiLit’) reading intervention for older low-progress readers. Cogent Education, 1: 962786.
Limbrick, L., Wheldall, K., & Madelaine, A. (2012). Do boys need different remedial reading instruction from girls? Australian Journal of Learning Difficulties, 17, 1-15.
Wheldall, K. (2009). Effective instruction for socially disadvantaged low-progress readers: The Schoolwise program. Australian Journal of Learning Difficulties, 14, 151-170.
Wheldall, K., Beaman, R., & Langstaff, E. (2010). ’Mind the gap’: Effective literacy instruction for indigenous low-progress readers. Australasian Journal of Special Education, 34, 1-16.
Martin & Pratt Nonword Reading Test mean scores for Phase 1 experimental (E1) and Phase 2 experimental (E2) groups at pre-test, post-test 1 and post-test 2.
Burt Word Reading Test mean scores for Phase 1 experimental (E1) and Phase 2 experimental (E2) groups at pre-test, post-test 1 and post-test 2.
Wheldall Assessment of Reading Passages (WARP) mean scores for Phase 1 experimental (E1) and Phase 2 experimental (E2) groups at pre-test, post-test 1 and post-test 2.
Neale Analysis of Reading Ability (Accuracy) mean scores for Phase 1 experimental (E1) and Phase 2 experimental (E2) groups at post-test 1 and post-test 2.
Neale Analysis of Reading Ability (Comprehension) mean scores for Phase 1 experimental (E1) and Phase 2 experimental (E2) groups at post-test 1 and post-test 2.
Average WARP score at fortnightly intervals over the two terms.