Efficacy of collaborative strategic reading with middle school students

Vaughn, Sharon; Klingner, Janette K.; Swanson, Elizabeth A.; Boardman, Alison G.; Roberts, Greg; Mohammed, Sarojani S.; & Stillman-Spisak, Stephanie J. (2011). Efficacy of collaborative strategic reading with middle school students. American Educational Research Journal, 48(4), 938-964.

Here is an appalling example of how a quest for experimental rigor can have high costs in terms of ethics. It is a cautionary tale about how we must never let the desire for the “perfect” research study get in the way of what really counts: student learning, and respect for teachers and learners and the precious time spent in the classroom. Although I have no real problems with the treatment being tested here, which is a multicomponent instructional model involving explicit comprehension strategy instruction and learning in structured cooperative groups, I do have many problems with the way this study was conducted. I understand the problems of control in research that takes place in authentic classroom settings. However, the quest to solve those control problems does not justify the type of extreme interference in the classrooms that I believe happened with this research study. There are some good reasons why researchers choose to implement less rigorous studies in schools and classrooms, even though the choice of interfering less means that we don’t have quite as strong causal evidence as we would if we achieved more control through more interference.

The reader must look at the article himself/herself to decide if I am over-reacting here. I’ll summarize the areas in the study which made me uncomfortable (and one portion of the report which amazed and appalled me particularly). In the first place, the authors somehow gained enough access to three Colorado and Texas school districts that they were able to randomly assign middle school students to various middle school language arts teachers. We are given no information in this report as to how that access was obtained. Why would a school district allow a research team to presume to assign students to teachers, probably for a whole year, without regard for which kinds of placements and groupings would actually be most beneficial for the students? Random assignment here is the “gold standard,” the thing that provides rigor, and the thing that makes this an experiment from which we can infer the effectiveness of the treatment. But was student welfare taken into account at all here? The school districts obviously allowed this, but why? And what about parental consent, or teacher consent, or even student assent, which may be an issue with adolescent students? The study was funded by a government grant, and the researchers are all connected to universities, so I must assume that the study procedures were reviewed and approved by some sort of institutional review board. Maybe if I knew more about how access and consent were gained, I’d feel better, but no information is provided on that score here.

My next concern arose when I read that these randomly grouped classes were also randomly assigned to treatment (taught using the Collaborative Strategic Reading (CSR) model) and control (“business as usual” middle school language arts instruction) conditions. Random assignment is another hallmark of “rigorous” experiments, and though I personally would have difficulty in being told exactly how to teach my class, I understand why it is needed. Provided informed consent was obtained from all parties, that would not bother me. Even what seems like intrusive researcher observations to determine “fidelity” to the various treatment conditions, and requirements that the teachers log in every classroom activity they engaged in (to the minute!), though they make me uncomfortable (and I’d refuse to consent to such intrusions if I thought I had any choice—did these teachers?), still, if consent was obtained, I can live with all that.

The portion that stopped me in my tracks, however, was the discussion of how “contamination” (why can’t better words than this be used for such occurrences?) between the treatment and control conditions was minimized (see pp. 950-951). First, the authors conceded that some of the key elements of the CSR intervention (e.g., writing gist statements and working in groups) may also have occurred during “business as usual” instruction. I guess a teacher choosing to use effective teaching methods is a bad thing (contamination!) if that teacher happens to have the luck to be part of the control group. The researchers felt that to insure rigor, they had to prevent any contamination from happening if they could. I will here quote, in its entirety, what for me was the most appalling portion of the article, so the reader can make her/his own judgment:

“To address possible areas of contamination, teachers met with research support staff early and frequently during the study to help differentiate instruction provided in CSR and comparison classes. In addition, during each booster session, we clarified specific points related to contamination of the comparison group. In other words, we addressed the question, ‘What is and is not allowed in typical practice classes?’ Whereas versions of certain reading strategies were preexisting within some typical practice classes, through proactive training by research staff to implement the CSR strategies with high fidelity and to reduce instances of contamination, differential instruction for treatment students and comparison students with respect to the target instructional practices was achieved. (pp. 950-951).”

I’d like to be a fly on the wall during a “booster session” and hear how the researchers “clarified specific points” with control group teachers who dared to “contaminate” the control condition by doing some of the forbidden CSR practices in their classrooms. Then I’d fly to the teacher’s lounge and probably really get an earful. The authors seem to be saying that if I am a control group teacher, and I am observed having my students working in collaborative groups to generate gist statements, because I think that will help them learn, I will have my wrist slapped (or will have specific points “clarified” with me, as the authors put it), because I did not comply with “what is and is not allowed” for the teaching condition in which I was placed randomly and which I did not choose. What gives these researchers the right to dictate instructional practice like this, even if standard informed consent procedures were observed?

After all this interference in the lives of 17 teachers and 782 middle school students, though the authors claim a positive impact for CSR over “business as usual” instruction (based on effect sizes), in actuality the differences were small, and none achieved statistical significance. Even the authors, at the article’s very conclusion, question the actual value of their work. Here is the final sentence of the article, in its entirety; make your own judgment: “We think it is worthwhile to ask whether these multicomponent approaches like CSR are the most impactful ways of influencing comprehension or whether interventions that focus more on vocabulary building and/or background knowledge may be even more efficacious (p. 960).” Oh, I get it now—another day, another research study!