The "Ed Tech" Study and why this is important for Classroom 2.0

Yesterday I posted about the pre-release headlines on a USDOE study that concluded that certain educational software packages did not increase standardized test scores, and why this will hurt efforts to promote ALL technology use in classrooms.

Today the study is out and here’s the link. It's called: Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort. And guess what, the first sentence of the summary already says it’s about “education technology.” That’s just plain sloppy.

I've glanced through it, and I'll read it again thoroughly. It's pretty much what I guessed yesterday, big publishers with big products that claim to raise test scores in math and reading. The "student achievement" they tested was based on standardized tests.

Oh well, too bad, test prep software doesn't work and standardized tests are bad assessments. They deserve each other. I'm not crying for these companies who have been promoting their effectiveness in increasing test scores, or the districts who pay millions of dollars hoping for a swift solution to avoid the hard work of teaching and learning.

So on the one hand, I'm glad the USDOE had the guts to publish this study. On the other hand, they allowed the publishers to hide behind aggregated results.

Classroom 2.0 advocates have to be clear that what we are talking about are student (and teacher) empowerment and agency, not tools, or else it becomes muddled with everything else that plugs in in a classroom. We also have a duty to be critical of "solutions" offered by vendors that co-opt the language but don't deliver. We have to be willing to say that out loud online, in schools, and to people outside of our own circle of friends.

0 members like this

Comment by nlowell on April 6, 2007 at 4:28pm: I just finisthed reading the report. It seems pretty typical.

Another NSD (No Significant Difference) study that can't really say if it's because both conditions are really good, really bad, or really mediocre.

None of the differences in pedagogy were really addressed, either, other than a note that when the students were using the products, the teachers were less likely to lecture.

The fundamental flaw in the study is in this statement from Recruiting Districts and Schools for the Study segment:

"The experimental design provides a basis for understanding whether software products improve achievement. Teachers in the treatment group were to implement a designated product as part of their reading or math instruction. Teachers in the control group were to teach reading or math as they would have normally, possibly using technology products already available to them. Because the only difference on average between groups is whether teachers were assigned to use study products, test-score differences could be attributed to being assigned to use a product, after allowing for sampling variability."

The flaw in this statement is that *how* one uses a product can make such a large difference in the efficacy. A whole range of possibilities exist for how any given teacher in any given class might implement it. Things like class size, availability of computers, time frames, etc, can have an effect on the outcomes and simply averaging together the classroom results only serves to hide the actual findings rather than illuminating them.

And that's even before we get into the experiential differences across teachers and products.

If one of my grad students submitted this study for grade, I'd be very disappointed.

Comment by Sylvia Martinez on April 6, 2007 at 4:36pm: I'm glad to hear you say that, because although the research methods seem fishy to me, that's not my area of expertise.

Comment by Sharon Betts on April 6, 2007 at 4:47pm: There was an extensive discussion of this report on the Maine Assoc. of Computer Educators list today. We all agree with you - sloppy and not stated well at all. The equated educational technology with computer assisted instructional software. Ok, if you read the report - I was not pleased with the air it received on NBC this morning. I did find it interesting that the teacher they used as an example of what should happen was working with integrated whiteboards.

Comment by Alice Mercer on April 6, 2007 at 10:32pm: Now this is interesting, I've just been found by an edbloggger in South Africa (http://www.khanya.co.za/blogs/index.php?entry=entry070406-102846). He came to your conclusion Sylvia which suggests either he's pretty perceptive (probably), or that answer was pretty obvious.

Comment by Carolyn Foote on April 7, 2007 at 7:11am: I blogged about this as well, because of a situation that happened recently at my school.

Whenever an article like this appears, it's boiled down to the most sensationalistic "headline" version, and some teachers who are reluctant to try new technologies latch on to it as their justification for not learning something new.

More of my concerns here: http://futura.edublogs.org/2007/04/05/baby-with-the-bathwater/

Thanks for the links. I would like to follow up on the methodology.