The difference between research and evaluation

An earlier post reviewed the recently published District Guide to Blended Learning Measurement from The Learning Accelerator, and promised a follow-up post regarding the distinction that the guide makes between research and evaluation.

This distinction is important, for reasons explained by Richard Culata and Katrina Stevens of the US Department of Education:

“Every app sounds world-changing in its app store description, but how do we know if an app really makes a difference for teaching and learning?

In the past, we’ve used traditional multi-year, one-shot research studies. These studies go something like this: one group of students gets to use the app (treatment group) while another group of students doesn’t (control group). Other variables are controlled for as best as possible. After a year or so, both groups of students are tested and compared. If the group that used the app did better on the assessment than the group that didn’t, we know with some degree of confidence that the app makes a difference. This traditional approach is appropriate in many circumstances, but just does not work well in the rapidly changing world of educational technology for a variety of reasons.”

These reasons that this approach often doesn’t work well are that research 1) takes too long, 2) is too expensive, and 3) is usually one-and-done, as opposed to being iterative. Instead, the authors write, “There is a pressing need for low-cost, quick turnaround evaluations.”

That is a good explanation of the shortcomings of formal research. The TLA guide takes the thinking even further with this table explaining research versus evaluation, which for simplicity I’m reproducing in full here:

As the table shows, compared to research, evaluations tend to be shorter, less expensive, and more closely tied to specific practices and outcomes. As the writers from the US Department of Education suggest, in the rapidly changing world of education technology, there is a greater need for short studies focused on specific online and blended learning programs.

The key question about technology in education is not “does it work?” The key questions are “can it work, and if so under what circumstances?” This is because blended learning and other technology implementations in education operate under many highly variable conditions related to teachers, students, instructional methods, and so forth. The technology being used is going to support a particular instructional approach, and variability in these other factors will almost certainly have a larger effect than the technology alone. But the technology is often important, even critical, when it allows a certain pedagogical approach to be employed.