A Measurement Agenda: Role of Edtech Developers?

As the implementation of blended and personalized learning grows, questions about effectiveness are becoming relevant to more students and families, teachers, and classrooms across the nation. Researchers and think tanks cannot generate these answers alone. A broad group of stakeholders must play a critical role, and edtech developers have a unique position in their ability to both contribute to the evidence as well as drive the sector towards evidence-based implementation.

The more developers rely on and share evidence of their products’ effectiveness, the more districts and other prospective clients will demand and use evidence in their decision-making.

This is why The Learning Accelerator (TLA) released the Measurement Agenda for Blended Learning. In it, we include several objectives for learning about, disseminating, competence in, and implementation of evidence-based practices in blended learning. We developed these objectives for multiple stakeholder groups, including researchers, educators, developers (industry), and others. While it may be clear why we have identified measurement objectives for researchers (often the default stakeholder group for all things measurement), developers may wonder what they have to contribute to understanding the effectiveness of blended learning.

The Big Disconnect?

The fundamental challenge to implementing evidence-based practices in blended and personalized learning is a disconnection between what we call the “evidence” cycle, carried out by researchers; and the “implementation” cycle, carried out by educators and others. In an ideal world, the evidence and implementation cycles would be unified into one complete cycle. While no single stakeholder group can achieve the unification of these cycles on its own — the Measurement Agenda includes objectives for no less than five stakeholder groups — edtech developers can act as a bridge between the cycles. Developers are especially well positioned to benefit from and engage in both evidence generation (to better understand, and improve their products), as well as implementation (to ensure their products are used by those and in ways that best support teaching and learning).

The Unique Role of Edtech Developers

The fact is, developers occupy a unique position in the implementation space. Working alongside educators and other stakeholders, developers hold an essential part of the puzzle, beyond student data. Developers also have evidence about instructional design and strategies, without which we would not be able to understand all we need to about blended learning effectiveness. Perhaps more importantly, developers can further the demand for evidence-based approaches in the sector, by walking the talk and taking the lead on defining “success” of their products as “evidence of effectiveness.” Developers can take specific steps, which are described below, but in general they can lead by example in order to drive the market to demand evidence as a routine aspect of product marketing. Rigorous evidence takes time, so we should set our expectations accordingly, understanding that we all need to work together with a sense of urgency in order to generate and implement the evidence our teachers and students need.

There are two approaches developers can use to improve our collective understanding of if, how, and when blended learning is effective:

1. Take an evidence-based approach to instructional design, product development, and marketing — developers interested in creating products for blended and personalized learning should proceed, not wait for perfect evidence of impact. However, they should implement in an evidence-based way, by drawing on the existing evidence about learning sciences, instructional design, personalized learning, and learning for mastery.

2. Encourage measurement of their products, both for product improvement and to add to the evidence and their marketing base — developers should open up their learning environments to enable measurement of blended learning implementation, and share what they find with prospective customers and others.

Four High-Impact Ways for Edtech Developers to Contribute

Developers of digital learning environments to be used in blended settings are critical in ensuring that evidence-based practices and decisions are supporting their learning communities. The community members and industry objectives included in our Measurement Agenda outline the knowledge and skills necessary for developers and others to support the effective implementation of evidence-based practices and to ensure data from their learning community is considered in the generation of new evidence. The more developers rely on and share evidence of their products’ effectiveness, the more districts and other prospective clients will demand and use evidence in their decision-making.

Specifically, here are the four highest-impact ways edtech developers can contribute:

1. To ensure that we are learning from all of the evidence we can, we urge developers and others to take a data-driven approach to understanding what is and isn’t working for teaching and learning in their community. As with educators, developers can rely on different research designs, measures, and methods are used to increase their level of confidence in their own findings (internal validity); understand more sophisticated implications about cause and effect (rigor); and generate findings that are applicable to broader classrooms and contexts (external validity).
2. Developers should also support the dissemination of evidence-based practices by demanding as well as enacting a data-driven approach to design and implementation, and by using an evidence-based framework for determining which instructional strategies within their digital learning environments best support effective teaching and learning in their community, for their students and teachers.
3. In order to further our understanding of effectiveness, developers must develop competence in novel skills to be able to apply the relevant evidence-base when designing and reviewing their products with the goal of maximizing the supports for effective teaching and learning, and facilitating data-based decisions for improvement in their learning community.
4. Finally, implementation of evidence-based practices is supported when developers demand continued measurement of blended learning practices (including their products) on an ongoing basis, and that appropriate adjustments be made as necessary. When developers and others seek and foster sharing of findings outside of their own learning communities, and encourage participation in measurement activities that can support causal claims and be more broadly applied to varying teaching and learning contexts, implementation of evidence-based practices will grow, and students will benefit.

Developers and other vendors or community members who are interested in contributing to our shared understanding of blended learning effectiveness can find details about specific objectives in our Blended Learning Measurement Agenda for Community Members and Industry.

Who Else Plays a Role in Measuring Blended Learning Success?

As mentioned above, edtech developers are necessary, but not sufficient, stakeholders to advance our understanding of blended learning effectiveness. The Measurement Agenda also includes specific, detailed objectives for researchers, educators, administrators and policymakers, and funders – this work applies to all of these groups, and all are necessary to complete this work. Implementing evidence-based practices requires these multiple stakeholders to take coordinated action utilizing a breadth of knowledge and skills traditionally associated with very different roles and responsibilities. We recognize the magnitude of the task, but we know that together, it can be accomplished.

If you are a member of any of these stakeholder groups, and are interested in learning more about how you can contribute, please stay tuned for future articles, or visit our Blended Learning Measurement Agenda landing page for more information. If you are engaged in work that illustrates or aligns with objectives outlined in our Measurement Agenda, please let me know @EdResearchWorks, #BLMeasurement or email me.