Back to index

Failure to Disrupt: Why Technology Alone Can’t Transform Education

Tags: #education #technology #innovation #social change #inequality

Authors: Justin Reich

Overview

In my book, Failure to Disrupt: Why Technology Alone Can’t Transform Education, I challenge the pervasive notion that technology will revolutionize education. While acknowledging the immense potential of learning technologies, I argue that they have consistently failed to live up to the hype of disruptive innovation. I attribute this to the enduring power of existing educational systems, which, due to their complexity and inherent conservatism, tend to absorb new technologies rather than be transformed by them. I trace the history of large-scale learning technologies, classifying them into three genres: instructor-guided, algorithm-guided, and peer-guided. Examining popular examples like MOOCs, adaptive tutors, and the Scratch programming platform, I illustrate how each genre has found a niche within existing educational systems but has not fundamentally altered them. Despite significant investments in these technologies, we haven’t seen the dramatic improvements in learning outcomes or access to education that many had hoped for. I delve into the reasons for this, identifying four intractable dilemmas that persistently plague learning at scale: the curse of the familiar, the edtech Matthew effect, the trap of routine assessment, and the toxic power of data and experiments. I explain how these challenges are not simply technical in nature, but are deeply intertwined with social, cultural, and political dynamics that shape our educational systems. My book is a call to educators, researchers, and policymakers to approach technology with a critical lens, recognizing its potential while also understanding its limitations. I advocate for a focus on tinkering and continuous improvement, embracing research and evidence-based practices, and building strong communities of teachers who are empowered to use technology in thoughtful and meaningful ways. While technology alone won’t transform education, we can leverage its power to create a more equitable and engaging future for learning by working collaboratively to address the persistent challenges of learning at scale. This book is written for anyone interested in the role of technology in education–teachers, technologists, school leaders, parents, and policymakers.

Book Outline

1. Instructor-Guided Learning at Scale: Massive Open Online Courses

This chapter explores the rise of Massive Open Online Courses (MOOCs), a technology touted as revolutionary for its potential to democratize learning. Early MOOCs boasted massive enrollment numbers but completion rates remained low. The core learning model of MOOCs is grounded in traditional instructionist pedagogy, which, combined with the challenges of automated assessment at scale, limits their ability to deliver on the initial promises of transformative change. Ultimately, MOOCs have primarily become supplemental resources for advanced learners rather than disruptive forces for systemic change.

Key concept: Three Big Bets for MOOCs: 1. MOOCs would transform the delivery model of higher education, led by a new generation of online providers. 2. MOOCs would dramatically expand global access to higher education. 3. Through research and continuous iteration, these new online courses would provide more engaging and effective learning experiences.

2. Algorithm-Guided Learning at Scale: Adaptive Tutors and Computer-Assisted Instruction

Algorithm-guided learning, epitomized by adaptive tutors and computer-assisted instruction (CAI), relies on algorithms to personalize instruction based on individual student performance. While this technology has a long history, its impact has been limited, primarily confined to mathematics and early reading, where automated assessment is more feasible. The most promising applications of adaptive tutors have been as supplemental learning tools in math classrooms, where they can lead to moderate gains in student learning.

Key concept: Reich’s Law: ‘People who do stuff do more stuff, and people who do stuff do better than people who don’t do stuff.’

3. Peer-Guided Learning at Scale: Networked Learning Communities

This chapter examines peer-guided learning, where learners navigate and learn from a network of experiences generated by a community of peers, as seen in early connectivist MOOCs (cMOOCs). This approach emphasizes social learning, interest-driven learning, and community building as central to the learning process. The challenge for schools adopting this model, as exemplified by the Scratch programming platform, lies in reconciling the open-ended, exploratory nature of peer-guided learning with the structured, assessment-driven culture of formal education.

Key concept: “Legitimate peripheral participation” is when a novice hangs out on the edge of a community of experts, looking for opportunities to move from the edge toward the middle.

4. Testing the Genres of Learning at Scale: Learning Games

This chapter explores learning games and the persistent challenge of transfer – the ability to apply knowledge and skills learned in a game to real-world contexts. Learning games generally fall into the three established learning-at-scale genres, but their success often depends on finding a balance between engaging game mechanics and strong connections to academic content. The chapter looks at examples like Math Blaster, Duolingo, Minecraft, and Zoombinis, each offering insights into the affordances and limitations of different approaches to learning games.

Key concept: “Chocolate-covered broccoli” describes learning games that add a layer of gamification elements, like points and badges, on top of traditional drill-and-practice activities.

5. The Curse of the Familiar

This chapter introduces the “curse of the familiar,” the tendency for technologies that replicate existing school practices to be more easily adopted but less likely to lead to meaningful changes in teaching and learning. This phenomenon arises because schools, as conservative institutions, often resist radical shifts in pedagogy. The chapter highlights how both charismatic technologists and educators contribute to this dynamic and argues that overcoming this hurdle requires fostering strong communities of teachers committed to pedagogical change.

Key concept: The “curse of the familiar” is the observation that technologies that look like typical elements in schools — like the practice problems on Khan Academy — scale much more easily than things that look very different from anything that has come before.

6. The EdTech Matthew Effect

This chapter explores the edtech Matthew effect, the phenomenon where new educational technologies, even those offered for free, tend to disproportionately benefit already-advantaged students. The chapter debunks three common myths about technology and equity: 1. Technology disrupts systems of inequality; 2. Free and open technologies will democratize education; 3. Expanding access will bridge digital divides. Instead, technology often reinforces existing social and cultural barriers to equitable educational opportunity.

Key concept: “For whoever has will be given more, and they will have an abundance. Whoever does not have, even what they have will be taken away from them.” (Matthew 25:29)

7. The Trap of Routine Assessment

This chapter explores the “trap of routine assessment,” highlighting how computers excel at assessing routine, structured tasks—tasks that are increasingly being automated. As a result, educational assessment often focuses on what computers can easily evaluate, neglecting more complex skills like unstructured problem solving and complex communication that are crucial for success in the 21st-century workforce. The chapter explores promising advancements in automated assessment, such as stealth assessment, but argues that relying solely on computational systems will continue to limit the scope of what we teach and learn.

Key concept: “Complex communication” are problems whose solutions require understanding a task through social interaction, or when the task itself involves educating, persuading, or engaging people in other complex ways.

8. The Toxic Power of Data and Experiments

This chapter addresses the potential and peril of data collection and experimentation in large-scale learning environments. The vast amounts of data generated by these systems offer valuable insights into learning processes, but they also pose risks to student privacy and autonomy. The chapter examines the history of educational data collection and argues for new ethical frameworks, such as contextual integrity, to guide data usage and privacy practices. It concludes by emphasizing the need for transparency, public engagement, and continuous evaluation to ensure that the benefits of data and experimentation outweigh the risks.

Key concept: “Toxic assets,” like radioactive materials, which can both save lives and cause cancer, must be used with great care.

-1. Conclusion: Preparing for the Next Learning-at-Scale Hype Cycle

Technology alone cannot solve the complex challenges of educational inequality, but technologies can contribute to system change. To mitigate the edtech Matthew effect, this chapter outlines four design principles for digital equity. By uniting around shared purpose, aligning home, school, and community efforts, connecting to the interests and identities of diverse learners, and studying and addressing the needs of specific subgroups, we can create technologies that work toward a more just and equitable future for learning.

Key concept: Four design principles for digital equity: 1. Unite around shared purpose. 2. Align home, school, and community. 3. Connect to the interests and identities of culturally diverse children and youth. 4. Measure and target the needs of subgroups.

Essential Questions

1. What are the core genres of learning at scale, and how have they impacted education?

Learning at scale encompasses a wide range of technologies and approaches that aim to reach large numbers of learners with limited expert guidance. The three core genres are instructor-guided (e.g., MOOCs), algorithm-guided (e.g., adaptive tutors), and peer-guided (e.g., Scratch). Each genre has its own strengths and limitations. While promising in theory, these technologies have largely failed to live up to the hype of revolutionizing education, instead often serving as supplements to existing practices rather than drivers of systemic change. Understanding these genres helps in evaluating new learning technologies and predicting their potential impact within existing educational systems.

2. What is the ‘curse of the familiar,’ and how does it impact the adoption of new learning technologies?

The ‘curse of the familiar’ refers to the tendency for educational systems to readily adopt technologies that replicate existing practices, while resisting those that require more substantial pedagogical shifts. This dynamic is driven by a combination of factors, including educator comfort with familiar routines, the constraints of standardized testing, and the pressure to maximize efficiency in resource-constrained environments. Overcoming this hurdle requires engaging educators in critical conversations about pedagogy, fostering communities of practice, and designing technologies that offer compelling entry points while also providing opportunities for deeper engagement and innovation.

3. What is the ‘edtech Matthew effect,’ and how can we address it?

The edtech Matthew effect refers to the tendency for new learning technologies, even free and widely available ones, to disproportionately benefit already-advantaged learners. This phenomenon is driven by a combination of factors, including disparities in access to technology and support resources, differences in home learning environments, and social and cultural dynamics that shape students’ engagement with learning opportunities. Addressing this challenge requires moving beyond a focus on expanding access to technology and toward a deeper understanding of the social, cultural, and economic barriers that limit equitable participation in learning at scale.

4. What is the ‘trap of routine assessment,’ and how does it impact teaching and learning?

The ‘trap of routine assessment’ highlights the limitations of using computers to evaluate complex human performance. Automated assessment tools excel at evaluating routine, structured tasks, but struggle to assess skills like critical thinking, creativity, and complex communication that are increasingly valued in the 21st-century workforce. This limits the scope of learning experiences that can be effectively measured and incentivized at scale, potentially narrowing the curriculum and reinforcing a focus on skills that are readily automatable. To move beyond this trap, we need to develop more sophisticated assessment tools and explore alternative approaches to evaluating learning, such as stealth assessment, that focus on capturing the process of learning rather than just the product.

5. How can we ethically and effectively harness the power of data and experimentation in learning at scale?

The abundance of data generated by large-scale learning environments offers researchers and educators unprecedented insights into learning processes. However, this data collection also poses significant risks to student privacy and autonomy. Striking a balance between maximizing the research benefits of data collection and minimizing the risks requires new ethical frameworks, like contextual integrity, that prioritize learner consent, data security, and responsible data usage practices. Transparency and public engagement are also crucial to ensure that learners and communities have a voice in shaping how their data is used and to build trust in these data-driven systems.

Key Takeaways

1. New is not always better.

Many new learning technologies simply digitize existing classroom practices, offering efficiency gains but little potential for transformative change. Understanding the history of education technology and recognizing common patterns of innovation and adoption can help educators make more informed decisions about which technologies are most likely to lead to meaningful improvements in teaching and learning.

Practical Application:

When evaluating new educational software, don’t just look at the flashy features or marketing claims. Dig deeper to understand the underlying pedagogical model and how it aligns with established teaching practices and research evidence. Consider whether the technology is truly innovative or simply a digital replica of existing methods.

2. Focus on what computers can’t do.

Computers excel at routine tasks that are increasingly being automated in the workforce. To prepare learners for a future where human work will be valued for its creativity, problem-solving, and communication skills, we need to design learning experiences and assessments that emphasize these uniquely human capabilities.

Practical Application:

When designing AI-powered learning systems, consider how to incorporate opportunities for learners to engage in complex communication tasks, such as collaborating on projects, explaining their reasoning, providing feedback to peers, or crafting persuasive arguments. Don’t just focus on tasks that are easily automatable.

3. Design with and for diverse communities.

Technology alone cannot address the complex social and cultural factors that contribute to educational inequality. To create truly equitable learning technologies, designers and developers need to actively work with diverse communities to ensure that their products are aligned with learners’ needs and values and do not inadvertently reinforce existing inequities.

Practical Application:

When developing an AI-based educational product, engage with diverse stakeholders, including educators, learners, and families from the communities you aim to serve. Conduct user research to understand their needs, values, and experiences with technology. Co-design features and functionalities with representatives from your target audiences.

4. Data is a toxic asset.

The vast amounts of data collected by large-scale learning platforms offer valuable opportunities for research and continuous improvement, but they also pose significant risks to student privacy. It is essential to treat learner data as a ‘toxic asset,’ to be handled with great care and ethical responsibility.

Practical Application:

If your company is collecting large amounts of learner data, implement robust data security measures, adopt privacy-preserving data storage and analysis techniques, and be transparent with learners about how their data is being used. Consider working with independent ethical review boards to guide data usage policies and practices.

Suggested Deep Dive

Chapter: The Trap of Routine Assessment

This chapter is particularly relevant to AI engineers because it explores the limitations of using computers to evaluate complex human performance, highlighting the challenges of developing assessment technologies that can reliably measure skills like critical thinking and creativity. This is a crucial consideration in the design of AI-powered learning systems, as the types of assessments we use shape the learning experiences we create.

Memorable Quotes

Introduction. 8

“The path of educational progress more closely resembles the flight of a butterfly than the flight of a bullet.”

Algorithm-Guided Learning at Scale. 54

“We think of it like a robot tutor in the sky that can semi-read your mind and figure out what your strengths and weaknesses are, down to the percentile.”

Peer-Guided Learning at Scale. 82

“Legitimate peripheral participation” is when a novice hangs out on the edge of a community of experts, looking for opportunities to move from the edge toward the middle—a kid hangs around the auto repair shop, watching the mechanics at work, until one day, a mechanic asks him to hold a bolt in place for a minute, and the next week he’s asked to actually tighten the bolt, then he’s hired a few hours a week, and from there, the journey commences.

Testing the Genres of Learning at Scale. 101

“Chocolate-covered broccoli” describes these kinds of games. The core activity in Math Blaster or XtraMath is no different from the core activity on a worksheet: solve arithmetic problems.

The EdTech Matthew Effect. 132

“For whoever has will be given more, and they will have an abundance. Whoever does not have, even what they have will be taken away from them.”

Comparative Analysis

Justin Reich’s Failure to Disrupt echoes arguments made by other prominent scholars who critique techno-solutionism in education, such as Larry Cuban (Oversold & Underused, The Flight of a Butterfly or the Path of a Bullet?) and Audrey Watters (Teaching Machines). All three authors caution against viewing technology as a panacea for educational challenges, emphasizing the complex interplay of social, cultural, and institutional factors in shaping teaching and learning. However, while Cuban and Watters often highlight the resilience of traditional schooling practices in the face of technological interventions, Reich offers a more nuanced perspective. He recognizes the potential of certain learning technologies, like adaptive tutors and peer-guided learning platforms, to modestly improve learning outcomes, especially when thoughtfully integrated into existing systems. His book is less a rejection of technology and more a call for pragmatism and strategic implementation grounded in research and a deep understanding of educational contexts.

Reflection

Justin Reich’s Failure to Disrupt offers a valuable and sobering perspective on the role of technology in education. While his critique of techno-solutionism is compelling, one could argue that he might be too quick to dismiss the potential for more radical, technology-driven transformation in the future. The history of technological innovation is full of examples where seemingly incremental advancements eventually led to profound societal shifts. Perhaps the true impact of today’s learning technologies will only become fully apparent in the decades to come. However, Reich’s focus on complexity, unevenness, and inequality in the adoption and impact of learning technologies is crucial. His work serves as an important reminder that technology is not a neutral force and that equitable implementation requires a deep understanding of the social, cultural, and economic contexts in which learning takes place. His call for continuous improvement, research, and community engagement is essential for ensuring that learning technologies are used to create a more equitable and effective future for education.

Flashcards

What are the three core genres of learning at scale?

Instructor-guided, algorithm-guided, and peer-guided.

What is the ‘curse of the familiar’?

Technologies that replicate familiar classroom practices are more likely to be adopted, but less likely to lead to meaningful change.

What is the ‘edtech Matthew effect’?

New educational technologies tend to disproportionately benefit already-advantaged learners.

What is the ‘trap of routine assessment’?

The tendency to focus educational assessment on what computers can easily evaluate, often neglecting more complex skills.

What is ‘contextual integrity’?

A framework for reasoning about privacy that considers the context of data collection and use, as well as user expectations and values.

What are the four design principles for digital equity?

Uniting around shared purpose, aligning home, school, and community efforts, connecting to the interests and identities of diverse learners, and studying and addressing the needs of specific subgroups.