What does learning science say about AI in the classroom?

Everyone has an opinion about AI in classrooms right now, but very few of those opinions are grounded in how kids learn.

In this episode, I sit down with Erin Mote, CEO and founder of InnovateEDU and co-founder of Brooklyn Laboratory Charter School, to cut through the noise and talk about what the evidence actually shows about AI as a learning tool.

This isn’t a conversation about ChatGPT, cheating, whether to ban devices, or what kids are doing with AI at home on their own time (which are separate topics worthy of their own discussion.)

We’re talking specifically about AI-powered tools designed for classroom use that are being pitched to your district and showing up in your school, and whether they hold up when measured against what we know about how kids learn.

In this episode, you’ll hear Erin share her thoughts on:

• Why “good enough” consumer tech is not good enough for students, and what to look for in tools that are actually built around learning
• The difference between sycophantic AI and Socratic AI, and why it matters for student outcomes
• What the early research is showing about AI as a learning tool, including what’s promising and what we still don’t know
• What to ask for and advocate for at the school and district level so these decisions aren’t made without you

If you’ve been feeling overwhelmed by the pace of change and unsure what to actually do, this episode will leave you with more clarity and more confidence in what you already know about how kids learn.

Listen to episode 351 below, or subscribe in your podcast app

Sponsored by the Stay Human Live Virtual Training

Why educators feel uncertain about AI right now

ANGELA: So, Erin, you recently published a piece in Forbes, and it opened with talking about how your dinner table has more chicken nuggets than the boardroom, but feels a lot alike lately. You’re the CEO of an ed tech nonprofit, and yet you still don’t feel like you know if you’re making the right calls for your own kids. What does that uncertainty feel like from where you sit?

ERIN: Well, you know, I wrote that piece in Forbes, Chicken Nuggets to Chatbots, and thanks to my editor for letting me use that headline. It really was a reflection of the conversations I’m having in boardrooms and advisory boards, in conversations with industry about how do we navigate tech use with young people, what’s developmentally appropriate, when should we be thinking about AI literacy, when should we be thinking about the use of these tools or not?

And at the same time, as a mom, just like probably many of your listeners who are parents, I am navigating questions about screen time and using AI tools in my household. One of the reasons I wrote that piece was to reassure people that even folks like me, who are 20 years in tech policy still have questions. As an industry, I think we have lots of questions and need research about generative AI and kids, and how they use it.

I also wanted to help people understand some of the choices I’m making with my own kids, which are side by side tech use, no cell phones, really closely monitoring use of apps and screen time, and also educating my 10-year-old about when he uses AI that it doesn’t always give you the right answer, that it’s like talking to an adult who will sometimes lie to you, and that when it starts to engage you in ways that feel like it’s trying to be your friend, to end the chat.

As parents, we can do a lot with parental controls and guardrails and guidelines, but the most effective thing we need to do is educate young people about this technology so they can be critical consumers of it and know when something isn’t real, or when a chatbot might be trying to become their friend, or when what they’re engaging with is not appropriate content.

I think we need to do the same for educators. Right now, we have a huge AI literacy gap where these tools are widespread in our classrooms, and yet access to professional learning opportunities is not ubiquitous. Many teachers don’t feel like they have access to those opportunities. So I hope it’s a clarion call for folks about how we need to call in AI literacy, and how we don’t just think about guardrails and guidelines from an industry or benchmark perspective, but what we need to do from a human perspective in terms of equipping humans with the skills to navigate this technology.

How old are your children?

Eight and ten, about to be eleven. Claire, my eight-year-old, has a really different relationship with technology and these tools.

Robert, my 10- almost 11-year-old, has dyslexia and dysgraphia and has been using word processing and technology as an on-ramp to accessibility in his own learning. He’s definitely more of a user. He also loves to design, build, and code games, loves to create with technology and programming. So he’s naturally inclined to use these tools, which makes it so important for me to really think through that literacy piece of being both a parent and an educator.

Yeah, there’s such a nuance to your response, and I really like how you’re considering the threat but also the promise. You have these limits, but you also see the potential. And I think that kind of nuanced response is so needed and also so difficult because people want easy answers. It’s like, “Well, let’s just ban it, no AI till 18”, or this is the future, and our kids are going to be left behind if we don’t go all in from kindergarten.

And we have to be developmentally appropriate. We as educators know a ton about the learning sciences and what the developmentally appropriate ways to engage here are.

As someone who was an educator leading a school through the widespread adoption of social media, I can raise my hand and say I think as a sector we abdicated responsibility. We said we’re going to block social media on school devices with filters, which is the right thing to do, and then we said, okay, that means it’s not something we have to deal with in schools.

I don’t think we did the job we should have done around media literacy and digital literacy when it came to social media. I don’t think we did the right calling in of families and communities. And so I’m really dedicated to that not being a mistake we repeat again here, because I think we all know we’re paying the consequences in a major way.

Yeah. And in some ways, this is even harder to wrap your head around because it changes so fast. Until November 2022, when ChatGPT was released, most of us were not thinking about generative AI in schools as something we’d have to deal with immediately, that it was going to change work processes and change the way kids approach their work pretty much overnight. And I think the fact that it continues to evolve so rapidly, people get scared, and they just shut down.

Yeah, and that’s a totally normal reaction. It’s actually physiological. When you’re overwhelmed, you come from a place of fear, and that’s the reason we all know the words fight or flight. Those are natural patterns when we’re overwhelmed by something.

I want to express empathy for folks who are feeling that way right now, and actually, that’s the vast majority of Americans. Only 25% of Americans are more hopeful and optimistic about this technology, according to Pew Research. So you’re not alone in that fear.

But I think you have to start trying to overcome it by thinking about how you build a set of skills that will stay constant no matter what the technology does or how it evolves. Can you interrogate the technology? Can you understand the inputs and the outputs? Can you critically evaluate what’s coming out of it? Does that sound the way I would sound?

There are just some basics that, no matter whether or not you decide to use this technology in your classroom, you need to understand about how it works so you can be equipped to make those decisions.

Why consumer AI isn’t good enough for schools

You wrote in your Forbes piece and also said in your keynote at the EdSafe AI event that good enough consumer tech is not good enough for our students. What is the difference between what we need to expect from a tool built for kids in educational settings and one built for general public use like ChatGPT?

So I think it’s really important we draw this distinction between consumer tech and ed tech. We just released a report at InnovateEDU, which took the top 150 ed tech tools and made this distinction. Purpose-built technology for classrooms is meant to elicit a learning experience rather than just engage.

We stacked out the certifications for privacy, security, interoperability, accessibility, and research-backed, evidence-based development of these tools. And in most domains, except where it was legally mandated for accessibility, consumer tech tools were in the single digits, zero to 2%, around these really critical domains, including evidence about whether they have a positive impact on student outcomes.

I’ll also say that when I look at consumer tech tools, particularly around chatbots and companions, as a parent and as an educator, I am not comfortable right now with the way they’re structured in terms of providing offboarding for students who might be using that chatbot and talking about bullying or suicidal ideation or other really sensitive matters.

I think we’re looking at a lot of consumer tools that don’t have sophisticated offboarding to put a young person who might be in crisis in connection with a human who can help support them. We know at least 25% of young people are using these tools to engage in conversation of an intimate nature without proper offboarding, without the emotional infrastructure they need. So right now, consumer chatbots and companions don’t belong in our classrooms.

Sycophantic AI vs. Socratic AI: What’s the difference?

What about AI chatbots that are tied to educational tools made for classroom use? How do you feel about those?

I think they have to be incredibly well-built and designed, aligned to the learning sciences. When we’re building these tools to engage in classrooms, we should be asking designers: Are you reminding young people that this is artificial intelligence? Are you scaffolding young people who might engage inappropriately to a human and keeping a human in the loop?

And then we can design tools that actually evoke support and motivation or evoke Socratic thinking. Right now, when you engage with a consumer chatbot, you might have had the experience where it says to you, “Wow, that’s such a smart idea”, “Let me help you develop this further”, “You’re brilliant”. If a chatbot is doing that or assuming human characteristics, it doesn’t belong in our classrooms.

But some chatbots simulate Socratic thinking, where a young person gives an idea, and the chatbot says, “Tell me more about that”, in a way that gets to the productive struggle of learning, doesn’t give them the answer, really pushes the dialogue back and forth. That’s the type of support and scaffolding that helps with motivation and engagement, helps young people work through an answer, might help them remember precursor knowledge that would be valuable in that moment, but does not give them the answer.

A really good way to test this is to deliberately give a chatbot the wrong answer and see if it affirms you. If it says that’s the right answer, that’s probably not something you want in your classroom.

What research says about AI and student learning

That framing of sycophantic versus Socratic is so brilliant, and I think it really resonates with educators because we understand the power of leading Socratic seminars and pushing thinking. When I look at the mixed results from studies about AI in classrooms, some show that AI will help a student get to a better finished product more quickly, but the same regions of the brain are not lighting up as if they were doing it on their own, and there’s no skill transfer. Just because they produce something better with AI ten times doesn’t mean the eleventh time on their own, they can do it. The studies that have shown a positive impact on student learning are what you’re talking about here with the Socratic method, where it’s pushing their thinking, asking them to explain, and making students do more of the work rather than just giving it to them.

Yeah, and the other thing we’re seeing from research — there’s a really good study out of Stanford — is that when AI tutors are used alongside active learning environments with a human teacher, it’s not necessarily moving core knowledge understanding but it’s moving motivation and engagement, which in the learning sciences can be powerful drivers of keeping young people engaged in the learning experience.

We’ve seen really interesting research where we put an AI tutor trained on a closed corpus of knowledge about a course side by side with a live active learning instructor, so a professor or TA, where there’s still a human in the loop, and the relationship-centered part of learning is still happening. But there’s a way for students to ask a question and deepen their knowledge so they’re not left behind as the content continues to move.

Those are things I get really excited about because I think about how we keep young people engaged in learning experiences — not just rote knowledge, but how do we help them understand what they know so they can take that knowledge and create with it. That is the transformation that can happen when we design learning experiences with intentionality anchored in what we know about the learning sciences and human relationships, and we see AI as a tool to help augment that active learning experience.

Why productive struggle still matters in an AI world

And I’m thinking about a student responding to this. The reason kids like AI to help with schoolwork is because it makes their work easier. It’s just going to tell them what they want to hear and give them answers, and that’s what kids like. And we’re talking about what is actually happening in schools. If kids are choosing to use ChatGPT at home as a friend or someone to talk to, that’s not within our control as educators. But what I hear you saying is that we’re talking about how we design learning experiences. Just because kids may be using AI in unsanctioned ways doesn’t mean we have to give up and say, Well, they’re just going to use ChatGPT so we might as well let them write everything with it. We don’t have the data to back that up as a proven instructional method. We don’t have that much time with kids. Do we want to let them just use ChatGPT to write during class time?

Yeah, all the learning happens in the first draft and the draft revisions. As an educator, I think there are ways to use AI tools to unlock learning experiences and dive deeper into passion-driven subject exploration. My son Robert does this all the time.

We’re building a desert ecosystem in our house right now and introducing new creatures into it every two weeks or so. We use AI to unlock and verify knowledge, like understanding when the harvester ant is ready to come out of the tube and go into the habitat. The learning experience Robert is having is around the ecosystem, around the project-based learning, around the experiential learning. He’s using AI the way I might have used an encyclopedia decades ago. It’s a knowledge source that still has to be verified.

I believe we’re in a space where we can help young people understand how to unlock what they want to create with AI. But if we try to simply take a rote memorization schooling experience and replace it with ChatGPT, I think this also helps us interrogate what the purpose of schooling is. What is the purpose of the learning experiences we’re creating in our schools? Because if you have the ability to access knowledge, then what do you do with that knowledge? How do you connect it across domains?

When people ask me what their kids should major in college, I say let me tell you all the things AI just can’t do, and those are the places you should be doubling down on skill development: debate and discourse, connecting knowledge across domains, interdisciplinary thinking.

The reality is this is an arrival technology, not an adoption technology. It’s going to massively disrupt existing systems and infrastructure. As we think about the teaching profession, staying anchored in relationships is the thing that actually moves the needle for kids. Understanding the learning sciences and then seeing AI as a tool in building and crafting those learning experiences, rather than something we have to program around or ban.

Where AI shows the most promise in education

How do you see AI being used well in classrooms? Are there any specific examples of ways it’s really moving the needle for kids in their learning?

I always come from an evidence base. I can think about anecdotal classrooms and districts I’ve visited where I see cool uses of AI, particularly in the creative arts and coding, but we don’t have a good evidence base yet about that so I’m not going to talk about specific tools or learning experiences.

But some of the places where I think we’re seeing positive outcomes that have early evidence: this idea of equipping young people with the ability to access prior knowledge and core knowledge alongside an active learning experience, using an AI tutor trained on a closed course syllabus or closed set of materials, what are called small language models, alongside an active teaching scenario. Building these really interesting hybrid experiences where the educator is focused on pushing the experiential and project-based learning and covering core knowledge, but not having to go back and reteach constantly, because there’s scaffolding and support for that.

The second place I’ve seen AI really improve performance is around just-in-time AI pedagogy interventions for educators and tutors who need support around prior knowledge, prerequisite skills, or differentiation.

And then at the high school and college level, I’ll name two places with incredible promise. The first is around making the invisible visible, giving young people access to information about financial aid, dual enrollment, internships. I’ve seen a number of AI tools that take the insurmountable job of a college counselor and make publicly available data about scholarships, resources, filling out the FAFSA, answering questions in real time. Those are amazing applications of generative AI in education, because we don’t have enough high school guidance counselors, so much access and opportunity have been driven by prior knowledge of where to find it, and it uses publicly available data with no PII.

And then at John Jay College in New York, along with Google, they deployed a student support generative AI tool that helped intervene for students who were getting off track, prompting them to go to office hours or get additional student support. It was really a generative AI tool about early intervention for students who might be missing classes or needed additional supports. Some of the promises right now is not just in classroom tools but in student support tools as well.

How AI can expand student support and access

That almost goes back to what we were saying earlier about chatbots, how kids are needing a mentor, they need advice.

Yeah, and this tackles something we’ve known in education for a really long time. At the charter school I founded, one of the things we did in New York City was make sure our high school guidance counseling ratio was 1 to 45 students. It took a massive investment, but we had 98% first-generation college students.

What we knew from evidence and research is that those students were going to need a special set of scaffolds and support to navigate the process of getting into a successful post-secondary pathway, from a financial resources perspective, from a college aperture perspective, understanding what was available to them and what they could afford.

What we know from research is that the first rung for a first-generation college student is the hardest to climb. Those are the places where I think we can take persistent problems that we know exist in our ecosystem and say, how could we intervene here so that young people have more access and opportunity?

Can schools use AI without making students guinea pigs?

I appreciate that you want to look for evidence-based practices and not just anecdotes. And I’m holding that with the tension of knowing that we won’t have any evidence if we don’t have some students be the guinea pigs for it. That’s so hard to sit with, that we don’t know the outcome of this and we’re using it in classrooms anyway. Particularly when we think back to the adoption of social media and cell phones, we have a whole generation of kids who were caught up in this new wave of technology, and we didn’t have the evidence yet. And I share your fears about doing that here. How do you navigate that tension?

Yeah, I do think we know some things. In literacy, for example, we know from the evidence base that we shouldn’t be trying to teach reading comprehension with digital textbooks or on screens, that when you use a book, reading comprehension is six to eight times better than if a child is trying to learn it on a screen. But decoding is a repetitive task where we’ve seen incredible outcomes with purpose-built educational technology. So there are things we can have educated hypotheses about when we think about learning experiences.

We know what pieces of the learning sciences help initiate productive struggle, help initiate metacognition, help get us to domains of learning. If we’re thinking about how we’re building tools, we need to be engineering benchmarks around what we know about the science of learning and development, and that can inform the way we use these tools in classrooms.

I don’t think we have to think about young people strictly as guinea pigs. We can lean on what we already know about neuroscience, about the learning sciences, about eliciting metacognition, to inform the way we build and architect these tools.

At EdSafe AI, we’re building right now a learning sciences benchmark for the foundational models. The reason we’re doing that is because all of the models right now are benchmarked against human replacement and task replacement. They’re not benchmarked against learning. So actually, the models are doing exactly what they’re programmed to do. We need to reorient the yardstick, and that’s why we’re building this learning sciences benchmark. We’ll release a paper in June and the benchmark in early fall, alongside learning scientists like Ryan Baker, Rose Luckin, and Pam Kanter, pioneers in this space, so that we can change the yardstick we’re using when we think about these tools in our classrooms.

And there have been some really exciting developments to interrogate these tools in a more rapid cycle way. In the UK, just in January, the government announced a 23 million pound investment in ed tech testbeds, rapid cycle trials of AI tools in classrooms, asking, Does this move the needle on student outcomes? Does this tool do what it says it does? And in 12-week cycles, they can have a pretty educated hypothesis. The same type of ed tech testbeds are happening in New York City, six tools focused on whether they’re anchored in purpose-built and research-backed design, in 12 to 15-week cycles rather than five-year RCTs.

And the work the Southern Education Foundation is doing around outcomes-based contracting, where districts enter into contracts with ed tech tool providers and only pay if they get the student outcomes the provider promised. Those are the procurement levers, research levers, and design levers we need to be pulling if we don’t want young people to have to wait and see whether these tools actually move the needle for learning.

Why learning science should guide AI adoption

That’s such an encouraging response, to know that there are things being done and other people prioritizing this. And I really love the framework of thinking about it through what we do know about neuroscience, what we do know about the science of learning, because I think that puts teachers back into a comfortable position. I am a subject matter expert. I know about this stuff. I understand how kids’ brains work, I understand how students learn, I understand what’s developmentally appropriate and what’s best practice in a way that’s unique to my experience because I’m living it every single day with these kids. And so if I understand the learning process, I can think about AI use through that lens rather than just what’s the new cool thing everybody’s trying.

Yeah, and it’s fun to play with image manipulation tools and make yourself into a raccoon, no doubt. But the thing I would say to educators and teachers in every classroom in this country is to have a foundational baseline level of AI literacy, no matter what. It’s not like text-to-speech replaced the need to read, and AI is not going to replace the need to read or write.

So have that baseline AI literacy so you understand how these tools work, that sometimes it’s like an adult who lies to you but seems really trustworthy. Test these tools, build that AI literacy, and then lean on the knowledge of your own pedagogy and practice. Think about the learning experiences you want to create and how you use AI to have peer-driven learning experiences, how you use AI to produce something that students then critique. There are lots of ways to think about the productive struggle and dissonance that really does generate authentic learning.

Educators know that, so remember it and lean into it. You’re not alone, but you do need that baseline AI literacy to meet this moment. And demand that from your principals and your districts. You shouldn’t have to do this on your own. You should be asking for resources and professional development and feeling empowered to do so.

If your district’s not doing that, there are resources out there from Google’s Coursera classes, ISTE, ASCD, and if you’re part of a union, NEA and AFT have resources. At InnovateEDU with EdSafe AI, we run National AI Literacy Day every March, and there are year-round lessons for your classroom as well as free synchronous and asynchronous professional development. You can put on your headphones, take a walk, and learn about the basics of AI literacy, listen to a student panel or parent panel about what they’re worried about. At ailiteracyday.org, we want to make it easier for you to find those opportunities.

What schools should tell families about AI

I’m glad you brought up parents and community. What conversations should schools be having with kids and families, and what should teachers help advocate for at the school and district level?

One of the things I always really encourage teachers and school districts to do is share with parents what technology and applications you’re using in your classrooms and schools. That transparency can be transformative in terms of the relationship, so parents can understand how these tools are being used or not used, and have visibility on the choices you’re making, like when you’re using a tool for something like decoding versus something that’s much higher level on the skill strand.

We have to be ambassadors of understanding with our parents and students, make sure we’re not in an adversarial relationship, and really have a calling in. At EdSafe AI, we have a number of resources, including one we developed with Common Sense Media and the National Parents Union: questions for parents to ask teachers about AI use in schools at parent-teacher conferences. I always tell educators and superintendents and state chiefs, can you answer those questions? It’s a really helpful activity to take those questions and see if, as an educator, you could answer them from a parent’s perspective.

We need to be having these conversations with parents, calling them in, providing resources. Through National AI Literacy Day, there are year-round resources where you can structure coffee conversations about what AI is, family learning opportunities where you take parents through basic activities to build understanding. You can do those activities with students too.

If we’re really reflective about what happened with social media, many of us would say we needed to do a better job of calling in our students and families. So let’s do that now. It’s not too late.

What teachers should ask for from district leaders

Yes, and you don’t have to do it alone. The resources are out there, the support is out there. And as you mentioned, it needs to be spearheaded at the school level so it’s not on the individual teacher.

Yeah, it should. We really need to be thinking about the structures we’re providing our educators, professional development, the types of space we make for educators to experiment with these tools.

And really importantly, I talk to educators all the time who are paying out of their own pocket so they don’t have to use free versions of these tools. That should not be the norm. I’m going to call in our district leaders and superintendents and say if you want AI and technology used in your classrooms, you’re going to have to pay for enterprise licenses.

That’s right, Erin. This has been a really positive conversation about a topic that can get dark and depressing very quickly.

I am someone who wants us to think about the promise and the peril of this technology and hold both of those at the same time.

I’d like to close out with a Takeaway Truth, something you want folks to remember in the week ahead. What is something you wish every educator listening understood about AI use in the classroom?

I actually want to say to every educator who’s listening, thank you. And just repeat: you’re not alone. There are a lot of resources out there to support you in this learning journey, including EdSafe AI and National AI Literacy Day. So do something. Get engaged, build that AI literacy. We’re here to do that alongside you.

Join me in July for this two day virtual training!