Are you an educator troubled by the looming question of how AI will impact writing instruction? Look no further. Furthermore, as this experiment unfolds, you'll be interested in the findings' powerful implications on student creativity and expression. Prepare to be inspired, challenged, and equipped to navigate the future of writing in a world where humans and AI learn to work together to communicate in better ways as Steve Dembo takes us inside his classroom.
Microsoft 's Free AI Course for Educators
A Great Student AI Writing Experiment with Steve Dembo
Our Podcast is on YouTube!
Hi listeners! We're trying an experiment! For this season of the 10 Minute TEacher, we're experimenting with full video (and human-reviewed captions, of course). We will release the audio via podcast, and the video will just be released on YouTube.
I look forward to seeing how useful this is (or not) and with your feedback and the statistics we will know if it is worth the effort going forward. I always love to hear your ideas at vicki at coolcatteacher dot com!
Table of Contents
00:00:00:02 – 00:00:03:00
This is the Ten Minute Teacher podcast, with your host, Vicki Davis.
00:00:03:00 – 00:00:29:20
A student, AI writing experiment with Steve Dembo, Episode 809.
Microsoft Education: Free AI Course – Sponsor
Today's sponsor is Microsoft Education. Stay tuned at the show's end to learn more about their (free) Artificial Intelligence Course for educators.
00:00:30:09 – 00:00:34:14
The papers where you had the four different groups of people. Would you talk about that experiment?
The Writing Experiment That Matters
00:00:35:01 – 00:00:54:13
We wanted to sort of test out not just whether people could identify whether something was written by a bot or not. You know, that's what a lot of people are doing, is they're just saying like, here are five papers. Can you figure out which one ChatGPT did? It's all well and good. The question is, as a tool, did this help make you a better writer or not?
Group 1: Write the Paper with Just Grammatical Correction and Typical Resources They Would Use
00:00:54:21 – 00:01:18:16
Does the work that it put out, is it comparable to the work that you would normally do as a student or not? And so on? So we split up the class into four groups. We chose a topic that they were already familiar with, something that they read in language arts or studied in social studies. One group was assigned to just write the paper or a one-page paper using Google, using Grammarly, and using whatever resources they normally would use.
00:01:19:04 – 00:01:42:18
But you write in the traditional way.
Group 2: ChatGPT for Pre-Writing Only
The second group they were allowed to use ChatGPT, but we were specifically using ChatGPT just to make sure that everything was comparable. They were allowed to use it for any part of the pre-writing process so they could use it to get ideas, to get an outline, to find resources, to identify quotes, to get a hook for their paper, or any part of the pre-writing process they wanted.
00:01:43:12 – 00:02:06:17
But then they had to write it themselves. And once they wrote it, they couldn't use ChatGPT anymore.
Group 3: ChatGPT for Post-Writing Only
Third group was post-writing. So they had to write it by hand, write it the traditional way, but then they could use ChatGPT to their heart's content afterward. So they could use it to have it check their grammar, to check the spelling, to maybe to upgrade it from an eighth-grade level to a high school level or whatever they wanted.
00:02:06:17 – 00:02:24:21
They could have them modify in any way, but they had to write it first, and then they could use the bot.
Group 4: ChatGPT Only with No Human Intervention/Supervision
And the fourth group they were tasked with only using the bot. They could only do any prompts. They wanted as many prompts as they wanted, but ultimately they had to copy and paste. They could not make any changes.
After The Papers Were Written, Blind Evaluations Happened by Other Students to Rank the Best
00:02:24:23 – 00:02:45:10
And then we took those four papers, gave it to the opposite class, to the grade level. There are two classes per grade level. So they flipped, and they just had to read through the papers and rank them. Which one they thought was the best work, not whether they thought it was human or bot. Regardless of that, what produced the highest quality work?
How Was the Highest Quality Work Produced?
00:02:45:21 – 00:03:09:23
And interestingly, almost across the board, not 100% across the board, but pretty darn close to it. The human-written papers outperformed the bot-written papers. In fact, typically, the entirely bot-written papers tended to do the worst. And there are a number of reasons. There's a number that is a lot we could really dig into the whys of it.
00:03:10:03 – 00:03:29:03
And there's a lot of different, you know, elements of why the bot-written. Is it just that ChatGPT couldn't do it? Is it because they weren't using the right prompts? Is it because they just sort of like we talked about a little earlier, if it is really easy, do they just take the easy way out and say it looks good enough?
00:03:29:08 – 00:03:49:12
Did they even really read through their own essays? I don't know. I mean, so there's a lot that we could dig into with it. But the net result was across the board. Students tend to prefer or think that the student-written essays, as in the entirely student written essays and the pre group where they used ChatGPT
00:03:49:14 – 00:03:53:12
But then they still wrote it themselves. They tended to think it was better.
00:03:53:15 – 00:03:55:07
So the Post group was the third?
When AI Corrects Grammar, It Can Often Rewrite the Paper Unless Prompted Correctly
00:03:55:09 – 00:04:23:13
Post group came in third. And one of the reasons for that is it was entirely human written, and then they would put it in, they'd say, Hey, correct my grammar. Now when it corrects grammar, it's not just finding a grammar mistake and correcting it. It basically rewrote it. Ultimately, what they came to the conclusion when we were reflecting, there's a vast difference between saying, Find my grammar mistakes, identify my grammar mistakes and correct my grammar.
00:04:24:00 – 00:04:39:05
Identifying grammar mistakes was sort of like grammar check Google Docs or like handing it to a parent or a tutor and saying, “Can you proofread this?” Whereas “correct my grammar mistakes.” It was saying, “Hey, take this and punch it up for me.”
Why AI Experiments Make Sense in Classrooms Now
00:04:39:08 – 00:05:09:08
I love about what you're doing. We're experimenting with ChatGPT and AI with our students because we're really entering an unknown new territory and the way we move forward is together. Not, you know, you see it. Everybody needs to block ChatGPT.
Okay, well, so what's going to happen when this is built into Microsoft Word or Excel or PowerPoint? Like it's so it's all everywhere!
What Do Educators Do When AI is Everywhere?
00:05:09:08 – 00:05:10:23
What do you do when it's everywhere?
00:05:11:03 – 00:05:37:17
It's a Pandora's box. We started off this entire unit with me, just asking the students if you got assigned a paper and you wanted to cheat on it, could you? Could you figure out? How do you know how you could? And 100% of students' hands went into the air. The reality is they could cheat before. Those students who want to cheat are going to find a way.
00:05:38:00 – 00:05:55:15
Is this a better way for them to cheat? Yes, it is. It's a quicker, easier. It's a cleaner. It's a definitely a higher-tech way for them to cheat. But I think what we're going to find is the students who are choosing to cheat are probably the ones who are choosing to do it anyway or prone to doing it anyway.
Why Some Students Said ChatGPT Should Be Blocked
00:05:55:21 – 00:06:16:22
And most students, they don't want to. Most students do have a degree of ethics. So, you know, I would say maybe about a quarter of the students raised their hand and said that we think this should be blocked at our school because we're I'm not sure whether we can be trusted with it. Now, interestingly enough, it wasn't when I said so you don't think you can be.
00:06:17:03 – 00:06:35:07
Oh, no, no, I can be trusted. I don't think he could be trusted with it. It was sort of a compelling concern about their fellow students more than just themselves. The reality is they've been able to they could do it. They know how to do it. And we can't waste our time getting into this sort of cat and mouse sort of game.
00:06:36:02 – 00:06:40:03
It is what it is. And it is; it's too powerful to just completely ignore.
The Integrity Gap and the Replaceability of Humans
00:06:40:04 – 00:07:03:12
Well, as I tell my students, if you choose to allow bots to do your work, you're saying that you are replaceable and you're unnecessary and have blogged about what I call the integrity gap. So there's always been an integrity gap. There always has. I mean, in the 1980s when I was in high school, there was integrity gap because there were folks who were giving each other their math homework.
00:07:03:13 – 00:07:24:10
And teachers see it when the kids are acing their math homework and failing all the tests, it's you got an integrity gap here. You have these kids that are honestly, genuinely doing their work, and you have those that are just trying to quote, “get the A” without getting the knowledge. And this human intelligence is more important.
00:07:24:10 – 00:07:43:23
And human intelligence, when you have a person who has integrity, I believe is going to be much more magnified because, you know, there's a there's an old proverb, “a good name is better to be having great riches,” and if I know that Steve is the kind of person that he may use these tools, but it's going to be what he writes.
00:07:43:23 – 00:08:07:19
Like you have a voice; you have a way that you talk. Known you forever. You're the one who got me on Twitter. Because when Steve says, Hey, this is a big deal, I'm listening to Steve because I know him, right? And so helping our kids understand that their integrity means something even more than ever, I would say in the world of A.I., like, how do you talk about the need for honesty?
How Do We Discuss the Need for Integrity with Students?
00:08:07:19 – 00:08:14:08
Because, you know, kids are tired of “I need to be honest because you said so.” So how do we discuss this integrity gap?
00:08:14:17 – 00:08:33:09
I think I think you put it on them. Then ultimately, we wind up wrapping this unit up by having a, you know, an almost week-long discussion where the students, each class, crafted their own ethical use policy. You know, what is ethical? What is unethical. Where do you draw these lines? What we just talked about with that or having a paper proofread by a tutor.
00:08:33:15 – 00:09:00:19
Is that ethical? Where does it become not ethical? The students will have these remarkably deep conversations. They will really think critically about it when you give them not just the opportunity to, but when they know that it's authentic and meaningful.
Students Crafting Their Ethical Policy for the Use of AI
When they found out that they were crafting the policy that was actually going to be sent home and they were going to have to sign, they took it seriously because this does mean something to them.
00:09:00:19 – 00:09:22:01
This is the standard to which they're going to be held. They were the ones who were ultimately creating the rules for themselves. And my hunch is if you do the same exercise in 50 different schools, we're going to get very different results. But it's sort of like back when we were, you know, in the old days, we were talking about the value of creating a Wiki textbook.
00:09:22:08 – 00:09:47:01
The value of creating a wiki textbook was not to ultimately have a textbook that you could use for reference. It was the act of actually creating it that was so significant. And I think it's the same thing with these, you know, these AI tools, the action of going through and discussing how does this fit into my student experience, what is ethical, what isn't ethical, Where are these lines going to be drawn?
00:09:47:03 – 00:09:56:21
They may ultimately come to a very similar place as all those other students, but the act of going through it and getting to there is what's significant and meaningful.
00:09:57:02 – 00:10:22:18
Thank you to my friend Steve Dembo for sharing his knowledge about A.I. in the classroom, both on this episode and at ISTE. Now, if you want to hear the full original 35-minute live stream, then go to my YouTube channel at Cool Cat Teacher and you can hear the full conversation that we had. Also, Steve will be starting at a new school this fall.
00:10:22:19 – 00:10:52:22
Good luck with that, Steve, and you'll definitely want to be following him and learning about how he's using A.I. in the classroom. I just want to encourage you, Let's focus on those who are actually using AI and have the conversations we need to have so we can use this tool wisely.
Free AI Course From Microsoft for Educators
Microsoft has an amazing artificial intelligence course for educators that will help you and your education team have the knowledge you need to understand the best practices for artificial intelligence in schools.
00:10:52:22 – 00:11:23:21
AI is changing rapidly, and you'll want your team to take advantage of this free course from Microsoft today. Go to aka.ms/AIforEducators. Now that aka.ms/AIforEducators sign up for this free course today so that you're ready to understand and use AI in your school in ways that are safe and work for schools.
00:11:24:05 – 00:11:30:00
I highly recommend this free AI course from Microsoft. So sign up today.
00:11:30:00 – 00:11:41:19
You've been listening to the 10 Minute Teacher podcast. If you like this program, you can find more coolcatteacher.com If you wish to see more content like this, you can find her on Facebook and Twitter @coolcatteacher. Thank you for listening.
Disclosure of Material Connection: This is a “sponsored podcast episode.” The company who sponsored it compensated me via a cash payment, gift, or something else of value to include a reference to their product. Regardless, I only recommend products or services I believe will be good for my readers and are from companies I can recommend. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.”
Never Miss a Podcast Episode
Subscribe to get our podcast episodes by email.