Student AI Writing Experiment

The Great Student AI Writing Experiment with Steve Dembo

Discover the fascinating world of AI in student writing as Steve Dembo's students embark on a groundbreaking experiment! Uncover the incredible capabilities of AI and witness how it stacks up against authentic student work. But that's not all – delve into an inspiring project that encourages students to explore the ethical dimensions of AI in the classroom.

Are you an educator troubled by the looming question of how AI will impact writing instruction? Look no further. Furthermore, as this experiment unfolds, you'll be interested in the findings' powerful implications on student creativity and expression. Prepare to be inspired, challenged, and equipped to navigate the future of writing in a world where humans and AI learn to work together to communicate in better ways as Steve Dembo takes us inside his classroom.

Microsoft 's Free AI Course for Educators

Microsoft has an amazing artificial intelligence course for educators that will help you and your education team have the knowledge you need to understand the best practices for artificial intelligence in schools. AI is changing rapidly, and you'll want your team to take advantage of this free course from Microsoft today. Go to aka.ms/AIforEducators. I highly recommend this free AI course from Microsoft. So sign up today.
Click Here to Learn More
10 Minute Teacher Podcast - Episode 809

A Great Student AI Writing Experiment with Steve Dembo

Steve Dembo

Subscribe to the Show

10 minute teacher podcas audible

Episode 809: Video with Closed Captions - Cool Cat Teacher YouTube

Our Podcast is on YouTube!

Hi listeners! We're trying an experiment! For this season of the 10 Minute TEacher, we're experimenting with full video (and human-reviewed captions, of course). We will release the audio via podcast, and the video will just be released on YouTube.

I look forward to seeing how useful this is (or not) and with your feedback and the statistics we will know if it is worth the effort going forward. I always love to hear your ideas at vicki at coolcatteacher dot com! 

Episode Table of Contents

Table of Contents

Resources Mentioned in Episode 809

Steve Dembo - Bio as Shared on ISTE 2023 Website

steve-dembo

steve-dembo

Director of Digital Innovation
Western Springs, District 101
@teach42
@Teach42
Steve Dembo is the middle school Computer Science/Artificial Intelligence Teacher and Director of Technology at Quest Academy in Palatine, IL. He is the co-author of the book Untangling the Web: 20 Tools to Power Up Your Teaching and the president of the Skokie/Morton Grove District 69 School Board, serving his third term. Dembo is a dynamic speaker on the capabilities of social networking, the power of educational technologies and the potential for digital content to empower teachers to improve student outcomes.

Transcript - Episode 809 with Steve Dembo

00:00:00:02 – 00:00:03:00

John Davis

This is the Ten Minute Teacher podcast, with your host, Vicki Davis.

 

00:00:03:00 – 00:00:29:20

Vicki Davis

A student, AI writing experiment with Steve Dembo, Episode 809. 

 

Microsoft Education: Free AI Course – Sponsor

Today's sponsor is Microsoft Education. Stay tuned at the show's end to learn more about their (free) Artificial Intelligence Course for educators

 

We are kicking off 80 days of A.I. with my wonderful friend Steve Dembo @teach42. Now, you did an experiment that I really want you to talk about with your students.

 

00:00:30:09 – 00:00:34:14

Vicki Davis

The papers where you had the four different groups of people. Would you talk about that experiment?

 

The Writing Experiment That Matters

00:00:35:01 – 00:00:54:13

Steve Dembo

We wanted to sort of test out not just whether people could identify whether something was written by a bot or not. You know, that's what a lot of people are doing, is they're just saying like, here are five papers. Can you figure out which one ChatGPT did? It's all well and good. The question is, as a tool, did this help make you a better writer or not?

 

Group 1: Write the Paper with Just Grammatical Correction and Typical Resources They Would Use

00:00:54:21 – 00:01:18:16

Steve Dembo

Does the work that it put out, is it comparable to the work that you would normally do as a student or not? And so on? So we split up the class into four groups. We chose a topic that they were already familiar with, something that they read in language arts or studied in social studies. One group was assigned to just write the paper or a one-page paper using Google, using Grammarly, and using whatever resources they normally would use.

 

00:01:19:04 – 00:01:42:18

Steve Dembo

But you write in the traditional way. 

 

Group 2: ChatGPT for Pre-Writing Only

The second group they were allowed to use ChatGPT, but we were specifically using ChatGPT just to make sure that everything was comparable. They were allowed to use it for any part of the pre-writing process so they could use it to get ideas, to get an outline, to find resources, to identify quotes, to get a hook for their paper, or any part of the pre-writing process they wanted.

 

00:01:43:12 – 00:02:06:17

Steve Dembo

But then they had to write it themselves. And once they wrote it, they couldn't use ChatGPT anymore. 

 

Group 3: ChatGPT for Post-Writing Only

Third group was post-writing. So they had to write it by hand, write it the traditional way, but then they could use ChatGPT to their heart's content afterward. So they could use it to have it check their grammar, to check the spelling, to maybe to upgrade it from an eighth-grade level to a high school level or whatever they wanted.

 

00:02:06:17 – 00:02:24:21

Steve Dembo

They could have them modify in any way, but they had to write it first, and then they could use the bot. 

 

Group 4: ChatGPT Only with No Human Intervention/Supervision

And the fourth group they were tasked with only using the bot. They could only do any prompts. They wanted as many prompts as they wanted, but ultimately they had to copy and paste. They could not make any changes.

 

After The Papers Were Written, Blind Evaluations Happened by Other Students to Rank the Best

00:02:24:23 – 00:02:45:10

Steve Dembo

And then we took those four papers, gave it to the opposite class, to the grade level. There are two classes per grade level. So they flipped, and they just had to read through the papers and rank them. Which one they thought was the best work, not whether they thought it was human or bot. Regardless of that, what produced the highest quality work?

 

How Was the Highest Quality Work Produced?

00:02:45:21 – 00:03:09:23

Steve Dembo

And interestingly, almost across the board, not 100% across the board, but pretty darn close to it. The human-written papers outperformed the bot-written papers. In fact, typically, the entirely bot-written papers tended to do the worst. And there are a number of reasons. There's a number that is a lot we could really dig into the whys of it.

 

00:03:10:03 – 00:03:29:03

Steve Dembo

And there's a lot of different, you know, elements of why the bot-written. Is it just that ChatGPT couldn't do it? Is it because they weren't using the right prompts? Is it because they just sort of like we talked about a little earlier, if it is really easy, do they just take the easy way out and say it looks good enough?

 

00:03:29:08 – 00:03:49:12

Steve Dembo

Did they even really read through their own essays? I don't know. I mean, so there's a lot that we could dig into with it. But the net result was across the board. Students tend to prefer or think that the student-written essays, as in the entirely student written essays and the pre group where they used ChatGPT

 

00:03:49:14 – 00:03:53:12

Steve Dembo

But then they still wrote it themselves. They tended to think it was better.

 

00:03:53:15 – 00:03:55:07

Vicki Davis

So the Post group was the third?

 

When AI Corrects Grammar, It Can Often Rewrite the Paper Unless Prompted Correctly

 

00:03:55:09 – 00:04:23:13

Steve Dembo

Post group came in third. And one of the reasons for that is it was entirely human written, and then they would put it in, they'd say, Hey, correct my grammar. Now when it corrects grammar, it's not just finding a grammar mistake and correcting it. It basically rewrote it. Ultimately, what they came to the conclusion when we were reflecting, there's a vast difference between saying, Find my grammar mistakes, identify my grammar mistakes and correct my grammar.

 

00:04:24:00 – 00:04:39:05

Steve Dembo

Identifying grammar mistakes was sort of like grammar check Google Docs or like handing it to a parent or a tutor and saying, “Can you proofread this?” Whereas “correct my grammar mistakes.” It was saying, “Hey, take this and punch it up for me.”

 

Why AI Experiments Make Sense in Classrooms Now

 

00:04:39:08 – 00:05:09:08

Vicki Davis

I love about what you're doing. We're experimenting with ChatGPT and AI with our students because we're really entering an unknown new territory and the way we move forward is together. Not, you know, you see it. Everybody needs to block ChatGPT. 

 

Okay, well, so what's going to happen when this is built into Microsoft Word or Excel or PowerPoint? Like it's so it's all everywhere!

 

What Do Educators Do When AI is Everywhere?

 

00:05:09:08 – 00:05:10:23

Vicki Davis

What do you do when it's everywhere?

 

00:05:11:03 – 00:05:37:17

Steve Dembo

It's a Pandora's box. We started off this entire unit with me, just asking the students if you got assigned a paper and you wanted to cheat on it, could you? Could you figure out? How do you know how you could? And 100% of students' hands went into the air. The reality is they could cheat before. Those students who want to cheat are going to find a way.

 

00:05:38:00 – 00:05:55:15

Steve Dembo

Is this a better way for them to cheat? Yes, it is. It's a quicker, easier. It's a cleaner. It's a definitely a higher-tech way for them to cheat. But I think what we're going to find is the students who are choosing to cheat are probably the ones who are choosing to do it anyway or prone to doing it anyway.

 

Why Some Students Said ChatGPT Should Be Blocked

 

00:05:55:21 – 00:06:16:22

Steve Dembo

And most students, they don't want to. Most students do have a degree of ethics. So, you know, I would say maybe about a quarter of the students raised their hand and said that we think this should be blocked at our school because we're I'm not sure whether we can be trusted with it. Now, interestingly enough, it wasn't when I said so you don't think you can be.

 

00:06:17:03 – 00:06:35:07

Steve Dembo

Oh, no, no, I can be trusted. I don't think he could be trusted with it. It was sort of a compelling concern about their fellow students more than just themselves. The reality is they've been able to they could do it. They know how to do it. And we can't waste our time getting into this sort of cat and mouse sort of game.

 

00:06:36:02 – 00:06:40:03

Steve Dembo

It is what it is. And it is; it's too powerful to just completely ignore.

 

The Integrity Gap and the Replaceability of Humans

 

00:06:40:04 – 00:07:03:12

Vicki Davis

Well, as I tell my students, if you choose to allow bots to do your work, you're saying that you are replaceable and you're unnecessary and have blogged about what I call the integrity gap. So there's always been an integrity gap. There always has. I mean, in the 1980s when I was in high school, there was integrity gap because there were folks who were giving each other their math homework.

 

00:07:03:13 – 00:07:24:10

Vicki Davis

And teachers see it when the kids are acing their math homework and failing all the tests, it's you got an integrity gap here. You have these kids that are honestly, genuinely doing their work, and you have those that are just trying to quote, “get the A” without getting the knowledge. And this human intelligence is more important.

 

00:07:24:10 – 00:07:43:23

Vicki Davis

And human intelligence, when you have a person who has integrity, I believe is going to be much more magnified because, you know, there's a there's an old proverb, “a good name is better to be having great riches,” and if I know that Steve is the kind of person that he may use these tools, but it's going to be what he writes.

 

00:07:43:23 – 00:08:07:19

Vicki Davis

Like you have a voice; you have a way that you talk. Known you forever. You're the one who got me on Twitter. Because when Steve says, Hey, this is a big deal, I'm listening to Steve because I know him, right? And so helping our kids understand that their integrity means something even more than ever, I would say in the world of A.I., like, how do you talk about the need for honesty?

 

How Do We Discuss the Need for Integrity with Students?

 

00:08:07:19 – 00:08:14:08

Vicki Davis

Because, you know, kids are tired of “I need to be honest because you said so.” So how do we discuss this integrity gap?

 

00:08:14:17 – 00:08:33:09

Steve Dembo

I think I think you put it on them. Then ultimately, we wind up wrapping this unit up by having a, you know, an almost week-long discussion where the students, each class, crafted their own ethical use policy. You know, what is ethical?  What is unethical. Where do you draw these lines? What we just talked about with that or having a paper proofread by a tutor.

 

00:08:33:15 – 00:09:00:19

Steve Dembo

Is that ethical? Where does it become not ethical? The students will have these remarkably deep conversations. They will really think critically about it when you give them not just the opportunity to, but when they know that it's authentic and meaningful. 

 

Students Crafting Their Ethical Policy for the Use of AI

 

When they found out that they were crafting the policy that was actually going to be sent home and they were going to have to sign, they took it seriously because this does mean something to them.

 

00:09:00:19 – 00:09:22:01

Steve Dembo

This is the standard to which they're going to be held. They were the ones who were ultimately creating the rules for themselves. And my hunch is if you do the same exercise in 50 different schools, we're going to get very different results. But it's sort of like back when we were, you know, in the old days, we were talking about the value of creating a Wiki textbook.

 

00:09:22:08 – 00:09:47:01

Steve Dembo

The value of creating a wiki textbook was not to ultimately have a textbook that you could use for reference. It was the act of actually creating it that was so significant. And I think it's the same thing with these, you know, these AI tools, the action of going through and discussing how does this fit into my student experience, what is ethical, what isn't ethical, Where are these lines going to be drawn?

 

00:09:47:03 – 00:09:56:21

Steve Dembo

They may ultimately come to a very similar place as all those other students, but the act of going through it and getting to there is what's significant and meaningful.

 

00:09:57:02 – 00:10:22:18

Vicki Davis

Thank you to my friend Steve Dembo for sharing his knowledge about A.I. in the classroom, both on this episode and at ISTE. Now, if you want to hear the full original 35-minute live stream, then go to my YouTube channel at Cool Cat Teacher and you can hear the full conversation that we had. Also, Steve will be starting at a new school this fall.

 

00:10:22:19 – 00:10:52:22

Vicki Davis

Good luck with that, Steve, and you'll definitely want to be following him and learning about how he's using A.I. in the classroom. I just want to encourage you, Let's focus on those who are actually using AI and have the conversations we need to have so we can use this tool wisely. 

 

Free AI Course From Microsoft for Educators

Microsoft has an amazing artificial intelligence course for educators that will help you and your education team have the knowledge you need to understand the best practices for artificial intelligence in schools. 



00:10:52:22 – 00:11:23:21

Vicki Davis

AI is changing rapidly, and you'll want your team to take advantage of this free course from Microsoft today. Go to aka.ms/AIforEducators. Now that aka.ms/AIforEducators sign up for this free course today so that you're ready to understand and use AI in your school in ways that are safe and work for schools.

 

00:11:24:05 – 00:11:30:00

Vicki Davis

 I highly recommend this free AI course from Microsoft. So sign up today.

 

00:11:30:00 – 00:11:41:19

John Davis

You've been listening to the 10 Minute Teacher podcast. If you like this program, you can find more coolcatteacher.com If you wish to see more content like this, you can find her on Facebook and Twitter @coolcatteacher. Thank you for listening.

Disclosure of Material Connection: This is a “sponsored podcast episode.” The company who sponsored it compensated me via a cash payment, gift, or something else of value to include a reference to their product. Regardless, I only recommend products or services I believe will be good for my readers and are from companies I can recommend. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.”

Never Miss a Podcast Episode

Subscribe to get our podcast episodes by email.

Powered by ConvertKit
Facebook
Twitter
LinkedIn
Pinterest
Picture of Vicki Davis

Vicki Davis

Vicki Davis is a full-time classroom teacher and IT Director in Georgia, USA. She is Mom of three, wife of one, and loves talking about the wise, transformational use of technology for teaching and doing good in the world. She hosts the 10 Minute Teacher Podcast which interviews teachers around the world about remarkable classroom practices to inspire and help teachers. Vicki focuses on what unites us -- a quest for truly remarkable life-changing teaching and learning. The goal of her work is to provide actionable, encouraging, relevant ideas for teachers that are grounded in the truth and shared with love. Vicki has been teaching since 2002 and blogging since 2005. Vicki has spoken around the world to inspire and help teachers reach their students. She is passionate about helping every child find purpose, passion, and meaning in life with a lifelong commitment to the joy and responsibility of learning. If you talk to Vicki for very long, she will encourage you to "Relate to Educate" or "innovate like a turtle" or to be "a remarkable teacher." She loves to talk to teachers who love their students and are trying to do their best. Twitter is her favorite place to share and she loves to make homemade sourdough bread and cinnamon rolls and enjoys running half marathons with her sisters. You can usually find her laughing with her students or digging into a book.

All Posts »

Leave a Reply

[script_19]

This site uses Akismet to reduce spam. Learn how your comment data is processed.

The Cool Cat Teacher Blog
Vicki Davis writes The Cool Cat Teacher Blog for classroom teachers everywhere
-
00:00
00:00
Update Required Flash plugin
-
00:00
00:00