1. Catch ChatGPT in mistakes
Think about a hobby or interest that you have, that you have experience with, and that you have strong opinions on. It might be strategies in some video game, types of dance moves in a particular style you do, regional cuisine from your grandparents’ village, which intersections in your hometown have the worst traffic, which classrooms at UCI are the best, whatever. Don’t pick a factual topic that you could learn about through reading books and papers - pick something that involves participating, or attending, or spectating, or being part of a community.
Start a new chat with ChatGPT (use a free account, not a paid one, see below). Come up with some question or argument on this topic and have ChatGPT summarize the arguments for or against. Carefully read its answer and see if there are any parts where you think it’s making mistakes. Think about which of the following types of mistakes it’s making:
- Hallucination - just making up things that are incorrect, that aren’t common beliefs
- Missing things - not including information that would be important to mention
- Perspective - giving one biased perspective rather than a general one, or conversely being too even-handed about perspectives that are incorrect
- Common misconceptions - repeating ideas that many people believe, though they are definitely incorrect
- Sycophancy - saying that you are totally right and your questions and comments are the best
- Others?
Don’t tell it about the mistakes, just keep asking follow-up questions, and see how far it digs itself into a hole, or if it fixes things. See how many different mistakes you can make it say. See if you can understand what sorts of things it is getting right or wrong - think about what you would say if a student turned this in for a class you were teaching.
At the top of the chat, there should be a button marked “Share” - click that, and “copy link”, and paste this link into a file. Also in this file, write a couple sentences explaining what kinds of mistakes ChatGPT is making. Submit that file.
If you haven’t used ChatGPT before
Go to https://chatgpt.com/ and create a free account using the burner e-mail address you created in class. Even if you have a paid account, log out of it, and use this free account. If you have a paid account and want to see the difference, you can try the same prompt you used above, and include the shared link to that one as well in the file you submit, as long as you clearly mark which is the free account, and which is the paid account, and include a few extra sentences commenting on the difference between the results.