My last post describes a Discord bot, named “crbot,” that I wrote for my college friends. Its name is short for “call-and-response bot.” It remembers simple commands that are taught by users. I patterned it after some basic functionality in a bot we have at Etsy. crbot has become a useful part of my friends’ channel. It’s been taught over 400 commands, and two of my friends have submitted patches.
But there were problems. We also used crbot to troll each other. Bug fixes were needed to curb bad behavior. The situation is better now, but I’ve had a nagging question: “should I have seen this coming, or could I have only fixed these problems by being reactive?” I couldn’t answer this without my friends’ perspective. So I asked them!
I was hoping for a small discussion. However, it blew up. With the bot as a staging ground, the subtext of the conversation was about our friendships. Many of us have known each other for a decade. Some, far longer. We’ve seen each other through major life events, like home purchases, children, and weddings. But I don’t remember any conversation where we’ve discussed, at length, how we perceive each other’s actions. And we only resorted to personal attacks once or twice. Go us!
To answer “Could I have seen this coming?,” we’re going to look at this in three parts:
- What happened? A story about how the bot was used and abused, and how it changed over time.
- What did my friends think? All of their insights from our discussion.
- Lessons learned. Could I have seen this coming?
I think it’s worth adding a disclaimer. These discussions have an implicit “within my group of friends” clause. The fixes work because my friends and I have real-life karma to burn when we mess with each other. Not because they’re some panacea.
What happened?
First bot version: ?learn, without ?unlearn
Channel #general
// Teach a bot a command.
Katie: ?learn brent https://www.example.com/brent.gif
crbot: Learned about brent
// Later, it's used.
Chris: my meeting went well today
Jake: ?brent
crbot: https://www.example.com/brent.gif
Discord unfurler: [gif of a kid giving a thumbsup]
Sidebar: Where did the word “unfurler” come from? Is it named because it “un-f”s the url? Or because it’s actually sailing related? The people need to know.
At launch, my friends’ initial reaction was mixed. Some immediately got it, and taught it inside jokes. Others said, “I don’t understand what this does, or why we’d want this.” One cleverly pointed it to a URL of a precipitation map, only to discover that Discord’s unfurler wasn’t re-caching the image within a useful timespan. By now, everyone has latched onto the bot’s ability to quickly recall gifs. Good enough for me.
At launch, crbot could learn new commands. But it could not forget them. This introduced a land grab, where my friends claimed as many names as possible. Scarcity encouraged usage, which was good for the bot. However, my friends took this opportunity to ?learn ridiculous or insulting things for each other’s names.
It got a little nasty. One message insulted someone’s intelligence. Another referenced the fact that someone is adopted. I’m speaking generically because the phrasing was over-the-top. There wasn’t a legitimate defense against this. You could claim all the commands that you might find insulting, if you were somehow able to foresee every way that you could be offended. For instance, people tried squatting on placeholders on the ?learns of their actual names and online handles. However, the bot is case-sensitive, so they’d need to protect all capitalized variations of their names.
If you’ve read “A Group Is Its Own Worst Enemy” by Clay Shirky, none of this is a surprise. Technology amplified our group’s negative interactions, and there was no moderation to fight back. The joy from trolling on crbot outweighed any reputation hit they took to their actual friendships, especially if they learned the commands in private. However, most of the things that I found really abusive were done in broad daylight.
global ?unlearn
To combat this, Ryan submitted the first user-contributed patch, to add ?unlearn. Now, users could make the bot forget responses.
Channel #general
// Show that the bot has not forgotten about dre
Jake: ?dre
crbot: I know who Dr. Dre is!
// Delete the ?dre command
Jake: ?unlearn dre
crbot: Forgot about dre
// The bot has forgotten about dre
Jake: ?dre
// No response
This helped a little. There was no longer an incentive to be the first to ?learn something. Now, you needed to be the last to learn it. This incentivizes high-quality ?learns. Your crummy commands are going to be replaced by better ones.
There was an adjustment period, where we figured out acceptable use. For instance, I replaced ?popcorn with a better gif, which touched off an argument about ?unlearn etiquette. There was a long sidebar about the life choices of people who post shitty gifs, when better ones exist. We settled on some guidelines, like “the person to ?unlearn a command should be the person who ?learned it.” We don’t always follow these rules. But it’s a good start.
?unlearn introduced a second problem. ?unlearn could be executed in a private channel. This introduced an attack where popular commands could be replaced in a direct message with crbot.
Direct Message with crbot
Attacker: ?unlearn ping
crbot: Forgot about ping
Attacker: ?learn ping Fuck you, ping yourself.
crbot: Learned about ping
Later, in #general
Jake: hey! there's a new release of the bot
Jake: ?ping
crbot: Fuck you, ping yourself.
Jake: :(
Jake: ?unlearn ping
crbot: Forgot about ping
Jake: ?learn ping pong
crbot: Learned about ping
Jake: ?ping
crbot: pong
As a design decision for crbot, I don’t log anything. Basically, I don’t want to respond to “who did X” bookkeeping questions, and I don’t want to write a system for others to do this. I don’t know who ?learned what, and I don’t care. This anonymity created a problem where there is no accountability for private ?unlearns. To this day, I still don’t know who did these. Nobody ever stepped forward. I would claim that I didn’t take part, but that’s what everybody else says, too
Public-only ?unlearn
We had a group discussion about ?unlearn, where I proposed that ?learn and ?unlearn could only be executed in public channels. My idea was that a public record would force everybody to properly balance the social forces at work. Andrew and Bryce argued that only ?unlearn should be prevented from being executed in private. This would force our real-life karma to be tied to removing someone else’s work. But ?learn should be allowed in private channels, since Easter eggs are fun. Plus, the command list is so large, that a new command will never be found, without being used publicly by the person who created it.
So, Ryan tweaked his ?unlearn implementation so it could only be executed in public channels. Now that a month has gone by, it has elegantly balanced ?unlearn and ?learn within our group of friends. The social forces at work have prevented further abuses of the system.
Hobbit bomb
One of our friends is often called a hobbit. I don’t know the details. Something about his feet.
Anyways, this led to ?hobbitbomb, which pastes a url of his picture. 48 times in one message. So, typing ?hobbitbomb once causes the Discord unfurler to inline the same image 48 times. The effect is that it takes up a massive amount of vertical screen real estate; it takes a long time to scroll past the bomb. It was used 7 times across a month (I used it a few of those times), and then effectively abandoned.
My friends’ reactions fell into 2 camps.
- So what?
- This makes Discord unusable. Also, this isn’t funny.
At one point, somebody decided that they’d had enough, and they ?unlearned ?hobbitbomb. The original poster, not to be deterred, created ?hobbitbombreturns, ?hobbitbombforever, and ?hobbitbombandrobin, all of which were duplicates of the original. A good meme has a healthy immune system.
Then, there was a lengthy detente, where the capability still existed to ?hobbitbomb, but nobody was using it. Finally, the command was brought up as a major source of frustration during our lengthy conversation on trolling in our channel. My friends settled on a social outcome: they limited the bomb size to 4 images (since 4 hobbits forms a fellowship). It still exists, it still requires scrolling, but it’s not extreme.
What did my friends think?
?unlearn abuse
Multiple patches were needed to balance ?learn and ?unlearn. Despite that, some people in the channel didn’t think that ?learn abuse was noteworthy. Their viewpoint was interesting to me. This required actual code fixes, so it must have been the biggest problem we faced. Right?
But when looking at individual instances, the problems caused by ?unlearn were minor. “I’m kind of annoyed right now,” or “I need to claim my username, so that nobody learns something weird for it.” This happened in low volume over time. For me, it added up to being the worst abuse of crbot. For other people, this was just something mildly annoying to deal with over time.
Hobbit bomb, and my own blind spots
Before the discussion, I hadn’t given ?hobbitbomb a second thought. “So what?” was my official position, and I wasn’t alone in having it. Scrolling seemed like a minor problem. But other people were seriously impacted. One friend felt it was the only abuse in the channel, and had to be reminded of all the ?unlearn abuse.
Before the bot, we had 2 tiers of users: moderators, and an owner. But when I added the bot to the channel, I created another implicit power position as the bot’s maintainer. I can change the bot to prevent behaviors from happening again. I can reject my friends’ patches if I don’t like them. I can still do private ?unlearn myself, since I have access to the Redis database where the commands are stored. And I can just shut off the bot someday, for any reason I want. This cuts both ways – I’m held in check, because the bot can be banned.
Anyways, the most interesting part of our discussion was finding out that I had a blindspot in how I handled this situation. I never thought that ?hobbitbomb was a problem, so I didn’t even file a bug ticket. I had been treating social problems like technical bugs, but this one hadn’t risen to the level of reporting yet. I needed to disconnect myself from my own feelings, and implement fixes based on my users’ complaints. As Chris put it, “the issue is its potential and how different people react to its use.”
Otherwise, you end up like Twitter, which had a long and storied harassment problem that has reportedly cost the company potential buyers. In my experience with crbot, users have great suggestions for fixing problems, and I’ve seen great user suggestions for Twitter. For instance, “I only want to be contacted by people who have accounts that are verified with a phone number. And when I block an account, I never want to see another account associated with that phone number.”
Technical solution, or compromise?
Since technical and social problems are related, I offered to fix ?hobbitbomb technically; I’d limit crbot’s output to a small number of URLs per response. ?hobbitbomb might be the only command that has multiple URLs per response, so it would have little impact. One of my friends pointed out that part of its utility is how annoying it is. So this would have the dual-impact of reducing pain, and reducing utility.
My friends rejected this offer, and decided to work towards a compromise. This was interesting to me; the core problem is still latent in the project. I may still implement my fix. If the bot were exposed to the public, I’d have to implement it, given how the Discord unfurler works. But on the other hand, I can think of a dozen ways to troll somebody with the bot, and I haven’t even finished my second cup of coffee. Plus, the premise of this whole chatroom is that we are friends. We have the option to create house rules, which might not be available in public forums.
During the discussion, we reduced the ?hobbitbomb payload from 48 to 4 images. This is enough to clear a screen, but doesn’t force people to scroll through multiple pages of hobbits. I don’t think that everybody was happy with this, since the ?hobbitbomb payload still exists. But both camps accepted it, and the great ?hobbitbomb war of 2017 was finally put to bed.
Social forces of friendship
Most of the problems with the bot were fixed with fairly light technical solutions, or house rules. For instance, public-only ?unlearn was the last time we saw ?unlearn abused in any capacity, even though there are still plenty of ways to cause mischief. And we have a few house rules; for instance, “don’t ?unlearn commands you didn’t ?learn.”
As Chris pointed out, this implies that everybody in the group assigns some weight to the combination of “we care about each other” and “we care about how we are perceived by each other.” This adds a hefty balancing force to our channel. It also means that all of my fixes for this channel are basically exclusive to this channel. There’s no way that this bot could be added to a public Discord channel. It would turn into a spammy white supremacist in 3 seconds.
Could I have seen this coming?
Or put a better way, “If I had to implement just one anti-trolling solution, before any of my friends ever used the bot, what would I implement?”
I imagined tons of problems that never arose. Nobody mass-unlearned all messages. Nobody mass-replaced all the messages. Nobody did a dictionary attack to ?learn everything. Nobody tried spamming the ?list functionality to get it blocked from using Github Gists. Nobody managed to break the parser, or found a way to get the bot to /leave the channel (not that they didn’t try). I didn’t need to add the ability to undo everything that a specific user had done to the bot. Once my friends saw the utility of crbot, there was little risk in the bot being ruined.
I did foresee spam as a problem. But I would have guessed that it’d be somebody repeating the same message, over and over again, to make the chat unusable. I never expected ?hobbitbomb, one message that was so large that some of my friends thought it broke scrolling. I’m not even sure this is fixable; even if I limited the number of images in a response to one, I imagine that one skinny + tall image can be equally annoying. I’m at the mercy of the unfurler here. Also, my traditional image of spam is something that comes in mass volume, not something that has a massive volume.
So, back to ?learn without ?unlearn. I should have seen that one coming. My idea was that this created scarcity, so people would be encouraged to use the bot. I didn’t imagine that people would use the opportunity to ?learn things that were abusive. Plus, the functionality for ?learn and ?unlearn are quite similar, so I could have quickly gotten it out the door, even if I still wanted to launch the bot with just ?learn. Launching without ?unlearn was too aggressive. Even with social pressures at work, we needed to have the ability to undo.
When reviewing the ?unlearn patch, I never guessed that private ?unlearn would be abused like it was. Honestly, a lot of this surprised me. This wasn’t even the general public; these were all problems that were surfaced by people I’ve known for a decade. If I can’t predict what they’re going to do, then it feels like there’s no hope to figure this out ahead of time, even if you have mental models like “private vs. public,” or “what is the capacity of people to tolerate spam?”
So my key takeaways from this project are pretty simple.
- Discuss bug fixes with impacted users. They have great opinions on your fixes, and will suggest better ideas than you had. Especially if the people are technical.
- Treat all user complaints like technical bug reports. Not just the ones you agree with. That doesn’t mean that all reports are important. But they deserve to have estimates for severity, scope of impact, and difficulty of the fix.
- Plan on devoting post-launch time to fixing social problems with technical fixes.Because you will, whether you plan on it or not.
- Every action needs to be undone. The most basic of moderation tools. Not even limiting the bot to my own friends obviated this.
- Balance public-only and public+private. Balance privacy and utility. When something involves your personal information, it should be default-private. When your actions interact with other users, it should be attributed to you.
Thanks to Andrew, Brad, Bryce, Chris, Drew, Eric, Katie, and Ryan for sharing their thoughts!