Tinder is asking its users a question we all may want to consider before dashing off a messageon social media: “Are you sure you want to send?”
The dating app announced last week it will use an AI algorithm to scan private messages and compare them against texts that have been reported for inappropriate language in the past. If a message looks like it could be inappropriate, the app will show users a prompt that asks them to think twice before hitting send.
Tinder has been testing out algorithms that scan private messages for inappropriate language since November. In January, it launched a feature that asks recipientsof potentially creepy messages “Does this bother you?” If a user says yes, the app will walk them through the process of reporting the message.
Tinder is at the forefront of social apps experimenting with the moderation ofprivate messages. Other platforms, like Twitter and Instagram, have introduced similar AI-powered content moderation features, but only for public posts. Applying those same algorithms to direct messages offers a promising way to combat harassment that normally flies under the radar—but it also raises concerns about user privacy.
Tinder leads the way on moderating private messages
Tinder isn’t the first platform to ask users to think before they post. In July 2019, Instagram began asking “Are you sure you want to post this?” when its algorithms detectedusers were about to post an unkind comment. Twitter began testing a similar feature in May 2020, which prompted users to think again before posting tweets its algorithms identified as offensive. TikTok began asking users to “reconsider” potentially bullying comments this March.
But it makes sense that Tinder would be among the first to focus on users’ private messages for its content moderation algorithms. In dating apps, virtually all interactions between users take place in direct messages (although it’s certainly possible for users to upload inappropriate photos or text to their public profiles). And surveys have shown a great deal of harassment happens behind the curtain of private messages: 39% of US Tinder users (including 57% of female users) said they experienced harassment on the app in a 2016 Consumer Research survey.
Tinder claims it has seen encouraging signs in its early experiments with moderating private messages. Its “Does this bother you?” feature has encouraged more people to speak out against creeps, with the number of reported messages rising 46% after the prompt debuted in January, the company said. That month, Tinder also began beta testing its “Are you sure?” feature for English- and Japanese-language users. After the feature rolled out, Tinder says its algorithms detected a 10% drop in inappropriate messages among those users.
Tinder’s approach could become a model for other major platforms like WhatsApp, which has faced calls from some researchers and watchdog groups to begin moderating private messages to stop the spread of misinformation. But WhatsApp and its parent company Facebook haven’t heeded those calls, in part because of concerns about user privacy.
The privacy implications of moderating direct messages
The main question to ask about an AI that monitors private messages is whether it’s a spy or an assistant, according to Jon Callas, director of technology projects at the privacy-focused Electronic Frontier Foundation. A spy monitors conversations secretly, involuntarily, and reports information back to some central authority (like, for instance, the algorithms Chinese intelligence authorities use to track dissent on WeChat). An assistant is transparent, voluntary, and doesn’t leak personally identifying data (like, for example, Autocorrect, the spellchecking software).
Tinder says its message scanner only runs on users’ devices. The company collects anonymous data about the words and phrases that commonly appear in reported messages, and stores a list of those sensitive words on every user’s phone. If a user attempts to send a message that contains one of those words, their phone will spot it and show the “Are you sure?” prompt, but no data about the incident gets sent back to Tinder’s servers. No human other than the recipient will ever see the message (unless the person decides to send it anyway and the recipient reports the message to Tinder).
“If they’re doing it on user’s devices and no [data] that gives away either person’s privacy is going back to a central server, so that it really is maintaining the social context of two people having a conversation, that sounds like a potentially reasonable system in terms of privacy,” Callas said. But he also said it’s important that Tinder be transparent with its users about the fact that it uses algorithms to scan their private messages, and should offer an opt-out for users who don’t feel comfortable being monitored.
Tinder doesn’t provide an opt-out, and it doesn’t explicitly warn its users about the moderation algorithms (although the company points out that users consent to the AI moderation by agreeing to the app’s terms of service). Ultimately, Tinder says it’s making a choice to prioritize curbing harassment over the strictest version of user privacy. “We are going to do everything we can to make people feel safe on Tinder,” said company spokesperson Sophie Sieck.
