Viral AI Tool Turns Out To Be Racist

Recently, people online have been asking an AI tool to see what it sees when it looks at their face. The results have been surprising, sometimes flattering, and often quite racist…

ImageNet Roulette uses a neural network to classify pictures of people uploaded to the site. You simply go to the site and paste the URL of a photo you want analysed (or just upload your own photo) and it will tell you what the algorithm sees in your photograph.

If you try it and get a bad result, remember that there are much worse things it can call you.

It’s also known to be quite insulting.

And as the title suggests, even slightly racist…

This tool was created by artist Trevor Paglen and co founder of New York University’s AI Institute Kate Crawford, and it uses an algorithm from one of the most “historically significant training sets” in AI. In 2009, computer scientists at Stanford and Princeton tried to train computers to recognise a wide range of objects. To do this, they amassed a huge database of photographs of everything from trees to goats. They then got people to sort the photos into categories.

The result was ImageNet.

“We want to shed light on what happens when technical systems are trained on problematic training data. AI classifications of people are rarely made visible to the people being classified. ImageNet Roulette provides a glimpse into that process – and to show the ways things can go wrong,” Paglen and Crawford explain on the tool’s website.

“ImageNet Roulette is meant in part to demonstrate how various kinds of politics propagate through technical systems, often without the creators of those systems even being aware of them.”

Essentially, the machines become racist and misogynistic because humans are racist and misogynistic.

“ImageNet contains a number of problematic, offensive, and bizarre categories, all drawn from WordNet. Some use misogynistic or racist terminology. Hence, the results ImageNet Roulette returns will also draw upon those categories.”