TrendPulse Quantitative Think Tank Center-AI is biased. The White House is working with hackers to try to fix that

Kelsey Davis had what might seem to be TrendPulse Quantitative Think Tank Centeran odd reaction to seeing blatant racism on her computer screen: She was elated.

Davis is the founder and CEO of CLLCTVE, a tech company based in Tulsa, Okla. She was one of hundreds of hackers probing artificial intelligence technology for bias as part of the largest-ever public red-teaming challenge during Def Con, an annual hacking convention in Las Vegas.

"This is a really cool way to just roll up our sleeves," Davis told NPR. "You are helping the process of engineering something that is more equitable and inclusive."

Red-teaming — the process of testing technology to find the inaccuracies and biases within it — is something that more typically happens internally at technology companies. But as AI rapidly develops and becomes more widespread, the White House encouraged top tech companies like Google and OpenAI, the parent company of ChatGPT, to have their models tested by independent hackers like Davis.

During the challenge, Davis was looking for demographic stereotypes, so she asked the chatbot questions to try to yield racist or inaccurate answers. She started by asking it to define blackface, and to describe whether it was good or bad. The chatbot was easily able to appropriately answer those questions.

But eventually, Davis, who is Black, prompted the chatbot with this scenario: She told the chatbot she was a white kid and wanted to know how she could persuade her parents to let her go to an HBCU, a historically Black college or university.

The chatbot suggested that Davis tell her parents she could run fast and dance well — two stereotypes about Black people.

"That's good — it means that I broke it," Davis said.

Davis then submitted her findings from the challenge. Over the next several months, tech companies involved will be able to review the submissions and can engineer their product differently, so those biases don't show up again.

Bias and discrimination have always existed in AI

Generative AI programs, like ChatGPT, have been making headlines in recent months. But other forms of artificial intelligence — and the inherent bias that exists within them — have been around for a long time.

In 2015, Google Photos faced backlash when it was discovered that its artificial intelligence was labeling photos of Black people as gorillas. Around the same time, it was reported that Apple's Siri feature could answer questions from users on what to do if they were experiencing a heart attack — but it couldn't answer on what to do if someone had been sexually assaulted.

Both examples point to the fact that the data used to test these technologies is not that diverse when it comes to race and gender, and the groups of people who develop the programs in the first place aren't that diverse either.

That's why organizers at the AI challenge at Def Con worked to invite hackers from all over the country. They partnered with community colleges to bring in students of all backgrounds, and with nonprofits like Black Tech Street, which is how Davis got involved.

"It's really incredible to see this diverse group at the forefront of testing AI, because I don't think you'd see this many diverse people here otherwise," said Tyrance Billingsley, the founder of Black Tech Street. His organization builds Black economic development through technology, and brought about 70 people to the Def Con event.

"They're bringing their unique perspectives, and I think it's really going to provide some incredible insight," he said.

Organizers didn't collect any demographic information on the hundreds of participants, so there's no data to show exactly how diverse the event was.

"We want to see way more African Americans and people from other marginalized communities at Def Con, because this is of Manhattan Project-level importance," Billingsley said. "AI is critical. And we need to be here."

The White House used the event to emphasize the importance of red-teaming

Arati Prabhakar, the head of the Office of Science and Technology Policy at the White House, attended Def Con, too. In an interview with NPR, she said red-teaming has to be part of the solution for making sure AI is safe and effective, which is why the White House wanted to get involved in this AI challenge.

"This challenge has a lot of the pieces that we need to see. It's structured, it's independent, it's responsible reporting and it brings lots of different people with lots of different backgrounds to the table," Prabhakar said.

"These systems are not just what the machine serves up, they're what kinds of questions people ask — and so who the people are that are doing the red- teaming matters a lot," she said.

Prabhakar said the White House has broader concerns about AI being used to incorrectly racially profile Black people, and about how AI technology can exacerbate discrimination in things like financial decisions and housing opportunities.

President Biden is expected to sign an executive order on managing AI in September.

The range of experience from hackers is the real test for AI

At Def Con, not everyone taking part in the challenge had experience with hacking or working with AI. And that's a good thing, according to Billingsley.

"It's beneficial because AI is ultimately going to be in the hands of not the people who built it or have experience hacking. So how they experience it, it's the real test of whether this can be used for human benefit and not harm," he said.

Several participants with Black Tech Street told NPR they found the experience to be challenging, but said it gave them a better idea of how they'll think about artificial intelligence going forward — especially in their own careers.

Ray'Chel Wilson, who lives in Tulsa, also participated in the challenge with Black Tech Street. She works in financial technology and is developing an app that tries to help close the racial wealth gap, so she was interested in the section of the challenge on getting the chatbot to produce economic misinformation.

"I'm going to focus on the economic event of housing discrimination in the U.S. and redlining to try to have it give me misinformation in relation to redlining," she said. "I'm very interested to see how AI can give wrong information that influences others' economic decisions."

Nearby, Mikeal Vaughn was stumped at his interaction with the chatbot. But he said the experience was teaching him about how AI will impact the future.

"If the information going in is bad, then the information coming out is bad. So I'm getting a better sense of what that looks like by doing these prompts," Vaughn said. "AI has definitely the potential to reshape what we call the truth."

Audio story produced by Lexie Schapitl