PHILADELPHIA (WPVI) — The legal system is increasingly being challenged by the rise of AI-generated deepfakes.

Melissa Sims claims her former boyfriend fabricated AI-generated text messages, which led to her wrongful imprisonment.

“It was a nightmare,” she expressed.

Sims recounted enduring two harrowing days inside a Florida jail.

“It felt straight out of a show like ‘Orange is the New Black’,” she recalled. “I was thrown into the general population.”

Her ordeal has captured significant media attention in Florida.

Sims and her boyfriend had recently moved there from Delaware County, Pennsylvania.

She said her nightmare began in November 2024 after she called the police during an argument with her boyfriend, when she said he allegedly ransacked her home.

“Next thing I know, I’m looking at him and he’s slapping himself in the face,” she said.

She said he also allegedly scratched himself. When police arrived, they arrested her for battery.

As part of her bond, the judge ordered Sims to stay away from her boyfriend and not speak to him.

Fast forward several months, and she said her boyfriend created an AI-generated text that called him names and made disparaging comments.

“I end up getting arrested for violating my bond,” she said. “No one verified the evidence.”

Judge Herbert Dixon says Sims’ ordeal is one of increasing frequency.

“Several years ago, it started out with just fake audio recordings,” he said. “And now it’s gotten to a point of fake video and fake images being produced.”

Dixon is a senior judge on the Superior Court of the District of Columbia. He also serves as a member of the Council on Criminal Justice alongside former Philadelphia Police Commissioner Charles Ramsey. Its mission is to advance policy in the criminal justice system.

“One of the things we’re trying to do is to develop a framework for the responsible use of artificial intelligence,” he said.

He believes prosecutors and police need to do a bit more due diligence in this age of AI before they bring charges.

Drexel University professor Rob D’Ovidio teaches AI forensics.

“This is scary to say, but we’re no longer going to be able to trust what we see in front of us,” he said.

He said AI-generated video, texts and other evidence can be difficult to spot. AI is simply getting too good.

“The challenge is the detection tools are not keeping up with those capabilities,” he said.

As an example, he created an AI-generated photo to show us. He input it into three different well-known AI detection software programs. All three spit back different results that ranged from 1% to 62% probability of the photo being synthetic or AI-generated.

“The standard nowadays is we trust unless proven otherwise, right? I think we have to flip the script and distrust until we verify,” he said.

As for Sims, her story has a happy ending.

After eight months of legal wrangling by her attorney, prosecutors dropped the bond violation charge against her. And, last month, she went to trial on the battery charge and was acquitted.

Sims shared her story on her journey to advocate for a new law to create AI evidence standards and penalties.

“If this can happen to me, it can happen to anyone,” she added.

In July, Pennsylvania Governor Josh Shapiro signed a new digital forgery law that makes it a felony to create AI deepfakes that injure, exploit or scam in the state.

WPVI-TV reached out to Sims’ ex-boyfriend but has not heard back.