Litigant Hid AI Prompts in Court Filings to Sway Ruling

A self-represented litigant, suspecting that a court was using artificial intelligence to help process filings, embedded hidden prompt injection text into his court documents in an attempt to influence the outcome of his case, according to a report from Ars Technica.

The report describes a judge who warned that pro se litigants, those representing themselves without an attorney, are increasingly turning to AI chatbots for legal help but often misusing the tools and growing desperate in the process. In this instance, the litigant went further than simply consulting a chatbot for advice. He reportedly inserted concealed instructions within his filings designed to manipulate any AI system that might process or evaluate the documents, in hopes of steering a favorable ruling.

Prompt injection is a technique in which hidden text or commands are embedded within content to manipulate how an AI system interprets or responds to it. The tactic has been documented in other contexts, including attempts to influence AI-based screening tools and automated review systems, but its reported use in a court filing highlights growing concerns among judges about how AI is intersecting with the legal process.

According to Ars Technica, the episode reflects a broader pattern in which judges are seeing more litigants without legal representation rely on AI tools, sometimes producing filings with fabricated case citations or other errors, and in this case attempting to directly manipulate any AI system involved in reviewing court documents.

Ars Technica did not name the court or the litigant in the portion of the report reviewed. The incident underscores mounting scrutiny from the judiciary over how generative AI is being used, and potentially misused, by parties navigating the court system without lawyers.

Based on reporting by arstechnica.com.