Can AI be trusted in court? Supreme Court’s warning opens debate on regulating legal AI
Indiadailyupdate.com – Recent revelations about the misuse of artificial intelligence in legal proceedings have sparked a critical conversation about its role in judicial systems. When the Supreme Court uncovered that two insolvency tribunals had cited case laws that didn’t exist, the incident was more than a simple error—it revealed a deeper issue: the growing reliance on AI for legal research and the potential consequences when its inaccuracies slip into court decisions.
The National Company Law Tribunal (NCLT) and the National Company Law Appellate Tribunal (NCLAT) were compelled to retract their rulings after the Supreme Court identified fabricated legal references. These made-up precedents, likely the result of AI-generated hallucinations, have now become a focal point for legal professionals in India. The case underscores the risks of unregulated AI use, particularly in contexts where accuracy is paramount.
A Growing Reliance on AI Tools
As AI becomes more integrated into legal workflows, tools designed to summarize judgments, draft pleadings, and streamline research are being widely adopted. While these systems offer efficiency, they also highlight a critical flaw: their ability to produce convincing yet incorrect information. This capability raises concerns about the reliability of AI in shaping legal arguments and decisions.
“Every court, law firm, and regulator is vulnerable when legal research is handled by systems prioritizing plausibility over truth,” said Saakar Yadav, founder of Lexlegis AI. “The future of legal AI depends on verified legal corpora, not just larger models.” His statement reflects a broader agreement among experts that AI should support, not substitute, the meticulous verification required in legal practice.
The incident has shifted the discourse from technology itself to the question of accountability. If a citation fabricated by AI appears in court records, who is responsible? The lawyer using the tool, the judge relying on its output, the tribunal reproducing it, or the company behind the AI model? This ambiguity has intensified calls for clearer guidelines on AI’s use in judicial settings.
Unlike traditional legal databases that index authenticated judgments, generative AI systems focus on predicting likely word sequences. This design choice makes them prone to inventing precedents without rigorous checks. Human oversight, therefore, remains essential to ensure the integrity of legal citations.
India’s Regulatory Gap
The controversy has exposed a lack of formal regulations in India’s legal system. Currently, there are no rules mandating disclosure of AI’s role in preparing legal submissions or standards for verifying AI-generated research before its use in court. In contrast, countries like the United States, Canada, Australia, and the United Kingdom are already implementing measures to address these risks.
Some U.S. judges now require lawyers to certify that all citations are independently verified, while AI-generated content must undergo human review. India’s legal experts suggest similar safeguards could be introduced, such as mandatory checks for AI citations, transparency in drafting processes, and training programs for legal professionals on AI capabilities and limitations.
The debate extends beyond efficiency. Judicial decisions rely on the legitimacy of accurate legal interpretation. If AI-generated authorities begin influencing rulings, they could erode public trust in the justice system. The Supreme Court’s ruling, therefore, serves as a warning about the challenges of integrating AI into institutions that depend heavily on credibility.
As AI becomes an everyday tool in legal practice, the key question now is not whether it should be allowed, but whether India’s justice system can ensure its responsible use. The case of the insolvency tribunals has set the stage for a transformative discussion on accountability in the digital age of law.
