Scale and Singapore launch AI benchmark for legal reasoning
Scale AI and Singapore’s IMDA have launched SG-LegalClarify, a benchmark testing whether AI models can reason about Singapore law and ask for missing facts.
Scale AI and Singapore’s Infocomm Media Development Authority (IMDA) have launched SG-LegalClarify, a benchmark designed to test whether AI systems can reason about Singapore law while recognising when they need more information. The companies describe it as the first research initiative under their collaboration on AI evaluation.1
The benchmark covers 1,860 scenarios across 10 areas of Singaporean law, including criminal, employment, property, family and judicial-process questions. Scale says 1,100 original questions were written by legal professionals. A further 760 scenarios deliberately leave out facts needed to reach a reliable answer, with paired versions that supply those details.1
That structure tests more than whether a model can recall a legal rule. It also asks whether the system can identify the relevant issue, ask for missing facts and apply the right rule to a specific situation. That matters in legal contexts because an answer can sound confident while still being wrong if the system has not established the facts that change the outcome.
What the benchmark found
Scale says it tested 19 AI models, including systems from OpenAI, Anthropic, Google DeepMind and DeepSeek, with and without web search. Without web access, the highest-performing model met 69.8% of the mandatory grading criteria, while the average was 53.8%. With web search, the average rose to 63.0% and the highest score reached 77.8%.1
The announcement identifies legal-rule recall as a particular weakness. Even with web access, models met only 56.5% of the applicable rule-recall criteria. Scale also says models failed to ask a clarifying question in about 39% of scenarios where information was missing. When models asked about every relevant missing fact in web-enabled trials, 55.6% reached the correct legal conclusion; 32.1% reached an incorrect conclusion and 12.3% gave no conclusion.1
The results are reported by Scale and have not yet been independently replicated. The announcement does not provide enough detail here to assess every model’s identity, the benchmark’s grading validity or how the planned public subset will perform under outside scrutiny. Those limitations do not erase the result, but they do mean the figures should be read as an early report rather than a settled league table.
Why Singapore law matters
Many AI evaluations focus on a limited set of languages and legal systems. Singapore’s legal framework draws on English common law while also incorporating Singapore-specific statutes and principles, creating a useful test of whether a model can apply general legal knowledge in a national context.
Scale’s examples make the practical issue clear: answering a question about a speeding penalty may require details about the vehicle, speed and an existing demerit-point record. An assistant that supplies a penalty before asking for those facts may produce a fluent answer that does not fit the person’s circumstances.
Scale and IMDA say they plan to release a public subset of SG-LegalClarify for further research. They are also exploring work covering agentic AI, model evaluation and AI safety.1
Our opinion
SG-LegalClarify is interesting because it tests the part of AI assistance that benchmarks often flatten: knowing when an answer is premature. The headline scores matter, but the missing-facts results are the more revealing measure. A system that asks the right question before answering can be more useful than one that produces a polished response immediately. The next test is transparency. Independent researchers need access to the public subset, scoring criteria and enough methodology to establish whether the benchmark measures legal reasoning rather than familiarity with Scale’s preferred evaluation format.