Signal Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

Summary

The paper introduces an expert-validated benchmark for evaluating LLM reliability on the Colombian legal system. It contains 1,042 items across ten areas of law and three question formats, built through a human-in-the-loop pipeline. Across 15 models, closed-question accuracy ranged from 0.577 to 0.905. No model exceeded 0.45 factual correctness on free-text legal answers. Models often sounded responsive while being wrong, and only about half of the norms they cited were correct. The authors conclude that current LLMs need expert supervision for Colombian legal tasks, and that grounding answers in authoritative sources is promising.