Signal Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Summary

A study argues that systems which have a language model draw a conclusion from many sources should keep the work of interpreting each source separate from the work of combining those interpretations. To do this, the authors propose a four-part evidence tuple made up of a hypothesis, a reliability bucket, a rationale, and a provenance. They identify a problem they call count-scale drift, in which simply thresholding a sum of unnormalized weights behaves like posterior thresholding whose cutoff point shifts as the number of sources consulted changes. They show that combining calibrated log-likelihood ratios instead fixes this problem. The approach was applied to a single longitudinal corpus under two settings, one after outcomes were known and one before. In that test, a small sequence encoder paired with a tree ensemble built on a censored survival loss reached an AUPRC of 0.921, beating a hand-built baseline's 0.805.