Signal Inducing Task Models from Computer-Use Traces

Summary

Naturally recorded computer-use traces, such as screenshots and mouse or keyboard actions, are a resource for deriving symbolic, verifiable and reusable models of how everyday work actually gets done. As computer-use agents move into real work, agents need to learn how tasks are actually carried out and organizations need to be able to audit and reuse that knowledge. This is difficult, however, because activity is observed only as low-level events and real work is multi-threaded, with several goals interleaved at once. Existing methods assume a task is given in advance or that work follows a single workflow, producing simple step-by-step summaries rather than structured task models. The authors introduce TMI, which discovers latent tasks within unconstrained traces, separates out concurrent activity, and for each task derives both a hierarchical model of how goals break down and a procedure model of how execution actually flowed. In controlled experiments, TMI recovered interleaved tasks with 0.974 agreement against ground-truth groupings, reconstructed 74.9 percent of observed execution steps, and skills derived from it improved accuracy on unseen tasks by 30 percent over the strongest existing baseline.