 |
ML Newsnew | past | best | rss | submit | about |
| Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data (arxiv.org) | |
| 2 points by liliumregale on Aug 29, 2025 | hide | past | pdf | 1 comment on HN | |
|
In plain words: Instead of tweaking prompts step by step, this treats a multi-step LLM workflow as one typed program and trains it with gradients, so each step's output fits what the next step needs. It beat prompt-tuning, doubling accuracy on financial questions from 12% to 24.7%. AbstractReliably composing Large Language Models (LLMs) for complex, multi-step workflows remains a significant challenge. The dominant paradigm -- optimizing discrete prompts in a pipeline -- is notoriously brittle and struggles to enforce the formal compliance required for structured tasks. We introduce Type-Compliant Adaptation Cascades (TACs), a framework that recasts workflow adaptation as learning typed probabilistic programs. TACs treat the entire workflow, which is composed of parameter-efficiently adapted LLMs and deterministic logic, as an unnormalized joint distribution. This enables principled, gradient-based training even with latent intermediate structures. We provide theoretical justification for our tractable optimization objective, proving that the optimization bias vanishes as the model learns type compliance. Empirically, TACs significantly outperform state-of-the-art prompt-optimization baselines. Gains are particularly pronounced on structured tasks, improving FinQA from $12.0\%$ to $24.7\%$ for a Qwen 3 8B model, MGSM-SymPy from $57.1\%$ to $75.9\%$ for a Gemma 2 27B model, MGSM from $1.6\%$ to $27.3\%$, and MuSR from $36.5\%$ to $62.6\%$ for a Gemma 7B model. TACs offer a robust and theoretically grounded paradigm for developing reliable, task-compliant LLM systems. | |
| add comment on HN |
The paper highlights a few key results that make this seem particularly practical:
* Significant gains over prompt-optimization: On a structured symbolic math generation task (MGSM-SymPy), their method achieved 75.9% accuracy, while a strong DSPy baseline with constrained decoding scored 57.1%. The paper shows that TACS consistently outperforms prompt-optimization baselines, especially on smaller models or highly structured tasks.
* Makes smaller models viable: It shows how a 7B model that initially produced invalid, unparsable outputs 83% of the time was trained to be type-compliant. After just one epoch, the parsing failure rate dropped to 1%. This suggests adaptation can enforce correctness where prompting alone fails.
* A more principled approach: The core idea is to move away from brittle "prompt-hacking". You define a workflow graph with explicit input/output types, and the framework trains the lightweight adapters to respect those types. This allows for principled training on latent variables (like chain-of-thought steps) without needing direct supervision for them.
It seems like a solid step towards making complex LLM compositions more of an engineering discipline. It's less about finding the "magic prompt" and more about training small, specialized modules to be verifiably correct components in a larger system