about
LLMs as Method Actors: A Model for Prompt Engineering and Architecture (arxiv.org)
2 points by gronky_ on Dec 3, 2024 | hide | past | pdf | 1 comment on HN

In plain words: Treat the AI as an actor, the prompt as its script and cues, and its answer as a performance — a way of writing prompts to get better results. On the Connections word puzzle, this framing let GPT-4o solve 86% of puzzles, versus 41% with the usual step-by-step thinking prompt.

Abstract

We introduce "Method Actors" as a mental model for guiding LLM prompt engineering and prompt architecture. Under this mental model, LLMs should be thought of as actors; prompts as scripts and cues; and LLM responses as performances. We apply this mental model to the task of improving LLM performance at playing Connections, a New York Times word puzzle game that prior research identified as a challenging benchmark for evaluating LLM reasoning. Our experiments with GPT-4o show that a "Method Actors" approach can significantly improve LLM performance over both a vanilla and "Chain of Thoughts" approach. A vanilla approach solves 27% of Connections puzzles in our dataset and a "Chain of Thoughts" approach solves 41% of puzzles, whereas our strongest "Method Actor" approach solves 86% of puzzles. We also test OpenAI's newest model designed specifically for complex reasoning tasks, o1-preview. When asked to solve a puzzle all at once, o1-preview solves 79% of Connections puzzles in our dataset, and when allowed to build puzzle solutions one guess at a time over multiple API calls, o1-preview solves 100% of the puzzles. Incorporating a "Method Actor" prompt architecture increases the percentage of puzzles that o1-preview solves perfectly from 76% to 87%.

Colin Doyle
arXiv:2411.05778 · cs.AI, cs.CL · submitted Nov 8, 2024 · updated Nov 11, 2024
abstract · pdf · html

add comment on HN

Every time I see the phrase "prompt engineering", I have to wonder what "tardy engineering" would be like.