Automatic Prompt Engineer: Does Auto Prompting Work?

Automatic Prompt Engineer is a 2022 paper, published at ICLR 2023, and the phrase has since spread to cover every tool that rewrites a prompt for you. The paper's idea is narrow and worth keeping narrow. Give a model a handful of input and output pairs, ask it to guess the instruction that produced them, generate a pile of those guesses, run each one on more pairs, keep the guess that scores best. The instruction is treated as a program and the search is over programs.

Does it work? On the paper's own benchmarks, yes, and there are three different numbers for how well, which I'll get to. In practice I think the answer is a condition rather than a verdict. Automatic prompt engineering works when you have a way to score an output and a set of examples to score it on. Take away the score and it degrades into a rewriter that makes your prompt longer and more confident, which the vendor selling one of these tools admits in its own docs. Everything below is about which side of that line you're on.

What The Paper Actually Runs

The loop has four moves, and the paper's own templates are short enough to quote.

First, proposal. The model is shown demonstrations and asked to complete a sentence. The forward template reads, in full, "I gave a friend an instruction and five inputs. The friend read the instruction and wrote an output for every one of the inputs. Here are the input-output pairs:" followed by the pairs and then "The instruction was". Whatever the model writes after that is a candidate. The paper generated 250 of them per task in its main experiments, using the largest InstructGPT model, and notes in its cost analysis that the bigger model was the cheaper way to find a good prompt despite the higher per-token price, because it needed fewer tries.

Second, scoring. Two functions are tested. Execution accuracy is exact match, a 0 or a 1 per example. Log probability is the softer one, the probability the target model assigns to the right answer given the candidate instruction, which gives a gradient of sorts when every candidate is scoring zero on exact match.

Third, filtering. This is the part the explainers skip and the part that makes it affordable. Every candidate is scored on a small subset of the training data first. Only candidates above a threshold get a second, non-overlapping subset. The process repeats until a small set remains, and only those are scored on the full training set. Bad instructions cost a few calls. Good ones cost the full evaluation.

Fourth, and optionally, resampling. The best candidates go back into the model with the prompt "Generate a variation of the following instruction while keeping the semantic meaning." The paper's own conclusion on this step is that the top instruction "tends to remain the same with more stages", so iterative search is switched off by default. I'd remember that when a tool sells you rounds.

The famous output of all this is a chain-of-thought prefix. The authors took "Let's think step by step", which had already lifted InstructGPT on MultiArith from 17.7 to 78.7, and searched for a prompt starting with "Let's" that maximised the likelihood of correct reasoning traces. APE returned "Let's work this out in a step by step way to be sure we have the right answer." That moved MultiArith from 78.7 to 82.0 and GSM8K from 40.7 to 43.0. A few points, on a model you can no longer call, and I find the second part of that sentence more important than the first.

Three Numbers For One Claim

The paper's headline is that model-written instructions match or beat human-written ones on the 24 instruction-induction tasks. Read across the results page and you'll find that claim quoted three ways, and the sources disagree with each other because they're reading different versions of the same paper.

None of those sources is wrong about what it read. The arXiv listing shows an abstract with the earlier figure while the PDF it links to carries the camera-ready one. If you're citing APE, cite the PDF and say which version, because "19 of 24" and "24 of 24" are different claims about the same method and both are in circulation with the same arXiv number attached.

I'd add that the human baseline in the interquartile mean is a fixed number, 0.749, in both readings that state one. What changed between versions is the APE side. That's consistent with the authors improving their own search between submission and camera-ready, and it's also exactly the kind of drift an automatic optimizer produces when you give it more budget. My reading, not theirs. The method's headline number is itself a function of how long it was run.

The Model It Ran On Is Gone

Every number above was produced on text-davinci-002. The GitHub repository that ranks third for this query still defaults both the proposal model and the evaluation model to that string, with num_prompts at 50 and eval_rounds at 20, MIT licensed, and around 1.4k stars on the day I looked. OpenAI's deprecations page lists text-davinci-002 as shut down on January 4, 2024, with gpt-3.5-turbo-instruct as the recommended replacement.

So the repo runs only after you change the model names, and once you do, you're no longer reproducing the paper. You're running the paper's method on a model with different defaults, and there's no published 24-of-24 for that. I don't think this makes the repo useless. It's a readable reference implementation of a loop that fits in a page. It does mean the "automatic prompt engineer github" search leads to code whose numbers you can't check, which is a different thing from code that works.

Auto Prompt Meaning, In 2026

The related searches ask for the meaning of "auto prompt", and the honest definition has widened since 2022. The February 2025 survey on the results page formalises the whole family as "a maximization problem over discrete, continuous, and hybrid prompt spaces" and sorts methods by what they optimise (instructions, soft prompts, or exemplars) and how (model-based optimisation, evolutionary search, gradients, or reinforcement learning). APE is the model-based, instruction-only, discrete corner of that map. OPRO, a 2023 paper, sits next to it and feeds the model its previous attempts with their scores so it can propose better ones, reporting gains "by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks" over human prompts.

What a reader can actually use today is a shorter list. Here's what each one wants from you, from its own documentation on the fetch date.

Two things jump out of that table for me. The first is that the OpenAI row has an expiry date on it. Their page says the dataset-backed optimizer is going away with Evals, and it gives the reason for using it in the same breath, that "the effectiveness of prompt optimization depends on the quality of your graders". The second is the difference between Google's zero-shot mode and everything else. It needs no data. Which means it can't be scoring anything, so what it's doing is applying prompting best practices to your text. That's a rewriter. Useful, sometimes, I'd guess, and not the thing the paper describes.

The observability and evaluation products that usually get bundled into this conversation are priced and compared in prompt engineering tools, and none of them are optimizers in the APE sense either.

Does It Work? Depends On What You Have

Here's the decision as I'd make it, by what you're holding before you start.

The DSPy documentation has the cleanest worked example of the first row, and I'll describe it rather than reproduce it. The task is a haiku from three inputs. The metric checks line count, syllables per line, whether the input words were repeated verbatim, first person, part-of-speech balance, adjective count, article count, and tense. With gpt-5.4-nano as the student and gpt-5.4 as the reflection model, their docs report the score climbing from 78.1% to 90.1% on that metric, with the unoptimised larger model scoring 82.4% on the same test. Their numbers on their own metric, so I read them as a demonstration of the mechanism, but the mechanism is the point. The small model with an optimised instruction beat the large model with the one-line instruction, and that's the trade every one of these tools is selling.

Now look at what the optimizer wrote. The instruction went from "Write a classical haiku given the provided inputs." to a document with sections, whose success criteria start with "Exact 5-7-5 syllable counts, one line per count." and go on to "Do not repeat the input season or mood words verbatim", "Do not use first-person pronouns", "Keep article use minimal". Every one of those lines is a metric check, translated into English. The optimizer didn't learn what a haiku is. It learned what the grader counts. When the grader is good, that's the same thing. When it isn't, you get the last row of the table.

Run The Loop By Hand

You don't need the repository to use the method. The paper's loop fits in three chat prompts, and running it once by hand teaches you more about what the tools do than reading about them. This is the paper's procedure, not a test I've published results for.

I will give you five input and output pairs. A single instruction was used to produce every output from its input. Propose ten different instructions that could have been that instruction. Make them genuinely different from each other in wording and in what they assume, not paraphrases. Output them as a numbered list and nothing else.

Input: [PASTE]
Output: [PASTE]
Input: [PASTE]
Output: [PASTE]
Input: [PASTE]
Output: [PASTE]
Input: [PASTE]
Output: [PASTE]
Input: [PASTE]
Output: [PASTE]

That's the forward template with the "friend" framing swapped for a direct request and ten candidates instead of 250, since ten is what a hand run can afford. "Genuinely different" is doing the work the paper's sampling temperature did. Without it you get ten rewordings of the first idea.

Here are ten candidate instructions and five new input and output pairs that none of the candidates have seen. For each candidate, apply it to each of the five inputs and write the output you would produce. Then compare your output to the given output and mark it MATCH or MISS. Report a table with one row per candidate: candidate number, matches out of five. Do not revise any candidate. Do not explain the misses.

Candidates:
[PASTE THE TEN]

Held-out pairs:
[PASTE FIVE NEW PAIRS]

This is execution accuracy on a held-out subset, and it's the step that matters. The pairs here must not be the pairs from the first prompt, or you're scoring memorisation. I'd keep them in a separate file so I can't mix them up. The model grading its own outputs is a real weakness, and it's the same weakness the paper's log-probability score has, so on anything that matters you'd run the candidates through the API and check the outputs with code. For a first pass, the table tells you whether the candidates even disagree with each other, which is the thing you need to know before spending money.

Take the two highest-scoring candidates below. For each, generate three variations that keep the meaning but change the wording, the ordering, or the level of detail. Output six numbered instructions and nothing else.

[PASTE THE TOP TWO WITH THEIR SCORES]

The resampling step. Run the scoring prompt again on these six, and if the winner is one of the original two, you've reproduced the paper's finding that resampling rarely changes the top result. Stop there. If a variation wins, keep it and stop anyway, because a hand loop past two rounds is where you start fitting to your five held-out pairs.

Where It Goes Wrong

The failure modes are documented by the people who build these tools, one reason to read their docs instead of the glossary pages that rank for this.

Reusing the validation set. DSPy's page says that if you omit a validation set, GEPA "reuses trainset for selection", and that this "deliberately allows prompts to overfit those examples". It's presented as a feature for inference-time search, and it is one. It's also the mistake I'd expect most people to make first, because it makes every score look better.

Metric gaming. The haiku instruction above is the gentle version. The severe version is an optimizer that discovers your grader rewards length, or the word "carefully", or a particular formatting habit, and writes an instruction that produces those. OpenAI's page tells you to build "narrowly-defined graders for each of the desired output properties where you see your prompt failing", which is the right advice and also an admission that the grader is where the work lives.

Model drift. The DSPy page puts it plainly. "Phrasing that works on today's model can stop working when the provider ships an update next month." An optimised instruction is tuned to a snapshot. The same fact sits behind OpenAI's own advice to pin a model snapshot and keep an evaluation suite, which is the part of their platform guide I'd read before any of this, and it's covered in openai prompt engineering.

Wording versus design. The November 2024 survey post on the results page makes a point I agree with, that these methods are "assistive in nature" and don't touch the decisions above the prompt, like whether the task needs retrieval or a different model. An optimizer will find the best instruction for a design that's wrong. The hand techniques it's searching over, the ones you'd otherwise apply yourself, are in chatgpt prompt engineering.

And regression on specific inputs. OpenAI's page again, on their own tool. "It's possible for the optimized prompt to perform worse than your original on specific inputs." An average going up is compatible with your most important case going down. A metric that's a mean hides that, which is why the held-out set should include the examples you'd be embarrassed to get wrong.

The Career Questions Google Attached To This

The People Also Ask box under this query is entirely about the job. How much a prompt engineer is paid, whether the role is still in demand, whether it's a good career, and two questions about which jobs survive AI and which engineer makes half a million a year. None of that is about the paper, and Google appears to be blending this query with "prompt engineer" as a title.

I'm not going to quote a salary, for the reason given at length in llm prompt engineering. The numbers online mix titles that share a word with roles that don't share a job. The one thing this article can add to the career question is a small irony. The paper's title says models are human-level prompt engineers, and the paper's method needs a human to write the score function, choose the examples, hold out a test set, and decide whether the winning instruction is acceptable. That's the job, and it didn't go anywhere. It moved one level up.

What I'd Do First

Write the metric. Not the prompt, the metric. If you can't write a function or a rubric that returns a number for an output, no optimizer can help you yet, and the zero-data tools will hand you a longer prompt with the same problem.

Then run the three-prompt loop above once, by hand, on ten pairs. You'll see the candidates, you'll see how little they disagree, and you'll see the winner. After that, the tools make sense, because you'll know exactly which part of the loop each one is charging you for.