Many prompts start with a role: "You are an expert marketer", "Act as a senior developer". The idea is that the model answers better when it is told who to be. A research team measured that, and the result is worth knowing before you write your next prompt.
Below is what the study tested, what it found and what it did not cover. Then the three things we add to a prompt instead of a role, each with a before and after, and the cases where a role is still worth a line.
1. What the study tested and found
The paper is "When 'A Helpful Assistant' Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models" by Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee and David Jurgens, published in Findings of EMNLP 2024 (arXiv 2311.10054).
What they tested. 162 roles, covering six kinds of personal relationship and eight areas of expertise. Each role was put in the system prompt of nine open models from four families: FLAN-T5-XXL, Llama-3-Instruct (8B and 70B), Mistral-7B-Instruct and Qwen2.5-Instruct (3B to 72B). Every combination answered 2,410 multiple-choice questions from MMLU, a standard exam-style benchmark, spread over 26 subjects. The control was the same question with no role at all.
What they found.
- Adding a role did not improve accuracy compared with no role. The authors describe the effect as none or slightly negative, and some roles lowered accuracy.
- If you could pick the best role for each question after seeing the results, accuracy would go up. But the methods they tried for picking that role in advance were often no better than choosing one at random. The authors conclude that the effect of a persona is largely unpredictable.
What it did not cover. The authors list these limits themselves:
- No closed models. GPT-3.5 and GPT-4 were not tested, so the paper says nothing direct about ChatGPT, Claude or Gemini.
- One kind of task. MMLU is multiple-choice factual questions. Writing, coding, open-ended answers and tone were not measured.
- 162 roles, not every possible role.
So the fair reading is narrow: on factual multiple-choice questions, in these open models, telling the model who to be did not make it more correct. It is not proof that roles never matter.
2. The three things that help instead
A role tells the model who to be. It does not tell it anything about your situation. These three parts do. This section is our recommendation, built on how we write prompts; it is not a finding of the study above. Where a source backs a part, it is linked.
Context: what the model can't know
The model knows a lot in general and nothing about your case. Context is who the answer is for, what it is for, and what you already have. Anthropic's prompting guide makes the same point: it compares the model to a capable new employee "who lacks context" on your norms, and says explaining the reason behind an instruction helps it give more targeted answers (Anthropic, Prompting best practices).
Before
You are an expert fitness coach. Give me a workout plan.After
Give me a workout plan. I'm 30, I work at a desk, and I can train
three times a week for 40 minutes at home with two dumbbells. I want
to get stronger, not train for a sport. I have a sore lower back, so
avoid anything that loads it heavily.The first prompt can only get a generic plan. The second rules out a gym, sets the time, the equipment and the goal, and names the one thing to avoid.
An example: show the answer you want
It is often easier to show the kind of answer you want than to describe it. Putting a few worked examples in the prompt is called few-shot prompting. The GPT-3 paper showed a large model can take on a new task from a few examples written into the prompt, with no retraining (Brown and colleagues, 2020). Anthropic's guide calls examples one of the most reliable ways to steer format, tone and structure, and suggests three to five for best results (Anthropic, Prompting best practices). For a one-off chat question, one good example is usually enough to make the point.
Before
You are a professional copywriter. Write product descriptions for my
three candles.After
Write product descriptions for my three candles: cedar, fig, sea salt.
Match the style of this one I already wrote:
"Lavender. Smells like clean sheets on a Sunday. Burns for 40 hours.
Soy wax, cotton wick."
Same length, same four short sentences, no exclamation marks.The example sets the length, the rhythm and the details to include far more precisely than "professional" ever could. One caution: the model copies examples closely, so if you give one, make it the kind of answer you would actually be happy with.
What a good answer looks like: say when it's done
Tell the model what a finished answer is: how long, in what format, and what it must include or leave out. Without that, it guesses, and its guess is usually long and general.
Before
Act as a senior software engineer. Review my code.After
Review this Python function. List at most five problems, most serious
first. For each one give the line, what is wrong in one sentence, and
the fixed line. Ignore style and naming; I only care about bugs and
edge cases.The second version says what counts as a problem, how many to report, in what order and in what shape. You can check the answer against it.
3. When a role still helps, and when it doesn't
The study measured accuracy on factual questions. It did not measure tone or style, and the authors note that a system role can still be used for reasons like language style. So we treat a role as a style setting, not an accuracy setting. This is our recommendation, not a finding of the study.
Where a role can still earn a line
- Tone and voice. "You are a patient tutor for a 12-year-old" changes how the answer sounds. Anthropic's guide says a role in the system prompt focuses behaviour and tone (Anthropic, Prompting best practices).
- Format and register. "Reply as a support agent writing to a customer" points the model at a familiar format: short, polite, one next step.
- Practice. In role-play, such as a mock interview or a hard conversation, the role is the task itself.
Where it doesn't
- Making the answer more correct. On the questions the study tested, a role did not do this. If you need a right answer, spend the words on context and a check, not on a title.
- Replacing the details. "You are an expert lawyer" does not tell the model your country, your contract or what you are trying to decide. The details do.
If you keep a role, keep it to one line and put the context, the example and the definition of a good answer after it.
One caution
A better prompt makes a better answer more likely. It does not make the model reliable. Check anything important (a medical, legal or money question) against a primary source.