Stop Tweaking Your Prompts. Start Managing Your Context

👁 116 reads

Anyone who has worked with language models for more than a week knows the moment. You write a prompt, get an answer that misses the point, so you rework the phrasing. You add “be precise,” “think step by step,” “you are an expert with twenty years of experience.” Fifth version, sixth version, and the output is still mediocre. The obvious conclusion: I must not be good at writing prompts, and I need to learn how.

I believed that for a long time. I test models every day, on my own tasks and at my own expense, so hundreds of conversations pass through my hands every week. The longer I do this, the more convinced I become that obsessing over any single prompt is overrated. The difference between people who genuinely get results from AI and those who constantly wrestle with it usually lies somewhere else entirely: in what you manage between prompts. In the context.

The model doesn’t see your prompt. It sees everything at once

Start with the mechanics, because everything else follows from them. The model doesn’t receive a “prompt.” It receives one long string of text containing the tool’s system instructions, the entire conversation so far with all your revisions and its previous replies, any files you’ve pasted in, search results, and only then, right at the end, your latest message. From the model’s perspective, all of that is a single document. It doesn’t draw a sharp line between “old noise” and “the current instruction” the way we imagine it does.

That reframes everything. Your carefully polished prompt is one paragraph inside material that may have grown to the length of a short story. You can spend hours refining that paragraph, but if the remaining pages are full of contradictory decisions, abandoned threads, and wrong answers that were never corrected, you are polishing the door handle on a room that’s already a mess.

Why long conversations get dumber

Everyone has seen it: a conversation that started brilliantly begins to fall apart after an hour. The model circles back to ideas you clearly abandoned, mixes up versions, answers a question from fifteen minutes ago instead of the one you just asked. This isn’t fatigue or caprice. It’s arithmetic.

First, every wrong answer stays in the window and works against you. The model reads its own mistakes as legitimate content, and what it has read anchors its next responses. Second, contradictory instructions don’t cancel each other out, they accumulate. If the conversation included “keep it brief” early on and “expand on this” later, both versions sit in the window and the model picks whichever it likes. Third, the more text there is, the more things compete for the attention of the mechanism that decides what matters. An important instruction buried in the middle of a long conversation simply loses out to what’s fresh and to what came first.

I have a simple, practical warning sign for this: if I find myself correcting the model on the same issue for the third time, I stop correcting. In my experience, further attempts to fix things within the same conversation rarely fix anything, because all the failed attempts are still sitting in the window, pulling the answers back toward the same mistakes.

A prompt is a sentence. Context is a working environment

The best analogy I know is a new employee on their first day. You can give them the most beautifully worded instruction imaginable, but if you sit them at a desk buried under someone else’s papers, without access to the right documents and with three contradictory notes from their predecessor, you’ll get shoddy work. Not because they’re incompetent, but because the working environment settled the matter before they even started.

It’s the same with a model. The quality of a response comes down to three questions: what the model knows (whether it has the right material), what it doesn’t know (whether something you consider obvious is missing because it only exists in your head), and what it’s seeing unnecessarily (how much noise has built up in the window). The prompt is fourth on that list. A good instruction in a bad context will lose. A decent instruction in a clean, well-prepared context will usually be enough.

Seven habits that do more than magic formulas

This isn’t theory. These are things I do every day, and they have genuinely changed how I work with AI.

1. Start a new conversation more often than instinct tells you to. A new session is a clean desk. Almost everyone holds on to conversations too long because they hate to lose the “progress.” But the progress worth keeping usually fits in a few sentences of summary. Everything else is dead weight.

2. Provide the material, not a description of the material. Instead of summarising a document from memory, paste the document. Instead of describing the code, show the code. The model works on what it sees, not on what you saw.

3. Cut failed attempts, don’t correct them. When a conversation has gone sideways, the most effective move is: ask for a short summary of whatever conclusions still hold up, open a new session, paste the summary, and carry on. The mistakes stay behind the door instead of riding along with you.

4. Keep standing instructions in a fixed place. If you’re writing the same thing about your preferred style, format, or constraints for the third time this week, that’s not work, that’s copying. Project instructions, guideline files, and system notes exist for this reason. Write them once and they apply to every conversation.

5. One task, one conversation. Mixing research, drafting, and invoice corrections in a single window is an invitation for threads to bleed into one another. Separate your contexts the way you separate browser tabs.

6. Before you step away, make the model save its state. “Summarise in bullet points what we’ve decided, what we’ve ruled out, and what still needs to be done.” That snapshot, pasted into a fresh session the next day, works better than returning to yesterday’s bloated conversation.

7. Mind the order. Models hold on best to what comes at the beginning and the end of the window. Key requirements are worth placing in both spots: once in the source material, once in the final prompt. The middle of a long conversation is the worst place to put anything important.

External memory instead of an endless conversation

There’s a level above all this, one you tend to reach naturally once the habits above become second nature: you stop treating the conversation as a place to store knowledge. Knowledge lives in files. The conversation is just a workspace.

In practice, this means keeping your own AI-facing notes: a project description file, a decisions file, a style file. At the start of each session you hand the model those files and get a repeatable starting point that has nothing to do with what happened yesterday. Agentic tools, such as coding assistants, already do this systematically by maintaining memory files and project instructions across sessions. But the same principle works in an ordinary chat window with no automation at all: short working sessions plus external, updated notes beat one eternal conversation that remembers everything, meaning increasingly less.

When the prompt still matters

To be clear: I’m not claiming the content of a prompt is irrelevant. Three things in prompts make a real difference. A precise goal rather than a vague one, because “write something about context” and “write an 800-word practical guide for people who use AI every day” are two different commissions. Output format, because you need to tell the model whether you want a table, a list, or flowing prose. And an example, because one good model answer can replace a whole paragraph of explanation.

But all of that is cheap and quick. You master it in an evening and there’s nothing left to refine. The rest of the celebrated “prompt engineering” canon, the incantations about experts, deep breaths, and creative threats, ages with every new generation of models, and at this point more has been written about it than ever comes out of it.

So if I had to reduce this whole shift to one sentence, it would be this: stop asking “how do I phrase this so the model understands” and start asking “what is the model seeing right now, and is it something worth working with.” The first question leads to a sixth version of the same prompt. The second leads to a clean desk, the right materials, and answers that don’t need fixing.

// read moreAI Agents That Don’t Ask: When AI Stops Advising and Starts ActingMagazineI Pay $40 for Two AI Subscriptions and Won’t Stop. Here’s WhyMagazineOpenAI COO Brad Lightcap Leaves to Start New VentureMagazine

Kinetyra