Six things we never do as AI engineers, and what we do instead

These are six things we never do when we build with AI. None of them is clever. They are boring, and that is why people skip them. Each one costs you later: a model that looked good but is not, a bill you did not plan for, or a password in the wrong place.

A few words, in plain terms, before we start:

• AI model: the program that writes the answers, like the one behind ChatGPT or Claude.
• Leaderboard: a public scoreboard that ranks AI models by their test scores.
• Eval set: your own list of test questions, with the right answers, that you check the AI against.
• Fine-tuning: retraining a model on your own examples so it learns a new habit.
• Agent: an AI that does a job step by step on its own, using tools.
• API key: the password your app uses to talk to an AI company's servers. Anyone who has it can spend your money.

Every number here comes from a page we opened on 18 October 2026, linked in each section and listed at the end. The habits and their order are our own practice and our opinion. None of the companies, papers or tools named here is connected to webridgeAI.

1. Pick a model from a leaderboard score

Why it bites: a public benchmark is a fixed set of questions, and fixed questions leak. Once they are on the internet they can end up in a model's training data, and a model that has seen the test scores higher without being better at the job.

There is evidence that this happens. Researchers at Scale AI wrote GSM1k, a fresh set of grade-school maths problems built to match the style and difficulty of the well-known GSM8k benchmark, so a model that really could do the maths should score about the same on both. In the published version of the paper (NeurIPS 2024), some models scored up to 8% lower on the fresh set, and the authors found signs that some had partly memorised GSM8k. Many models, especially the frontier ones, showed little sign of this, so it is not true of every model. An earlier preprint of the same paper reported drops of up to 13%.

Leaderboards built on votes have their own problems. In "The Leaderboard Illusion" (2025), researchers studying Chatbot Arena reported that some providers privately tested many variants before release (27 for Meta before Llama 4) and that extra Arena data could give large gains on the Arena itself, which they argue leads to overfitting to the leaderboard rather than general quality.

What we do instead: treat the leaderboard as a shortlist, then test the two or three finalists on our own questions. That is what section 2 is for.

2. Change anything with no eval set

Why it bites: a language model's answers vary from run to run, and a new model version can behave differently from the last. Without a fixed set of test questions, every prompt change or model swap is judged on a few answers you happened to read, and a fix in one place quietly breaks another.

OpenAI's own model optimization guide puts evals first in the loop, suggests you might even write them before you write prompts, and says to run them on test data "representative of real world inputs".

What we do instead: before we build, we write down real questions the system will get, with the answer we expect, and keep them in a file. Every change runs against the whole file. A few dozen is enough to start; ours grow every time a real user finds a new failure.

# evals.jsonl, one test per line:
# {"question": "What is the refund window?", "must_include": "30 days"}
 
import json
 
def ask(question: str) -> str:
    """Call your model or your whole pipeline here and return its answer."""
    raise NotImplementedError
 
def run_evals(path: str = "evals.jsonl") -> None:
    tests = [json.loads(line) for line in open(path) if line.strip()]
    failures = []
    for t in tests:
        answer = ask(t["question"])
        if t["must_include"].lower() not in answer.lower():
            failures.append((t["question"], answer))
    print(f"{len(tests) - len(failures)} of {len(tests)} passed")
    for question, answer in failures:
        print("FAIL:", question, "->", answer[:120])
 
if __name__ == "__main__":
    run_evals()

A substring check is crude, and that is fine for a start. When answers get longer, replace must_include with a stricter check or a second model that grades against a written rubric, and keep the file.

3. Fine-tune before trying a better prompt

Why it bites: fine-tuning means collecting and cleaning training examples, paying for training runs and owning a custom model from then on. Much of the time the problem was the instructions or the missing context, which costs minutes to fix.

OpenAI's model optimization guide says "the prompt engineering process may be all you need in order to get great results for your use case", calls fine-tuning "a time-consuming process", and treats it as something for some use cases. When we read the page, it also said OpenAI is winding down its own fine-tuning platform and that the platform is no longer open to new users.

What we do instead: climb this ladder one rung at a time, and run the eval set from section 2 at every rung so you know whether the step helped.

Fine-tuning is the right tool for some jobs, and we do it. We just do it last, with an eval set that proves the first three rungs were not enough.

4. Run an agent with no step limit and no spend limit

Why it bites: an agent is a model calling tools in a loop, and every pass round the loop is another paid model call. If the model never decides it is done (a tool keeps failing, or it keeps trying the same fix), the loop keeps running and keeps billing.

Anthropic's guide Building effective agents notes that agents bring "higher costs, and the potential for compounding errors", and that it is "common to include stopping conditions (such as a maximum number of iterations) to maintain control". OpenAI's Agents SDK has this built in: its max_turns setting caps the number of model calls in a run and raises a MaxTurnsExceeded error when the cap is hit.

What we do instead: two limits in code, one in the account.

MAX_STEPS = 10          # model calls per task
MAX_COST_USD = 0.50     # stop well before a surprise
 
def run_agent(task: str) -> str:
    messages = [{"role": "user", "content": task}]
    spent = 0.0
    for step in range(MAX_STEPS):
        reply, cost = call_model(messages)   # your model call; return the reply and its cost
        spent += cost
        if reply.is_final:
            return reply.text
        if spent > MAX_COST_USD:
            return f"Stopped: spent {spent:.2f} USD without finishing."
        messages += run_tools(reply)         # run the tools it asked for, append the results
    return f"Stopped: no answer after {MAX_STEPS} steps."

The third limit lives in your provider's console. OpenAI's API key safety page recommends monthly spend limits with hard enforcement turned on, and points out that spend alerts alone do not stop API traffic.

If you want to see a whole agent with a step limit, our guide to building a working AI agent in Python has one in about 120 lines.

5. Put an API key in code

Why it bites: once a key is in a public repository, automated scanners can find it within minutes, and anyone holding it can spend on your account.

• In a honeypot experiment by Comparitech, researchers published AWS keys on GitHub and logged what happened. The first key was abused within one minute. In a second test, the attacker made more than 1,000 calls to launch servers within a minute of the key going public, apparently to run up charges.
• GitGuardian's State of Secrets Sprawl 2025 counted 23.8 million secrets leaked on public GitHub in 2024, and found that 70% of secrets leaked in 2022 were still valid.

Deleting the file afterwards does not help much, because the key stays in the git history. GitHub's own guide to removing sensitive data says that the first step is to revoke or rotate the secret.

What we do instead: keys live in environment variables, loaded from a .env file that git never sees. OpenAI's API key safety page says the same: never commit a key to your repository, use environment variables, and never put a key in a browser or mobile app.

# .env  (never committed)
OPENAI_API_KEY=your-key-here
ANTHROPIC_API_KEY=your-key-here

# .gitignore  (add this before your first commit)
.env
.env.*
!.env.example

# pip install python-dotenv
import os
from dotenv import load_dotenv
 
load_dotenv()                            # reads .env into the environment
api_key = os.environ["OPENAI_API_KEY"]   # fails loudly if it is missing

Commit a .env.example with the variable names and no values, so the next person knows what to set.

Three more habits that make a leak hurt less:

• Check before you push. git grep -nE "sk-[A-Za-z0-9]|AKIA[0-9A-Z]{16}" finds the common OpenAI-style and AWS key shapes in your working tree.
• Leave push protection on. GitHub turns on push protection for users by default, which blocks pushes that contain recognised secrets to public repositories. Repository-level push protection is off by default and has to be enabled by an admin. See about push protection.
• Give keys an end date. OpenAI recommends one key per person, keys with an expiry date and a regular rotation process.

If a key has leaked: revoke it in the provider's console first, then create a new one, then clean the history.

6. Paste private data into a consumer chatbot

Why it bites: what you paste may be used to train future models, depending on the product and your settings. Customer records, contracts and internal code are not yours alone to share.

OpenAI's data controls page explains that a setting called "Improve the model for everyone" decides whether your conversations train OpenAI's models, and that you can turn it off in Settings, Data controls. Temporary chats are not used to improve the models but may be kept for up to 30 days for safety. By default, OpenAI does not train on content from ChatGPT Business, Enterprise, Edu or ChatGPT for Healthcare workspaces.

What we do instead:

• Read the data terms of the exact plan you are on before any real data goes in. Other providers have their own settings; check them the same way.
• Use a business workspace or the API with a company account for company data, not a personal chatbot login.
• Strip names, emails, phone numbers and account numbers before you paste when you can. A model rarely needs them to do the task.
• If you are not sure whether you are allowed to share it, you are not.

The checklist

Run it before you ship anything built on a model.

Honest notes

• The six habits, their order and the fixes are our practice and our opinion. Your team may draw the lines elsewhere.
• The snippets are starting points written to be read, not drop-in production code. ask, call_model and run_tools are where your own model calls go.
• Product settings and docs change. Everything quoted from OpenAI, Anthropic and GitHub is as their pages read on 18 October 2026.
• The honeypot timings come from one company's experiment with AWS keys; other providers and setups will differ. The point that holds everywhere is that discovery is automated and fast.
• We have no arrangement with any company, paper or tool named here, and nothing here is sponsored.

Sources

All read on 18 October 2026.

• A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k), Zhang and others, NeurIPS 2024: NeurIPS proceedings
• The same paper, earlier preprint: arXiv 2405.00332
• The Leaderboard Illusion, Singh and others, 2025: arXiv 2504.20879
• Model optimization, OpenAI: OpenAI docs, model optimization
• Building effective agents, Anthropic, December 2024: Anthropic engineering
• Running agents (max_turns), OpenAI Agents SDK: Agents SDK docs, running agents
• Best practices for API key safety, OpenAI: OpenAI Help Center, API key safety
• AWS credentials found on GitHub and abused within 1 minute, Comparitech: Comparitech, GitHub honeypot
• The State of Secrets Sprawl 2025, GitGuardian: GitGuardian blog
• About push protection, GitHub: GitHub docs, push protection
• Removing sensitive data from a repository, GitHub: GitHub docs, removing sensitive data
• Data controls in ChatGPT, OpenAI: OpenAI Help Center, data controls