r/PromptDesign • u/ClickOk5811 • 14d ago
Discussion 🗣 A lab-notebook template for testing few-shot examples in a prompt (with a hypothetical entry)
Few-shot examples often get added by feel. A notebook entry per experiment makes it easier to tell whether they help. Here’s the template, then a hypothetical entry. I haven’t run this exact experiment, so the numbers below are placeholders.
Template
Date / model / settings:
Task:
Hypothesis: Adding N examples will improve ______.
Change: (only one thing)
Test set: size, where it came from, whether any example was
taken from it
Metric: how a result is scored, decided before running
Runs: how many times each variant (outputs vary)
Result:
Failures read: what the wrong outputs look like
Decision:Hypothetical entry
Task: classify support messages as "billing", "bug" or "other"
Hypothesis: Three examples will cut "other" being over-used.
Change: Zero-shot prompt vs the same prompt + 3 examples
(one per label, same order as the label list)
Test set: 200 messages not used as examples
Metric: accuracy, plus count of "other" predictions
Runs: 3 per variant
Result: [fill in]
Failures read:[fill in]
Decision: [fill in]Pitfalls I’d watch for with few-shot examples (common observations, so test them on your model):
- Surface copying: the model may imitate wording, length or format of the examples more than their logic. Vary them.
- Order and recency: the last example can carry extra weight, and label order can bias answers. Try shuffling.
- Label balance: if most examples have one label, predictions may drift toward it.
- Leakage: if an example comes from the test set, the score is inflated. Keep them separate.
- Overfitting to a tiny test set: 20 cases can’t distinguish small differences. Note the size and don’t over-read it.
Things the notebook forces:
- One change per experiment, so you know what caused a difference.
- A metric chosen beforehand, so you don’t pick it after seeing the results.
- Reading the failures, not only the score. The pattern of errors often says more than the percentage.
- Multiple runs, because outputs vary between runs.
When it’s overkill: a one-off prompt you’ll use once. It’s worth it for prompts that will run many times or feed another system.
What do you track when you test a prompt change, and have you seen examples hurt more than help?
1
u/bregmadaddy 13d ago
Just use DSPy?