Narrow beats clever
An assistant that will attempt anything fails in ways nobody can predict or reproduce. An assistant that does one job refuses the rest, and refusal is a feature.
Narrow scope also makes quality measurable. You cannot test everything, but you can test one job thoroughly.
Ground it in your own material
Most useful features are not asking a model what it knows, they are asking it to read your documents and answer from those. That changes the failure mode from invention to retrieval.
It also changes where the effort goes. Finding the right passage is harder than writing the answer, and it is where these projects actually succeed or fail.
Show the working
An answer with a link to the source it came from is checkable. Without it, users either trust everything or nothing, and both are bad outcomes.
Citations do more than build confidence. They give you a cheap way to spot a wrong answer, because the wrong source is visible immediately.
- Which document the answer came from
- A way to open that document at the right place
- An explicit answer when nothing was found
- A route to a human that does not restart the conversation
Put a person where reversal is expensive
Drafting a reply is safe. Sending it is not. Suggesting a refund is safe. Issuing it is not. The pattern is the same everywhere: let the model prepare, let a person release.
Confident wrong answers are a property of the technology rather than a defect awaiting a fix. Design the workflow to survive them instead of waiting for them to stop.
Build the test set before you build the feature
Collect real questions with answers you know to be correct, written down before launch. Fifty is enough to be useful and small enough to actually assemble.
Without that set, every change is judged on whichever answer someone tried last, and quality becomes a matter of mood. With it, you can tell whether a change helped.
Watch what people ask, not what you expected
The questions users bring are never the ones planned for. That gap is the most valuable output of the first month, and it is only available if the questions are recorded.
Read them weekly at first. They will tell you what to add to the source material, which is almost always a faster fix than touching the model.