My research agent's demo looked perfect. The eval said 8 out of 15.
Building Research Scout & Verifier, a citation-or-nothing research assistant, for my FlyRank AI Fluency capstone: one design decision, one bug the eval caught, and one limitation I haven't fixed.
For the last eight weeks I've been working through the FlyRank General AI Fluency track. The capstone is a small tool I now use for my own research: Research Scout & Verifier.
The idea is simple. You ask a question, and it answers only from your own notes. Every claim comes with a file:line citation. If your notes don't support an answer, it prints NOT FOUND and stops.
The decision: no language model at run time
My first design in Week 5 was a Claude Project with web search, PDF uploads and Zotero. When I built it, I cut almost all of that. What I shipped is one Python file that reads Markdown notes, matches topic words line by line, and either cites the line or refuses.
The reason was one rule from my design doc: never fabricate a citation. A model in the loop can always produce a plausible-looking reference. A matcher that can only print lines that exist in your files can't. I gave up cleverness to get traceability. (I built the code itself with Claude as a coding partner. The agent it produced just doesn't call a model.)
The bug my demo hid
In Week 5 I recorded a clean demo. I asked it two questions, it gave good citations for one and said NOT FOUND for the other, and everything looked right.
This week I wrote the eval I had planned from the start: 15 questions, 9 that my notes answer and 6 that they don't. I ran it against the Week 5 version and the new one side by side.
| Week 5 MVP | v2 | |
|---|---|---|
| Cases passed | 8/15 | 13/15 |
| Off-topic questions reported as "grounded" | 5/6 | 0/6 |
Five of six off-topic questions, including "Who won the 2022 World Cup?", came back from the Week 5 version as FROM-MY-FILES with citations. The matcher counted the word "the" as evidence, and every line in my notes contains "the".
My demo hadn't shown this because I had only asked questions I already knew it could handle. The fix was a stopword filter and a regression test. It took ten minutes. Finding the problem needed the eval.
The limitation I haven't fixed
v2 still fails 2 of the 15 cases, and both fail the same way. If I ask "Can it fabricate citations?", it says NOT FOUND, even though my notes say claims "must stay unverified rather than being guessed". The words don't overlap, so the matcher misses it.
It fails safe, because it says "I don't know" rather than guessing. But it will under-report what's in your notes. The fix is retrieval by meaning (embeddings) while still printing only verbatim lines, and that's what I'm building next.
What I'm taking away
- Write the eval before the feature, and run it against the old version too.
- Cut scope until you can check everything.
- Say the limitation before someone else finds it.
The code, the setup steps, the full eval table and all seven known limitations are in the project README. I'm happy to walk anyone through it.