Same local model, twice the right answers: what the MuninAI paper changes

A new arXiv paper puts the open-weight model a research group already runs inside an agent loop over its own papers. Accuracy on 199 literature questions went from 38.7 % to 87.4 %, and the usual top-5 retrieval scored worse than no retrieval at all.

Every day a small pipeline of mine reads about 200 new AI papers from arXiv and scores each one for how much it matters to someone who builds with these models. The best ones become 40-second videos. This one stayed with me longer than 40 seconds, because it says out loud something many teams find out the hard way: the loop around a model can matter more than the model.

The paper is MuninAI: A Self-Hosted Agentic Framework for Independent Research Groups by Markus Fischer, Jens Meiler and Matthias Elgeti (arXiv 2610.23480, 16 October 2026, CC BY 4.0). The code and the evaluation suite are on GitHub.

The problem

A research group that runs its own model, for privacy or cost, usually cannot get it to use the group's own papers well. The common fix is retrieval: find the five passages that look closest to the question, paste them in, answer once.

Munin does something else. It indexes the group's papers and their citation graph, and wraps the same open-weight model in an agent harness: a loop with tools for searching, reading full papers and computing (the authors list about 45 tools and four sub-agents for sources, search, compute and deep research, each with its own context). The model searches, reads the whole paper, and either quotes the line that answers the question or says the answer is not there.

The authors' claim is that the harness, not the specific model, is what makes it useful.

The numbers

All figures are the authors' own, on 199 literature questions (LitQA2), with Qwen3.8-27B, the model their system runs on:

Set-up Right answers
The model alone, no tools 38.7 %
Top-5 passages pasted in, one answer 21.1 %
The same model in the Munin harness 87.4 %

Two things stand out.

The plain retrieval set-up was worse than nothing. Pasting in the five closest passages scored 21.1 %, well below the model on its own. The paper reports that this held on every backbone they tried. Five passages that look similar to the question are not the passage that answers it, and a model given them seems to trust them.

The model did not change. The jump from 38.7 % to 87.4 % comes from the loop: search, read in full, quote or say no.

The catch

The 87.4 % run could also reach the web. With all outbound network access switched off, so the model saw only the group's own papers, the same model scored 66.3 %. That is still far above the 38.7 % baseline, and it narrowly clears the published 66.0 % of PaperQA2 (which, the authors note, was trained on the same question set, so that comparison favours PaperQA2), but it is not 87.4 %.

If your reason for a self-hosted model is that nothing leaves the building, 66.3 % is your number. The paper reports both, which is what makes it easy to trust.

Smaller hardware

The authors also ran gpt-oss-20b, a model that runs on one consumer graphics card (in their set-up, one card using 24 GB of memory). It went from 40.7 % alone to 56.3 % in the harness, and 48.2 % with the network off. A real gain, smaller than on the larger model.

Saying "it isn't there"

Two tests in the paper are worth more to me than the headline.

  • Remove the source. The authors took the 49 source papers behind the questions (2,247 chunks) out of the index and asked again. On the questions it had answered correctly before, the system now said it could not find the answer about nine times in ten on the Qwen models (0.889 and 0.875) and eight in ten on gpt-oss-20b (0.800).
  • Ask about papers that do not exist. They asked about 300 made-up references: 100 per model, 80 with DOIs that do not exist and 20 described only in words. The system cited none of them from the local library.

A system that is right 87 % of the time but cannot say "not here" is hard to use for research. These two tests are the ones I would copy first.

What it changes in practice

  1. Measure top-5 retrieval against no retrieval before you ship it. This paper is one more data point that it can do harm. It costs one evaluation run to find out.
  2. Try a better loop before a bigger model. Reading the full source and quoting it is slower than one shot, but here it more than doubled the right answers with the same weights.
  3. Report the offline number next to the online one. A tool that can browse the web is a different system from one that cannot, and a privacy argument rests on the second.
  4. Test that it abstains. Remove the source, invent a reference, and see what comes back.

I do something close to point 4 in my own video pipeline. Every card of a paper script names the exact paragraph it comes from, and I check each number against the paper before anything is posted. It's the same principle: an answer is only as good as the line it can point to.

Where a question may go is also why I route every AI call in my own projects by role rather than by model. That rule is described in Ask for a role, not a model.

The 40-second version

Watch the Short on YouTube.

Paper: Fischer, Meiler, Elgeti. MuninAI: A Self-Hosted Agentic Framework for Independent Research Groups. arXiv:2610.23480, 2026. arxiv.org/abs/2610.23480

More posts

All posts →
  1. 21 Nov 2026

    Same agent, rebuilt workspace, a quarter of the bill: what EnvOpt showsA new arXiv paper leaves the model and the harness alone and lets another AI rebuild the workspace the agent works in. On 200 test tasks, the same Claude Haiku 4.5 agent cost $42.36 before and $11.55 after, and passed more tasks.

    AI · 7 min
  2. 21 Nov 2026

    CLAUDE.md vs AGENTS.md: which one Claude Code reads, and why I keep only oneWhen a repository has both files, Claude Code reads CLAUDE.md and leaves AGENTS.md alone unless you import it. I found five of my repositories set up the wrong way round: the real instructions in AGENTS.md, and a CLAUDE.md that only pointed at it.

    How I build · 5 min
  3. 20 Nov 2026

    Running several Claude Code agents at once: worktrees, handoffs and one check per phaseWhat broke when I put many Claude Code agents on the same repositories, and the four rules that stayed: one worktree per agent, small file-disjoint jobs, a rewritten handoff, and one full check per phase.

    How I build · 5 min