Every day a small pipeline of mine reads about 200 new AI papers from arXiv and scores each one for how much it matters to someone who builds with these models. The best ones become 40-second videos. This one stayed with me longer than 40 seconds, because it says out loud something many teams find out the hard way: the loop around a model can matter more than the model.
The paper is MuninAI: A Self-Hosted Agentic Framework for Independent Research Groups by Markus Fischer, Jens Meiler and Matthias Elgeti (arXiv 2610.23480, 16 October 2026, CC BY 4.0). The code and the evaluation suite are on GitHub.
The problem
A research group that runs its own model, for privacy or cost, usually cannot get it to use the group's own papers well. The common fix is retrieval: find the five passages that look closest to the question, paste them in, answer once.
Munin does something else. It indexes the group's papers and their citation graph, and wraps the same open-weight model in an agent harness: a loop with tools for searching, reading full papers and computing (the authors list about 45 tools and four sub-agents for sources, search, compute and deep research, each with its own context). The model searches, reads the whole paper, and either quotes the line that answers the question or says the answer is not there.
The authors' claim is that the harness, not the specific model, is what makes it useful.
The numbers
All figures are the authors' own, on 199 literature questions (LitQA2), with Qwen3.8-27B, the model their system runs on:
| Set-up | Right answers |
|---|---|
| The model alone, no tools | 38.7 % |
| Top-5 passages pasted in, one answer | 21.1 % |
| The same model in the Munin harness | 87.4 % |
Two things stand out.
The plain retrieval set-up was worse than nothing. Pasting in the five closest passages scored 21.1 %, well below the model on its own. The paper reports that this held on every backbone they tried. Five passages that look similar to the question are not the passage that answers it, and a model given them seems to trust them.
The model did not change. The jump from 38.7 % to 87.4 % comes from the loop: search, read in full, quote or say no.
The catch
The 87.4 % run could also reach the web. With all outbound network access switched off, so the model saw only the group's own papers, the same model scored 66.3 %. That is still far above the 38.7 % baseline, and it narrowly clears the published 66.0 % of PaperQA2 (which, the authors note, was trained on the same question set, so that comparison favours PaperQA2), but it is not 87.4 %.
If your reason for a self-hosted model is that nothing leaves the building, 66.3 % is your number. The paper reports both, which is what makes it easy to trust.
Smaller hardware
The authors also ran gpt-oss-20b, a model that runs on one consumer graphics card (in their set-up, one card using 24 GB of memory). It went from 40.7 % alone to 56.3 % in the harness, and 48.2 % with the network off. A real gain, smaller than on the larger model.
Saying "it isn't there"
Two tests in the paper are worth more to me than the headline.
- Remove the source. The authors took the 49 source papers behind the questions (2,247 chunks) out of the index and asked again. On the questions it had answered correctly before, the system now said it could not find the answer about nine times in ten on the Qwen models (0.889 and 0.875) and eight in ten on gpt-oss-20b (0.800).
- Ask about papers that do not exist. They asked about 300 made-up references: 100 per model, 80 with DOIs that do not exist and 20 described only in words. The system cited none of them from the local library.
A system that is right 87 % of the time but cannot say "not here" is hard to use for research. These two tests are the ones I would copy first.
What it changes in practice
- Measure top-5 retrieval against no retrieval before you ship it. This paper is one more data point that it can do harm. It costs one evaluation run to find out.
- Try a better loop before a bigger model. Reading the full source and quoting it is slower than one shot, but here it more than doubled the right answers with the same weights.
- Report the offline number next to the online one. A tool that can browse the web is a different system from one that cannot, and a privacy argument rests on the second.
- Test that it abstains. Remove the source, invent a reference, and see what comes back.
I do something close to point 4 in my own video pipeline. Every card of a paper script names the exact paragraph it comes from, and I check each number against the paper before anything is posted. It's the same principle: an answer is only as good as the line it can point to.
Where a question may go is also why I route every AI call in my own projects by role rather than by model. That rule is described in Ask for a role, not a model.
The 40-second version
Paper: Fischer, Meiler, Elgeti. MuninAI: A Self-Hosted Agentic Framework for Independent Research Groups. arXiv:2610.23480, 2026. arxiv.org/abs/2610.23480