Method

The rule behind the whole project: a language model only turns paper text into structured notes, and (for possible disagreements) answers one narrow question about one pair of claims at a time. Everything that counts, ranks or decides what to show is ordinary code that anyone can read in the public source.

The pipeline

1. Collect. Papers on one subject are gathered from OpenAlex and sorted into on-topic, borderline and off-topic by fixed keyword rules (venue, field, and whether the subject's key terms appear together). Off-topic papers are dropped.
2. Read. A Google Gemini model reads each paper — the full text where an open-access copy could be found, otherwise the abstract — and records its claims, stated limits, methods and suggested next steps.
3. Compare. Code looks for questions a paper raised that no later paper in the library took up, and for limits that recur. For disagreements, code picks out pairs of claims whose wording is very similar; only those pairs are ever checked, and the model is asked about each pair on its own.
4. Order. Code ranks results by fixed rules. The model's own confidence is never used in a score.
5. Check. Flagged disagreements are checked by hand (below). Doubts about our own checks are published, not hidden.

What each library contains

How the libraries grow

Diet and all-cause mortality and Algorithmic fairness in machine learning and Social media and adolescent mental health are set up to grow once a week. An automated run looks in OpenAlex for three kinds of new paper: papers that cite both sides of a checked disagreement, papers that cite a paper whose open question is unanswered, and recent papers that cite the library. It keeps the papers the library's labelling rules accept, removes duplicates, and has a language model take notes on them as above. Code then runs the same checks again. A run changes the site only if every automated test passes; otherwise nothing is published.

• A newly found disagreement is shown as “Flagged by the system, not yet checked” and is not counted anywhere on this site until it has been checked by hand.
• A method-transfer lead is shown only after the language-model check has looked at it.
• Spending on the language model is capped per week and in total, in code, before any call is made.

Language-model reliability does not grow: it is kept fixed so that a planned test (whether its results pointed to research that was later done) has a fixed starting point.

What was found in each library

Every library goes through the same four checks with the same thresholds. For the method-transfer check, the number of topic groups grows with the size of the library, and the strongest leads (at most 15 per library) are checked one by one by a language model, which labels each as substantive, trivial or not addressing the problem. A fifth, experimental check (papers linked only through a third paper) is switched off in every library until it can be validated.

What was checked by hand, and by whom

All hand checks so far were done by the person who built the project, not by independent experts. A blind review packet for the diet disagreements has been prepared for outside experts but has not been sent.

Verdicts we now doubt

• Diet & mortality: We doubt our own "disagreement" verdict on red meat / stroke: one paper reports no link while the other gives a risk per 100 g a day — the same kind of difference in how results were measured that got the triglyceride pair set aside. The verdict is unchanged; the pair has been added to a packet for expert review (not yet sent).
• Diet & mortality: We doubt our own "disagreement" verdict on red meat / type 2 diabetes: one paper is about unprocessed red meat and the other about red meat in general, which may not be the same exposure. The verdict is unchanged; the pair has been added to a packet for expert review (not yet sent).
• ML fairness: 123 of this library's classifier verdicts are for pairs outside the ones we report: 113 were made with a looser matching setting than the library's documented one (similarity below 0.80; iteration 3 ran at 0.72 with up to 4 matches per claim), and 10 no longer rank among each claim's top 2 matches. None of them is a disagreement, and none is counted.

Has this been tested?

Not yet validated. The pre-registered test was inconclusive: too few open questions to tell a useful ranking from chance, and on the numbers it has, the engine did not do better than ranking by citation count.

The test was written down before it was run: freeze each library at an earlier year, let the tool rank the questions it finds open, then check whether papers citing each question's source paper took it up in the next two years. 88 questions were tested and 15 were taken up; a test able to tell a useful ranking from chance would need at least 209.

“Taken up” is a proxy: a citing paper whose summary is close in meaning to the question. Citing is not the same as answering. Full report: Validation v1 (protocol committed before the run, 1bcdb4b).

Known limits

• Partial reading. Language-model reliability: 46 of 113 papers read from the abstract only; Diet and all-cause mortality: 84 of 101 papers read from the abstract only; Algorithmic fairness in machine learning: 53 of 109 papers read from the abstract only; Social media and adolescent mental health: 82 of 106 papers read from the abstract only. A limit or next step mentioned only in the body of a paper can be missed.
• Only what is in the library. A question can look unanswered here because the paper that answered it is not in the collection.
• Only similar pairs are compared. Two findings that disagree but are worded differently are never picked out, so they are never checked. Zero disagreements means none were found among the pairs checked, not that none exist.
• The model can misread. Notes are extracted by a language model and can drop a condition or overstate a finding. Each result links to the paper so you can check.
• Not yet validated. Whether the listed questions predict later research has not been tested.

The detailed working notes are on the library page. Numbers on this page are generated from the project's data files by a script in the source code.