Duplicate check v2 — title-similarity pass (2026-10-04)
Free; no API calls. Produced by
python -m backend.app.corpus.duplicate_check --v2 (writes
data/duplicate_check_v2.json); the tables are generated by
python -m backend.app.corpus.doc_numbers. Rule: pass 5 in
docs/merge-policy.md. Supersedes the corpus counts in
duplicate-check.md.
Report-only run on the published libraries
Every pair with the same first-author surname, years within 2, distinct titles, and title-token Jaccard of 0.6 or more (a reporting floor below the 0.75 rule), with the outcome of each condition:
| set | pair (or papers scanned) | years | title Jaccard | numbers A | numbers B | abstract Jaccard | rule outcome |
|---|---|---|---|---|---|---|---|
| llm-calibration | W7131427313 / W7147133480 | 2025 / 2026 | 0.7778 | — | — | 0.2417 | blocked: abstract_jaccard |
| diet-and-mortality | 100 | — | — | — | — | — | no candidates |
| ml-fairness | 99 | — | — | — | — | — | no candidates |
| ml-fairness (before the 2026-10-04 merge) | W2974817986 / W4289438483 | 2019 / 2018 | 0.8 | 360 | 360 | 0.6178 | merge |
Wider sweep used to choose the thresholds
Same report over every raw OpenAlex record seen while building the two new libraries (kept and rejected), and LLM calibration's 113 papers:
| set | pair (or records scanned) | years | title Jaccard | numbers A | numbers B | abstract Jaccard | rule outcome |
|---|---|---|---|---|---|---|---|
| llm-calibration | W7131427313 / W7147133480 | 2025 / 2026 | 0.7778 | — | — | 0.2417 | blocked: abstract_jaccard |
| diet-and-mortality | W1617145133 / W3143437408 | 2016 / 2015 | 0.7742 | 19902015 2015 79 | 188 19902013 2013 79 | — | blocked: numeric_tokens |
| diet-and-mortality | W4249178369 / W4255618880 | 2017 / 2018 | 0.6364 | 4 diabetes2018 | 5 diabetes2019 | 1.0 | blocked: title_jaccard |
| diet-and-mortality | W4311263988 / W4405187504 | 2022 / 2024 | 0.8824 | 5 diabetes2023 | 5 diabetes2025 | 0.9577 | blocked: numeric_tokens |
| ml-fairness | W2974817986 / W4289438483 | 2019 / 2018 | 0.8 | 360 | 360 | 0.6178 | merge |
Pairs with identical normalized titles are not listed: passes 2 and 3 already handle them (they need a shared author).
What was done with each flagged pair
| pair | outcome | action |
|---|---|---|
AI Fairness 360: W4289438483 (arXiv 2018) and W2974817986 (IBM J. Res. Dev. 2019) | passes every condition | Merged (owner decision; 2019 journal version kept, merged_from recorded; data/domains/ml-fairness/merge_log.json). |
LLM calibration: W7131427313 (2025 review) and W7147133480 (2026 empirical study) | blocked: abstract overlap | Not merged. Hand-checked in duplicate-check.md: different works. |
| Diet sweep: GBD comparative risk assessment, 2016 vs 2015 records | blocked: numeric tokens | Not in the published library; nothing to do. |
| Diet sweep: ADA Standards of Care chapters (2017 vs 2018; 2022 vs 2024 records) | blocked: title overlap / numeric tokens | Not in the published library; nothing to do. Different annual editions. |
Uncertain pairs needing a decision: none. No pair in any published library is left for the rule to merge.
Before and after, per library
| library | papers before | papers after | shortlisted pairs before | after | supports verdicts before | after | flagged before | after |
|---|---|---|---|---|---|---|---|---|
| LLM calibration | 113 | 113 | 437 | 437 | unchanged | unchanged | 2 | 2 |
| Diet & mortality | 101 | 101 | 152 | 152 | unchanged | unchanged | 10 | 10 |
| ML fairness | 100 | 99 | 95 | 91 | 82 | 78 | 0 | 0 |
| Social media & teens | 106 | 106 | 214 | 214 | unchanged | unchanged | 8 | 8 |