Biolit

biolit

mcp-name: io.github.rachadele/biolit

LLM-assisted biomedical literature screening and structured extraction. Accepts PubMed alert emails and mixed lists of PMIDs, DOIs, and GEO accessions in any combination. Retrieves full text from PMC, Europe PMC, bioRxiv/medRxiv, Unpaywall, and Semantic Scholar. Supports multiple LLM providers and exposes all functionality as an MCP server.

Setup

Requirements: Python 3.8+

Install from PyPI:

pip install biolit

Or install from source for development:

pip install -e .

Copy .env.example to .env and add your API key:

cp .env.example .env# edit .env and set ANTHROPIC_API_KEY (or OPENAI_API_KEY)

On macOS, you can store the key in the system keychain instead of .env. biolit consults the keychain by service name only (no account required):

security add-generic-password -s ANTHROPIC_API_KEY -w# or for OpenAI:security add-generic-password -s OPENAI_API_KEY -w

Omit -w <value> to be prompted for the key without echoing it. The keychain is checked first; the env var is used only as a fallback (so a stale value in .env cannot mask a working keychain entry).

Usage

The tool accepts a PubMed alert email (.eml) or a plain-text file of identifiers, as well as inline identifiers via --ids. Identifiers can be PMIDs, DOIs, or GEO accessions — mixed lists are supported in a single run.

Use --default to run with schizophrenia genomics defaults (no prompts):

biolit docs/alert.eml --defaultbiolit docs/pmids.txt --defaultbiolit docs/geo_accessions.txt --defaultbiolit --ids 41795042,41792186,GSE53987 --defaultbiolit --ids 10.1101/2025.03.17.25324098 --default

Or specify criterion and fields as flags:

biolit identifiers.txt \  --criterion "Is this about treatment-resistant schizophrenia?" \  --fields "methodology, sample_size, treatment, outcomes"

Add --markdown (or --md) to also write a prose .md summary alongside the CSV. Each record gets a markdown section with ### field subsections; records that failed or were skipped appear as stub entries:

biolit refs.bib --config my_config.json --markdownbiolit refs.bib --config my_config.json --markdown --markdown-max-tokens 2048

Add --batch to issue screening, extraction, and markdown rendering through the provider's Message Batches / Batch API instead of one call per record. Per-request cost drops by ~50%, but each batch blocks on completion (typically several minutes per stage; up to 6 hours), so it's intended for the bulk weekly-alert case rather than one-off lookups. Anthropic and OpenAI only — falls back to sequential calls on Ollama or on OpenAI-compatible endpoints with a custom base_url. Also accepted as "batch": true in a config file.

biolit docs/alert.eml --default --batchbiolit docs/alert.eml --default --batch --markdown   # batches markdown too

Or use a JSON config file to store reusable parameters (CLI flags take precedence). The config can include ids or input_file (path to an .eml, .bib, or identifier list), and "markdown": true to enable markdown output:

biolit alert.eml --config my_config.jsonbiolit refs.bib --config my_config.json   # DOIs extracted from .bib automaticallybiolit --config my_config.json            # ids or input_file supplied by config

The fields key in a config file can be a comma-separated string or a JSON object mapping field names to extraction descriptions. When a string is used, an extra LLM call converts the field names into descriptions before extraction. When a dict is used, that call is skipped — the descriptions are passed directly to the model:

{  "fields": {    "tf_name": "HGNC symbol of the transcription factor perturbed in this experiment",    "organism": "scientific name of the organism used",    "platform": "GPL accession of the microarray platform"  }}

Omit --criterion to skip screening (all records are extracted). Omit --fields to use the default fields (methodology, sample_type, causal_claims, summary):

# fetch + extract with defaults (no screening)biolit alert.eml
# fetch + screen only, then extract with defaultsbiolit alert.eml --criterion "Is this about treatment-resistant schizophrenia?"

Single-record screening

Use biolit screen to quickly check one paper or GEO record for relevance without running the full extraction pipeline:

biolit screen --pmid 41627908 --defaultbiolit screen --accession GSE53987 --defaultbiolit screen --doi 10.64898/2026.02.16.706214 --defaultbiolit screen --pmid 41627908 --criterion "Is this about treatment-resistant schizophrenia?"

Output is a single line to stdout:

RELEVANT [abstract] — Paper uses GWAS to investigate schizophrenia risk loci.

Mixed identifier lists

PMIDs, DOIs, and GEO accessions can be freely mixed in a file or via --ids. Each identifier is auto-detected by format:

• 41795042 → PMID (all digits)
• 10.1101/2025.03.17.25324098 → DOI (starts with 10.)
• GSE53987 → GEO accession (starts with GSE, GDS, GSM, or GPL)

biolit --ids 41795042,GSE53987,10.1101/2025.03.17.25324098 --default

GEO records additionally include a linked_pmids column. All record types share pmid, doi, and geo_accession columns (null when not applicable).

Full-text retrieval

Full-text retrieval runs automatically for every PMID and DOI (including preprints). For GEO records, the pipeline attempts full-text retrieval via each linked PMID in order, falling back to the GEO record metadata if no linked paper has accessible full text. The pipeline tries each source in order:

1. PMC JATS XML (open access)
2. Europe PMC JATS XML (broader open-access coverage)
3. Preprint XML (bioRxiv / medRxiv)
4. Unpaywall PDF (requires --unpaywall-email)
5. Semantic Scholar open-access PDF
6. OpenAlex green-OA PDF (author manuscripts Unpaywall/S2 miss; key-less)
7. Europe PMC open-access full-text PDF (OA subset)
8. CORE aggregated green-OA PDF (opt-in; needs CORE_API_KEY)
9. Publisher landing-page scrape (the citation_pdf_url meta tag; key-less)
10. Custom resolvers (institutional OpenURL / library proxy; opt-in via BIOLIT_CUSTOM_RESOLVERS)
11. Publisher landing-page HTML full text (the citation_fulltext_html_url meta tag; key-less)
12. Abstract fallback

Steps 6-9 are all open-access-only (green-OA author manuscripts, institutional-repository copies, and the publisher's own advertised OA PDF link) — never a paywall bypass. OpenAlex, Europe PMC, and the landing-page scrape need no key; CORE is a no-op unless CORE_API_KEY is set. The landing-page scrape (step 9) follows the DOI to the article page and reads the citation_pdf_url link the publisher itself embeds (the Highwire / Google-Scholar standard) — this catches OA PDFs the aggregator APIs mislabel or never index. bioRxiv / medRxiv are skipped there (their servers block agents; the preprint step above covers them). Step 10 is the seam for your own authorized access — see Custom full-text fetchers. Step 11 is the HTML counterpart of step 9: when no downloadable PDF exists at all, it extracts the article body text from the publisher's full HTML page (the citation_fulltext_html_url Highwire signal that PLOS / eLife / BMC / Frontiers and many society journals set) — recovering Methods text for OA papers the PDF chain can never reach. When even that misses, the abstract fallback records a paper_status classification (bot_blocked / js_shell / abstract) in the per-record artifacts so a caller can read why full text was not reached.

To enable Unpaywall (step 4), pass your email:

biolit alert.eml --default --unpaywall-email [email protected]

Limit which sections are sent to the LLM:

biolit alert.eml --default --sections methods,results

LLM providers

The tool supports Anthropic (default), OpenAI, and local Ollama models:

# OpenAIbiolit pmids.txt --default --provider openai --model gpt-4o
# Ollama (local)biolit pmids.txt --default --provider ollama --model llama3

You can also set LLM_PROVIDER and LLM_MODEL as environment variables.

Output

Each run creates a timestamped directory (e.g. run_20260313_142000/) containing:

• results.csv — one row per relevant record
• results.md — prose markdown summary (written when --markdown or "markdown": true in config)
• artifacts/<id>/ — per-record folder with the text sent to the LLM, metadata, and any retrieved full-text files

Records that fail at any pipeline stage (fetch error, not found, no content, screening or extraction error) are excluded from the CSV but appear in the markdown as stub entries with a failure note.

With default fields, the CSV columns are:

GEO records additionally include a linked_pmids column listing all associated PubMed IDs.

The CSV can be imported directly into Google Sheets (File → Import).

MCP server

biolit ships an MCP server that exposes the pipeline as tools for any MCP-compatible client (Claude Desktop, Claude CLI, OpenAI Agents SDK, etc.).

Start the server:

biolit-mcp# or pick a provider/model explicitly (overrides LLM_PROVIDER / LLM_MODEL env vars):biolit-mcp --provider openai --model gpt-4o-mini

Or test interactively with the MCP inspector:

mcp dev biolit/mcp_server.py

Configure Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{  "mcpServers": {    "biolit": {      "command": "biolit-mcp",      "args": ["--provider", "openai"]    }  }}

Restart Claude Desktop. The tools will appear in the tool picker. Drop args to use the default Anthropic provider (or set LLM_PROVIDER / LLM_MODEL env vars instead).

Configure Claude CLI

Add a .mcp.json in your project root:

{  "mcpServers": {    "biolit": {      "command": "biolit-mcp",      "args": ["--provider", "openai"]    }  }}

Available tools

Batch pipeline (equivalent to the biolit CLI):

Low-level (for custom workflows):

Use as a Python library

The pipeline functions are importable directly:

from biolit.pipeline import run, screen_paper, fetch_recordfrom biolit.llm import get_llm_client
client = get_llm_client("anthropic")
# Batch pipeline — PMIDs, DOIs, and GEO accessions can be mixed freely# criterion and fields_description are optional; omit either to skip that step# markdown=True writes results.md alongside the CSV# Returns (csv_path, record_count)csv_path, count = run(client, ids=["41627908", "GSE53987", "10.1101/2025.03.17.25324098"],    criterion="...", fields_description="methodology, summary", output_path="results.csv",    markdown=True)
# Fetch + write metadata only (no LLM calls)csv_path, count = run(client, ids=["41627908", "GSE53987"])
# Fetch a single record (auto-detects PMID / DOI / GEO)paper = fetch_record("10.1101/2025.03.17.25324098")
# Screen pre-fetched textresult = screen_paper(client, paper, "Is this about schizophrenia genomics?", paper["abstract"])# {"relevant": True, "reason": "..."}

Custom full-text fetchers

The built-in chain (PMC → Europe PMC → preprint → Unpaywall → Semantic Scholar → OpenAlex → Europe PMC OA PDF → CORE → landing-page PDF scrape → custom resolvers → landing-page HTML → abstract) leaves coverage gaps for closed-access or recently-published work. You can plug in additional sources of full text — a Zotero library, a flat directory of PDFs, an institutional full-text database — without forking biolit.

Landing-page scrape and custom resolvers (built-in, opt-in via env vars)

Two built-in chain steps mirror Zotero's "Find Full Text" file resolvers and are configured the same way as Unpaywall / CORE — through environment variables, no code changes.

Landing-page scrape (step 9). Always on, key-less. Resolves the DOI to the publisher's article page and reads the PDF link the page itself advertises — <meta name="citation_pdf_url"> first (the Highwire / Google-Scholar standard most publishers embed), then <link rel="alternate" type="application/pdf">, Open Graph / Twitter-card PDF pointers, and obvious *.pdf anchors. A <meta http-equiv="refresh"> redirect is followed once; every candidate is verified to start with the %PDF magic. This follows the publisher's own advertised OA link — not a paywall bypass — and catches OA PDFs the aggregator APIs mislabel or never index. bioRxiv / medRxiv are skipped (their servers block agents; the preprint step covers them). Source label: landing_page_pdf.

# Optional: override the browser User-Agent used for the landing-page# request (some publishers serve a stub or 403 to non-browser agents).export BIOLIT_LANDING_USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

Landing-page HTML full text (step 11). Always on, key-less. The HTML counterpart of step 9. When no downloadable PDF was found anywhere, this follows the DOI to the article page, reads the publisher's <meta name="citation_fulltext_html_url"> "a full HTML version exists" signal (set by PLOS / eLife / BMC / Frontiers and many society journals), and extracts the article body text — dropping nav / scripts / styles / chrome and preferring an <article> / <main> container. This recovers Methods text for OA papers that have no PDF at all. Bot-challenge pages (Cloudflare "Just a moment") and JavaScript-rendered shells are rejected rather than returned as "text". bioRxiv / medRxiv are skipped (the preprint step covers them). Source label: landing_page_html. It shares the BIOLIT_LANDING_USER_AGENT override above.

When the whole chain misses, the abstract fallback classifies why full text was not reached (bot_blocked / js_shell / abstract) and records it as paper_status in the per-record artifacts, so a downstream caller can distinguish "blocked by a bot challenge" from "publisher serves only a JS shell" from "genuinely abstract-only".

# Optional: visible-character floor below which a fetched landing page is# treated as a JS-rendered shell (no server-rendered article body) rather# than full text. Default 2000.export BIOLIT_JS_SHELL_CHAR_THRESHOLD=2000

Custom resolvers (step 10). Opt-in. This is the seam for your own authorized access — typically your institution's OpenURL endpoint or a library EZproxy URL pattern. Set BIOLIT_CUSTOM_RESOLVERS to a JSON array of resolver entries. Each entry needs a url_template with {doi} / {url} placeholders; the URL is fetched with your configured headers and returned if it is a real PDF. When the resolved URL is an OpenURL / proxy landing page rather than a direct PDF, set "scrape": true to run its HTML through the landing-page scraper above. With nothing configured this step is a no-op. Source label: custom_resolver_pdf.

export BIOLIT_CUSTOM_RESOLVERS='[  {    "url_template": "https://proxy.lib.example.edu/login?url=https://doi.org/{doi}",    "headers": {"Cookie": "ezproxy=YOUR_SESSION"},    "scrape": true  },  {    "url_template": "https://resolver.example.edu/openurl?id=doi:{doi_encoded}&svc=fulltext"  }]'

Resolver-entry schema (only url_template is required):

biolit never hardcodes, requests, or stores credentials. Custom resolvers follow your configured URL patterns using your own authorized access (proxy login, session cookies, etc.). You are responsible for ensuring your use complies with your institution's and the publisher's terms of service.

Reference fetchers (opt-in via env vars)

Three ship with biolit and self-register on import when the relevant environment variables are set. Default priorities (lower = tried earlier) are bibtex=2.0, local_pdf=3.0, zotero=5.0.

BibTeX. Looks up papers by DOI, PMID, or citekey in a .bib export, reads the path from each entry's file = {...} field, and parses the PDF directly. Best fit for users who maintain a Better-BibTeX (or equivalent) auto-export — lookups are offline, in-memory, and exact, with no network round-trip and no dependence on the Zotero search index. Works around the Zotero web API's q-search not indexing the structured DOI field, which makes DOI lookups via that API unreliable for items where the DOI doesn't appear in indexed attachment full-text. Supports both BBT semicolon-separated file lists and the classic JabRef description:path:type format. The bib file is re-parsed automatically when its mtime changes.

export BIOLIT_BIBTEX=~/Zotero/My\ Library.bib# Optional:export BIOLIT_BIBTEX_PRIORITY=2.0   # lower = tried earlier (default 2.0)

Zotero. Searches the user's Zotero library by DOI then PMID, resolves attachment search hits up to their parent items, finds an attached PDF, downloads it, and parses it with biolit's PDF parser. When the Zotero /file API endpoint returns 404 (linked_file attachments, or imported attachments on accounts without sync), falls back to reading the PDF from local Zotero storage at $ZOTERO_DATA_DIR/storage/<key>/<filename> (default data dir ~/Zotero). Note: Zotero's web API q-search does not index the structured DOI field, so the BibTeX fetcher above is more reliable when both are available.

export ZOTERO_API_KEY=...export ZOTERO_USER_ID=...           # or ZOTERO_GROUP_ID for a group library# Optional:export ZOTERO_PRIORITY=5.0          # lower = tried earlier (default 5.0)export ZOTERO_DATA_DIR=~/Zotero     # only needed if Zotero is not at ~/Zotero

On macOS, any of ZOTERO_API_KEY, ZOTERO_USER_ID, and ZOTERO_GROUP_ID that are not in the environment fall back to the macOS keychain (security find-generic-password -s <NAME> -w), matching the resolution order used for LLM API keys. This means hosts like Claude Code that don't shell-source your profile can still pick up Zotero credentials without an env block in .mcp.json.

Local PDF directory. Looks up papers by DOI in a pre-built JSON index. Filenames are arbitrary — DOIs are extracted from each PDF's /Info metadata dict and (failing that) its first-page text.

Build (or update) the index. Re-running is cheap — by default only new or changed PDFs are re-extracted:

python -m biolit.fetchers.local_pdf --dir ~/Paperspython -m biolit.fetchers.local_pdf --dir ~/Papers --rebuild   # force full re-extraction

Then point biolit at the same directory:

export BIOLIT_LOCAL_PDF_DIR=~/Papersexport BIOLIT_LOCAL_PDF_PRIORITY=3.0  # default 3.0

The fetcher itself never builds the index — it only consults it. PDFs without an extractable DOI are listed in the index's unindexed_sample for visibility.

When configured, the text_source field in CSV/markdown output is bibtex_pdf, zotero_pdf, or local_pdf for hits from these sources. The raw bytes are persisted into artifacts/<id>/bibtex_pdf / zotero_pdf / local_pdf exactly like the built-in PMC/Europe PMC artifacts.

Writing your own fetcher

A fetcher is any callable that takes a FetchContext and returns either a FetchResult (when it found something) or None (when it didn't).

from biolit.fetchers import FetchContext, FetchResult, register_fetcher
def my_internal_db_fetcher(ctx: FetchContext) -> FetchResult | None:    pmid = ctx.paper.get("pmid")    if not pmid:        return None    text = my_db.lookup_fulltext(pmid)  # whatever you have    if not text:        return None    return FetchResult(text=text, source="internal_db", artifacts={})
register_fetcher(my_internal_db_fetcher, priority=1.0, name="internal_db")

Register before the first call to run / screen_by_* (e.g. at module import time). Registered fetchers are tried before the built-in chain in priority order; the first one to return a non-empty FetchResult.text wins. Exceptions inside a fetcher are logged to stderr and the next fetcher is tried.

Validation

An independent evaluation of the GEO screening and metadata extraction workflow is available at rachadele/biolit-eval. It uses a bootstrap resampling pipeline to estimate precision, recall, and F1 against a manually curated ground truth of 509 GEO accessions labelled for transcription factor perturbation experiments.

Known Limitations

• Papers without abstracts or accessible full text are skipped silently.
• GEO records attempt full-text retrieval via linked PMIDs. text_source will be geo_linked_fulltext, geo_linked_abstract, or geo_record depending on what was accessible.
• bioRxiv/medRxiv JATS XML is frequently blocked by Cloudflare regardless of headers. The pipeline falls back to the title and abstract from the bioRxiv API (text_source: preprint_abstract).
• The Semantic Scholar API allows roughly 100 unauthenticated requests per day. Set SEMANTIC_SCHOLAR_API_KEY in .env for higher limits.