Skip to content

SKILL.md

Latest commit

 

History

History
112 lines (83 loc) · 5.03 KB

SKILL.md

File metadata and controls

112 lines (83 loc) · 5.03 KB
Raw
name paper-downloader
description Download academic papers from a manuscript's reference list. Use this skill whenever the user asks to download, retrieve, or fetch references from a paper, thesis, or manuscript — especially when they provide a PDF and a reference number range (e.g., "download references 82-99 from this paper"). Also trigger when the user says "帮我下载这篇论文的参考文献", "下载这篇文献里 的引用", "get the cited papers from this PDF", or any request involving batch-downloading academic articles by reference number from a document. This skill handles DOI lookup via Crossref API, Sci-Hub downloading with automatic domain failover, and verification of downloaded PDFs.

Paper Downloader

Download academic papers referenced in a manuscript's bibliography. Given a PDF and a reference number range, this skill extracts the references, resolves DOIs via Crossref, downloads PDFs from Sci-Hub with automatic failover across multiple domains, and verifies each download.

Workflow

Step 1: Extract references from the PDF

Use pdfplumber to read the PDF and locate the references section. The references section is typically at the end of the document, often under a heading like "参考文献", "References", or "Bibliography".

pip install pdfplumber --break-system-packages

Read the PDF page by page starting from the last 20-30 pages to find the reference entries. Identify the target reference numbers and extract the full citation text for each.

Step 2: Resolve DOIs via Crossref API

For each reference, search the Crossref API to find matching DOIs:

import urllib.request, urllib.parse, json

def search_crossref(title):
    query = urllib.parse.quote(title)
    url = f"https://api.crossref.org/works?query={query}&rows=3"
    req = urllib.request.Request(url, headers={"User-Agent": "PaperDownloader/1.0 (mailto:example@example.com)"})
    resp = urllib.request.urlopen(req, timeout=15)
    data = json.loads(resp.read())
    return data.get("message", {}).get("items", [])

Match results by comparing titles. Be polite to the API — add a 0.5 second delay between requests.

Some references may be Chinese dissertations (typically marked with [D]) or other non-DOI publications. These won't have DOIs in Crossref. Flag them separately.

Step 3: Download from Sci-Hub

Use the bundled scripts/download_papers.py script for downloading. It handles:

  • Multiple Sci-Hub domain failover (sci-hub.ru → sci-hub.se → sci-hub.st)
  • Protocol-relative PDF URLs (//sci-hub.cat/... vs /storage/...)
  • PDF validation (checks %PDF- header and %%EOF trailer)
  • Duplicate detection (skips already-downloaded files)

Run it like this:

python3 scripts/download_papers.py \
  --outdir <output_directory> \
  --dois "doi1,doi2,doi3,..." \
  --prefix <two-digit-ref-number-start>

The script reads a comma-separated DOI list and outputs numbered PDFs (82.pdf, 83.pdf, etc.).

If running directly without the script, follow these rules when constructing Sci-Hub URLs:

  1. Fetch https://sci-hub.ru/<doi> to get the HTML page
  2. Extract the PDF path from <meta name="citation_pdf_url" content="...">
  3. Handle the URL format carefully:
    • //sci-hub.cat/storage/... → prepend https:
    • /storage/... → prepend https://sci-hub.ru
    • Full URL → use as-is
  4. Download with a Referer header set to the Sci-Hub page URL
  5. Verify: file size > 5000 bytes, starts with %PDF-

Step 4: Handle failures

For references that fail to download:

  1. Retry with alternate Sci-Hub domains: sci-hub.ru → sci-hub.se → sci-hub.st
  2. Try alternate URL construction: if sci-hub.cat storage URLs fail, try the same path on sci-hub.ru
  3. Chinese dissertations: These are typically only available via CNKI/Wanfang with institutional access. Sci-Hub does not index them. Generate CNKI/Wanfang search links for the user to access manually.
  4. Other failures: Log the error details for the final report.

Step 5: Generate a download report

After all downloads complete, create a README.txt (or DOWNLOAD_REPORT.md) in the output directory containing:

  • List of all requested references with their full citations
  • Status for each: ✅ Downloaded (with file size) or ❌ Failed (with reason)
  • For Chinese dissertations: CNKI/Wanfang links for manual download

Important notes

  • Sci-Hub domains change frequently. See references/scihub_domains.md for the current known-working domains.
  • Always use SSL context that skips certificate verification for Sci-Hub (ssl.create_default_context() with check_hostname=False).
  • Add 2-second delays between Sci-Hub downloads to avoid being rate-limited.
  • Chinese dissertations (博士论文/硕士论文, marked with [D] in the reference) are NOT available on Sci-Hub. Don't waste time trying — flag them immediately and provide CNKI/Wanfang links instead.
  • Some journals (especially older ones, pre-2000) may not be on Sci-Hub. If the DOI resolves but Sci-Hub returns no PDF, try Google Scholar as a fallback.