A fully automated multi-source job scraping pipeline for London tech roles with visa sponsorship. Aggregates jobs from 6+ sources, scores them against 6 CV profiles, deduplicates, and generates a ranked daily report.
- Fetches jobs from Indeed, LinkedIn, Adzuna, Reed, Jobicy, RemoteOK (+ optional JSearch)
- Normalises all jobs to a canonical schema with annual GBP salary
- Deduplicates by URL across sources and across runs
- Scores each job: keyword match (50%) + salary (25%) + visa signal (25%)
- Picks the best-fit CV for each job out of 6 profiles
- Stores results in SQLite and generates a colour-coded HTML + CSV report
- Tracks application status through a CLI
findwithjobspy/
├── main.py # CLI entrypoint
├── config.yaml # all tuneable parameters
├── requirements.txt
├── first.py # thin wrapper kept for manual testing
│
├── jobhunt/
│ ├── db.py # SQLite schema + all DB ops
│ ├── dedup.py # URL deduplication
│ ├── scorer.py # keyword + salary + visa scoring
│ ├── reporter.py # Jinja2 HTML + CSV report
│ └── sources/
│ ├── __init__.py # scrape_all() orchestrator
│ ├── base.py # BaseSource ABC + normalisation helpers
│ ├── jobspy_source.py # Indeed + LinkedIn (via python-jobspy)
│ ├── adzuna_source.py # Adzuna REST API
│ ├── reed_source.py # Reed.co.uk REST API
│ ├── jobicy_source.py # Jobicy REST API
│ ├── remoteok_source.py # RemoteOK REST API
│ └── jsearch_source.py # JSearch RapidAPI (optional, paid)
│
├── data/
│ └── jobs.db # SQLite — gitignored
└── reports/ # generated HTML/CSV reports
pip install -r requirements.txtEdit config.yaml. The free sources (jobspy, jobicy, remoteok) work with no keys. For paid/registered sources:
sources:
adzuna: { enabled: true, app_id: "YOUR_ADZUNA_APP_ID", app_key: "YOUR_ADZUNA_APP_KEY" }
reed: { enabled: true, api_key: "YOUR_REED_API_KEY" }
jsearch: { enabled: false, api_key: "YOUR_RAPIDAPI_KEY" } # $15/mo, off by default- Adzuna: register at https://developer.adzuna.com — free
- Reed: register at https://www.reed.co.uk/developers/jobseeker — free
- JSearch: subscribe on RapidAPI — $15/mo, enable only when needed
If a source has placeholder keys and enabled: true, it will fail gracefully and log the error without crashing the whole run.
python main.py scrapeThis will:
- Fetch from all enabled sources for all search terms
- Score every new job
- Insert new jobs into
data/jobs.db - Generate
reports/report_YYYYMMDD_HHMMSS.htmland.csv - Open the HTML report in your browser (macOS)
python main.py scrape --dry-runpython main.py scrape --sources adzuna,reed
python main.py scrape --sources jobspypython main.py scrape --terms "ml engineer"
python main.py scrape --terms "backend engineer,python developer"python main.py scrape --report-fmt csv # CSV only
python main.py scrape --report-fmt none # no report after scrape# Default: top 50 not-applied jobs, score >= 0.15, HTML
python main.py report
# More options
python main.py report --min-score 0.5
python main.py report --cv cv_backend_python
python main.py report --status all
python main.py report --top 100 --fmt both
python main.py report --no-open # don't auto-open browserReport row colours:
- Green — score >= 0.70 (strong match)
- Yellow — score 0.40–0.69 (moderate match)
- White — score < 0.40 (weak match)
python main.py track list
python main.py track list --status applied
python main.py track list --cv cv_llm_rag
python main.py track list --min-score 0.6 --top 20python main.py track update 42 --status applied
python main.py track update 42 --status applied --note "sent cv_backend_python"
python main.py track update 17 --status interview --note "technical screen 2026-05-01"
python main.py track update 17 --status offer
python main.py track update 5 --status rejected
python main.py track update 99 --status ignoredValid statuses: not_applied | applied | interview | offer | rejected | ignored
python main.py track statsOutput includes:
- Jobs by status with average and max score
- Interview rate and offer rate
- Scrape run history (total runs, jobs fetched, jobs inserted, last run time)
score_total = keyword_score × 0.50
+ salary_score × 0.25
+ visa_score × 0.25
Keyword score (0–1): best match across all 6 CV profiles
raw = (tier1_hits × 1.0) + (tier2_hits × 0.4)
score = raw / max_possible (capped at 1.0)
Salary score (0–1):
< £41,700 → 0.0 (below visa floor)
None → 0.3 (salary not advertised)
≥ £41,700 → linear ramp to 1.0 at £120,000
Visa score (0 / 0.5 / 1):
positive keywords ("visa sponsorship", "skilled worker", "tier 2", …) → 1.0
negative keywords ("cannot sponsor", "no visa", "right to work required", …) → 0.0
neither → 0.5
| Profile | Best for |
|---|---|
cv_backend_python |
Python backend, FastAPI, Flask, REST APIs, PostgreSQL |
cv_fullstack |
React + Python, full-stack roles |
cv_ai_ml |
PyTorch, scikit-learn, ML research/engineering |
cv_llm_rag |
RAG pipelines, LangChain, vector DBs, GenAI |
cv_data_scientist |
Statistical analysis, Jupyter, NLP, pandas |
cv_mlops |
Docker, Kubernetes, MLflow, model deployment |
Add or edit profiles in config.yaml under role_profiles. Each profile has tier1 (weight 1.0) and tier2 (weight 0.4) keywords.
scrape:
hours_old: 48 # only fetch jobs posted in last N hours
results_wanted: 100 # per source per search term
country_indeed: "UK"
linkedin_fetch_description: true
search_terms:
- "software engineer"
- "python developer"
- "backend engineer"
- "ml engineer"
- "llm engineer"
- "data scientist"
filters:
location_must_contain: ["london", "remote"] # keep only London or remote
salary_floor_gbp: 41700 # minimum salary threshold
salary_floor_strict: false # true = drop jobs with no salary data
scoring:
weights: { keyword: 0.50, salary: 0.25, visa: 0.25 }
salary_cap_gbp: 120000 # salary score = 1.0 at this point
sources:
jobspy: { enabled: true }
adzuna: { enabled: true, app_id: "YOUR_ID", app_key: "YOUR_KEY" }
reed: { enabled: true, api_key: "YOUR_KEY" }
jobicy: { enabled: true }
remoteok: { enabled: true }
jsearch: { enabled: false, api_key: "YOUR_KEY" } # enable when paid| Source | Cost | Auth | Notes |
|---|---|---|---|
| Indeed + LinkedIn (jobspy) | Free | None | High reliability |
| Adzuna | Free | app_id + app_key | UK-native, strong coverage |
| Reed.co.uk | Free | API key | UK-only board |
| Jobicy | Free | None | Remote jobs, geo=uk filter |
| RemoteOK | Free | None | Remote-only |
| JSearch (RapidAPI) | $15/mo | RapidAPI key | Google for Jobs aggregator, off by default |
To run automatically every morning at 08:00, add to crontab -e:
0 8 * * * cd /path/to/findwithjobspy && /usr/bin/python3 main.py scrape --report-fmt html >> /tmp/jobhunt.log 2>&1Use the full path to your virtualenv Python if applicable:
0 8 * * * cd /path/to/findwithjobspy && /path/to/.venv/bin/python main.py scrape >> /tmp/jobhunt.log 2>&1View the log: tail -f /tmp/jobhunt.log
The pipeline is fully idempotent — job_url UNIQUE constraint plus the dedup layer means re-runs insert 0 duplicates.
CREATE TABLE jobs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_url TEXT UNIQUE NOT NULL,
title TEXT,
company TEXT,
location TEXT,
is_remote INTEGER,
description TEXT,
job_type TEXT,
salary_min REAL,
salary_max REAL,
salary_annual_gbp REAL,
salary_currency TEXT,
salary_interval TEXT,
source TEXT,
date_posted TEXT,
date_scraped TEXT NOT NULL,
score_total REAL,
score_keyword REAL,
score_salary REAL,
score_visa REAL,
cv_match TEXT,
status TEXT DEFAULT 'not_applied',
status_updated TEXT,
notes TEXT
);Source fails silently: Each source failure is logged to stderr and the scrape_runs table. Run python main.py track stats to see error counts.
No jobs returned: Check config.yaml - hours_old: 48 is strict. Try increasing to 168 (1 week) for first run.
Adzuna/Reed returns 401: API key is wrong or not set. Check config.yaml.
LinkedIn returns 0 results: python-jobspy LinkedIn scraping can be rate-limited. Reduce results_wanted or add a delay.
DB locked error: Another process is using data/jobs.db. Only run one scrape at a time.