A Spider for WhoScored

This is a scrapy project for WhoScored web scraping.

Overview

Requirements

• Python 3.6+
• Scrapy 1.6.0
• Fully tested on Linux, but it should works on Windows, Mac OSX, BSD

Usage

Run Sentry

Initial postgres with senty first:

1. Generate secret key first:

docker run --rm sentry config generate-secret-key

2. Use the secret key to create a database in postgres:

docker run --detach \
    --name sentry-redis-init \
    --volume $PWD/redis-data:/data \
    redis
docker run --detach \
    --name sentry-postgres-init \
    --env POSTGRES_PASSWORD=secret \
    --env POSTGRES_USER=sentry \
    --volume $PWD/postgres-data:/var/lib/postgresql/data \
    postgres
docker run --interactive --tty --rm \
    --env SENTRY_SECRET_KEY='<secret-key>' \
    --link sentry-postgres-init:postgres \
    --link sentry-redis-init:redis \
    sentry upgrade

Then input the superusername and password

3. Stop the redis and postgres:

docker stop sentry-postgres-init sentry-redis-init && docker rm sentry-postgres-init senty-redis-init

4. Edit the env files to add the superusername, password and database related information

5. Start sentry with docker-compose.yml:

docker-compose up --detach && docker-compose logs --follow

Run MongoDB for httpcache

Run Percona Server MongoDB for cookies

Debug with mitmproxy

Start mitmproxy

Enable httpproxy middleware

Debug in mitmproxy

Server-side Replay

Follow Features of mitmproxy:

https://docs.mitmproxy.org/stable/overview-features/#server-side-replay

TODO