Implementaton of SerpBase provider#341
Open
Dietermartens wants to merge 4 commits into
Open
Conversation
Restore single quotes and spaces in types.d.ts, utils/scraper.ts, and scrapers/index.ts; align serpbase.ts with existing scraper modules; keep README table line within 80 characters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds SerpBase.dev as a selectable SERP scraper in Settings, and extends the scraper client so providers that use POST + JSON (not only GET)
Closes #338.
Changes
SerpBase provider
New scraper module: scrapers/services/serpbase.ts
Registered in scrapers/index.ts
Uses POST https://api.serpbase.dev/google/search with X-API-Key and a JSON body (q, hl, gl, page)
Maps organic results (url / link, rank / position) into SerpBear’s format
Documented in the README SERP services table
Scraper transport (reusable for future providers)
SerpBase requires POST; existing scrapers stay on GET. ScraperSettings and getScraperClient are extended with:
scrapeMethod?: 'GET' | 'POST' (default 'GET')
requestBody?(...) — JSON body for POST requests
API error handling is improved for SerpBase-style responses (status !== 0, error field) in scrapeSinglePage and scrapeKeywordFromGoogle.
Notes
SerpBase /google/search does not expose city or mobile device params in their docs; keywords with city/mobile still scrape, but without those targeting options.
SerpBase is not added to the parallel refresh list (same as Serper); refreshes run sequentially with scrape_delay.