Personal
GSoC 2026 Explorer
Every Google Summer of Code 2026 organization and its project ideas in one searchable place, which meant writing eight different extractors, because every organization publishes its ideas somewhere else.
Choosing where to apply for Google Summer of Code means reading 184 organizations’ project-ideas pages. This puts all of them in one place with full-text search, and commits the whole corpus as Markdown so it can be read by something other than a browser.
- Coverage
- 184 organizations, 180 with ideas content
- Corpus
- 574 sub-pages · 940 Markdown files
- Site
- Statically exported, client-side fuzzy search
- Repository
- VincenzoImp/gsoc-2026-explorer
The problem
The organization list comes from a clean public API. The ideas do not. Each organization points at a URL of its own choosing, and those URLs are GitHub blobs, GitHub wikis, gists, GitLab pages, Google Docs, issue trackers, plain HTML, and pages that render only after JavaScript runs.
Eight formats, one desired output. And a scraper pointed at a hundred and eighty arbitrary web pages does not merely fail: it succeeds at fetching the wrong thing, and stores it.
What I built
A scraper that classifies each URL and dispatches to the right extractor: raw fetches for GitHub and GitLab sources, an export path for Google Docs, the GitHub CLI for issue trackers, readability extraction for generic HTML, and a headless browser for anything that needs JavaScript. Everything ends up as normalised Markdown, committed to the repository.
184
organizations
574
sub-pages crawled
It also follows links out of an ideas page and fetches those as sub-pages, with slug generation, deduplication and relative-link resolution, so an organization that splits its ideas across a dozen wiki pages is captured whole rather than as a table of contents.
The site is a static export with a prebuilt search index and weighted fuzzy matching, so the whole thing is a folder of files with no backend.
Hard parts
Knowing when you fetched a page and not the page. The most valuable part of the scraper is a list of patterns that identify content which is not content: Cloudflare interstitials, proof-of-work challenges, “verify you are human”, login walls, content-management chrome, and 404 bodies. Each is labelled so a rejection is legible in the log. Without that check, an anti-bot page is just a successful fetch with unusual text, and it enters the corpus silently.
Sanitising at the right end. Markdown sanitisation initially happened at render time and broke legitimate scraped content, so it moved to scrape time: clean the corpus once, on the way in, rather than every time it is displayed.