Personal
GSoC 2026 Explorer
A searchable snapshot of Google Summer of Code 2026 organizations and their project ideas, collected from heterogeneous websites and normalized into a reusable Markdown corpus.
The recorded snapshot covers 184 Google Summer of Code 2026 organizations, with ideas content collected for 180 of them. The site makes that corpus searchable and the repository retains it as Markdown for reuse outside the browser.
- Coverage
- 184 organizations, 180 with ideas content
- Corpus
- 574 sub-pages · 940 Markdown files
- Site
- Statically exported, client-side fuzzy search
- Repository
- VincenzoImp/gsoc-2026-explorer
The problem
The organization list comes from a clean public API. The ideas do not. Each organization points at a URL of its own choosing, and those URLs are GitHub blobs, GitHub wikis, gists, GitLab pages, Google Docs, issue trackers, plain HTML, and pages that render only after JavaScript runs.
Eight formats, one desired output. And a scraper pointed at a hundred and eighty arbitrary web pages does not merely fail: it succeeds at fetching the wrong thing, and stores it.
What I built
A scraper that classifies each URL and dispatches to the right extractor: raw fetches for GitHub and GitLab sources, an export path for Google Docs, the GitHub CLI for issue trackers, readability extraction for generic HTML, and a headless browser for anything that needs JavaScript. Everything ends up as normalised Markdown, committed to the repository.
184
organizations
574
sub-pages crawled
It also follows links out of an ideas page and fetches those as sub-pages, with slug generation, deduplication and relative-link resolution, so an organization that splits its ideas across a dozen wiki pages can be represented by more than its table of contents. Failed or inaccessible pages still limit coverage.
The site is a static export with a prebuilt search index and weighted fuzzy matching, so the whole thing is a folder of files with no backend.
Hard parts
Knowing when you fetched a page and not the page. The most valuable part of the scraper is a list of patterns that identify content which is not content: Cloudflare interstitials, proof-of-work challenges, “verify you are human”, login walls, content-management chrome, and 404 bodies. Each is labelled so a rejection is legible in the log. Without that check, an anti-bot page is just a successful fetch with unusual text, and it enters the corpus silently.
Sanitising at the right end. Markdown sanitisation initially happened at render time and broke legitimate scraped content, so it moved to scrape time: clean the corpus once, on the way in, rather than every time it is displayed.