← All projects

Structured Data Collection in Python

Repeatable Python workflows for collecting and parsing web data into structured datasets, using Selenium and requests.

  • Python
  • Selenium
  • requests
  • Web Scraping

Overview

Useful information on the web is spread across sources that were never designed to be read programmatically. This work built repeatable collection and parsing workflows that turn that into structured data suitable for analysis.

Technical approach

  • Browser automation: Selenium for pages that only assemble their content once JavaScript has run
  • HTTP collection: requests where a page could be fetched directly, avoiding the cost of a full browser
  • Output: clean datasets exported for downstream analysis

What it built

Practical experience with the parts of web data collection that decide whether it is worth having: choosing between browser automation and direct HTTP, and structuring the output so the data is actually analysable rather than merely captured.