← All projects

Web Data Collection & Automation

Repeatable Python workflows for collecting and parsing web data into structured datasets, using Selenium, BeautifulSoup and requests.

  • Python
  • Selenium
  • BeautifulSoup
  • requests
  • Web Scraping

Overview

Useful information on the web is spread across sources that were never designed to be read programmatically. This work built repeatable collection and parsing workflows that turn that into structured data suitable for analysis.

Technical approach

  • Browser automation: Selenium for pages that only assemble their content once JavaScript has run
  • Parsing: BeautifulSoup for extracting content from HTML into typed records
  • HTTP collection: requests where a page could be fetched directly, avoiding the cost of a full browser
  • Output: clean datasets exported for downstream analysis

What it built

Practical experience with the parts of web data collection that decide whether it is worth having: choosing between browser automation and direct HTTP, writing parsing logic that survives markup changes, and structuring the output so the data is actually analysable rather than merely captured.