HTML scraper for tech news
A Selenium tool that works through a spreadsheet of URLs on its own and turns public web pages into structured data.
- Year
- 2024
- Tools
- Python, Selenium, Excel, JSON, CSV
The aim
Build a tool that scrapes public HTML content from a wide range of technology news websites.
How it works
Target URLs go in an Excel spreadsheet. The tool works through them on its own, so you can leave it running while you do other work. It copes with different page layouts and tags, recovers when a site’s structure changes, and saves the content in a structured format (JSON, CSV or a database). Results can be filtered by category or key attributes.
We use it for data mining in bulk and stay within data laws and each site’s terms throughout.
What we delivered
- A modular Python scraper.
- Documentation on using and maintaining it.
- A sample dataset scraped from selected sites.
The outcome
Fast, reliable collection of news content to feed analysis and research. It’s the same pipeline that supplies BettMAT.