Data Mine Scraper

Sep – Oct 2023

While I was part of Purdue’s The Data Mine, I built tdmscrape to avoid formatting each project notebook by hand. It was my first Go project. It preserves the assignment’s questions and subquestions from The Examples Book and adds empty code cells and response areas. Here’s a picture!

Data Mine Scraper architectureA Go CLI scrapes assignment HTML and collects notebook details, then combines them into Markdown cells and uses Pandoc to create a Jupyter notebook.tdmscrapeScrape The ExamplesBookColly & goqueryQuery user for: name +project detailsBubble TeaParse HTML response &generate MarkdownPandoc.md → .ipynbJupyter notebook
Flowchart of Logic in the scraping tool

The scraper uses Colly and goquery to process assignment pages.