Web Scraper

Worker(s): tug Reward: 1 Stars WIP

Build a web crawler that given a url will walk through all the links for that given domain and download each page, rewriting the links to be local as it goes. This web crawler should be written as a thread that writes to Clay.

A completed app should:

Take a url or set or urls, connect to the given url over HTTP and download each page, following links on the same domain and rewriting the links to be local as it goes. This should function like wget with the --mirror, --convert-links, and --page-requisites options.

Resources

wget

Contribution Guidelines

Do not begin work until your request to claim this bounty is accepted. We will assign a designer to work with you on the interface.

Submit your code as a PR to the Urbit repository.

Milestones

PR is merged

1 stars Your pull request is merged.