YCrawler is a web crawler that is useful for grabbing all user supplied input related to a given website and will save the output. It has proxy and log file support.
You can download it from the following link: https://packetstormsecurity.com/files/download/98546/yCrawler.tar.gz
Source: https://packetstormsecurity.com/files/98546/yCrawler-Web-Crawling-Utility.html