crawl-e

CRAWL-E is a web crawling framework that seamlessly supports distributed crawling across multiple threads as well as multiple machines.

CRAWL-E was designed to crawl the web fast fast as possible with as little development time as possible. It is only a framework, and requires the development of a Handler module in order to function properly.

The CRAWL-E developers are very familiar with how TCP and HTTP works and using that knowledge have written a web crawler intended to maximize TCP throughput. This benefit is realized when crawling web servers that utilize persistent HTTP connections as numerous requests will be made over a single TCP connection thus increasing the throughput.

Other features of CRAWL-E are multiple HTTP request method support, the most basic being GET, POST, PUT, DELETE, HEAD.

CRAWL-E has been utilized in the data collection of:

Can Social Networks Improve e-Commerce: a Study on Social Marketplaces, Gayatri Swamynathan, Christo Wilson, Bryce Boe, Kevin C. Almeroth and Ben Y. Zhao, WOSN'08 User Interactions in Social Networks and their Implications, Christo Wilson, Bryce Boe, Alessandra Sala, Krishna P. N. Puttaswamy and Ben Y. Zhao, EuroSys'09