Select a tag to browse associated projects and drill deeper into the tag cloud.
OpenWebSpider - The Open Source Web Spider And Search Engine The OpenWebSpider project was born from the idea that internet is free and all informations must be freely available for all users! Using all free software and being Open Source, OpenWebSpider would be the base for a new Search engine ... [More]
Check websites and HTML documents for broken links. * recursive and multithreaded checking * output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats * HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support * restriction of link ... [More]
Spidr is a versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Anemone is a Ruby library that makes it quick and painless to write programs that spider a website. It provides a simple DSL for performing actions on every page of a site, skipping certain URLs, and calculating the shortest path to a given page on a site. The multi-threaded design makes Anemone ... [More]