Larbin

Top > All Packages in Directory > Larbin

Larbin - Web crawler

Larbin is an HTTP Web crawler that can fetch more than 5 million pages a day on a standard PC (pentium II 300, 128 Mo SDRAM and a 10 Mbit ethernet card, with a good network). Larbin uses standard libraries, plus adns. The program is multithreaded but prefers using select instead of a lot of threads (for efficiency purposes). The advantage of Larbin over wget or ht://dig is that it is much faster (because it opens a lot of connexions at a time) and very easy to customize).

Common uses include: a crawler for a standard search engine, a crawler for a specialized search engine (xml, images, mp3...), and to provide statistics about servers or page contents).

Obtaining

Web page http://larbin.sourceforge.net/index-eng.html
Source tarball http://prdownloads.sourceforge.net/larbin/larbin-2.6.3.tar.gz?download
Version 2.6.3 (stable) released on 2002-07-16

Licensed under The GNU General Public License, Version 2.

This is not a GNU package.

Documentation

User guide included

Support contacts

Help List	<sebastien@ailleret.com>
Developer List	<sebastien@ailleret.com>
Bug List	<sebastien@ailleret.com>

Project contacts

Maintainers	Sebastien Ailleret <sebastien@ailleret.com>
Developers	Sebastien Ailleret <sebastien@ailleret.com> see CREDITS file in the distribution for the full listing

Related information

Interfaces	web
Source languages	C++
Related programs	Wget, ht://Dig

Entry information

License verified by	Janet Casey <jcasey@gnu.org> on 2001-04-04
Entry compiled by	Janet Casey <jcasey@gnu.org>

Categories

Top > All Packages in Directory
Top > Network Applications > Tools

The copyright licensing notice below applies to this text. The software described in this text has its own copyright notice and license, which can usually be found in the distribution itself.

Permission is granted to copy, distribute, and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of this license is included in the file COPYING.DOC.

Please report any problems in this page to bug-directory@gnu.org, or find out how you can help fix them.

The FSF provides this directory as a service to the free software community. Please consider donating to the FSF to help support this project.