Larbin

Jul 20, 2023

HTTP crawler with an easy interface

Larbin is a powerful web crawler also called [web] robot, spider…. It is intended to fetch a large number of web pages to fill the database of a search engine. With a network fast enough, Larbin is able to fetch more than 100 million pages on a standard PC.

Larbin was initially developed for the XYLEME project in the VERSO team at INRIA. The goal of Larbin was to go and fetch XML pages on the web to fill the database of an xml-oriented search engine.

The following can be done with Larbin

o A crawler for a search engine
o A crawler for a specialized search enginer xml, images, mp3...
o Statistics on the web about servers or page contents

Larbin is created by Sebastien Ailleret

See also http//larbin.sourceforge.net/ See also https//www.sourceforge.net/projects/larbin



Checkout these related ports:
  • Zope213 - Object-based web application platform Version 2.13
  • Zola - Fast static site generator
  • Zgrab2 - Fast Go application scanner
  • Zerowait-httpd - Lightweight and fast http server
  • Zenphoto - Simpler web photo gallery
  • Zend-framework - Framework for developing PHP web applications
  • Yuicompressor - The Yahoo! JavaScript and CSS Compressor
  • Ytdl - YouTube downloader written in Go
  • Yt-dlp - Command-line program for downloading videos from various platforms
  • Youtube_dl - Program for downloading videos from various services
  • Yourls - Your Own URL Shortener
  • You-get - Dumb downloader that scrapes the web
  • Yaws - Web server for dynamic content written in Erlang
  • Yarr - Yet another rss reader
  • Yarn - Package manager for node, alternative to npm (meta port)