The FreeBSD Ports Archive
FreeBSD www  : crawl
A small, efficient web crawler with advanced features
The crawl utility starts a depth-first traversal of the web at the
specified URLs. It stores all JPEG images that match the configured
constraints. Crawl is fairly fast and allows for graceful termination.
After terminating crawl, it is possible to restart it at exactly
the same spot where it was terminated. Crawl keeps a persistent
database that allows multiple crawls without revisiting sites.
The main reason for writing crawl was the lack of simple open source
web crawlers. Crawl is only a few thousand lines of code and fairly
easy to debug and customize.
Some of the main features:
- Saves encountered JPEG images
- Image selection based on regular expressions and size contrainsts
- Resume previous crawl after graceful termination
- Persistent database of visited URLs
- Very small and efficient code
- Supports robots.txt
http://www.monkey.org/~provos/crawl/
|
crawl history
v. 1.22
date: 2007/09/21 20:21:30; author: mnag; state: Exp; lines: +2 -2
- Change libevent lib and bump PORTREVISION since devel/libevent are updated.
v. 1.21
date: 2007/04/06 18:28:46; author: mnag; state: Exp; lines: +2 -2
- Bump PORTREVISION and change LIB_DEPENDS to reflect libevent update.
v. 1.20
date: 2006/12/18 15:15:00; author: leeym; state: Exp; lines: +4 -4
- utilize USE_BDB
v. 1.19
date: 2006/12/05 13:29:01; author: mnag; state: Exp; lines: +4 -4
- Bump PORT_REVISION and change LIB_DEPENDS to reflect update in devel/libevent
- Fix many wrong BUILD_DEPENDS. Thanks to ldd(1)
v. 1.18
date: 2006/05/13 04:41:09; author: edwin; state: Exp; lines: +0 -1
Remove USE_REINPLACE from categories starting with W
v. 1.17
date: 2006/04/14 20:45:43; author: linimon; state: Exp; lines: +1 -1
Reset petef due to no response to email. We hope to see him back sometime.
Hat: portmgr
v. 1.16
date: 2005/07/19 02:18:32; author: kris; state: Exp; lines: +0 -4
Now builds on 6.x
v. 1.15
date: 2005/07/01 22:47:21; author: jylefort; state: Exp; lines: +1 -1
Chase the libevent update.
Reported by: pointyhat
v. 1.14
date: 2005/06/02 20:14:55; author: ume; state: Exp; lines: +1 -0
Our getaddrinfo(3) never returns EAI_NODATA on 5.2-RELEASE and
later as RFC 3493 deprecated it. So, we have to see EAI_NONAME
instead.
Approved by: petef
v. 1.13
date: 2005/04/10 21:35:28; author: kris; state: Exp; lines: +7 -1
BROKEN on FreeBSD 6.0: Does not compile
v. 1.12
date: 2004/03/12 13:43:32; author: petef; state: Exp; lines: +1 -7
Unbreak on -current.
v. 1.11
date: 2004/02/06 08:13:11; author: kris; state: Exp; lines: +7 -1
BROKEN on 5.x: does not compile
v. 1.10
date: 2003/12/21 18:39:12; author: petef; state: Exp; lines: +5 -11
Update to 0.4.
v. 1.9
date: 2003/05/05 03:16:00; author: petef; state: Exp; lines: +12 -5
- fix build on -current
- no longer need to depend on libmd5 from libwww, libmd does fine
- install an example crawl.conf
PR: 51696
Submitted by: Rui Lopes
v. 1.8
date: 2003/03/07 06:12:36; author: ade; state: Exp; lines: +1 -0
Clear moonlight beckons.
Requiem mors pacem pkg-comment,
And be calm ports tree.
E Nomini Patri, E Fili, E Spiritu Sancti.
v. 1.7
date: 2002/11/21 23:56:43; author: edwin; state: Exp; lines: +3 -1
PERL -> REINPLACE_CMD
Noticed on: bento
v. 1.6
date: 2002/08/22 19:13:44; author: ade; state: Exp; lines: +2 -1
BerkeleyDB cleanup - stage 2
Update databases/db3 to 3.3.11, and fix the few ports that need sorting
after the shlib version update, and a slight API change from 3.2.x->3.3.x
v. 1.5
date: 2002/02/04 00:39:19; author: petef; state: Exp; lines: +6 -2
Update to 0.3
v. 1.4
date: 2001/12/23 10:14:50; author: petef; state: Exp; lines: +4 -3
- update to 0.2
- remove redundant GNU_CONFIGURE (this is implied by USE_AUTOCONF)
v. 1.3
date: 2001/08/22 23:07:06; author: petef; state: Exp; lines: +2 -2
Change my email address to petef@FreeBSD.org for the ports I maintain.
Reviewed by: ade
v. 1.2
date: 2001/07/04 06:23:38; author: dwcjr; state: Exp; lines: +2 -2
Update to 1.0b
PR: 28683
Submitted by: maintainer
v. 1.1
date: 2001/06/23 16:09:49; author: dwcjr; state: Exp;
Add crawl, a worm that searches for jpegs
PR: 28304
Submitted by: Pete Fritchman
=============================================================================
v. 1.22
date: 2007/09/21 20:21:30; author: mnag; state: Exp; lines: +2 -2
- Change libevent lib and bump PORTREVISION since devel/libevent are updated.
v. 1.21
date: 2007/04/06 18:28:46; author: mnag; state: Exp; lines: +2 -2
- Bump PORTREVISION and change LIB_DEPENDS to reflect libevent update.
v. 1.20
date: 2006/12/18 15:15:00; author: leeym; state: Exp; lines: +4 -4
- utilize USE_BDB
v. 1.19
date: 2006/12/05 13:29:01; author: mnag; state: Exp; lines: +4 -4
- Bump PORT_REVISION and change LIB_DEPENDS to reflect update in devel/libevent
- Fix many wrong BUILD_DEPENDS. Thanks to ldd(1)
v. 1.18
date: 2006/05/13 04:41:09; author: edwin; state: Exp; lines: +0 -1
Remove USE_REINPLACE from categories starting with W
v. 1.17
date: 2006/04/14 20:45:43; author: linimon; state: Exp; lines: +1 -1
Reset petef due to no response to email. We hope to see him back sometime.
Hat: portmgr
v. 1.16
date: 2005/07/19 02:18:32; author: kris; state: Exp; lines: +0 -4
Now builds on 6.x
v. 1.15
date: 2005/07/01 22:47:21; author: jylefort; state: Exp; lines: +1 -1
Chase the libevent update.
Reported by: pointyhat
v. 1.14
date: 2005/06/02 20:14:55; author: ume; state: Exp; lines: +1 -0
Our getaddrinfo(3) never returns EAI_NODATA on 5.2-RELEASE and
later as RFC 3493 deprecated it. So, we have to see EAI_NONAME
instead.
Approved by: petef
v. 1.13
date: 2005/04/10 21:35:28; author: kris; state: Exp; lines: +7 -1
BROKEN on FreeBSD 6.0: Does not compile
v. 1.12
date: 2004/03/12 13:43:32; author: petef; state: Exp; lines: +1 -7
Unbreak on -current.
v. 1.11
date: 2004/02/06 08:13:11; author: kris; state: Exp; lines: +7 -1
BROKEN on 5.x: does not compile
v. 1.10
date: 2003/12/21 18:39:12; author: petef; state: Exp; lines: +5 -11
Update to 0.4.
v. 1.9
date: 2003/05/05 03:16:00; author: petef; state: Exp; lines: +12 -5
- fix build on -current
- no longer need to depend on libmd5 from libwww, libmd does fine
- install an example crawl.conf
PR: 51696
Submitted by: Rui Lopes
v. 1.8
date: 2003/03/07 06:12:36; author: ade; state: Exp; lines: +1 -0
Clear moonlight beckons.
Requiem mors pacem pkg-comment,
And be calm ports tree.
E Nomini Patri, E Fili, E Spiritu Sancti.
v. 1.7
date: 2002/11/21 23:56:43; author: edwin; state: Exp; lines: +3 -1
PERL -> REINPLACE_CMD
Noticed on: bento
v. 1.6
date: 2002/08/22 19:13:44; author: ade; state: Exp; lines: +2 -1
BerkeleyDB cleanup - stage 2
Update databases/db3 to 3.3.11, and fix the few ports that need sorting
after the shlib version update, and a slight API change from 3.2.x->3.3.x
v. 1.5
date: 2002/02/04 00:39:19; author: petef; state: Exp; lines: +6 -2
Update to 0.3
v. 1.4
date: 2001/12/23 10:14:50; author: petef; state: Exp; lines: +4 -3
- update to 0.2
- remove redundant GNU_CONFIGURE (this is implied by USE_AUTOCONF)
v. 1.3
date: 2001/08/22 23:07:06; author: petef; state: Exp; lines: +2 -2
Change my email address to petef@FreeBSD.org for the ports I maintain.
Reviewed by: ade
v. 1.2
date: 2001/07/04 06:23:38; author: dwcjr; state: Exp; lines: +2 -2
Update to 1.0b
PR: 28683
Submitted by: maintainer
v. 1.1
date: 2001/06/23 16:09:49; author: dwcjr; state: Exp;
Add crawl, a worm that searches for jpegs
PR: 28304
Submitted by: Pete Fritchman
=============================================================================
v. 1.22
date: 2007/09/21 20:21:30; author: mnag; state: Exp; lines: +2 -2
- Change libevent lib and bump PORTREVISION since devel/libevent are updated.
v. 1.21
date: 2007/04/06 18:28:46; author: mnag; state: Exp; lines: +2 -2
- Bump PORTREVISION and change LIB_DEPENDS to reflect libevent update.
v. 1.20
date: 2006/12/18 15:15:00; author: leeym; state: Exp; lines: +4 -4
- utilize USE_BDB
v. 1.19
date: 2006/12/05 13:29:01; author: mnag; state: Exp; lines: +4 -4
- Bump PORT_REVISION and change LIB_DEPENDS to reflect update in devel/libevent
- Fix many wrong BUILD_DEPENDS. Thanks to ldd(1)
v. 1.18
date: 2006/05/13 04:41:09; author: edwin; state: Exp; lines: +0 -1
Remove USE_REINPLACE from categories starting with W
v. 1.17
date: 2006/04/14 20:45:43; author: linimon; state: Exp; lines: +1 -1
Reset petef due to no response to email. We hope to see him back sometime.
Hat: portmgr
v. 1.16
date: 2005/07/19 02:18:32; author: kris; state: Exp; lines: +0 -4
Now builds on 6.x
v. 1.15
date: 2005/07/01 22:47:21; author: jylefort; state: Exp; lines: +1 -1
Chase the libevent update.
Reported by: pointyhat
v. 1.14
date: 2005/06/02 20:14:55; author: ume; state: Exp; lines: +1 -0
Our getaddrinfo(3) never returns EAI_NODATA on 5.2-RELEASE and
later as RFC 3493 deprecated it. So, we have to see EAI_NONAME
instead.
Approved by: petef
v. 1.13
date: 2005/04/10 21:35:28; author: kris; state: Exp; lines: +7 -1
BROKEN on FreeBSD 6.0: Does not compile
v. 1.12
date: 2004/03/12 13:43:32; author: petef; state: Exp; lines: +1 -7
Unbreak on -current.
v. 1.11
date: 2004/02/06 08:13:11; author: kris; state: Exp; lines: +7 -1
BROKEN on 5.x: does not compile
v. 1.10
date: 2003/12/21 18:39:12; author: petef; state: Exp; lines: +5 -11
Update to 0.4.
v. 1.9
date: 2003/05/05 03:16:00; author: petef; state: Exp; lines: +12 -5
- fix build on -current
- no longer need to depend on libmd5 from libwww, libmd does fine
- install an example crawl.conf
PR: 51696
Submitted by: Rui Lopes
v. 1.8
date: 2003/03/07 06:12:36; author: ade; state: Exp; lines: +1 -0
Clear moonlight beckons.
Requiem mors pacem pkg-comment,
And be calm ports tree.
E Nomini Patri, E Fili, E Spiritu Sancti.
v. 1.7
date: 2002/11/21 23:56:43; author: edwin; state: Exp; lines: +3 -1
PERL -> REINPLACE_CMD
Noticed on: bento
v. 1.6
date: 2002/08/22 19:13:44; author: ade; state: Exp; lines: +2 -1
BerkeleyDB cleanup - stage 2
Update databases/db3 to 3.3.11, and fix the few ports that need sorting
after the shlib version update, and a slight API change from 3.2.x->3.3.x
v. 1.5
date: 2002/02/04 00:39:19; author: petef; state: Exp; lines: +6 -2
Update to 0.3
v. 1.4
date: 2001/12/23 10:14:50; author: petef; state: Exp; lines: +4 -3
- update to 0.2
- remove redundant GNU_CONFIGURE (this is implied by USE_AUTOCONF)
v. 1.3
date: 2001/08/22 23:07:06; author: petef; state: Exp; lines: +2 -2
Change my email address to petef@FreeBSD.org for the ports I maintain.
Reviewed by: ade
v. 1.2
date: 2001/07/04 06:23:38; author: dwcjr; state: Exp; lines: +2 -2
Update to 1.0b
PR: 28683
Submitted by: maintainer
v. 1.1
date: 2001/06/23 16:09:49; author: dwcjr; state: Exp;
Add crawl, a worm that searches for jpegs
PR: 28304
Submitted by: Pete Fritchman
=============================================================================
|
| |
|