Py-dns-crawler

Jul 20, 2023

Crawler for getting info about DNS domains and services

Despite the name, the crawler gets info for more services than just DNS

  • DNS
    • all A/AAAA records for the 2nd level domain and www.subdomain, annotated with GeoIP
    • TXT records with SPF and DMARC parsed for easier filtering
    • TLSA for the 2nd level domain and www.subdomain
    • MX
    • DNSSEC validation
    • nameservers
      • each server IP annotated with GeoIP
      • HOSTNAME.BIND, VERSION.BIND, AUTHORS.BIND and fortune also for all IPs
    • users can add custom additional RRs in the config file
  • E-mail for every server from MX
    • SMTP server banners optional, ports are configurable
    • TLSA records
  • Web
    • HTTP status & headers inc. parsed cookies for ports 80 & 443 on each IP from A/AAAA records
    • certificate info for HTTPS optionally with an entire cert chain
    • webpage content optional
    • everything of the above is saved for each step in the redirect history – the crawler follows redirects until it gets a non-redirecting status or hits a configurable limit


Checkout these related ports:
  • Zonenotify - DNS notify sender written in C
  • Zonecheck - Perform consistency checks on DNS zones
  • Zkt -
  • Yandex-ddns - Use Yandex DNS as a dynamic DNS provider
  • Yadifa - Authoritative nameserver with DNSSEC capabilities
  • Wrapsrv - DNS SRV record command line wrapper
  • Whoseip - Get information about IP addresses country code and network
  • Wdns - Low-level DNS library
  • Walker - Recover zone file information from servers that use DNSSEC
  • Void-zones-tools - Prepare a list of void zones that can be readily fed into Unbound
  • Vizone - Updates the serial number in one or more zonefiles
  • Vhostcname - Synchronize DNS with server names and aliases from Apache vhost
  • Utdns - Proxy UDP/DNS to TCP/DNS
  • Updatedd - Dynamic DNS Update Client supporting multiple services
  • Unbound - Validating, recursive, and caching DNS resolver