Sub-domain enumeration

Usr6 · November 12, 2017

As a penetration tester or a bug bounty hunter, most of the times you are given a single domain or a set of domains when you start a security assessment. You’ll have to perform extensive reconnaissance to find interesting assets like servers, web applications, domains that belong to the target organisation so that you can increase your chances of finding vulnerabilities.

We wrote an extensive blog post on Open Source Intelligence Gathering techniques that are typically used in the reconnaissance phase.

Sub-domain enumeration is an essential part of the reconnaissance phase. This blog post covers various sub-domain enumeration techniques in a crisp and concise manner.

A gitbook will be released as a follow up for this blog post on the same topic where we cover these techniques in-depth. We covered some of these techniques in the “Esoteric sub-domain enumeration techniques” talk given at Bugcrowd LevelUp conference 2017.

What is sub-domain enumeration?

Sub-domain enumeration is the process of finding sub-domains for one or more domain(s). It is an essential part of the reconnaissance phase.

Why sub-domain enumeration?

Sub-domain enumeration can reveal a lot of domains/sub-domains that are in scope of a security assessment which in turn increases the chances of finding vulnerabilities
Finding applications running on hidden, forgotten sub-domains may lead to uncovering critical vulnerabilities
Often times the same vulnerabilities tend to be present across different domains/applications of the same organization

The famous Yahoo! Voices hack happened due to a vulnerable application deployed on a yahoo.com sub-domain

Sub-domain enumeration techniques

1. Search engines like Google and Bing supports various advanced search operators to refine search queries. These operators are often referred to as “Google dorks”.

We can use “site:” operator in Google search to find all the sub-domains that Google has found for a domain. Google also supports additional minus operator to exclude sub-domains that we are not interested in “site:*.wikimedia.org -www -store -jobs -uk”

Using site operator in Google search to find sub-domains

Bing search engine supports some advanced search operators as well. Like Google, Bing also supports a “site:” operator that you might want to check for any additional results apart from the Google search

Finding sub-domains using “site:” operator in Bing

2. There are a lot of the third party services that aggregate massive DNS datasets and look through them to retrieve sub-domains for a given domain.

VirusTotal runs its own passive DNS replication service, built by storing DNS resolutions performed when visiting URLs submitted by users. In order to retrieve the information of a domain you just have to put domain name in the search bar

Searching for sub-domains using virustotal

sub-domains found using VirusTotal

DNSdumpster is another interesting tools that can find potentially large number of sub-domains for a given domain

Searching for sub-domains using DNSdumpster

Sublist3r is a popular tool that’ll enumerate sub-domains using various sources. Sublist3r enumerates sub-domains using many search engines such as Google, Yahoo, Bing, Baidu, and Ask. Sublist3r also enumerates sub-domains using Netcraft, Virustotal, ThreatCrowd, DNSdumpster, and ReverseDNS.

sub-domain enumeration using Sublist3r

3. Certificate Transparency(CT) is a project under which a Certificate Authority(CA) has to publish every SSL/TLS certificate they issue to a public log. An SSL/TLS certificate usually contains domain names, sub-domain names and email addresses. This makes them a treasure trove of information for attackers. I wrote a series of technical blog posts on Certificate Transparency where I covered this technique in-depth, you can read the series here.

The easiest way to lookup certificates issued for a domain is to use search engines that collect the CT logs and let’s anyone search through them. Few of the popular ones are listed below -

Finding sub-domains of an organisation’s primary domain using crt.sh

We wrote few scripts to simplify the process of finding sub-domains using CT log search engines. The scripts are available in our github repo — https://github.com/appsecco/bugcrowd-levelup-subdomain-enumeration

Interesting sub-domain entry from CT logs for uber.com

The downside of using CT for sub-domain enumeration is that the domain names found in the CT logs may not exist anymore and thus they can’t be resolved to an IP address. You can use tools like massdns in conjunction with CT logs to quickly identify resolvable domain names.

# ct.py - extracts domain names from CT Logs(shipped with massdns)
# massdns - will find resolvable domains & adds them to a file

./ct.py icann.org | ./bin/massdns -r resolvers.txt -t A -q -a -o -w icann_resolvable_domains.txt -

Using massdns to find resolvable domain names

4. Dictionary based enumeration is another technique to find sub-domains with generic names. DNSRecon is a powerful DNS enumeration tool, one of it’s feature is to conduct dictionary based sub-domain enumeration using a pre-defined wordlist.

$ python dnsrecon.py -n ns1.insecuredns.com -d insecuredns.com -D subdomains-top1mil-5000.txt -t brt

Dictionary based enumeration using DNSRecon

5. Permutation scanning is another interesting technique to identify sub-domains. In this technique, we identify new sub-domains using permutations, alterations and mutations of already known domains/sub-domains.

Altdns is a tool that allows for the discovery of sub-domains that conform to patterns

$ python altdns.py -i icann.domains -o data_output -w icann.words -r -s results_output.txt

Finding sub-domains that match certain permutations/alterations using AltDNS

6. Finding Autonomous System (AS) Numbers will help us identify netblocks belonging to an organization which in-turn may have valid domains.

Resolve the IP address of a given domain using dig or host
There are tools to find ASN given an IP address — https://asn.cymru.com/cgi-bin/whois.cgi
There are tools to find ASN given a domain name — http://bgp.he.net/

Finding AS Number using IP address

The ASN numbers found can be used to find netblocks of the domain. There are Nmap scripts to achieve that — https://nmap.org/nsedoc/scripts/targets-asn.html

$ nmap --script targets-asn --script-args targets-asn.asn=17012 > netblocks.txt

Finding netblocks using AS numbers — NSE script

7. Zone transfer is a type of DNS transaction where a DNS server passes a copy of full or part of it’s zone file to another DNS server. If zone transfers are not securely configured, anyone can initiate a zone transfer against a nameserver and get a copy of the zone file. By design, zone file contains a lot of information about the zone and the hosts that reside in the zone.

$ dig +multi AXFR @ns1.insecuredns.com insecuredns.com

Successful zone transfer using DIG tool against a nameserver for a domain

8. Due to the way non-existent domains are handled in DNSSEC, it is possible to “walk” the DNSSEC zones and enumerate all the domains in that zone. You can learn more about this technique from here.

For DNSSEC zones that use NSEC records, zone walking can be performed using tools like ldns-walk

$ ldns-walk @ns1.insecuredns.com insecuredns.com

Zone walking DNSSEC zone with NSEC records

Some DNSSEC zones use NSEC3 records which uses hashed domain names to prevent attackers from gathering the plain text domain names. An attacker can collect all the sub-domain hashes and crack the hashes offline
Tools like nsec3walker, nsec3map help us automate the collecting NSEC3 hashes and cracking the hashes. Once you install nsec3walker, you can use the following commands to enumerate sub-domains of NSEC3 protected zone

# Collect NSEC3 hashes of a domain
$ ./collect icann.org > icann.org.collect

# Undo the hashing, expose the sub-domain information.
$ ./unhash < icann.org.collect > icann.org.unhash

# Listing only the sub-domain part from the unhashed data
$ cat icann.org.unhash | grep "icann" | awk '{print $2;}'
del.icann.org.
access.icann.org.
charts.icann.org.
communications.icann.org.
fellowship.icann.org.
files.icann.org.
forms.icann.org.
mail.icann.org.
maintenance.icann.org.
new.icann.org.
public.icann.org.
research.icann.org.

9. There are projects that gather Internet wide scan data and make it available to researchers and the security community. The datasets published by this projects are a treasure trove of sub-domain information. Although finding sub-domains in this massive datasets is like finding a needle in the haystack, it is worth the effort.

Forward DNS dataset is published as part of Project Sonar. This data is created by extracting domain names from a number of sources and then sending an ANY query for each domain. The data format is a gzip-compressed JSON file. We can parse the dataset to find sub-domains for a given domain. The dataset is massive though(20+GB compressed, 300+GB uncompressed)

# Command to parse & extract sub-domains for a given domain

$ curl -silent https://scans.io/data/rapid7/sonar.fdns_v2/20170417-fdns.json.gz | pigz -dc | grep “.icann.org” | jq

Enumerating sub-domains using FDNS dataset

Sub-domain enumeration techniques — A comparison

We ran few of the discussed techniques against icann.org and compared the results. The bar chart below shows the number of unique, resolvable sub-domains each technique found for icann.org. Feel free to get in touch with us to know the methods we used to gather this information.

Number of unique, resolvable sub-domains each technique found for icann.org

Sub-domain enumeration — Reference

We created a simple reference for sub-domain enumeration techniques, tools and sources. This reference is created using a Github gist, feel free to fork, customise it— https://gist.github.com/yamakira/2a36d3ae077558ac446e4a89143c69ab

Quick reference for sub-domain enumeration

References

Thank you for reading this article. If you enjoyed it please let us know by clicking that little clap icon below.

Sursa: https://blog.appsecco.com/a-penetration-testers-guide-to-sub-domain-enumeration-7d842d5570f6

Sign In

Sub-domain enumeration

Recommended Posts

Usr6

What is sub-domain enumeration?

Why sub-domain enumeration?

Sub-domain enumeration techniques

Sub-domain enumeration techniques — A comparison

Sub-domain enumeration — Reference

References

Thank you for reading this article. If you enjoyed it please let us know by clicking that little clap icon below.

Join the conversation

Browse

Activity

Pages