August 2021 Survey Report

August Survey:

Our August survey represents our first official release of our crawling research. These reports will serve as a way for us not only to discuss the results of our crawls but the technical process of creating this technology. Working on this project has been an incredible adventure so far and we look forward to you taking this journey with us!

Survey Results:

High Level Survey Reports:

From a top level, our first HTTP crawl has yielded a great deal of interesting statistics about the adoption rates of many technologies across the internet. It is important to note that only the HTTP crawler is deployed in production crawling so these results are limited to that in the secondary crawl.

Port Ping Response Crawl Response
21 13016656 Pending Dev
80 60773485 19387484
443 53606644 2083897
7878 3385803 27962
8008 9079938 81633
8080 12263264 2291785
8888 5602088 1154780
9200 3868575 Pending Dev
9981 4798149 Pending Dev
11001 4908024 Pending Dev
47808 3204075 Pending Dev
50001 8383239 Pending Dev

Server Type Adoption Rates

Adoption rates are always an interesting look at the state of the internet. Technologies are rapidly replaced and obsolesced and the landscape is massively dynamic.

Server Type Count
nginx 3813748
unknown 3765476
Apache 3660319
Microsoft-IIS/10.0 733627
Microsoft-IIS/8.5 580386
DNVRS-Webs 557033
Microsoft-IIS/7.5 435336
Webs 403379
Apache/2.4.29 (Ubuntu) 399222
nginx/1.18.0 324084
Apache/2.4.41 (Ubuntu) 317789
nginx/1.14.0 (Ubuntu) 292996
SonicWALL 282804
micro_httpd 262466
Apache/2.4.18 (Ubuntu) 260325
nginx/1.16.1 259588
nginx/1.18.0 (Ubuntu) 254281
Apache-Coyote/1.1 212352
Nginx Microsoft-HTTPAPI/2.0 194509
Boa/0.94.14rc21 193325
LiteSpeed 182980
Apache/2.4.38 (Debian) 173570
Apache/2 161466
Apache/2.2.15 (CentOS) 160465
Apache/2.4.25 (Debian) 158716
IdeaWebServer/3.0.0 155090
nginx/1.10.3 (Ubuntu) 151313
nginx/1.14.2 121512
GoAhead-Webs 121471
nginx/1.20.0 120744
web 118241
nginx/1.20.1 114829
Apache/2.4.7 (Ubuntu) 101990

Responsible Disclose For August

Our first crawl has yielded over 1000 vulnerable high value targets that have been responsibly disclosed to government agencies and the end customers. For the time being these disclosures will remain unreleased publicly until we can insure that all vulnerable machines are patched.

Technical Challenges

Since in general, our technology is still under heavy development, this month has exposed a variety of interesting challenges. To start, our stack is primarily Golang, Bash, ArangoDB, and NGINX. This has allowed us to scale at a tremendous rate with some a variety of growing pains.

Zmap Json Bug

This bug ended up resulting in us needing to throw away all our results after discovering a major flaw in the zmap Json module. Anyone using this in their research undoubtedly must throw away all their results as the IP’s returned are not real responses. This bug has been reported to the zmap team but it is unclear how or when this will be patched. We have fallen back on the CSV module which seems still to be reporting IP responses correctly. Thankfully this only resulted in a small rework of our distributed zmap worker system.

Distributed Locking at Scale

This so far has been our most challenging issue. We use ArangoDB as a Command and Control surface for all our workers. For the most part this is a highly efficient method for managing thousands of jobs distributed across hundreds of workers and sub-processes. We have had some cases where workers, for whatever reason, become perfectly synchronized. Basically this means that a job is acquired at the exact same time by two different workers before the job can be locked. This is mostly my own oversight in not realizing that a synchronized api would be 100% necessary when scaling past a certain point. This has been mostly mitigated by randomizing a wait time before a worker gets a job, but this is only a bandage.

To fix this a few options are possible. A foxx API may be the answer but I’m not convinced that this will totally solve the issue at a larger scale. I think another possible option may be better logic on the worker end to ensure that it is not locked in tandem with another worker. Overall this will be an interesting challenge and a chance to maximise the power of ArangoDB.

Colly Deadlocking Issue (Potential Golang Source Bug)

Often during crawls we deal with honey pots and anti-crawl mechanisms. This has, for some reason, become an extreme issue on port 8008. My current guess is that honeypots create endless connections that are deadlocking a few jobs infinitely. I have set very aggressive timeouts which you can see below, but even still typically in a job of 10k IP’s, 4-15 will fully deadlock and c.Wait() never completes. We have enabled a variety of deadlines to help kill this job but even in the lowest level Golang ‘HTTP’ package, the workers never are able to close.

This issue has been reported to the Colly developers but again it seems unclear what the fix is. For the time being we have resorted to crashing workers after a timeout and skipping failed bricks of IP’s. While this solution is not optimal it has allowed our crawlers to successfully recover and skip failed blocks.

// Create Colly Collector
       c := colly.NewCollector(
       c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 500})
       c.SetRequestTimeout(5 * time.Second)
               Proxy: http.ProxyFromEnvironment,
               DialContext: (&net.Dialer{
                       Timeout:   5 * time.Second,
                       KeepAlive: 5 * time.Second,
                       DualStack: true,
               MaxIdleConns:          500,
               IdleConnTimeout:       5 * time.Second,
               TLSHandshakeTimeout:   5 * time.Second,
               ExpectContinueTimeout: 1 * time.Second,