Adding compression support can be very simple -- if your spider is coded in Perl using LWP::UserAgent, then the
addition of a single line of code will enable compression support.
$ua->default_header('Accept-Encoding' => 'gzip');
and then you need to make sure that you always refer to 'decoded_content' when dealing with the response object.
For other languages, all
you need to do is to add
Accept-encoding: gzipto the HTTP request that you send, and then be prepared to deal with a 'content-encoding: gzip' in the response.
Happily, some of the large spiders do support compression -- the googlebot and Yahoo Slurp do (to name but two). Since I started prodding crawler implementors, a couple have implemented compression (one within hours), and another reported that it was a bug that it didn't work -- which would be fixed shortly.
Crawlers which do more than 5% of the total (uncompressed) crawling activity are marked in bold below.
| Crawler | Last IP used |
|---|---|
| ia_archiver" "pond.gladstonefamily.net | 54.173.35.129 |
| ia_archiver" "pond1.gladstonefamily.net | 54.173.35.129 |
| masscan/1.0 (https://github.com/robertdavidgraham/masscan)" "- | 185.101.33.2 |
| Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)" "pond1.gladstonefamily.net | 207.241.231.163 |
| Mozilla/5.0 (compatible; Dataprovider.com)" "www.gladstonefamily.net | 167.114.209.38 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "73.253.74.102 | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "blog.gladstonefamily.net | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "blog1.gladstonefamily.net | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "c-73-253-74-102.hsd1.ma.comcast.net | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "charon.gladstonefamily.net | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "gladstonefamily.net | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "pond.gladstonefamily.net | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "pond1.gladstonefamily.net | 194.153.113.100 |
| Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)" "www.gladstonefamily.net | 194.153.113.100 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0" "gladstonefamily.net | 82.165.116.138 |
| Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 OPR/36.0.2130.32" "gladstonefamily.net | 163.172.7.237 |
| Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 OPR/36.0.2130.32" "pond1.gladstonefamily.net | 95.211.211.232 |
| Wget/1.19.4 (linux-gnu)" "pond1.gladstonefamily.net | 75.163.211.48 |