Spamhaus recently witnessed one of the biggest Distributed Denial of Service attacks on its infrastructure. The Spamhaus Project is an international non-profit organization that has a mission to provide spamless internet networks. They are even involved in working with the law enforcement agencies to fight fraudster spam gangs all over the world. This project was founded in 1988 and is based in Geneva, Switzerland and London, UK. The team comprises of 38 investigators and forensic specialists in different countries. The organization maintains spam-blocking databases and lists such as the:
- DNS Based Blocklist – This is a list of IP addresses published through the internet as a zone file that may be used by DNS Server Software or as a live DNS Zone which may be queried for real time DNS resolution.
- Spamhaus Block List- This list targets verified spammers, spam gangs and spam support services. Its focus is to maintain a list of the above-mentioned fraudsters.
- Exploits Block List – This list targets illegal exploits, proxies, worms, viruses, infected systems and other Trojans. Hence, it maintains a list of open proxies and compromised computers/servers that may be used for spam campaigns.
- Policy Block List – It lists dynamic and static IP addresses that should not be sending mails to third party servers –for example: core routers, company servers and unused IP addresses.
- Domain Block List – It lists spam domains, spam sources, URLs, known spammer gangs and virus related sites.
- Spamhaus White List – This is a white listing of IP addresses that allow incoming mail traffic to be segregated into roughly three categories like good, bad and unknown. Only verified and clean senders are approved and put up in this white list.
- Domain White list – This is a white list of the domain names maintained by Spamhaus.
This efficient team of investigators assists in inquiries into spam senders, botnet and malware attacks and other black operations. Spamhaus has one of the largest DNS infrastructures in the world and covers over 1.4 Billion users for spam protection.
What happened to this large infrastructure was that it became a victim of the world’s biggest denial of service attack that nearly brought down its services. Now, when Spamhaus was under attack it turned to Cloudfare, a content delivery network and distributed DNS service provider. Cloudfare successfully mitigated the attack and hence Spamhaus was still online with its services. The perimeter of the attack included the organization’s website as well as the Tier 1 internet exchanges.
The attack started with a 10 Gbps of attack force mainly coming from open DNS recursors. The next day, March 19, the attack increased to 90 Gbps and stayed prevalent to around the next two days. From then on March 22, the attack bandwidth reached 120 Gbps causing a very high load on the network. Cloudfare’s Anycast technology came to the rescue, which managed the huge load and distributed it over its data centers all over the world. Since the Internet is an interconnection of different networks and they each maintain a connection through different routes, Cloudfare used this model to distribute its network load by connecting to large networks and other carriers through which the traffic is routed. Also, it connects to its Internet exchanges where a large number of networks meet in a central point. With reference from Cloudfare’s architecture, we have Internet Exchanges in major cities. Cloudfare connects to these exchanges such as the London Internet Exchange, Amsterdam Internet Exchange, Frankfurt Internet Exchanges and few others. The globally used website networks like Facebook, Google also connect through these exchanges. When the attackers were unable to fight out this load distribution technique employed by Cloudfare they started hitting these exchanges for breaking the networks.
We have three types of networks on a large scale –Tier 1, Tier 2 and Tier 3. Tier 1 network is one that can reach every other network on the Internet without purchasing IP transits and paying settlements. Therefore, it is a transit free network that peers with every other Tier 1 network. Tier 2 network peers with some networks, but still purchases IP transits or pays for settlements to reach out to a portion of the Internet. The Tier 3 network is one that solely purchases transits from other networks to reach the internet. At the core, if the Tier 1 networks fail then a warning bell gets sounding. Now supposedly we have multiple hops in our way from the Anycast network to the final distributed network. If an attacker tried to attack the last hop in the network then the attack would become spread over the distributed network that would zero the end result of the attack. Instead of this, if an attacker tried to attack the hops before the last hop to the distributed network the attack would be channelized to only one pass way until it is introduced to the distributed network and this would cause choking. For the Cloudfare people, it was a very tough time to manage around 300 Gbps of traffic hitting the network. This had a direct link to the whole inter-network since the in line network was on fire and there is always a peak to maximum limit which these network devices may manage. If we step over this threshold, we have a sort of buffer overflow scenario. According to reports by Cloudfare, there was tremendous amount of traffic in Europe, which was facing the peak of attacks. Amidst all this, we had on the same hand controversial talks of Cloudfare boasting its service for DDoS protection for this big attack that they were successfully able to mitigate. One such blog post was on Gizmodo which posted the whole hot controversy as “That Internet War Apocalypse is a Lie” and just a false hype. It claimed that Cloudfare was largely exaggerating the attack to affect the whole internet backbone. For Gizmodo, this appeared as a ploy through which Cloudfare tried to market its services since the internet records did not show any evidences of the services being slow over the period. In addition, the internet backbone is capable of supporting traffics worth Tbps bandwidths and it is hard to imagine that only a Gbps attack could bring it down.
On investigating into the issue, the flaw was found due to a vulnerability in the configuration of the DNS servers available across the internet. In server terminology, there is an expression known as “resolvers.” What this does is basically it replies with information about a particular address, domain or a site name and sends it back to the receiver. The loophole present on these DNS servers available on the internet was that they were configured with the open resolver functionality. If anyone asks for any information from these DNS servers, it will run a query and return the results to that particular entity. The hackers misused this functionality of the DNS servers, and to further add to the mishap they mixed it up with spoofed address for these DNS queries. An attacker spoofed an IP address for a particular website/domain that they wanted to hit and sent a query to multiple DNS servers with the open resolver functionality. On receiving this query, we had these open resolvers sending in loads of information to the spoofed IP address that had been attacked, and thereby flooding its network. When we have multiple servers echoing the same information to a particular destination we can all imagine the amount of clogging it may cause. After this incident, The Open Resolver Project committee began investigating the issue and tracking the servers known open to these vulnerabilities. It is estimated that over 27 million servers were tracked with the open resolver functionality all across the internet. The only patch was to fix this vulnerability through secure configuration.
It is suspected that the hosting provider Cyberpunker had sponsored this attack after the Spamhaus committee blacklisted it. Sven Olaf Kamphuis, a self-proclaimed internet freedom fighter, was suspected by the New York Times as he recently talked about bringing down the services of Spamhaus. Sven denied from this fact although still he is a strong suspect since he is also known as the prince of spam. There are still investigations going on to get to the actual suspect with some proof of evidence. Regardless of this fact, an attack of 300 Gbps or rather a larger future attack could definitely rocket out the whole internet network. To muster our defenses we have to focus on the fact of how important security is and the need to patching and securing publicly exposed infrastructures. We also have Best Current Practices 38 [BCP 38], a set of guidelines that would limit the effectiveness of some DDoS attacks. These are very old guidelines, but are yet to be implemented effectively. In order to have a safe internetwork we need cooperation and support from all the ISPs, enterprises and small-scale networks, which indeed requires a lot of effort. During the past several months, there have been such incidences of DDoS attacks against US financial institutions like Bank of America, Citibank and JPMorgan Chase.
Presently we have solutions for DDoS attacks such as SYN, ICMP flooding but these are not very effective against application-based attacks. Hence, as the attack techniques are being enhanced, similarly we have to update our infrastructure to beef up our defenses against such attacks.