In this post we (UNIXY) are going to share our experience fending off a large Distributed Denial of Service (DDoS) attack for a client. Generally, Website owners deal with DDoSattacks on their own. There are equipment and solutions vendors cater tothese owners and guarantee protection against these kind of attacks upto a certain threshold. The cost of hiring these vendors can range fromthousands to hundreds of thousand or millions of dollars depending onthe severity of the attack.
Our goal was to build a solution with the least amount of funds possible. This solution is scalable and can handle the worst attacks.The client’s dedicated server is not a special server but a simple quadcore Xeon managed server running the LAMP stack.The DDoS riposte described in this article can scale to stop a 10Gbpsattack or more. The good news is this solution does not require changinganything on the dedicated serveritself of the constellation. The server could be running just about anysoftware stack. This configuration will work just fine with almost allcases effortlessly..
- Distributed Denial of Service – The Social
Before we delve into the glorious technical details, there is an important aspect of DDoS attacks that one should know about; that is thesocial dynamics that lead to the attack. The more one understands aboutthe the social aspect of a DDoS attack the easier it becomes to preventor stop it. Because once a DDoS has started, priorities shift quitedramatically and rational for making wise decisions becomes flawed.
DDoS comic
DDoS attacks do not occur randomly. They are targeted and come with a motive. The motive could be revenge but most of the time the motive isfinancial. The individual or groups that conduct the DDoS attacks aremost of the time hired to complete the job. They have the resources andknow-how to orchestrate the attack while hoping to avoid getting caughtby the authorities. They have no emotional attachment to the DDoS attackitself; they have no hard feelings towards the victim. They just getpaid for what they do and nonchalantly, but meticulously, execute.
As explained, DDoS attacks are preceded by an email, post, or phone call, from the individual or group with interest, to the victim. It isalways recommended to treat strangers you meet online or offlineprofessionally and politely. The smallest altercation can lead to anegative reaction, which can escalate actions. In the face of anonymousthreats against your business or organization, remain calm and composed.
DDoS Offer in Forum
There are public markets online (please don’t ask for links) where wannabe DDoS perpetrators get to hire the attackers. Pricing varies from$5/hr to $10 for a simple non-distributed DoS attack. A DDoS, however,tends to be more expensive depending on the sheer amount of data orpackets that needs to be delivered at the target. It can range from$20/hr to $100/hr. The word used to in the circles in lieu of DDoS is to“drop;” meaning to drop a certain Web site or network off the Internet.It really means to either overwhelm the target with enough traffic thatthe equipment fails or to force upstream providers to “null route” thedestination IP at the network level. The end result is that the IP getsdropped from the routing tables and the server to stop responding to allrequests.
The fact that DDoS is not cheap has got to be comforting to an extent. It means that it is only a matter of time before the DDoS“client” runs out of cash. This in itself is encouraging. Keep that inmind should you begin to lose patience. Perseverance is omnipotent.Denial of service attacks are considered a crime and are punishable byFederal law in the US and by the police in the UK. As we will explain inthe technical part of this article, DDoS attacks are almost impossibleto trace to back to the individual or group that are orchestrating theattack. Because of the distributed nature, it requires cooperation fromseveral network engineers that work for upstream providers.
Distributed Denial of Service – The Technicals
First things first, What is a DoS? what is the difference between a DoS and DDoS? A Denial of Service (DoS) is an attack originating fromone source or one system that results in the service in question beingunavailable to its legitimate users. It denies its very users accesseither because the service runs out of available resources or has beentricked to deny access to legitimate users. For example, a DoS attack ona Web server can cause it to run out of resources and stop respondingto requests. A DDoS, on the other hand, is a more sophisticated attacksince the attack originates from hundreds or thousands or nodes.
A DDoS attack is almost impossible to trace back to the source due to its distributed nature. DDoS orchestrators call the nodes andcontroller system a “bot.” With a few commands, the bot owner caninstruct infected nodes from around the world to attack a target. Thebot systems are hosted and controlled via the Internet Relay Chat(IRC) system or via a direct connection port connection. The nodes usedto attack the target are made of compromised Windows and Linux nodesfrom around the world.
Before we present our solution, we need to discuss the two types of DDoS attacks that exist. On one hand you have attacks arebandwidth-based and seek to saturate the connectivity link. On the otherhand, you have attacks that are packet-based and seek to saturate theprocessing capability of the equipment. In other words, they seek tooverwhelm the processing power of the CPU and memory or fabricof the routers or switches. All equipment has hard limits when it comesto their ability to handle a certain number of packets per second.Routers and switches are no exception.
Capacity of networking equipment - Mbps vs pps
For example, take the above specification for a Cisco 6500 firewall. Each module is able to handle 5Gbps or 2.8 million pps. This firewall sure looks like it can handle a 5Gbpsattack. Great! However, should there be a packet-based DDoS attack, onewould only need a 1.5Gbps payload to saturate it. That’s 2.8 million pps* 64 Bytes = 1.5Gbps. So bandwidth capacity means nothing by itself andsmall packets can cause havoc.
Our client was facing a 2Gbps DDoS attack that is packet based. It sought to force routing equipment along the way to start droppinglegitimate packets. This caused the upstream to null route the IP toalleviate the burden on other customers that are behind the link. Thisis the typical reaction from all upstreams as they seek to protect theirmany other customers from feeling the pinch of the attack. We weregiven one last chance to “fix” things before the IP could be routed backin. Here is how we were able to fend off the attack and keep the serverrunning.
We have deployed what we call a “constellation” of reverse proxy VM or VPS nodes running the high performance Web server Nginx. The VM nodes werepurchased from several providers given they are located at separatefacilities. Essentially, we are off-loading and “splitting” both packetprocessing and bandwidth consumption across several data centerfacilities (physical routers & carriers).
Nginx constellation
The configuration of the Nginx nodes is a typical reverse proxy configuration with the usual extra kernel security configuration. So fora 2Gbps attack and with 20 VM nodes, the bandwidth consumption per nodeis a maximum of 2GBps / 20 = 100Mbps. That’s a 100Mbps load per VMnode, which is reasonable enough and is below the threshold for gettingone’s IP null routed by the provider. One could add more and more Nginxnodes to the constellation without issues.
So how is 20 VM nodes going to be affordable? VM prices have dropped dramatically over the last year. For the above configuration, a VM cancost between $5/mo and $10/mo. That’s an average of $8*20 = $160/Mo.Knowing that most DDoS attackers have the attention span of a gold fish,the $160 is all you need to send your attacker and his accomplicepacking.
Let’s talk more about the Nginx constellation configuration. The Nginx front-end nodes will run in proxymode caching static files and requests. The more aggressive the DDoSthe higher the time-to-live for cache objects should be. This preventsthe Nginx nodes from proxy-passing requests to the quad core node.Although, if the main node has idle CPU and plenty of memory it wouldn’thurt to put it to good use to alleviate the burden on the Nginx frontnodes. Your domain’s A records is going to be the IP of the Nginx frontnodes configured in round robin fashion. DNS round robin has itsshortcomings in terms of not having control over how long (bad) recordsget cached by resolvers around the world. But in this case, it does notmatter much. Just be sure to set high TTL for the records so your DNSserver does not collapse under the enormous volume.
Nginx DDoS Constellation
There are tons of online tutorials that go over the installation of Nginx as a reverse proxy so be sure to read up on it. But we will listsome of the peculiar settings that are needed to handle a large scaleDDoS. Of importance is the number of Nginx worker processes and workerconnections. Those values will need to adjusted gradually and higher tohandle different kind of attacks depending the VM resource allocation.But you should set them at least as high as the following:
worker_processes 8;
events {
.
.
worker_connections 4096; # Be sure to set ulimit -n 4096 or more
.
.
}
Keep in mind that one still needs to gear up for the event by setting kernel and system variables on the Nginx nodes. Simple things likeper-IP rate limiting, flooding rate limits, and syn cookies should beenabled without a question. Here are some measures you can implement:
net.ipv4.tcp_syncookies = 1
# source validation / reversed path
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
kernel.pid_max = 65536
net.ipv4.ip_local_port_range = 9000 65000
Recap.
In brief, here are the elements that constitute our solution:
- Nginx reverse proxy constellation
- DNS round robin records
- Security at the Nginx front end level
- Know the social and technical dynamics behind DDoS attacks
