DDOS, or distributed denial of service attacks are becoming more common place on the internet. There are a large list of articles online telling you everything you need to know about what these attacks are, so we’re not going to re-invent the wheel.
According to Digital Attack Map a DDOS attack is an attempt to make an online service unavailable by overwhelming it with traffic from multiple sources.
They go on to say that you can currently buy a DDoS for as little as a $150, there are more than 2000 daily attacks, and 1/3 of all server downtime is due to a DDoS attack. So they are incredibly serious and very difficult to avoid, in our case we had a 10gb attack that lasted just short of 2 hours started on the 24th January 2018. This attack was aimed at one of our shared hosting servers’ shared IP addresses and effectively overloaded its ability to handle the data incoming.
We did prepare for this…
We had prepared as much as we thought but even with a Fortigate firewall with specs including 200Mbps Threat Protection and 2.5 Gbps protection, 4 separate Gb cables to each server combined into a active/passive bind, and monthly maintenance windows, there was no way to prevent such a devastating attack.
Reviewing the firewalls we pulled the following images to show you the effect of the attack;
You can see the difference in the traffic to the server, it was drastic compared to our normal traffic, but the firewall wasn’t struggling.
You can also see on these images that we’re used to constant attempts to access the server and bypass our security. In fact if you look at earlier in the day there was a more focused attempt to access everything and it didn’t affect our services.
The server affected noticed the spike in traffic, as shown in this graph, it wasn’t struggling but the traffic was clearly increasing.
The server that was attacked also registered the increase network spike but it was still only half of the physical limit of the devices.
So if everything was working fine, why did everything stop?
The problem we faced was capacity, we share the racks in the data centre with other customers and with an attack of this scale the data centre registered the attack and immediately noticed the degradation of service for the entire cabinet. If one server is being attacked it can affect all the servers in the rack, meaning the data centre have to make difficult choice to protect 35 other servers over the one being attacked.
By disabling the attacked site/server the attack has nowhere to go and simply stops. The DDoS is effectively still running but it doesn’t have a target, so the bot net running it will just keep trying for as long as it was paid to run.
So what can we do in the future to avoid this?
The difficult reality is avoiding it is near impossible but mitigating it is possible. DDoS are a normality of internet life, what we need to do is educate and improve our security for next time. We have started investigating our data centre setup to see if we can do something else next time, moving equipment, further redundancy, anything physical we can do and any investment we can take. These all take time and testing but we’ll get it done.
There’s a lot a user can do (yes you the reader), Cloudflare offer caching and DDoS protection for websites as standard, and we offer Cloudflare as standard on our servers. If you turn this one, it re-directs your domain name to them and caches your site on their systems, this means if the attack is aimed at you, Cloudflare will be hit first and can handle the attack for a few hours, hopefully by which time the attack ends and no one notices. If the attack is against us, your website is cached at Cloudflare and is still able to run (in a limited fashion) meaning no one knows.
We’re also going to split our customers up into IP groups, this means that if the attack happens again, we’ll only loose a few customers not them all. We’ll be organising this over the coming weeks.
Finally, a few of us had Office 365 as our email provided. This meant that our websites were down but our email was still running. We could reply, deal with the data centre and still manage the support desk.