Spectacular DDoS on one of my hosts systems

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Post Reply
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Spectacular DDoS on one of my hosts systems

Post by RobertGonzalez »

I just go this email a few hours ago. Was this a strategic and powerful attack or what?
Re: Network Outage - December 20, 2007 - Incident Update

Dear Customer,

We experienced a major network attack starting 08:30 PM on December 20, 2007, and resulted in a network outage that lasted for 3 hours.
The outage originated with a massive DDOS (Distributed Denial of Service) attack against one of our web servers.
At this point, we were seeing 50-80% packet loss to our network, and the origin/destination of attack was being investigated.
Quite soon, the attack went up to 800,000 pps (packet per second) and 500 megabit per second steady almost exhausting our channels to our upstreams.
The attack died by itself for about 4 minutes and returned with almost double the capacity at 1 million packets per second and sustained 900mbps on inbound data. This even went up intermittently to 1.5 million packets per second and 1.3 Gigabit per second of traffic.
Due to this huge inflow of traffic, our local blocks to mitigate this attack weren't successful and we had to seek help from our ISPs. This added to the resolution time since we had to liaison with multiple external units to get destination identified and an appropriate block instated. At approx 10:10PM, we were able to get the destination identified and block was immediately placed at our ISPs. However, despite being physically redundant, our core fiber link didn't come up even when the inbound traffic had settled to our normal rate. This added another hour of outage while we coordinated with the fiber team to get the fault located and corrected. This fault was related to the massive inflow on traffic during the DDOS.

For business continuity, we plan to add an alternative redundant fiber link in addition to this our current (redundant) fiber link before end of first quarter of 2008. Other suitable measures that can help in such cases will also be considered and implemented.

As per the last update, the DDOS attack was still on at sustained 400mbps of traffic, however being blocked far from our network, we continue to run safe and fine.
The server being attacked has also been successfully migrated to a new IP Address, allowing our customers to run their website without being affected anymore with the issue.

Thank you for your patience and cooperation throughout the issue.
We would also like to take this moment to wish you and yours a very Merry Christmas, a great New Year, and safe travel during this holiday season.

Regards,
Customer Support
matthijs
DevNet Master
Posts: 3360
Joined: Thu Oct 06, 2005 3:57 pm

Post by matthijs »

1.5 million packets per second and 1.3 Gigabit per second of traffic
Wow, that data center must have been smoking 8O
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Post by RobertGonzalez »

Yeah, the numbers seemed really frickin' high to me.
alex.barylski
DevNet Evangelist
Posts: 6267
Joined: Tue Dec 21, 2004 5:00 pm
Location: Winnipeg

Post by alex.barylski »

Everah wrote:Yeah, the numbers seemed really frickin' high to me.
For a hosting company...I'm not sure thats high...the last shared hosting company I was with claimed to serve a 15 tera-bytes a month...and they piggy backed on someone elses servers.

I'm thinking most big data centers are probably into the peta-bytes and beyond...what that averages out to in seconds...who knows. :P

It would be interesting to see their traffic charts in a line graph...just to see what the norm is...cause without telling you that...any numbers shown to you could be just smoke and mirrors.
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Post by Benjamin »

And who was on that IP?
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Post by Jenk »

The problem with gaging size of DDoS attacks if relevance of size, so if your host is someone the size of GoDaddy then that attack wouldn't be considered that big, but if it were a small independent hosting whom only have one rack of rented datacenter floorspace, then it would be friggin huge :)
User avatar
s.dot
Tranquility In Moderation
Posts: 5001
Joined: Sun Feb 06, 2005 7:18 pm
Location: Indiana

Post by s.dot »

Must've took a lot of machines to pull that one off.
Set Search Time - A google chrome extension. When you search only results from the past year (or set time period) are displayed. Helps tremendously when using new technologies to avoid outdated results.
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Post by John Cartwright »

Atleast you have some good support guys! Which host you using Everah?
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Post by RobertGonzalez »

This was my JodoHost account.
Post Reply