Re: Network Outage - December 20, 2007 - Incident Update
Dear Customer,
We experienced a major network attack starting 08:30 PM on December 20, 2007, and resulted in a network outage that lasted for 3 hours.
The outage originated with a massive DDOS (Distributed Denial of Service) attack against one of our web servers.
At this point, we were seeing 50-80% packet loss to our network, and the origin/destination of attack was being investigated.
Quite soon, the attack went up to 800,000 pps (packet per second) and 500 megabit per second steady almost exhausting our channels to our upstreams.
The attack died by itself for about 4 minutes and returned with almost double the capacity at 1 million packets per second and sustained 900mbps on inbound data. This even went up intermittently to 1.5 million packets per second and 1.3 Gigabit per second of traffic.
Due to this huge inflow of traffic, our local blocks to mitigate this attack weren't successful and we had to seek help from our ISPs. This added to the resolution time since we had to liaison with multiple external units to get destination identified and an appropriate block instated. At approx 10:10PM, we were able to get the destination identified and block was immediately placed at our ISPs. However, despite being physically redundant, our core fiber link didn't come up even when the inbound traffic had settled to our normal rate. This added another hour of outage while we coordinated with the fiber team to get the fault located and corrected. This fault was related to the massive inflow on traffic during the DDOS.
For business continuity, we plan to add an alternative redundant fiber link in addition to this our current (redundant) fiber link before end of first quarter of 2008. Other suitable measures that can help in such cases will also be considered and implemented.
As per the last update, the DDOS attack was still on at sustained 400mbps of traffic, however being blocked far from our network, we continue to run safe and fine.
The server being attacked has also been successfully migrated to a new IP Address, allowing our customers to run their website without being affected anymore with the issue.
Thank you for your patience and cooperation throughout the issue.
We would also like to take this moment to wish you and yours a very Merry Christmas, a great New Year, and safe travel during this holiday season.
Regards,
Customer Support
Spectacular DDoS on one of my hosts systems
Moderator: General Moderators
- RobertGonzalez
- Site Administrator
- Posts: 14293
- Joined: Tue Sep 09, 2003 6:04 pm
- Location: Fremont, CA, USA
Spectacular DDoS on one of my hosts systems
I just go this email a few hours ago. Was this a strategic and powerful attack or what?
- RobertGonzalez
- Site Administrator
- Posts: 14293
- Joined: Tue Sep 09, 2003 6:04 pm
- Location: Fremont, CA, USA
-
alex.barylski
- DevNet Evangelist
- Posts: 6267
- Joined: Tue Dec 21, 2004 5:00 pm
- Location: Winnipeg
For a hosting company...I'm not sure thats high...the last shared hosting company I was with claimed to serve a 15 tera-bytes a month...and they piggy backed on someone elses servers.Everah wrote:Yeah, the numbers seemed really frickin' high to me.
I'm thinking most big data centers are probably into the peta-bytes and beyond...what that averages out to in seconds...who knows.
It would be interesting to see their traffic charts in a line graph...just to see what the norm is...cause without telling you that...any numbers shown to you could be just smoke and mirrors.
Must've took a lot of machines to pull that one off.
Set Search Time - A google chrome extension. When you search only results from the past year (or set time period) are displayed. Helps tremendously when using new technologies to avoid outdated results.
- John Cartwright
- Site Admin
- Posts: 11470
- Joined: Tue Dec 23, 2003 2:10 am
- Location: Toronto
- Contact:
- RobertGonzalez
- Site Administrator
- Posts: 14293
- Joined: Tue Sep 09, 2003 6:04 pm
- Location: Fremont, CA, USA
This was my JodoHost account.