Page 1 of 2
High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 3:38 am
by jaoudestudios
I noticed that the other day my server had a massive inbound traffic of over 160GB in a matter of a few hours. This is very strange considering on an average month I use about 30GB total.
I am not sure what logs to check, how can I hunt this down? To try and find out what it was. At least which IP it came from or which user etc.
I am a bit concerned as it is excessively high.
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 3:50 am
by Benjamin
Inbound? I would check FTP logs and disk space.
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 4:27 am
by jaoudestudios
What port does ftp work on? I think it is block as I only have a few ports open on my firewall (http, ssh).
Diskspace is a good idea, thanks
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 4:43 am
by Benjamin
Default FTP port is 21.
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 4:50 am
by jaoudestudios
Thanks. I checked my firewall and 21 is not on the safe list (blocked).
So it could not have been via ftp.
I did check the disk space as you mentioned, and no extra diskspace has been taken. Which is a good thing I guess, but still confusing.
That morning with the high bandwidth I did download the source for php and a few devel libs, then compiled php few times, but I cant see that causing the issue?
Any other ideas?
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 5:45 am
by jaoudestudios
I have pages and pages of stuff like this, what does it mean?
Code: Select all
Jun 6 21:59:58 host-78-129-250-11 sshd[32519]: Connection closed by 87.117.237.68
Jun 6 22:00:54 host-78-129-250-11 sshd[32547]: Connection closed by 87.117.237.68
Jun 6 22:01:57 host-78-129-250-11 sshd[32574]: Connection closed by 87.117.237.68
Jun 6 22:02:56 host-78-129-250-11 sshd[32593]: Connection closed by 87.117.237.68
Jun 6 22:03:58 host-78-129-250-11 sshd[32616]: Connection closed by 87.117.237.68
Jun 6 22:04:58 host-78-129-250-11 sshd[32620]: Connection closed by 87.117.237.68
Jun 6 22:05:58 host-78-129-250-11 sshd[32654]: Connection closed by 87.117.237.68
Jun 6 22:06:57 host-78-129-250-11 sshd[32673]: Connection closed by 87.117.237.68
And this seem like an invalid user attempt, but why port 52998 using ssh, I though ssh used port 22, or are they just tring their luck with another port?
Code: Select all
Jun 6 22:51:32 host-78-129-250-11 sshd[1462]: pam_succeed_if(sshd:auth): error retrieving information about user service
Jun 6 22:51:32 host-78-129-250-11 sshd[1459]: Failed password for adm from 64.132.224.199 port 52998 ssh2
Jun 6 22:51:32 host-78-129-250-11 sshd[1463]: Invalid user student from 64.132.224.199
Jun 6 22:51:32 host-78-129-250-11 sshd[1465]: Invalid user student from 64.132.224.199
Jun 6 22:51:32 host-78-129-250-11 sshd[1466]: input_userauth_request: invalid user student
Jun 6 22:51:32 host-78-129-250-11 sshd[1467]: input_userauth_request: invalid user student
Jun 6 22:51:32 host-78-129-250-11 sshd[1463]: pam_unix(sshd:auth): check pass; user unknown
There are pages of the above too (probably hundreds in the space of 2 hours)
Cheers
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 5:50 am
by Weirdan
Somebody ran a bruteforce attack on your ssh.
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 6:03 am
by Tobey
Generating 160gb of traffic in a couple of hours is very uncommon if you don't have a big website running on your server. There might be someone who is trying to break into your system by attempting to log in with a lot of different passwords, but 160gb is still way too much for that. I don't know what software you have running, but if I were you I would immediately look for bugs that your php scripts might have and take the system down if there was still a lot of traffic in the next hours/days. In my opinion, the most likely reason, in case someone broke into your server, is that someone set up a FTP server and uploaded warez. That's the only thing I have seen so far where such huge amount of data came in.
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 6:16 am
by jaoudestudios
Thanks for your feedback.
The only ports open on the server are ssh, http, mail, pop, smtp. So no one can upload apart from me via ssh.
It is the 1st time it has happened and it has not happened since, so it was a one off occasion. i only have a few websites on the server and none of them are big, none of them have been updated in a while so I dont think it will be a run away script - server sits idle most of the time. Disk space has not changed, I can account for all of it, so does not appear to have any additional files on there.
But you think 160GB is alot for someone trying a brute force attack?
I could limit access via ssh to my IP only, would this help? or would it still show up in the logs as an attempted login?
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 7:03 am
by Benjamin
Sounds more like a DOS attack to me.
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 7:05 am
by jaoudestudios
If so how can I prevent it?
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 7:12 am
by Tobey
Yep, I think it is way too much for that..by the way, is it a dedicated server?
Just because tools like
df say that diskspace has not changed, it does not automatically mean that nothing has happened, or could have happened. Someone could have installed a rootkit that hides everything.
And even if your firewall blocks incoming connections, one can still upload files using a reverse proxy, or just downloading files by connecting from your server to another one. There are countless ways of transferring data

Well, of course, all this does not mean that something bad happened to your server, but there is always a chance.
Restricting ssh access to your ip would work, if nobody has gained access yet, but there are better things. On my server I also block anything except HTTP. Lighttpd is running in a jail so even if there is a bug in one of my scripts, nothing dangerous can happen to my system. To log in I have knockd running and also a little program that I wrote on my own that looks up a dynamic domain name (no-ip.org) and allows connections from that ip. So everyone but me can only access HTTP

Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 7:34 am
by jaoudestudios
Tobey wrote:Yep, I think it is way too much for that..by the way, is it a dedicated server?
Yep, does this make a difference?
Tobey wrote:Just because tools like df say that diskspace has not changed, it does not automatically mean that nothing has happened, or could have happened. Someone could have installed a rootkit that hides everything.
According to the history no one else has logged in, I guess they could have cleared this though. But then they would have to be root and root can not log into my box directly (extra security).
Tobey wrote:Lighttpd is running in a jail so even if there is a bug in one of my scripts, nothing dangerous can happen to my system. To log in I have knockd running and also a little program that I wrote on my own that looks up a dynamic domain name (no-ip.org) and allows connections from that ip. So everyone but me can only access HTTP

I have not looked into this before or was not aware of it, sounds complicated though?

Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 7:56 am
by Tobey
At first I wasn't sure if you were talking about a dedicated server or just shared webspace, but since you have root access and everything, nevermind

Clearing the login history is one of the easiest things and also getting root access is not that hard. Just take a look at bugtraq or milw0rm, there are many "local root" exploits floating around. In the worst case a hacker would exploit vulnerabilities that have not been reported yet to the developers. That's why I like to keep every port shut. Of course, there could be a bug in iptables, but the risk is hopefully very low.
Putting lighttpd into a jail took me some time, but there is a good
howto available. You can find knockd and a good documentation over
here.
Re: High inbound traffic - which logs to check?
Posted: Fri Jun 12, 2009 9:46 am
by mikemike
sounds like it might be a distributed DOS? Did the server go down, even for only a couple of minutes?