High inbound traffic - which logs to check?
Moderator: General Moderators
- jaoudestudios
- DevNet Resident
- Posts: 1483
- Joined: Wed Jun 18, 2008 8:32 am
- Location: Surrey
High inbound traffic - which logs to check?
I noticed that the other day my server had a massive inbound traffic of over 160GB in a matter of a few hours. This is very strange considering on an average month I use about 30GB total.
I am not sure what logs to check, how can I hunt this down? To try and find out what it was. At least which IP it came from or which user etc.
I am a bit concerned as it is excessively high.
I am not sure what logs to check, how can I hunt this down? To try and find out what it was. At least which IP it came from or which user etc.
I am a bit concerned as it is excessively high.
Re: High inbound traffic - which logs to check?
Inbound? I would check FTP logs and disk space.
- jaoudestudios
- DevNet Resident
- Posts: 1483
- Joined: Wed Jun 18, 2008 8:32 am
- Location: Surrey
Re: High inbound traffic - which logs to check?
What port does ftp work on? I think it is block as I only have a few ports open on my firewall (http, ssh).
Diskspace is a good idea, thanks
Diskspace is a good idea, thanks
Re: High inbound traffic - which logs to check?
Default FTP port is 21.
- jaoudestudios
- DevNet Resident
- Posts: 1483
- Joined: Wed Jun 18, 2008 8:32 am
- Location: Surrey
Re: High inbound traffic - which logs to check?
Thanks. I checked my firewall and 21 is not on the safe list (blocked).
So it could not have been via ftp.
I did check the disk space as you mentioned, and no extra diskspace has been taken. Which is a good thing I guess, but still confusing.
That morning with the high bandwidth I did download the source for php and a few devel libs, then compiled php few times, but I cant see that causing the issue?
Any other ideas?
So it could not have been via ftp.
I did check the disk space as you mentioned, and no extra diskspace has been taken. Which is a good thing I guess, but still confusing.
That morning with the high bandwidth I did download the source for php and a few devel libs, then compiled php few times, but I cant see that causing the issue?
Any other ideas?
- jaoudestudios
- DevNet Resident
- Posts: 1483
- Joined: Wed Jun 18, 2008 8:32 am
- Location: Surrey
Re: High inbound traffic - which logs to check?
I have pages and pages of stuff like this, what does it mean?
And this seem like an invalid user attempt, but why port 52998 using ssh, I though ssh used port 22, or are they just tring their luck with another port?
There are pages of the above too (probably hundreds in the space of 2 hours)
Cheers
Code: Select all
Jun 6 21:59:58 host-78-129-250-11 sshd[32519]: Connection closed by 87.117.237.68
Jun 6 22:00:54 host-78-129-250-11 sshd[32547]: Connection closed by 87.117.237.68
Jun 6 22:01:57 host-78-129-250-11 sshd[32574]: Connection closed by 87.117.237.68
Jun 6 22:02:56 host-78-129-250-11 sshd[32593]: Connection closed by 87.117.237.68
Jun 6 22:03:58 host-78-129-250-11 sshd[32616]: Connection closed by 87.117.237.68
Jun 6 22:04:58 host-78-129-250-11 sshd[32620]: Connection closed by 87.117.237.68
Jun 6 22:05:58 host-78-129-250-11 sshd[32654]: Connection closed by 87.117.237.68
Jun 6 22:06:57 host-78-129-250-11 sshd[32673]: Connection closed by 87.117.237.68
Code: Select all
Jun 6 22:51:32 host-78-129-250-11 sshd[1462]: pam_succeed_if(sshd:auth): error retrieving information about user service
Jun 6 22:51:32 host-78-129-250-11 sshd[1459]: Failed password for adm from 64.132.224.199 port 52998 ssh2
Jun 6 22:51:32 host-78-129-250-11 sshd[1463]: Invalid user student from 64.132.224.199
Jun 6 22:51:32 host-78-129-250-11 sshd[1465]: Invalid user student from 64.132.224.199
Jun 6 22:51:32 host-78-129-250-11 sshd[1466]: input_userauth_request: invalid user student
Jun 6 22:51:32 host-78-129-250-11 sshd[1467]: input_userauth_request: invalid user student
Jun 6 22:51:32 host-78-129-250-11 sshd[1463]: pam_unix(sshd:auth): check pass; user unknown
Cheers
Re: High inbound traffic - which logs to check?
Somebody ran a bruteforce attack on your ssh.
Re: High inbound traffic - which logs to check?
Generating 160gb of traffic in a couple of hours is very uncommon if you don't have a big website running on your server. There might be someone who is trying to break into your system by attempting to log in with a lot of different passwords, but 160gb is still way too much for that. I don't know what software you have running, but if I were you I would immediately look for bugs that your php scripts might have and take the system down if there was still a lot of traffic in the next hours/days. In my opinion, the most likely reason, in case someone broke into your server, is that someone set up a FTP server and uploaded warez. That's the only thing I have seen so far where such huge amount of data came in.
- jaoudestudios
- DevNet Resident
- Posts: 1483
- Joined: Wed Jun 18, 2008 8:32 am
- Location: Surrey
Re: High inbound traffic - which logs to check?
Thanks for your feedback.
The only ports open on the server are ssh, http, mail, pop, smtp. So no one can upload apart from me via ssh.
It is the 1st time it has happened and it has not happened since, so it was a one off occasion. i only have a few websites on the server and none of them are big, none of them have been updated in a while so I dont think it will be a run away script - server sits idle most of the time. Disk space has not changed, I can account for all of it, so does not appear to have any additional files on there.
But you think 160GB is alot for someone trying a brute force attack?
I could limit access via ssh to my IP only, would this help? or would it still show up in the logs as an attempted login?
The only ports open on the server are ssh, http, mail, pop, smtp. So no one can upload apart from me via ssh.
It is the 1st time it has happened and it has not happened since, so it was a one off occasion. i only have a few websites on the server and none of them are big, none of them have been updated in a while so I dont think it will be a run away script - server sits idle most of the time. Disk space has not changed, I can account for all of it, so does not appear to have any additional files on there.
But you think 160GB is alot for someone trying a brute force attack?
I could limit access via ssh to my IP only, would this help? or would it still show up in the logs as an attempted login?
Re: High inbound traffic - which logs to check?
Sounds more like a DOS attack to me.
- jaoudestudios
- DevNet Resident
- Posts: 1483
- Joined: Wed Jun 18, 2008 8:32 am
- Location: Surrey
Re: High inbound traffic - which logs to check?
If so how can I prevent it?
Re: High inbound traffic - which logs to check?
Yep, I think it is way too much for that..by the way, is it a dedicated server?
Just because tools like df say that diskspace has not changed, it does not automatically mean that nothing has happened, or could have happened. Someone could have installed a rootkit that hides everything.
And even if your firewall blocks incoming connections, one can still upload files using a reverse proxy, or just downloading files by connecting from your server to another one. There are countless ways of transferring data
Well, of course, all this does not mean that something bad happened to your server, but there is always a chance.
Restricting ssh access to your ip would work, if nobody has gained access yet, but there are better things. On my server I also block anything except HTTP. Lighttpd is running in a jail so even if there is a bug in one of my scripts, nothing dangerous can happen to my system. To log in I have knockd running and also a little program that I wrote on my own that looks up a dynamic domain name (no-ip.org) and allows connections from that ip. So everyone but me can only access HTTP
Just because tools like df say that diskspace has not changed, it does not automatically mean that nothing has happened, or could have happened. Someone could have installed a rootkit that hides everything.
And even if your firewall blocks incoming connections, one can still upload files using a reverse proxy, or just downloading files by connecting from your server to another one. There are countless ways of transferring data
Well, of course, all this does not mean that something bad happened to your server, but there is always a chance.
Restricting ssh access to your ip would work, if nobody has gained access yet, but there are better things. On my server I also block anything except HTTP. Lighttpd is running in a jail so even if there is a bug in one of my scripts, nothing dangerous can happen to my system. To log in I have knockd running and also a little program that I wrote on my own that looks up a dynamic domain name (no-ip.org) and allows connections from that ip. So everyone but me can only access HTTP
- jaoudestudios
- DevNet Resident
- Posts: 1483
- Joined: Wed Jun 18, 2008 8:32 am
- Location: Surrey
Re: High inbound traffic - which logs to check?
Yep, does this make a difference?Tobey wrote:Yep, I think it is way too much for that..by the way, is it a dedicated server?
According to the history no one else has logged in, I guess they could have cleared this though. But then they would have to be root and root can not log into my box directly (extra security).Tobey wrote:Just because tools like df say that diskspace has not changed, it does not automatically mean that nothing has happened, or could have happened. Someone could have installed a rootkit that hides everything.
I have not looked into this before or was not aware of it, sounds complicated though?Tobey wrote:Lighttpd is running in a jail so even if there is a bug in one of my scripts, nothing dangerous can happen to my system. To log in I have knockd running and also a little program that I wrote on my own that looks up a dynamic domain name (no-ip.org) and allows connections from that ip. So everyone but me can only access HTTP
Re: High inbound traffic - which logs to check?
At first I wasn't sure if you were talking about a dedicated server or just shared webspace, but since you have root access and everything, nevermind 
Clearing the login history is one of the easiest things and also getting root access is not that hard. Just take a look at bugtraq or milw0rm, there are many "local root" exploits floating around. In the worst case a hacker would exploit vulnerabilities that have not been reported yet to the developers. That's why I like to keep every port shut. Of course, there could be a bug in iptables, but the risk is hopefully very low.
Putting lighttpd into a jail took me some time, but there is a good howto available. You can find knockd and a good documentation over here.
Clearing the login history is one of the easiest things and also getting root access is not that hard. Just take a look at bugtraq or milw0rm, there are many "local root" exploits floating around. In the worst case a hacker would exploit vulnerabilities that have not been reported yet to the developers. That's why I like to keep every port shut. Of course, there could be a bug in iptables, but the risk is hopefully very low.
Putting lighttpd into a jail took me some time, but there is a good howto available. You can find knockd and a good documentation over here.
Re: High inbound traffic - which logs to check?
sounds like it might be a distributed DOS? Did the server go down, even for only a couple of minutes?