High inbound traffic - which logs to check?

Whether you are using Linux on the desktop or as a server, it's still good that you're using Linux. Linux related questions go here.

Moderator: General Moderators

User avatar
jaoudestudios
DevNet Resident
Posts: 1483
Joined: Wed Jun 18, 2008 8:32 am
Location: Surrey

High inbound traffic - which logs to check?

Post by jaoudestudios »

I noticed that the other day my server had a massive inbound traffic of over 160GB in a matter of a few hours. This is very strange considering on an average month I use about 30GB total.

I am not sure what logs to check, how can I hunt this down? To try and find out what it was. At least which IP it came from or which user etc.

I am a bit concerned as it is excessively high.
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: High inbound traffic - which logs to check?

Post by Benjamin »

Inbound? I would check FTP logs and disk space.
User avatar
jaoudestudios
DevNet Resident
Posts: 1483
Joined: Wed Jun 18, 2008 8:32 am
Location: Surrey

Re: High inbound traffic - which logs to check?

Post by jaoudestudios »

What port does ftp work on? I think it is block as I only have a few ports open on my firewall (http, ssh).

Diskspace is a good idea, thanks
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: High inbound traffic - which logs to check?

Post by Benjamin »

Default FTP port is 21.
User avatar
jaoudestudios
DevNet Resident
Posts: 1483
Joined: Wed Jun 18, 2008 8:32 am
Location: Surrey

Re: High inbound traffic - which logs to check?

Post by jaoudestudios »

Thanks. I checked my firewall and 21 is not on the safe list (blocked).

So it could not have been via ftp.

I did check the disk space as you mentioned, and no extra diskspace has been taken. Which is a good thing I guess, but still confusing.

That morning with the high bandwidth I did download the source for php and a few devel libs, then compiled php few times, but I cant see that causing the issue? :?

Any other ideas?
User avatar
jaoudestudios
DevNet Resident
Posts: 1483
Joined: Wed Jun 18, 2008 8:32 am
Location: Surrey

Re: High inbound traffic - which logs to check?

Post by jaoudestudios »

I have pages and pages of stuff like this, what does it mean?

Code: Select all

 
Jun  6 21:59:58 host-78-129-250-11 sshd[32519]: Connection closed by 87.117.237.68
Jun  6 22:00:54 host-78-129-250-11 sshd[32547]: Connection closed by 87.117.237.68
Jun  6 22:01:57 host-78-129-250-11 sshd[32574]: Connection closed by 87.117.237.68
Jun  6 22:02:56 host-78-129-250-11 sshd[32593]: Connection closed by 87.117.237.68
Jun  6 22:03:58 host-78-129-250-11 sshd[32616]: Connection closed by 87.117.237.68
Jun  6 22:04:58 host-78-129-250-11 sshd[32620]: Connection closed by 87.117.237.68
Jun  6 22:05:58 host-78-129-250-11 sshd[32654]: Connection closed by 87.117.237.68
Jun  6 22:06:57 host-78-129-250-11 sshd[32673]: Connection closed by 87.117.237.68
 
And this seem like an invalid user attempt, but why port 52998 using ssh, I though ssh used port 22, or are they just tring their luck with another port?

Code: Select all

 
Jun  6 22:51:32 host-78-129-250-11 sshd[1462]: pam_succeed_if(sshd:auth): error retrieving information about user service
Jun  6 22:51:32 host-78-129-250-11 sshd[1459]: Failed password for adm from 64.132.224.199 port 52998 ssh2
Jun  6 22:51:32 host-78-129-250-11 sshd[1463]: Invalid user student from 64.132.224.199
Jun  6 22:51:32 host-78-129-250-11 sshd[1465]: Invalid user student from 64.132.224.199
Jun  6 22:51:32 host-78-129-250-11 sshd[1466]: input_userauth_request: invalid user student
Jun  6 22:51:32 host-78-129-250-11 sshd[1467]: input_userauth_request: invalid user student
Jun  6 22:51:32 host-78-129-250-11 sshd[1463]: pam_unix(sshd:auth): check pass; user unknown
 
There are pages of the above too (probably hundreds in the space of 2 hours)

Cheers
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Re: High inbound traffic - which logs to check?

Post by Weirdan »

Somebody ran a bruteforce attack on your ssh.
Tobey
Forum Newbie
Posts: 12
Joined: Thu May 14, 2009 11:40 am
Location: Germany

Re: High inbound traffic - which logs to check?

Post by Tobey »

Generating 160gb of traffic in a couple of hours is very uncommon if you don't have a big website running on your server. There might be someone who is trying to break into your system by attempting to log in with a lot of different passwords, but 160gb is still way too much for that. I don't know what software you have running, but if I were you I would immediately look for bugs that your php scripts might have and take the system down if there was still a lot of traffic in the next hours/days. In my opinion, the most likely reason, in case someone broke into your server, is that someone set up a FTP server and uploaded warez. That's the only thing I have seen so far where such huge amount of data came in.
User avatar
jaoudestudios
DevNet Resident
Posts: 1483
Joined: Wed Jun 18, 2008 8:32 am
Location: Surrey

Re: High inbound traffic - which logs to check?

Post by jaoudestudios »

Thanks for your feedback.

The only ports open on the server are ssh, http, mail, pop, smtp. So no one can upload apart from me via ssh.

It is the 1st time it has happened and it has not happened since, so it was a one off occasion. i only have a few websites on the server and none of them are big, none of them have been updated in a while so I dont think it will be a run away script - server sits idle most of the time. Disk space has not changed, I can account for all of it, so does not appear to have any additional files on there.

But you think 160GB is alot for someone trying a brute force attack?

I could limit access via ssh to my IP only, would this help? or would it still show up in the logs as an attempted login?
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: High inbound traffic - which logs to check?

Post by Benjamin »

Sounds more like a DOS attack to me.
User avatar
jaoudestudios
DevNet Resident
Posts: 1483
Joined: Wed Jun 18, 2008 8:32 am
Location: Surrey

Re: High inbound traffic - which logs to check?

Post by jaoudestudios »

If so how can I prevent it?
Tobey
Forum Newbie
Posts: 12
Joined: Thu May 14, 2009 11:40 am
Location: Germany

Re: High inbound traffic - which logs to check?

Post by Tobey »

Yep, I think it is way too much for that..by the way, is it a dedicated server?
Just because tools like df say that diskspace has not changed, it does not automatically mean that nothing has happened, or could have happened. Someone could have installed a rootkit that hides everything.
And even if your firewall blocks incoming connections, one can still upload files using a reverse proxy, or just downloading files by connecting from your server to another one. There are countless ways of transferring data ;)
Well, of course, all this does not mean that something bad happened to your server, but there is always a chance.
Restricting ssh access to your ip would work, if nobody has gained access yet, but there are better things. On my server I also block anything except HTTP. Lighttpd is running in a jail so even if there is a bug in one of my scripts, nothing dangerous can happen to my system. To log in I have knockd running and also a little program that I wrote on my own that looks up a dynamic domain name (no-ip.org) and allows connections from that ip. So everyone but me can only access HTTP :)
User avatar
jaoudestudios
DevNet Resident
Posts: 1483
Joined: Wed Jun 18, 2008 8:32 am
Location: Surrey

Re: High inbound traffic - which logs to check?

Post by jaoudestudios »

Tobey wrote:Yep, I think it is way too much for that..by the way, is it a dedicated server?
Yep, does this make a difference?
Tobey wrote:Just because tools like df say that diskspace has not changed, it does not automatically mean that nothing has happened, or could have happened. Someone could have installed a rootkit that hides everything.
According to the history no one else has logged in, I guess they could have cleared this though. But then they would have to be root and root can not log into my box directly (extra security).
Tobey wrote:Lighttpd is running in a jail so even if there is a bug in one of my scripts, nothing dangerous can happen to my system. To log in I have knockd running and also a little program that I wrote on my own that looks up a dynamic domain name (no-ip.org) and allows connections from that ip. So everyone but me can only access HTTP :)
I have not looked into this before or was not aware of it, sounds complicated though? :?
Tobey
Forum Newbie
Posts: 12
Joined: Thu May 14, 2009 11:40 am
Location: Germany

Re: High inbound traffic - which logs to check?

Post by Tobey »

At first I wasn't sure if you were talking about a dedicated server or just shared webspace, but since you have root access and everything, nevermind ;)
Clearing the login history is one of the easiest things and also getting root access is not that hard. Just take a look at bugtraq or milw0rm, there are many "local root" exploits floating around. In the worst case a hacker would exploit vulnerabilities that have not been reported yet to the developers. That's why I like to keep every port shut. Of course, there could be a bug in iptables, but the risk is hopefully very low.
Putting lighttpd into a jail took me some time, but there is a good howto available. You can find knockd and a good documentation over here.
User avatar
mikemike
Forum Contributor
Posts: 355
Joined: Sun May 24, 2009 5:37 pm
Location: Chester, UK

Re: High inbound traffic - which logs to check?

Post by mikemike »

sounds like it might be a distributed DOS? Did the server go down, even for only a couple of minutes?
Post Reply