Server hacking....

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Post Reply
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Server hacking....

Post by Chris Corbyn »

I often look in my server logs and see lots of random failed login attempts. I never see what the point in people trying it is because nobody even know what's on my server (and that's not a lot in fact)... certainly nothing worth stealing or breaking.

It doesn't worry me because I use very strong passwords, don't allow plaintext passwords over SSH, don't allow root logins over SSH and chroot FTP users into ~/. Nobody ever actually breaks in but I'm amazed just how often I see attempts.

Today I leave a `tail -f' of /var/log/messages running and come back to find this... (not surprisingly).

Code: Select all

Jan  3 13:48:23 d11wtq sshd[14098]: Failed password for invalid user 1956 from 80.235.105.114 port 37819 ssh2
Jan  3 13:48:24 d11wtq sshd[14100]: Invalid user 1957 from 80.235.105.114
Jan  3 13:48:24 d11wtq sshd(pam_unix)[14100]: check pass; user unknown
Jan  3 13:48:24 d11wtq sshd(pam_unix)[14100]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:26 d11wtq sshd[14100]: Failed password for invalid user 1957 from 80.235.105.114 port 38242 ssh2
Jan  3 13:48:27 d11wtq sshd[14102]: Invalid user 1958 from 80.235.105.114
Jan  3 13:48:27 d11wtq sshd(pam_unix)[14102]: check pass; user unknown
Jan  3 13:48:27 d11wtq sshd(pam_unix)[14102]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:29 d11wtq sshd[14102]: Failed password for invalid user 1958 from 80.235.105.114 port 38679 ssh2
Jan  3 13:48:30 d11wtq sshd[14104]: Invalid user 1959 from 80.235.105.114
Jan  3 13:48:30 d11wtq sshd(pam_unix)[14104]: check pass; user unknown
Jan  3 13:48:30 d11wtq sshd(pam_unix)[14104]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:33 d11wtq sshd[14104]: Failed password for invalid user 1959 from 80.235.105.114 port 39099 ssh2
Jan  3 13:48:33 d11wtq sshd[14106]: Invalid user 1960 from 80.235.105.114
Jan  3 13:48:33 d11wtq sshd(pam_unix)[14106]: check pass; user unknown
Jan  3 13:48:33 d11wtq sshd(pam_unix)[14106]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:36 d11wtq sshd[14106]: Failed password for invalid user 1960 from 80.235.105.114 port 39527 ssh2
Jan  3 13:48:36 d11wtq sshd[14108]: Invalid user 1961 from 80.235.105.114
Jan  3 13:48:36 d11wtq sshd(pam_unix)[14108]: check pass; user unknown
Jan  3 13:48:36 d11wtq sshd(pam_unix)[14108]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:39 d11wtq sshd[14108]: Failed password for invalid user 1961 from 80.235.105.114 port 39953 ssh2
Jan  3 13:48:39 d11wtq sshd[14110]: Invalid user 1962 from 80.235.105.114
Jan  3 13:48:39 d11wtq sshd(pam_unix)[14110]: check pass; user unknown
Jan  3 13:48:39 d11wtq sshd(pam_unix)[14110]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:42 d11wtq sshd[14110]: Failed password for invalid user 1962 from 80.235.105.114 port 40328 ssh2
Jan  3 13:48:43 d11wtq sshd[14112]: Invalid user 1963 from 80.235.105.114
Jan  3 13:48:43 d11wtq sshd(pam_unix)[14112]: check pass; user unknown
Jan  3 13:48:43 d11wtq sshd(pam_unix)[14112]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:45 d11wtq sshd[14112]: Failed password for invalid user 1963 from 80.235.105.114 port 40764 ssh2
Jan  3 13:48:46 d11wtq sshd[14114]: Invalid user 1964 from 80.235.105.114
Jan  3 13:48:46 d11wtq sshd(pam_unix)[14114]: check pass; user unknown
Jan  3 13:48:46 d11wtq sshd(pam_unix)[14114]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:48:48 d11wtq sshd[14114]: Failed password for invalid user 1964 from 80.235.105.114 port 41213 ssh2
Jan  3 13:48:58 d11wtq sshd[14116]: Invalid user 1965 from 80.235.105.114
Jan  3 13:48:58 d11wtq sshd(pam_unix)[14116]: check pass; user unknown
Jan  3 13:48:58 d11wtq sshd(pam_unix)[14116]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.polytex.ee
Jan  3 13:49:00 d11wtq sshd[14116]: Failed password for invalid user 1965 from 80.235.105.114 port 41651 ssh2
Jan  3 14:46:54 d11wtq xinetd[1206]: START: imap2 pid=14135 from=68.82.20.28
Jan  3 14:46:54 d11wtq imapd[14135]: port 143 service init from 68.82.20.28
Jan  3 14:46:54 d11wtq imapd[14135]: Command stream end of file, while reading line user=??? domain=??? host=pcp03108393pcs.rte20201.de.comcast.net [68.82.20.28]
Jan  3 14:46:54 d11wtq xinetd[1206]: EXIT: imap2 pid=14135 duration=0(sec)
Jan  3 15:14:22 d11wtq xinetd[1206]: START: smtp pid=14145 from=71.48.212.231
Jan  3 15:14:35 d11wtq spamd[13534]: connection from localhost [127.0.0.1] at port 1251
Jan  3 15:14:35 d11wtq spamd[13534]: info: setuid to nobody succeeded
Jan  3 15:14:35 d11wtq spamd[13534]: Creating default_prefs [//.spamassassin/user_prefs]
Jan  3 15:14:35 d11wtq spamd[13534]: Cannot write to //.spamassassin/user_prefs: No such file or directory
Jan  3 15:14:35 d11wtq spamd[13534]: Couldn't create readable default_prefs for [//.spamassassin/user_prefs]
Jan  3 15:14:35 d11wtq spamd[13534]: checking message <000001c61078$65ff7c00$0100007f@work-3> for nobody:65534.
Jan  3 15:14:37 d11wtq spamd[13534]: clean message (3.4/5.0) for nobody:65534 in 2.2 seconds, 6415 bytes.
Jan  3 15:14:37 d11wtq spamd[13534]: result: .  3 - DATE_IN_PAST_06_12,HTML_FONT_BIG,HTML_MESSAGE,RCVD_IN_NJABL_DUL,RCVD_IN_SORBS_DUL,URIBL_SBL,URIBL_WS_SURBL scantime=2.2,size=6415,mid=<000001c61078$65ff7c00$0100007f@work-3>,autolearn=disabled
Jan  3 15:14:37 d11wtq exim[14145]: 2006-01-03 15:14:37 1Etnrk-0003g9-BU <= alexander@griffield.biz H=nc-71-48-212-231.dhcp.sprint-hsd.net (friend) [71.48.212.231] P=esmtp S=6470 id=000001c61078$65ff7c00$0100007f@work-3
Jan  3 15:14:37 d11wtq exim[14146]: 2006-01-03 15:14:37 1Etnrk-0003g9-BU => enquiries <enquiries@chriscorbyn.co.uk> R=localuser T=local_delivery
Jan  3 15:14:37 d11wtq exim[14146]: 2006-01-03 15:14:37 1Etnrk-0003g9-BU == enquiries@chriscorbyn.co.uk R=write_spam_05 T=write_spam defer (13): Permission denied: failed to create directories for /home/d11wtq/Mail/chriscorbyn.co.uk/spam: Permission denied
Jan  3 15:14:56 d11wtq sshd[14150]: reverse mapping checking getaddrinfo for corporativos_245185-3.etb.net.co failed - POSSIBLE BREAKIN ATTEMPT!
Jan  3 15:14:56 d11wtq sshd(pam_unix)[14150]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=201.245.185.3  user=root
Jan  3 15:14:59 d11wtq sshd[14150]: Failed password for root from 201.245.185.3 port 35057 ssh2
Jan  3 15:15:05 d11wtq sshd[14152]: reverse mapping checking getaddrinfo for corporativos_245185-3.etb.net.co failed - POSSIBLE BREAKIN ATTEMPT!
Jan  3 15:15:05 d11wtq sshd(pam_unix)[14152]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=201.245.185.3  user=root
Jan  3 15:15:08 d11wtq sshd[14152]: Failed password for root from 201.245.185.3 port 35356 ssh2
Jan  3 15:15:15 d11wtq sshd[14154]: reverse mapping checking getaddrinfo for corporativos_245185-3.etb.net.co failed - POSSIBLE BREAKIN ATTEMPT!
Jan  3 15:15:15 d11wtq sshd(pam_unix)[14154]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=201.245.185.3  user=root
Jan  3 15:15:17 d11wtq sshd[14154]: Failed password for root from 201.245.185.3 port 35633 ssh2
Jan  3 15:15:27 d11wtq sshd[14156]: reverse mapping checking getaddrinfo for corporativos_245185-3.etb.net.co failed - POSSIBLE BREAKIN ATTEMPT!
Jan  3 15:15:27 d11wtq sshd(pam_unix)[14156]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=201.245.185.3  user=root
Jan  3 15:15:29 d11wtq sshd[14156]: Failed password for root from 201.245.185.3 port 35969 ssh2
Jan  3 17:38:09 d11wtq proftpd[14212]: localhost (lns-bzn-31-82-252-215-230.adsl.proxad.net[82.252.215.230]) - FTP session opened.
Jan  3 17:38:10 d11wtq proftpd[14212]: localhost (lns-bzn-31-82-252-215-230.adsl.proxad.net[82.252.215.230]) - USER ftp (Login failed): Invalid shell: '/bin/false'
Jan  3 17:38:10 d11wtq proftpd[14212]: localhost (lns-bzn-31-82-252-215-230.adsl.proxad.net[82.252.215.230]) - FTP session closed.
The list is much longer but still.

Probably a bot, or several bots doing it.

It makes me wonder just how many people get rooted without even knowing it because they just have god awful security proedures on their meachines.
Charles256
DevNet Resident
Posts: 1375
Joined: Fri Sep 16, 2005 9:06 pm

Post by Charles256 »

what log was that in?
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Charles256 wrote:what log was that in?
/var/log/messages
User avatar
trukfixer
Forum Contributor
Posts: 174
Joined: Fri May 21, 2004 3:14 pm
Location: Miami, Florida, USA

Post by trukfixer »

yeah it is common

/var/log/messages and /var/log/auth.log (on some machines) often shows that - Ive dropped in plenty enough of iptables drops when I come across these- Often they cause very little load at all on the machine , so they are not really noticable, and yes these are automated bots attempting to crack in..

Despite your confidence at strong passwords, never, ever assume that you have not been 0wned..

If someone were bright enough to crack root and 0wn your box, they could also eliminate all traces of their dirty deeds by simply editing the logs and removing the details , and with enough ingenuity, they can put in their own root kit, partition off a small amount of your drive,leave it unmounted so you will never see it, and login and mount it and use that at will whenever they please.. even if you change your root pass , they can still get in.. they can install their own keys, disguising themselves as a user that actually does belong on the box (I.E. simply edit , for example, one of the wheel group users and change /etc/passwd shell from /bin/falselogin to /bin/bash and set up a hidden chroot environment , give it root privs or su to root privs, and then it wont matter in the least that you change root pass or any other pass- and these are extremely difficult to eradicate without screwing up SOMETHING on the machine, forcing you to take the box down for a couple hours while you remove entire applications, and their dependencies, and set it back up with clean, fresh apps.. and even that will not give you any guarantees - they easily could have set up *several* accounts with su privs.. so I cant stress enough, never ever become overconfident in the strength of your passwords... )

not even tripwire would really be effective unless you install tripwire immediately with a new fresh install of the O.S.

Never be so confident as to think "no one is going to be able to log in to THIS sucker" Key only SSH helps, as well as restricing ANY ssh access to the one user who can su to root , and no one else.. , but it is *not* foolproof :)

There are many other ways of cracking root and/or escalating privs than just a shell login - Root could be achieved via FTP logins as well, as well as any other open port on your machine (yes even port 80, though that'd be quite an accomplishment, and a LOT of things would have to be *just right* for that to happen, but it can and does happen.... )

Either way - any time I am sitting there tailing logs, I will more likely be doing

Code: Select all

tail -f /var/log/auth.log
then I just note the ip and drop in

Code: Select all

iptables -I INPUT  -s  xxx.xxx.xxx.xxx/32 -j DROP
into iptables rules and kill the attack... but there is not much else you can really do to prevent it - I even get these on machines that dont even allow challenge/response password auth - it's just a log of ssh "answering" ssh requests...
Charles256
DevNet Resident
Posts: 1375
Joined: Fri Sep 16, 2005 9:06 pm

Post by Charles256 »

do you get those kind of logs in a windows environment?
User avatar
trukfixer
Forum Contributor
Posts: 174
Joined: Fri May 21, 2004 3:14 pm
Location: Miami, Florida, USA

Post by trukfixer »

Charles256 wrote:do you get those kind of logs in a windows environment?
No idea- I would think Windows would have some sort of logging, but I dont even use windows for my Desktop PC, let alone a server.. :) I would say any logs they do have are probably GUI based.. (Bleccch!) hehehe
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

trukfixer wrote:Either way - any time I am sitting there tailing logs, I will more likely be doing

Code: Select all

tail -f /var/log/auth.log
then I just note the ip and drop in

Code: Select all

iptables -I INPUT  -s  xxx.xxx.xxx.xxx/32 -j DROP
into iptables rules and kill the attack... but there is not much else you can really do to prevent it - I even get these on machines that dont even allow challenge/response password auth - it's just a log of ssh "answering" ssh requests...
I'm not worried enough to do that although I may take a deeper look at my FTP configuration and see if I can tie it down a little more.

nmap 'ing my server only reveals a handful of needed open ports to be able to run the services I have (IMAP, HTTP, FTP, SSH, MySQL).

Could you not write a bash script to run by cron every hour or so to do what you're doing with iptables? A bit of grepping and logging but it should be fairly simple :) You could actually log all the multiple offenders in a database too.
User avatar
trukfixer
Forum Contributor
Posts: 174
Joined: Fri May 21, 2004 3:14 pm
Location: Miami, Florida, USA

Post by trukfixer »

as far as making it a shell script - we'd considered it, but it's hardly worth the effort... it doesnt happen all that often, and there's always teh chance of dropping an iptable block on an important IP .. so we only really do that when we're just happening to be checking a machine for one reason or another and just notice it ..

One of my habits on any machine I log in to is checking auth.log , messages, and dmesg, and cat /etc/passwd to look for recent changes, and check mysql-slow.log, and then have a peek at the top filesizes (ls -laS ) in apache logs looking for any unusual website activity logs, and a careful look at /tmp and /var/tmp and a couple other directories for www-data or 'nobody' owned files .. and see if they contain any of teh common exploits - if I dont know what a file is, I check it out, strings it if it is binary, or cat it if it isnt.. looking for common scripts like udp.pl , etc , and if any are found, I will cd to /var/www and start an rgrep for exec(), system() and passthru() functions -

very often, we'll find a shell script that some hacker uploaded via an exploitable script (such as phpBB, or VBulletin or recently, WordPress, among many many insecure php or cgi applications that allow for it..)

those scripts are usually deleted (I have a growing collection , every time I find an unique script like that, I keep a copy and see if I can find ways to break it or prevent it fom working , without breaking legit usage by legit scripts .. just for practice.. ) It also gives me an incredible insight into web security ...

Oh well..

I just wanted to mention a bit of real world experience from "in the trenches" :)

Bri!
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Post by josh »

trukfixer,

even with the ssh restricted to the sysadmin's account (one with su privs), brute forces and the like are still possible, what do you do to prevent that besides blocking IPs when you notice em (and of course having a strong password). Are there ways to set it up so after 5-10 failed log in's it locks that user out for a set amount of time and emails you? How would something like this be done, is it even necessary?

You mentioned rooting box's through open ports, how does that work and how does it affect me (the average guy running Apache mysql and php, with mediocre knowledge of Linux)? Which (firewalls?) do you recommend for this kind of thing?
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

jshpro2 wrote:trukfixer,

even with the ssh restricted to the sysadmin's account (one with su privs), brute forces and the like are still possible, what do you do to prevent that besides blocking IPs when you notice em (and of course having a strong password). Are there ways to set it up so after 5-10 failed log in's it locks that user out for a set amount of time and emails you? How would something like this be done, is it even necessary?

You mentioned rooting box's through open ports, how does that work and how does it affect me (the average guy running Apache mysql and php, with mediocre knowledge of Linux)? Which (firewalls?) do you recommend for this kind of thing?
You can get plugins for PAM that can do all sorts of useful things for the login process....

Brute force is practically useless with Key-only logins too :)

Run nmap on your server from another machine and see if there's more open than you think. I'm pretty sure the security holes in Apache are pretty useles to attackers wanting to root your box. Not sure about MySQL. FTP can be extremely insecure if you don't take precautions. It the moment I just use a chroot for each user over FTP which help restrict access to important areas but it still could be better locked down.

Not sure why I responded :? You asked for trukfixer :P..... Sorry :oops:
User avatar
trukfixer
Forum Contributor
Posts: 174
Joined: Fri May 21, 2004 3:14 pm
Location: Miami, Florida, USA

Post by trukfixer »

hehehe - yeah - like d11 said- ssh key auth is the way to go - yes we are aware that even with strong passwords, it is possible for a user to brute force or crack a machine, if they happen to "get lucky" and find a username that works, and happen to brute force the password correctly .. (its a highly random issue) but again- most machines, we will allow key only logins for ssh, and only a minimal number of users actually have a shell account..

Anyhow- we run Debian with grsecurity kernel patch for one thing, and we have a custom firewall, our network doesnt respond to pings, among a few things we do .. security is not any one thing that you do to lock down your box - it's a lot of little things that you can do that all add up .. if I were to list every step we take on a machine that is done with the express purpose of securing a box, I'd have a list about 3 browser screens long, if not longer :) - my best advice for anyone who really wants to get serious about protecting their box from being hacked, is, hang out at a couple security forums (securiteam for one, I subscribe to the mailing list there..) , and do some reading.. get to know your box from / (root directory) and know what is where .. pay special attention to where the most common problems will show up first - typically, /tmp and/or /var/tmp (where sessions are written, and file uploads are stored, etc ), and the /home directory where users are chrooted (normally /home/username, but can be aanywhere) and know where all of your logs are located.. visit your machine often, get to know where to find stuff, learn to use teh command line utilities like ps,cat,awk,sed,grep,less,more,tail,mysqladmin , etc , and know how to start and stop all services (/etc/init.d/mysql stop/start /etc/init.d/postfix/stop/start, apachectl restart , /sbin/firewall start , etc , etc ..
different flavors of Linux have differences in how services are started and controlled from root ..

learn how to read messages, dmesg, last logins, auth log, mysql logs, and other server logs.... Get to know what is normal on your box , so when things happen out of the ordinary, you'll notice them quickly , and often, patches on packages may need to be applied before the distro builders have an upgrade, so it'd also be good to learn how to use patch and the machine's package system (dpkg, rpm, whatever) as well as learning how to do a few of the iptables drops and listings, and how to read teh process lists, I could go on and on all day .. but in a nutshell, security is 3 parts active administration and 1 part common sense, more than any specific software that you can install and say "whew, OK, now my box wont get hacked" (I personally dont believe there *IS* any software that can make that possible hahaha) .. so either one needs to get to knoww Linux inside and out (from /) , or one would have to rely on a trained, seasoned systems admin to do that..

Basically , where I work, our clients just put up the site and run their business, and leave all the security and tuning and maintenance to us... which works out pretty good :)

Our track record, however , seems to indicate we've been doing something right, cause except for one instance of a box getting rooted , we've never had anything more serious that script kiddies and a few IRC bots breaking in via apache...(I think that one instance of being rootkitted was more due to the person that leased the machine being a little too free with his machine passwords, giving it out to several different coders he had hired to do work on and off, cause the server's archived logs didnt show any logins that were *not* authorized, and this was one box that allowed password auth )

Bri!
timvw
DevNet Master
Posts: 4897
Joined: Mon Jan 19, 2004 11:11 pm
Location: Leuven, Belgium

Post by timvw »

another way to nail your kernel is applying the grsec patches, mounting the partition with /bin as read-only, ...

It appears Panama Jack was faster to hint all this :)
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Post by josh »

d11 -

I was addressing trukfixer, but this is an open discussion, your input is appreciated

I should really be paying more attention to security, and I've really been slacking on learning my box like you said. (I just realized I don't even know how to upgrade my kernal). I'll be bookmarking this thread and googling a lot of your suggestions. Some of that stuff I already do (checking logs, etc..) so maybe I'm not doing sooo bad (after all my linux experience only goes back 2 months). Regardless I have a lot of research to do.

You had mentioned chrooting the FTP users, this is smart... but if the attacker was able to exploit your FTP in the first place couldn't they create a cgi / php script to get around that anyways?
User avatar
m3mn0n
PHP Evangelist
Posts: 3548
Joined: Tue Aug 13, 2002 3:35 pm
Location: Calgary, Canada

Post by m3mn0n »

Wow. Amazing things you bring up here guys.

I really need to get my hands on a Unix/Linux security book. :wink:
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

jshpro2 wrote:You had mentioned chrooting the FTP users, this is smart... but if the attacker was able to exploit your FTP in the first place couldn't they create a cgi / php script to get around that anyways?
Sorry I missed that last sentence before....

Anything that runs chrooted isn't completely bulletproof no... but I very much doubt it's as easy as that to break out of a chroot environment. It's like being in a cage only you can't see what's on the outside...

Lot's of networked applications have settings in the config files to handle chrooting themselves. This way, if anyone manages to somehow gain access to your server on that particular port they are stuck in that one directory so the damage they do will be little to none. I also run PowerDNS in a chroot environment... an attacker, if they broke in would be able to screw up my NS yes but they wouldn't be able to get to "/" . At least the slave DNS servers would carry on functioning in this case.

Just little things :)
Post Reply