Hi,
Thanks for all the details and suggestions. I just had a long meeting with my technical team and following are the updates.
1) SYN cookies is already Enable but not making much difference.
2) I will provide the packet details soon for the request to give the extra idea for the attack.
3) Following are the details I have got from my linux admin. This will help you to trace the issue in better way.
Problem: Apache SYN_RECV
OS - RHEL5
kernels - 2.6.18-92.1.22.el5-x86_64
2.6.18-92.el5-x86_64
rpms:-
kernel-devel-2.6.18-92.el5
kernel-headers-2.6.18-92.1.22.el5
kernel-devel-2.6.18-92.1.22.el5
kernel-2.6.18-92.1.22.el5
kernel-2.6.18-92.el5
OS Type:
cat /etc/issue
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
> cat /proc/version
Linux version 2.6.18-92.1.22.el5 (
mockbuild@hs20-bc2-5.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Fri Dec 5 09:28:22 EST 2008
We are providing 403 code for the URL request. Following is part of Access Log
94.70.118.139 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)"
89.216.230.148 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; sr; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
82.81.54.226 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB5; CT2077543_4.5.191.15)"
85.229.15.86 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; .NET CLR 2.0.50727; CT2088752_4.5.188.7)"
92.237.189.17 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
84.106.127.218 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; eSobiSubscriber 2.0.4.16; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)"
87.93.30.98 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.188.7)"
93.86.61.247 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.0" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; CT2077543_4.5.188.7)"
91.152.228.27 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; Creative ZENcast v2.00.13; CT2088347_4.5.188.7)"
94.69.164.32 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.1.4322; .NET CLR 2.0.50727; CT2088700_4.5.191.15)"
82.201.180.177 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6; CT2077543_4.5.189.28)"
83.248.2.230 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SIMBAR={46BC3752-9118-483D-8E88-CD3E89FCD192}; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2088752_4.5.188.7)"
99.235.137.30 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; CT2077543_4.5.191.15)"
216.155.136.84 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 1.1.4322; CT2077543_4.5.188.7)"
217.123.166.205 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)"
86.96.227.88 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)"
203.115.189.77 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; CT2077543_1.5.48.2; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
203.82.79.102 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; CT2088347_4.5.188.7)"
88.195.52.126 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.189.24)"
77.81.114.171 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; SIMBAR={B471FCBA-22ED-11DE-91A3-00196693641D}; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; CT2077543_4.5.191.15)"
92.84.250.65 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; CT2077543_4.5.191.15)"
netstat:
tcp 0 0 domain.com:http 85-156-91-20.elisa-mo:55168 SYN_RECV
tcp 0 0 domain.com:http 220.255.7.227:27183 SYN_RECV
tcp 0 0 domain.com:http 5e03cbc4.bb.sky.com:51086 SYN_RECV
tcp 0 0 domain.com:http 79.126.234.198:18139 SYN_RECV
tcp 0 0 domain.com:http 78.148.175.148:11115 SYN_RECV
tcp 0 0 domain.com:http 83-154-143-68.rev.lib:61479 SYN_RECV
tcp 0 0 domain.com:http ABTS-North-Static-248:54775 SYN_RECV
tcp 0 0 domain.com:http 90-230-131-95-no130.tb:1134 SYN_RECV
tcp 0 0 domain.com:http static-host119-73-6-2:49538 SYN_RECV
tcp 0 0 domain.com:http 222.127.130.238:gtp-control SYN_RECV
tcp 0 0 domain.com:http acl1-1571bts.gw.smartbr:g5m SYN_RECV
tcp 0 0 domain.com:http athedsl-282427.home.o:60002 SYN_RECV
tcp 0 0 domain.com:http CPE-58-166-77-138.nsw:60067 SYN_RECV
tcp 0 0 domain.com:http C-59-101-99-107.syd.c:51097 SYN_RECV
tcp 0 0 domain.com:http ti0111a380-2667.bb.on:60993 SYN_RECV
tcp 0 0 domain.com:http 92.81.2.242:60451 SYN_RECV
tcp 0 0 domain.com:http 118.100.120.248:pserver SYN_RECV
tcp 0 0 domain.com:http triband-del-59.178.84:50140 SYN_RECV
tcp 0 0 domain.com:http cpc4-leds5-0-0-cust82:obrpd SYN_RECV
tcp 0 0 domain.com:http ALyon-153-1-8-78.w86-:59494 SYN_RECV
tcp 0 0 domain.com:http 120.28.199.183:3comnetman SYN_RECV
tcp 0 0 domain.com:http h248.4.16.98.dynamic.:60758 SYN_RECV
tcp 0 0 domain.com:http 89.211.205.59:64217 SYN_RECV
tcp 0 0 domain.com:http CPE-124-187-26-30.qld:ff-sm SYN_RECV
tcp 0 0 domain.com:http frw.Gloworld.com:59104 SYN_RECV
tcp 0 0 domain.com:http 220.255.7.182:winpoplanmess SYN_RECV
tcp 0 0 domain.com:http srisaionline180.excell:1232 SYN_RECV
tcp 0 0 domain.com:http CPE-60-230-16-150.vic:52611 SYN_RECV
tcp 0 0 domain.com:http 203.82.91.102:41318 SYN_RECV
tcp 0 0 domain.com:http 69.171.165.50:32454 SYN_RECV
tcp 0 0 domain.com:http dsl-TN-static-195.:corbaloc SYN_RECV
tcp 0 0 domain.com:http 210.186.66.179:49330 SYN_RECV
tcp 0 0 domain.com:http ABTS-North-D:xinuexpansion3 SYN_RECV
tcp 0 0 domain.com:http c122-106-133-46.livrp:49273 SYN_RECV
tcp 0 0 domain.com:http 173.subnet125-1:nssalertmgr SYN_RECV
tcp 0 0 domain.com:http 121.246.52.30.dynamic:63977 SYN_RECV
tcp 0 0 domain.com:http mobile-3G-dyn-BC-179-1:4464 SYN_RECV
tcp 0 0 domain.com:http crd48.neoplus.adsl.t:aminet SYN_RECV
Following we have done till now is mentioned below for the configurations.
###############
sysctl.conf
##############
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
# Enable IP spoofing protection, turn on Source Address Verification
net.ipv4.conf.all.rp_filter = 1
# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1
# 65536 seems to be the max it will take
net.ipv4.ip_conntrack_max = 1048576
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1
#############
fwsnort, bfd burnintest chkrootkit ddos faf lsm nobody_check sim apf
#############
modsecurity-apache
LoadModule evasive20_module /usr/lib64/httpd/modules/mod_evasive20.so
<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 3
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 30
</IfModule>
LoadModule security_module /usr/lib64/httpd/modules/mod_security.so
<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 3
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 30
</IfModule>
####################
Again,
Hope this helps you to see the issue in details. I have also put the latest configurations to keep site going on.
If it may be where server has some maleware or spyware, please let me know the solution and we can scan the pc for that too. Besides these, please let me know for any suggestion you think will be helpful.
Thanks again for your time and have a wonderful day.
Sam