Yesterday our firewall at work started randomly dropping connections. It has probably happened a few times before, but been seemingly resolved before I had a chance to look at it.
Weirdly, some people in the office reported no problems, while others couldn't even get DNS lookups to work properly.
Looking at the firewall, I found this in /var/log/syslog (this is a Debian box - other places to look is /var/log/secure or /var/log/messages):
Apr 29 12:35:12 XXX syslog-ng[363]: STATS: dropped 113
Apr 29 12:45:12 XXX syslog-ng[363]: STATS: dropped 120
Apr 29 12:55:12 XXX syslog-ng[363]: STATS: dropped 116
Uh, oh. This is our iptables firewall dropping packets like crazy. When this happens, it could be someone trying to do something nasty, but then our connections shouldn't be dropping.
As it turns out, there's also a number of other possible causes, and this time here's the reason:
root@XXX:~# cat /proc/net/ip_conntrack | wc -l
11586
root@fw1:~# cat /proc/sys/net/ipv4/ip_conntrack_max
12000
The limit of 12000 connections to be tracked was likely hit regularly around that time, causing random packets to be dropped because the firewall wasn't allowed to keep the connection information, and so didn't find anything to indicate the packet should be allowed through.
The simplest way of fixing this is to simply echo a new value into ip_conntrack_max:
echo 25000 >/proc/sys/net/ipv4/ip_conntrack_max
Make sure to put it in your disto-appropriate script to ensure it survives a reboot..
You may also want to
read up some more on how to choose appropriate long term values for ip_conntrack and the associated hash table, rather than just randomly picking values...