Here's a nice study on the performance of Linux's network firewalling/packet mangling layer:
Conclusions (mine, based on the study):
- Netfilter/iptables is up to par with other filter solutions when it comes to plain routing.
- Netfilter/iptables is not up to par with other filter solutions when it comes to connection tracking (basically just getting all the traffic through netfilter and keeping track of it) and filtering
- When a chain has many rules, netfilter/iptables filtering performance drops significantly. Chain modifications (adding rules) performance also degrades significantly. This starts at 256 rules, so don't use more.
The problems seem to stem from the way Netfilter stores and processes the rules:
It is well known that netfilter/iptables does not scale well if one wants to use large number of rules in a single chain. The reason of the problem lies in the fact that the rules are processed in netfilter/iptables one after another, linearly.