Marius Ducea discussed three ways how to block brute force attacks under Linux using iptables, PAM and fail2ban. All three posts are quite detailed and well written.
iptables is a tool for packet filtering and NAT. You can use it to setup a firewall with shell scripts. A shell script with just a few lines is enough to block an IP address attempting to establish too many connection within a certain time frame.
PAM (Pluggable Authentication Modules) is an API used by many Unix systems for authentication. Marius described how to use PAM to block the IP address of an attacker after three failed logon attempts on a ssh server.
fail2ban can be used to update firewall rules based on log files scans. Marius demonstrated how to secure an ssh server with fail2ban using iptables.
The last two options are certainly more sophisticated than the mere iptables solution. Unfortunately, it takes more time to configure them as you might have to install the tools first.
My favorite solution is fail2ban since you can use it with almost any application, plus there are packages for most Linux distributions. PAM is more difficult to setup as you probably will have to install it from sources.
Marius described the installation and configuration of all three options in detail: