Like any server exposed to internet, our server also gets its fair share of attacks from all over the world. Since I am not sharing the IP address of the server here, I feel it safe to share the experience.

The server is behind a firewall which redirects SSH connections to the BBB server which hosts all our Git repositories. The BBB runs on Ubuntu and has its own firewall (ufw) running too. We have disabled password based authentication in ssh and use 4096 bit keys.

Initially we could see attempt after attempt by bots to try different user names and passwords. The server would disconnect the rogue clients immediately on getting wrong user name. But the bots seemed to try brute force method to find appropriate user names.

To limit this we used ufw's throttling feature. This feature blacklists an IP for some time if too many connections are attempted from the same IP in a short span of time. This brought down the number of attempts considerably. But this was just not enough. After some time, the attempts from the same IP would come back again only to be blacklisted again. Then we decided to write a helper script to harden the firewall on BBB.

The script written was just a plain bash script invoked every hour as a cron job. The script goes through the log files to see any unsuccessful log-ins and stores the IP addresses in a SQLite database. The script has logic to remove false negatives also. At the end, it invokes ufw to add more DENY rules for those addresses. Actually we use X.Y.0.0 mask to create deny rule. We go for this range because almost all of the attacks seem to originate from countries which we can safely put in our blacklist.

Without the above setup, the log-in attempts would run into thousands within a couple of days. With it, they don't reach even 100 in a week!