If you only support public key or two factor authentication you won't need to worry about brute force attacks. Most SSH brute force attacks are dictionary based using common usernames.
You'll most likely be ok. The main attack vector is a chunk of regularly used usernames and a small selection of passwords. These are quite successful against shared hosting where password quality is hard to control properly.
If you've got a "garden variety" server with a strong password, I wouldn't worry.
I've got a laptop slung on the end of my ADSL line that has had literally millions of attempts.
If you are worried, you can install fail2ban which will block repeated attacks at the firewall level.
Virtually all SSH attacks are of the nature mysql:mysql or mysql:password, so you should be safe as long as you can trust your users not to be stupid. The attackers prefer quantity over quality when looking for targets.
And if you use SSH keys you should be totally safe.
Remember to apply security fixes though since the automatic attacks also probe for ancient versions of SSH servers.
If you're worried about ssh brute force or just don't like all the noise in your logs, moving the port tends to drop off about 95% of them. In addition, running iptables tarpit rules (or your OS equivalent) tends to kill the rest fairly quickly.
I put info like this in a password manager for sanity (Keepass, I work on a PC).
One easy way to manage it is make a folder for each hostname, and add things like mysql root password, ssh port, public IP, pivate IP as different entries relating to the all aspects of managing the host.
