To check the new account's password at creation time against existing salted hashes, you'd have to hash the new password with each existing password's salt. If you are using something like bcrypt or scrypt which is designed to be slow at this, that might take a while if you have a lot of existing accounts.
Maybe a Bloom filter approach? Besides the salted slow hash you store of each password, also put the password in a Bloom filter. Check new passwords against the Bloom filter. You'll get some false positives that way, but maybe that is acceptable.
I'm seriously tempted, if I ever have to implement a password system again, to allow up to 256 characters, just store an unsalted SHA256 hash, and tell people on signup that they should be using a password manager with a long random password if they care about security of their account.
Maybe a Bloom filter approach? Besides the salted slow hash you store of each password, also put the password in a Bloom filter. Check new passwords against the Bloom filter. You'll get some false positives that way, but maybe that is acceptable.
I'm seriously tempted, if I ever have to implement a password system again, to allow up to 256 characters, just store an unsalted SHA256 hash, and tell people on signup that they should be using a password manager with a long random password if they care about security of their account.