In OS X Apple (for one reason or another) artificially limits the maximum RSA key size for certificates to 4k (4098 bits). Most users will never notice any issues since most sites use 4k RSA keys or lower. However – for the handful of people who use/own websites that use 8/16/32 etc bit RSA keys, this artificial limit can cause problems. Below I’ll run though how to change the key size limit with 1 quick line in Terminal. As ever, please ensure you’ve read and agree to the disclaimer (linked here) before continuing.
Changing the maximum RSA key size on OS X.
1) Firstly you’ll need to open up Terminal.
2) Type in the following command (be sure to use the correct command for your release of OS X. In OS X Mavericks (10.9) Apple changed the location of where the key limit preference needs to be written to. The commands below set the RSA key limit to 16k/16384 bits and thus should be future-proof (unless Apple change the location of where the preference needs to be written to again). After typing/copying in the command and pressing enter, it should ask for your password and go straight ahead and change the RSA key size limit for you.
Mountain Lion and below (up to and including 10.8):
sudo defaults write /Library/Preferences/com.apple.crypto RSAMaxKeySize -int 16384
Mavericks and above (10.9+):
sudo defaults write /Library/Preferences/com.apple.security RSAMaxKeySize -int 16384
