Firefox and newer no longer SSL support keys less than 1024 bits. This means you need to create a new set of keys for Webmin.
The easiest way to fix this is from Webmin using a different browser (like current versions of Chrome/Chromium) that let you bypass this type of error. Just login to the Webmin web UI and select:
Webmin -> Webmin Configuration -> SSL Encryption -> Self Signed Certificate.
Fill in the form and then click the
button. This will update your self-signed certificate for Webmin and you will now be able to access the page from Firefox (With the usual browser warning about an untrusted connection)