Keystores contain certificates used by Java-based applications to authenticate and encrypt HTTPS traffic. VMware Cloud Foundation (VCF) leverages a keystore and automates a significant part of the interaction with it. As a result, management of the keystore is often overlooked.
In this article, I’ll discuss the keystore used by VCF, why you would need to manage it, and demonstrate some of the commands you might use.
VMware Cloud Foundation leverages Java-based applications extensively for a wide range of purposes. These applications often have a need to communicate securely to another component on the network. Commonly, the Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocol is leveraged for this purpose. This protocol (as well as others) leverages a private/public key method. The public key is used to establish a secure communication session with a web server and the private key is used to digitally sign the web pages so that they can be validated as certified pages.
The public key is contained within a certificate. The certificate can include other information, such as details about the domain or the owner of the site. The certificate is signed by a Certificate Authority (CA) and this allows content to be trusted.
I’m over-simplifying this a bit, but for our purposes it’s just important to understand that these certificates are used by the VCF SDDC Manager to securely communicate with other components. This might be with a vCenter Server instance or another web-based service.
As there are several components that the SDDC Manager needs to communicate with, it needs a place to store all these certificates. This is where the keystore comes into play.
On a VCF system, there are a couple of keystores including:
- /etc/vmware/vcf/commonsvcs/trusted_certificates.store
- /etc/alternatives/jre/lib/security/cacerts
- /usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/cacerts
Each of the keystores is protected with a password, which you will need in order to perform any actions on the keystore. The default password for a keystore is ‘changeit’. You will find that this password is used for the two keystore files named ‘cacerts’ above. The password for the third keystore is stored in plain text in the file /etc/vmware/vcf/commonsvcs/trusted_certificates.key. Simply look at the contents of that file to see what the password is.
Before going further, we should talk about why you may need to work with the keystores…
The most common reason is because you want to remove old certificates from the system. The certificates may have expired or are no longer valid. In any case, this is just general administrative housekeeping.
The second most common reason is because you want to add in a certificate manually. For example, recently I was approached by someone who was setting up a fake VCF depot. They kept getting an error that the certificate that was being used didn’t match with the name of the system that was being connected to. To make a long story short, they just needed to add in the certificate of the web server they were using to the keystore.
Another example of this is when you need to add in the certificate to support a shared Single Sign-On domain.
Now that you know why you might want to work with the keystores, the next question is how do you do it? Well, there are two methods that can be used. First, let’s do it the hard way…
Once you know where the keystore is and what the password for the keystore is, you can use the keytool utility to work with the keystore. This utility will allow you to examine the contents of the keystore, add keystore entries, and delete the keystore entries.
To display the contents of the keystore, you would use a command like:
keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store
This will prompt you for the keystore password and then will show you a verbose list of all the keystore entries.
You can also specify the password on the command line, by using the -storepass option like this:
keytool -list -keystore /usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/cacerts -storepass changeit
If you want to add a public certificate, you can use the keytool utility like this:
keytool -importcert -alias new_cert -file new_cert.csr -keystore /usr/lib/jvm/OpenJDK8-1.8.0/jre/lib/security/cacerts -storepass changeit
Notice that every entry will have an alias associated to it. This is just another name that you can use to easily identify the entries.
There are several more command options for the keytool utility, such as the options to delete a entry in the keystore. You can use the -help option to see a list of all the options.
Use of the keytool utility will work with all the Java keystores. However, VMware Cloud Foundation provides another method in the form of a utility called ‘sddcmanager-ssl-util.sh’ located under /opt/vmware/vcf/operationsmanager/scripts/cli/. Through this utility, you can add, delete, and list keystore entries.
Using the -help option, you can see how the command syntax should look like:
As this is just a shell script, you can look at the contents and you’ll see that it’s using the same keytool utility we talked about earlier!
I hope this helps you to understand how you can manage the keystore entries in your environment!
Have fun and remember – don’t start randomly deleting certificates from the keystores or bad things will happen.