Most platforms today implement a password aging system that requires a user to change the passwords used within a specific interval or have access automatically disabled. VMware is no different. By default, password aging is enabled on most VMware products, including vCenter, NSX, and so on.

In this article, I’ll discuss what can happen when passwords expire within a VMware Cloud Foundation (VCF) environment and demonstrate how you can avoid issues.

A little while ago, I needed to use a lab environment that was running VCF version 3.10.x. When I logged into the SDDC Manager, instead of the UI that I was expecting, I received the following:

Obviously, that’s not right…

So began the troubleshooting effort to try and identify what had happened.  

I went immediately to the log files on the SDDC Manager to look for clues. In the log for the SDDC Manager UI, I found the following entries:

With VCF 3.10.x, the Platform Service Controllers (PSCs) were not embedded and were deployed as a set of separate virtual machines. The log message didn’t quite tell me what the exact issue was, but there was certainly an issue with the PSCs.

When I tried to access the PSC, the root cause of the issue became clear – the password for the PSC had expired.

The password aging is a standard security practice. The idea is that users are forced to change their passwords during a given time interval or their access will be disabled.

VCF is comprised of a multitude of products, such as vCenter, NSX, and so on. To provide the automation and services that the SDDC Manager provides, it needs to communicate to all these components. This is done through a variety of methods, such as through API calls or by use of SSH. Failures in this communication can cause a variety of issues, such as the issue I was observing with not being able to access the UI.

Most commonly, a failure with the communication occurs because someone changes the password manually on a component. For example, a vSphere admin may change the password on vCenter like they normally would – forgetting that this was a VCF environment. This would cause a issue as the SDDC Manager maintains a list of specific accounts it uses to communicate with the components in the environment. If the SDDC Manager doesn’t know about the password change then it won’t be able to perform the actions it needs to.

This is also the case when the passwords expire.

To fix the issue then, all that is needed is to change the password for the expired account and make sure that it’s the same password used by the SDDC Manager. This is where you’ll likely run into the next challenge….

In conjunction with password aging, other security features such as password complexity and password history are also implemented. VCF automates the process of changing the password either to a randomly generated one or to one defined by the user. However, this process doesn’t work when the password has already expired.

With VCF 4.x, there is an option to rectify a password. You can use this when a password has been changed outside of the SDDC Manager to update the information stored by the SDDC Manager. This would have allowed me to simply change the expired password and then update the SDDC Manager. However, this was a VCF 3.x environment where that feature was not available. This required me to change the password and follow the password complexity and history rules. These rules prevented me from just changing the password to what the password was before, simply adding a character to the end, or using a simple word (like ‘password’) for the password. And because the password history was set to remember the last five passwords, the password would have to be changed at least five times before I could get the password back to what it was to make the SDDC Manger happy again.

As luck would have it, the lab environment I was working in had expired passwords for both PSCs, a couple of vCenter instances, and the NSX components. Additionally, there were three other lab environments that were in the same state.

I decided it was a good time to write a script to automate this for me.

There are several ways that this could be handled, but I decided to use expect as it makes some of the tasks of dealing with an interactive SSH session easier. The script is very basic in operation – There’s no config files and it’s sloppy. But it works…

While I was making the script so that it would handle resetting an expired password, I also added the ability to disable the password aging and to wipe out the password history. Wiping out the password history allowed me to immediately set the password back to what the SDDC Manager thought the password should be. An expired password still had to be set in order to allow the commands to be executed, but I use a temporary password to accomplish this.  Removing the password aging just prevents this from being an issue later.

Of course, you would *never* want to do this in a production environment. In my case though, I was just working in a self-contained lab environment where this wasn’t so much of an issue. Even if you do use it, be aware of two things:

• You will have to modify the script to suit your environment.
• There’s relatively no error handling.

That said, I’m providing it here in case it helps someone else!

Until next time!

(P.S., Special thanks to Kevin Tebear for coming up with the name for the script!)