Looking for a method to rotate the passwords used for all of the VMware Cloud Foundation (VCF) service accounts? You came to the right place!

In this article, I’ll talk a bit about the service accounts and provide you a script so that you can easily rotate all the passwords at will!

For those who don’t know, VMware Cloud Foundation has this concept of ‘service accounts’. These are special accounts that are used to communicate between the various components within VCF. For example, a service account may communicate between the SDDC Manager and a vCenter Server instance.

Service accounts are automatically created during events like bring-up, host commissioning, and workload domain creation.

These accounts have a limited set of privileges assigned to them and are treated differently than normal user accounts. One of these differences is that the passwords used for these accounts are randomly generated. Unlike typical user accounts, an administrator cannot go in and set a specific password for these accounts. Administrators can, however, go in and rotate the service account passwords.

Using the SDDC Manager UI, you can select a component such as the vCenter as shown below. From here, you can filter on ‘svc’ for the username to see all the service accounts for that component. You can even select the service accounts and rotate the passwords right from the UI!

What if you want to change the passwords for all of the service accounts?

If you want to do this, using the UI might not be the best solution. This is because you would have to select each component, then each service account, then rotate. You would then repeat the process until all the service accounts for all the components have been rotated.

A better solution would be to leverage the API.

Using the API will allow you to perform various actions, such as listing out all the service accounts or performing a rotation of the password. We can even use the APIs to help us automate the tasks.

But before you go further, it’s important to note that you may not need to continue at all. I say this because most of the service accounts have a mechanism in place to automatically rotate a given service account every 30 days, by default. If that timeframe doesn’t comply with your security policies, then you can use the API to change this to a more suitable value for you.

To do this on the command line, you would use a command like this one:

$ curl -k -X PATCH -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://172.19.11.59/v1/credentials -d '{
"operationType" : "UPDATE_AUTO_ROTATE_POLICY",
"elements" : [ {
  "resourceName" : "sfo-w01-vc01.sfo.rainpole.io",
  "resourceType" : "VCENTER",
  "credentials" : [ {
    "credentialType" : "SSO",
    "username" : "svc-sfo-vcf01-sfo-w01-vc01@vsphere.local"
  } ]
} ],
"autoRotatePolicy" : {
  "frequencyInDays" : 20,
  "enableAutoRotatePolicy" : true
  }
}'

In this example, I would be specifying that the auto rotation of the password should occur every 20 days, instead of the default of 30 days.

To verify this, you can use a API command to list out the information for all the service accounts like this:

$ curl -k -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://172.19.11.59/v1/credentials?accountType=SERVICE | jq

 This will return a list of all the service accounts. Here is just a small snippet:

{
      "id": "44147672-616d-4feb-867e-2242c0696e80",
      "credentialType": "SSH",
      "accountType": "SERVICE",
      "username": "svc-vcf-sfo01-w01-esx04",
      "creationTimestamp": "2023-09-26T03:02:44.096Z",
      "modificationTimestamp": "2023-09-26T03:02:44.096Z",
      "resource": {
        "resourceId": "db18c5b0-81e1-4b3a-a98b-aed587e5851f",
        "resourceName": "sfo01-w01-esx04.sfo.rainpole.io",
        "resourceIp": "172.19.31.104",
        "resourceType": "ESXI",
        "domainName": "sfo-w01"
      }
    },
    {
      "id": "07d1c900-f2fa-4b4a-a3f9-132e0b95c096",
      "credentialType": "SSO",
      "accountType": "SERVICE",
      "username": "svc-sfo-m01-nsx01-sfo-m01-vc01@vsphere.local",
      "creationTimestamp": "2023-09-25T22:26:28.720Z",
      "modificationTimestamp": "2023-09-25T22:26:28.720Z",
      "resource": {
        "resourceId": "7fce531f-3b81-4cac-a3fa-5ece92c09c54",
        "resourceName": "sfo-m01-vc01.sfo.rainpole.io",
        "resourceIp": "172.19.11.70",
        "resourceType": "VCENTER",
        "domainName": "sfo-m01"
      },
      "autoRotatePolicy": {
        "frequencyInDays": 20,
        "nextSchedule": "2024-07-21T00:00:00.090Z"
      }
    },

Looking under the autoRotatePolicy for the vCenter service account, you can see the frequencyInDays key showing that that particular service account will get automatically rotated every 20 days.

You might have noticed that the ESXi service account in the example above does not have a auto rotation policy defined. If you attempt to enable this, this is what you will see:

$ curl -k -X PATCH -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://172.19.11.59/v1/credentials -d '{
 "operationType" : "UPDATE_AUTO_ROTATE_POLICY",
 "elements" : [ {
   "resourceName" : "sfo01-w01-esx04.sfo.rainpole.io",
   "resourceType" : "ESXI",
   "credentials" : [ {
     "credentialType" : "SSH",
     "username" : "svc-vcf-sfo01-w01-esx04"
   } ]
 } ],
 "autoRotatePolicy" : {
   "frequencyInDays" : 30,
   "enableAutoRotatePolicy" : true
   }
 }'
{"errorCode":"PASSWORD_MANAGER_AUTO_ROTATE_INPUT_SPEC_RESOURCE_TYPE_NOT_SUPPORTED","arguments":["ESXI"],"message":"Resource type: ESXI not supported for auto rotate.","referenceToken":"FU50SR"}

As you can see, the error message states that the ESXi Resource types are not supported by the auto rotate feature. This is because historically, the passwords for the ESXI service accounts were set to 99999, or to never expire. Thus, there was no need to automatically rotate the password. You can expect this behavior to change in a future version of VCF, but for now you will have to rotate the passwords for these resource types manually.

To rotate a service account using the API, you would use a command like the following:

$ curl -k -X PATCH -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://172.19.11.59/v1/credentials -d '{
"operationType" : "ROTATE",
"elements" : [ {
  "resourceName" : "sfo-w01-vc01.sfo.rainpole.io",
  "resourceType" : "VCENTER",
  "credentials" : [ {
    "credentialType" : "SSO",
    "username" : "svc-sfo-vcf01-sfo-w01-vc01@vsphere.local"
  } ]
} ]
}'

The output from this command will be something like:

{"id":"a601501c-6b1d-4caa-8ee3-067b7640fe4f","status":"IN_PROGRESS"}

This gives you the task ID for the rotation task. You can then use this ID to get the status of the task:

$ curl -k -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://172.16.11.59/v1/credentials/tasks/a601501c-6b1d-4caa-8ee3-067b7640fe4f | jq

You may be tempted to use the command I showed earlier to list all the service accounts to look at the  modificationTimestamp field to see if this updates. The problem with this is that this field will only get updated for service accounts that the SDDC Manager actually uses. There are some accounts that exist that the SDDC Manager doesn’t use. As a good security practice, it doesn’t store the passwords for service accounts it doesn’t use. As a result, the modificationTimestamp field only get updated for the service accounts the SDDC Manager stores the password for – Even if the password rotation task completed successfully.

Of course, if we can do all this with the API from the command line, why not make a script to do it for us?

The following is a basic script that does just that. It will go through all of the service accounts and rotate the password. Of course, you’ll need to change the user defined variables to match your environment first!

#!/bin/bash

#==================================================
# Title         : rotate_all_svc_accounts.sh
# Desciption    : Shell script to manually rotate
#                 all of the
#                 VMware Cloud Foundation
#                 service account passwords.
# Author        : Tom Stephens
# Date          : 19 Jul 24
# Notes         : Tested w/ VCF 5.x
#==================================================

#==================================================
# User Defined Variables
#
# Please define as applicable to your environment
#--------------------------------------------------
USERNAME=administrator@vsphere.local
PASSWORD="MyPassword123$"
SDDC_MGR_IP=172.16.11.59
#==================================================

# Define colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m'

# Get VCF API Token
TOKEN=`curl -s -X POST -H "Content-Type: application/json" -d '{"username":"'$USERNAME'","password":"'$PASSWORD'"}' --insecure localhost/v1/tokens | jq -r '.accessToken'`

# Get list of all Service Accounts
SVC_ACCT_JSON=`curl -s -k -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://$SDDC_MGR_IP/v1/credentials?accountType=SERVICE | jq`

# Find the number of service accounts
COUNT=`echo $SVC_ACCT_JSON | jq -r '.elements[] .id' | wc -l`

# Iterate through all the service accounts
# Rotate password and wait until status until going to next account
for (( i=0 ; i<$COUNT ; i++ ));
do
  echo ""
  ITEM_COUNT=$((i+1))
  echo -e "${RED}New Resource${NC} - $ITEM_COUNT of $COUNT"
  RES_USER_NAME=`echo $SVC_ACCT_JSON | jq -r ".elements[$i] .username"`
  CRED_TYPE=`echo $SVC_ACCT_JSON | jq -r ".elements[$i] .credentialType"`
  RESOURCE_NAME=`echo $SVC_ACCT_JSON | jq -r ".elements[$i] .resource .resourceName"`
  RESOURCE_TYPE=`echo $SVC_ACCT_JSON | jq -r ".elements[$i] .resource .resourceType"`
  echo "     resourceName: $RESOURCE_NAME resourceType: $RESOURCE_TYPE username: $RES_USER_NAME credentialType: $CRED_TYPE"
  echo "     Attempting to rotate password..."

  # Rotate password
  ROTATE_OUT=`curl -s -k -X PATCH -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://$SDDC_MGR_IP/v1/credentials -d '{"operationType" : "ROTATE", "elements" : [ {"resourceName" : "'$RESOURCE_NAME'","resourceType" : "'$RESOURCE_TYPE'","credentials" : [ {"credentialType" : "'$CRED_TYPE'","username" : "'$RES_USER_NAME'"} ]} ]}'`

  TASK_ID=`echo $ROTATE_OUT | jq -r ".id"`
  echo "          Task $TASK_ID initiated."

  # Check Task Status
  echo "          Checking task status of task..."
  TASK_STATUS="UNKNOWN"
  echo "               Current task status is: $TASK_STATUS"

  # Wait until the task completes before moving to the next account
  while [ $TASK_STATUS = "PENDING" ] || [ $TASK_STATUS = "UNKNOWN" ] || [ $TASK_STATUS = "IN_PROGRESS" ];
  do
    TASK_STATUS=`curl -s -k -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://$SDDC_MGR_IP/v1/credentials/tasks/$TASK_ID  | jq -r ".status"`
  done

  # Display status of completed task
  if [ $TASK_STATUS = "SUCCESSFUL" ]
  then
    echo -e "                Task Status: ${GREEN}$TASK_STATUS${NC}"
  else
    echo -e "                Task Status: ${RED}$TASK_STATUS${NC}"
  fi

done

This is what the script will look like when you are executing it. Keep in mind that some of the tasks may take a bit of time, so just be patient and wait.

Hope this helps you understand a bit more about the VCF service accounts and give you some options for managing them!