Authentication Failure in vSphere 6.0

After recently deploying a new vSphere 6.0 environment, one of my colleagues was unable to log into vCenter.

vCenter Singed Sign-On login screen with "Authentication failure" error.
The SSO login screen simply returned “Authentication failure.” when attempting to log in.

Access to both the Web Client and C# client was failing. No one else in our organization was having this problem.

This environment was set up with the vCenter Server Appliance 6.0, and has an external Platform Services Controller cluster. We were using an Active Directory identity provider for logins.

After reviewing the SSO logs and doing some searching, I came across this VMware KB:

Unable to add Active Directory users or groups to vCenter Server Appliance or vRealize Automation permissions (2127213)

This KB sorta, kinda matched up with what I was seeing in the logs, and it applies to vCSA 6.0, and has some actual steps to try, so it seemed like a good lead. Essentially, the recommendations there are to search the logs for connection problems to specific domain controllers, and to make sure that both forward and reverse DNS resolution is functioning properly. This is good advice, and I confirmed that things were all good with DNS.

At this point, since there was nothing else to do, I decided to try a packet capture. As it happens, tcpdump is not installed by default on vCSA. You need to follow the instructions here to enable it (and also netcat):

Using tcpdump on vCenter Server Appliance (2084896)

Basically, run this from a privileged shell:

# /etc/vmware/gss-support/

Finally, after capturing packets while reproducing the error, and also during a successful login using another account, I did some analysis.

When a user logs in using an AD account, the Platform Services Controller (via Likewise, if you are on vCSA), performs the Kerberos dance with a domain controller. Kerberos is complicated and there are other resources that explain it much better than I ever could, but the gist is this:

  1. PSC sends an AS-REQ request to the DC over UDP
  2. DC responds with KRB5KDC_ERR_PREAUTH_REQUIRED over UDP
  3. PSC sends a new AS-REQ request to the DC, this time with a padata value (preauthorization data) over UDP
  4. DC responds with KRB5KDC_ERR-RESPONSE_TOO_BIG over UDP
  5. PSC sends yet another AS-REQ to the DC, this time over TCP
  6. DC responds with a AS-REP containing the TGT (ticket-granting-ticket), over TCP
  7. PSC sends a TGS-REQ to the DC to get a service ticket over TCP
  8. DC responds with the service ticket in a TGS-REP, which also includes Privilege Account Certificate data; this is a Microsoft-specific extension that basically includes details about the user, including their AD group membership

There are more steps after 8, but the are not relevant here.

The critical thing that I noticed was that, for the user having trouble, there was never a KRB5KDC_ERR_PREAUTH_REQUIRED challenge returned from the DC, the conversation never switched over to TCP, and the TGS-REP was missing the PAC data. Without this data, Likewise cannot determine the user’s group membership, and thus cannot decide whether the user is authorized.

So, what controls whether an account requires Kerberos preauthorization? It’s the DONT_REQUIRE_PREAUTH flag in the userAccountControl attribute, which you can read about here:

How to use the UserAccountControl flags to manipulate user account properties

We removed that flag from the user’s account, which for some reason was set for him but no one else. After that, problem solved.

Route 53 Dynamic DNS with VyOS

The Amazon Web Services team has a great writeup on Medium about how to roll your own dynamic DNS provider using Route 53. Actually, it uses a smattering of AWS offerings beyond just DNS to offload most of the complexity and just require a simple script on the client side.

In this post, we describe how to build your own dynamic DNS system with a small script and several AWS services. There are other systems that provide similar solutions; however, building a serverless system using nothing but AWS services and a few lines of code is simple, cost-effective, and an example of how to build your own serverless solutions on AWS.

Definitely read their writeup if you have even a remote interest in such things, because it’s very well done and explains the whole idea thoroughly.

To jump to the punchline, the client system must run a simple script that does three things:

  1. Gets the current IP address
  2. Builds a URI to update the IP using a shared secret
  3. Performs an HTTP GET on that URI

That’s all that the client has to do. Obviously, for this to be effective over time, it needs to run on a regular basis so that changes to the client’s IP are reflected in DNS in a timely fashion.

Once you’ve copied their update script onto your VyOS router (and ensured the file is executable), set up a scheduled task to run it periodically:

$ configure
# set system task-scheduler task AWS-DDNS executable arguments " SHAREDSECRETHERE"
# set system task-scheduler task AWS-DDNS executable path /path/to/your/script.bash
# set system task-scheduler task AWS-DDNS interval 5m
# commit
# save

Custom PowerShell Prompt: Display Connected vCenter Servers

Several people that I work with have custom bash prompts that they use to visually indicate which Git branch they are in, and the state of that branch. It occurred to me that this would be very useful in PowerCLI; I often avoid connecting to more than one vSphere endpoint at a time because I’m afraid of the consequences of a forgotten -Server argument.

With this inspiration, I decided to implement a PowerShell prompt that shows the hostnames of all connected PowerCLI endpoints. Here’s an example of the prompt I achieved:

prod-vc dr-vc nyc-esxi-099 PowerShell>

This prompt shows that I’m connected to the prod-vc and dr-vc vCenter servers, as well as an ESXi host, nyc-esxi-099. I also shortened the present working directory indication to be just the name of the lowest directory in the hierarchy, bash-style. This keeps the prompt short, leaving lots of room for way-too-long one-liner PowerShell commands. (I typically use C:\Users\pjorg\Documents\Scripts\PowerShell as my working directory, thus PowerShell is shown above.)

The key to achieving this is in the global:prompt function in PowerShell. This function is called every time the prompt is drawn, so overwriting the default with your own code allows you to modify what is printed. Here’s mine:

# Modify the prompt function to change the console prompt.
# Save the previous function, to allow restoring it back.
$originalPromptFunction = $function:prompt
function global:prompt{
# change prompt text
if(Test-Path variable:global:defaultviservers) {
$global:DefaultVIServers | %{ Write-Host $_.Name.Split('.')[0].Trim() -NoNewLine -foregroundcolor green;Write-Host ' ' -NoNewLine }
Write-Host (Get-Location).Path.Split("\")[-1] -NoNewLine
return "> "

Pretty simple, really. To break it apart:

# Modify the prompt function to change the console prompt.
# Save the previous function, to allow restoring it back.
$originalPromptFunction = $function:prompt

Some initial comments to explain what we’re doing, and save the preexisting prompt function into a variable for safekeeping.

# change prompt text
if(Test-Path variable:global:defaultviservers) {
$global:DefaultVIServers | %{ Write-Host $_.Name.Split('.')[0].Trim() -NoNewLine -foregroundcolor green;Write-Host ' ' -NoNewLine }

Check if the $global:DefaultVIServers variable is set (this contains all connected vSphere endpoints). If it is, then iterate through each member of the array and print the part of the hostname prior to the first dot in green, then add a space afterward. Don’t append a newline character to the end of the string (by default, Write-Host does).

Write-Host (Get-Location).Path.Split("\")[-1] -NoNewLine
return "> "

Finally, print the portion of the current working directory path that is after the last backslash to the screen. Again, no newline. Finish up with the greater-than symbol, for tradition.

You can run this ad-hoc to overwrite the default prompt every time you open a PowerCLI session, but a better way might be to drop it into C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-PowerCLIEnvironment_Custom.ps1, which the PowerCLI session setup process runs every time you start it.

Now, the list of vCenter Servers that you are connected to will always be clearly visible. Still, don’t forget that -Server argument…

PowerCLI One-Liner: Rename Default “datastore1” Datastores in Bulk

The ESXi installer, by default, creates a datastore on the local disk called “datastore1” on all new hosts. When you add a bunch of new hosts to vCenter, it will resolve the duplicate names by appending a sequence number in parenthesis, like so:

datastore1 (1)
datastore1 (2)
datastore1 (3)

That’s pretty ugly. In my environments, we rename these and use the hostname of the owning host to make the names unique. Our format is a prefix consisting of z-, the unqualified hostname, and then a suffix _boot. So, if a host is named, the datastore name would be:


The prefix ensures that these datastores sort to the bottom of the list, which is nice.

Problem is, if you are adding more than just one or two hosts, it can be time consuming to go rename each datastore individually. Enter PowerCLI!

Get-Datastore -Name datastore1* | %{ $n = 'z-' + (Get-VMHost -Id $_.ExtensionData.Host[0].Key[0]).Name.Split('.')[0] + '_boot';Set-Datastore -Datastore $_ -Name $n }

This could definitely be broken into multiple lines for clarity, but I’m a fan of one-liners. Here’s what’s going on there:

  1. Get an array of datastores with names starting with “datastore1”
  2. For each datastore in that array:
    • Set a variable $n that takes the part of the owning-host’s FQDN before the first dot and attaches the prefix and suffix to it, and
    • Rename the datastore with the Set-Datastore cmdlet

vRealize Orchestrator: Get the datastore containing a VM’s configuration files

I’ve recently had reason to make use of the built-in “Add disk” workflow in vRealize Orchestrator (formerly vCenter Orchestrator) as part of an overall VM provisioning workflow. One of the inputs that “Add disk” requires is a VC:Datastore object where the disk should be placed.

Typically (in my experience), you want new disks to be placed on the same datastore as the VM configuration files. However, there don’t appear to be any out-of-the-box solutions to this particular problem—you have to figure out which datastore is hosting a VM’s config files yourself.

Fortunately, the vSphere API makes this information available in the VirtualMachineFileInfo data object of every VM.

Let’s take a look inside this object with PowerCLI:

PowerCLI> $vm = Get-VM MyVM
PowerCLI> $vm.ExtensionData.Config.Files

VmPathName : [my_datastore] MyVM/MyVM.vmx
SnapshotDirectory : [my_datastore] MyVM/
SuspendDirectory : [my_datastore] MyVM/
LogDirectory : [my_datastore] MyVM/
FtMetadataDirectory :

We can see that the full path to the VMX file is stored in VmPathName. Particularly, the information we need (the name of the datastore) is inside the two square brackets. All that is necessary is to extract the string between those two brackets, and then search inventory for a datastore with that name.

From here, it’s relatively straightforward to create a vRO Action that will accept a VC:VirtualMachine as an input, and return a VC:Datastore.

// INPUT: VC:VirtualMachine in variable 'vm'

var vmxPath = vm.config.files.vmPathName;
var startOfDatastoreName = vmxPath.indexOf('[')+1;
var endOfDatastoreName = vmxPath.indexOf(']');
var vmxDatastore = vmxPath.substring(startOfDatastoreName,endOfDatastoreName);

var xpath = "xpath:name='" + vmxDatastore + "'";
var datastores = VcPlugin.getAllDatastores(null, xpath);

if(datastores.length === 0) {
	throw "Unable to find a datastore named '" + vmxDatastore + "'";
else if (datastores.length > 1) {
	throw "Found more than one datastore named '" + vmxDatastore + "'";

return datastores[0];

ESXi Host Serial Number with PowerCLI (v5.0 or later)

I was recently asked to pull serial numbers for each of the ESXi hosts in my environment. I initially assumed that this would be a property in the VMHost object in PowerCLI, and would thus be a trivial task.

Guess what? It isn’t. Continue reading “ESXi Host Serial Number with PowerCLI (v5.0 or later)”

FiOS “Three-Router” with VyOS and ESXi, Part 10: Port Forwarding

At this point the network should be mostly functional. Hosts on the internal home network ( should have Internet access, and the Verizon router should be also be accessing the FiOS ISP network via secondary router impersonation scheme.

Some FiOS features still won’t be working, though. These include remote web access to your DVR, on-screen caller ID, and possibly others. The reason these features still don’t work is that they depend on Verizon’s ability to initiate connections to the Verizon hardware router from the WAN. As it stands right now, all traffic that is not initiated from behind the primary router is dropped by the FROM-EXTERNAL firewall rule set.

The solution to this is to create rules allowing the correct ports to pass through the FROM-EXTERNAL firewall, and also to create NAT rules that will correctly map traffic arriving on those ports to the Verizon router. Continue reading “FiOS “Three-Router” with VyOS and ESXi, Part 10: Port Forwarding”

FiOS “Three-Router” with VyOS and ESXi, Part 9: Secondary Router Configuration

The secondary router requires much less configuration to get working, though there is a slight twist. Let’s briefly rediscuss what the role of the secondary router is.

In essence, the secondary router’s job is to sit between the Verizon hardware router and the primary router, allowing the VZ hardware to believe it is connected directly to the FiOS ISP network. It does this by assigning (via DHCP) the VZ router the same public IP that the FiOS ISP network has assigned the primary router for actual Internet connectivity, and then performing Network Address Translation to convert packets originating from the VZ router to correct IP segment for the home network. They can then be handled by the primary router just like any other internal host’s traffic. Continue reading “FiOS “Three-Router” with VyOS and ESXi, Part 9: Secondary Router Configuration”

FiOS “Three-Router” with VyOS and ESXi, Part 8: Network Address Translation

At this point, our primary router has connectivity to the Internet, and we have a functioning home network with DHCP and DNS. Now we need to implement Network Address Translation (NAT) so that all of the hosts on the Home Network can share Internet access through the primary router’s one public IP.

Sharing one public IP with many clients to is called source NAT. This is because the translation is occurring at the source of the traffic—our home network. The other type, destination NAT, is better known in the consumer space as port forwarding. We’ll discuss that later.

The behavior we are looking for is simple. We want traffic that is bound for the Internet to be received by the router, have its source address changed to that of the router itself (instead of the private address), and then forwarded on to its destination. When a response is received, the opposite should happen so that the response makes its way to the host on the net that originally requested it.

We need only one source NAT rule for this. Continue reading “FiOS “Three-Router” with VyOS and ESXi, Part 8: Network Address Translation”