Jekyll2018-07-03T17:20:09-04:00https://pjorg.com/Peter D. JorgensenPeter D. Jorgensenpdjorgensen@gmail.comAuthentication Failure in vSphere 6.02016-12-06T13:06:58-05:002016-12-06T13:06:58-05:00https://pjorg.com/authentication-failure-in-vsphere-6-0<p>After recently deploying a new vSphere 6.0 environment, one of my colleagues was unable to log into vCenter.<figure id="attachment_169" style="max-width: 752px" class="wp-caption aligncenter"></figure></p>
<figure>
<img src="/img/AuthenticationFailure.png" alt="vCenter Singed Sign-On login screen with "Authentication failure" error." srcset="/img/AuthenticationFailure-300x75.png 300w, /img/AuthenticationFailure-768x191.png 768w, /img/AuthenticationFailure.png 781w" sizes="(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px" />
<figcaption>The SSO login screen simply returned “Authentication failure.” when attempting to log in.</figcaption>
</figure>
<p>Access to both the Web Client and C# client was failing. No one else in our organization was having this problem.</p>
<p>This environment was set up with the vCenter Server Appliance 6.0, and has an external Platform Services Controller cluster. We were using an Active Directory identity provider for logins.</p>
<p>After reviewing the SSO logs and doing some searching, I came across this VMware KB:</p>
<p><a href="https://kb.vmware.com/kb/2127213">Unable to add Active Directory users or groups to vCenter Server Appliance or vRealize Automation permissions (2127213)</a></p>
<p>This KB sorta, kinda matched up with what I was seeing in the logs, and it applies to vCSA 6.0, and has some actual steps to try, so it seemed like a good lead. Essentially, the recommendations there are to search the logs for connection problems to specific domain controllers, and to make sure that both forward and reverse DNS resolution is functioning properly. This is good advice, and I confirmed that things were all good with DNS.</p>
<p>At this point, since there was nothing else to do, I decided to try a packet capture. As it happens, <code class="highlighter-rouge">tcpdump</code> is not installed by default on vCSA. You need to follow the instructions here to enable it (and also <code class="highlighter-rouge">netcat</code>):</p>
<p><a href="https://kb.vmware.com/kb/2084896">Using tcpdump on vCenter Server Appliance (2084896)</a></p>
<p>Basically, run this from a privileged shell:</p>
<pre># /etc/vmware/gss-support/install.sh</pre>
<p>Finally, after capturing packets while reproducing the error, and also during a successful login using another account, I did some analysis.</p>
<p>When a user logs in using an AD account, the Platform Services Controller (via Likewise, if you are on vCSA), performs the Kerberos dance with a domain controller. Kerberos is complicated and there are <a href="http://www.roguelynn.com/words/explain-like-im-5-kerberos/">other resources that explain it much better than I ever could</a>, but the gist is this:</p>
<ol>
<li>PSC sends an <code class="highlighter-rouge">AS-REQ</code> request to the DC over UDP</li>
<li>DC responds with <code class="highlighter-rouge">KRB5KDC_ERR_PREAUTH_REQUIRED</code> over UDP</li>
<li>PSC sends a new <code class="highlighter-rouge">AS-REQ</code> request to the DC, this time with a padata value (preauthorization data) over UDP</li>
<li>DC responds with <code class="highlighter-rouge">KRB5KDC_ERR-RESPONSE_TOO_BIG</code> over UDP</li>
<li>PSC sends yet another <code class="highlighter-rouge">AS-REQ</code> to the DC, this time over TCP</li>
<li>DC responds with a <code class="highlighter-rouge">AS-REP</code> containing the TGT (ticket-granting-ticket), over TCP</li>
<li>PSC sends a <code class="highlighter-rouge">TGS-REQ</code> to the DC to get a service ticket over TCP</li>
<li>DC responds with the service ticket in a <code class="highlighter-rouge">TGS-REP</code>, which also includes Privilege Account Certificate data; this is a Microsoft-specific extension that basically includes details about the user, including their AD group membership</li>
</ol>
<p>There are more steps after 8, but the are not relevant here.</p>
<p>The critical thing that I noticed was that, for the user having trouble, there was never a <code class="highlighter-rouge">KRB5KDC_ERR_PREAUTH_REQUIRED</code> challenge returned from the DC, the conversation never switched over to TCP, and the <code class="highlighter-rouge">TGS-REP</code> was missing the PAC data. Without this data, Likewise cannot determine the user’s group membership, and thus cannot decide whether the user is authorized.</p>
<p>So, what controls whether an account requires Kerberos preauthorization? It’s the <code class="highlighter-rouge">DONT_REQUIRE_PREAUTH</code> flag in the <code class="highlighter-rouge">userAccountControl</code> attribute, which you can read about here:</p>
<p><a href="https://support.microsoft.com/en-us/kb/305144">How to use the UserAccountControl flags to manipulate user account properties</a></p>
<p>We removed that flag from the user’s account, which for some reason was set for him but no one else. After that, problem solved.</p>pjorgAfter recently deploying a new vSphere 6.0 environment, one of my colleagues was unable to log into vCenter.Route 53 Dynamic DNS with VyOS2016-02-24T13:39:15-05:002016-02-24T13:39:15-05:00https://pjorg.com/route-53-dynamic-dns-with-vyos<p>The Amazon Web Services team has <a href="https://medium.com/aws-activate-startup-blog/building-a-serverless-dynamic-dns-system-with-aws-a32256f0a1d8#.lfoz43hlt">a great writeup on Medium</a> about how to roll your own dynamic DNS provider using <a href="https://aws.amazon.com/route53/">Route 53</a>. Actually, it uses a smattering of AWS offerings beyond just DNS to offload most of the complexity and just require a simple script on the client side.</p>
<blockquote>
<p>In this post, we describe how to build your own dynamic DNS system with a small script and several AWS services. There are other systems that provide similar solutions; however, building a serverless system using nothing but AWS services and a few lines of code is simple, cost-effective, and an example of how to build your own serverless solutions on AWS.</p>
</blockquote>
<p>Definitely read their writeup if you have even a remote interest in such things, because it’s very well done and explains the whole idea thoroughly.</p>
<p>To jump to the punchline, the client system must run a simple script that does three things:</p>
<ol>
<li>Gets the current IP address</li>
<li>Builds a URI to update the IP using a shared secret</li>
<li>Performs an HTTP GET on that URI</li>
</ol>
<p>That’s all that the client has to do. Obviously, for this to be effective over time, it needs to run on a regular basis so that changes to the client’s IP are reflected in DNS in a timely fashion.</p>
<p>Once you’ve copied <a href="https://github.com/awslabs/route53-dynamic-dns-with-lambda/blob/master/dynamic_dns_lambda_client.sh">their update script</a> onto your VyOS router (and ensured the file is executable), set up a scheduled task to run it periodically:</p>
<pre>$ configure
# set system task-scheduler task AWS-DDNS executable arguments "your-hostname.dyn.your-domain.com. SHAREDSECRETHERE your-api-endpoint.amazonaws.com/prod"
# set system task-scheduler task AWS-DDNS executable path /path/to/your/script.bash
# set system task-scheduler task AWS-DDNS interval 5m
# commit
# save</pre>pjorgThe Amazon Web Services team has a great writeup on Medium about how to roll your own dynamic DNS provider using Route 53. Actually, it uses a smattering of AWS offerings beyond just DNS to offload most of the complexity and just require a simple script on the client side.Custom PowerShell Prompt: Display Connected vCenter Servers2015-10-24T16:26:25-04:002015-10-24T16:26:25-04:00https://pjorg.com/custom-powershell-prompt-connected-vcenter-servers<p>Several people that I work with have custom bash prompts that they use to visually indicate which Git branch they are in, and the state of that branch. It occurred to me that this would be very useful in PowerCLI; I often avoid connecting to more than one vSphere endpoint at a time because I’m afraid of the consequences of a forgotten <code class="highlighter-rouge">-Server</code> argument.</p>
<p>With this inspiration, I decided to implement a PowerShell prompt that shows the hostnames of all connected PowerCLI endpoints. Here’s an example of the prompt I achieved:</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell">prod-vc dr-vc nyc-esxi-099 PowerShell></code></pre></figure>
<p>This prompt shows that I’m connected to the <code class="highlighter-rouge">prod-vc</code> and <code class="highlighter-rouge">dr-vc</code> vCenter servers, as well as an ESXi host, <code class="highlighter-rouge">nyc-esxi-099</code>. I also shortened the present working directory indication to be just the name of the lowest directory in the hierarchy, bash-style. This keeps the prompt short, leaving lots of room for way-too-long one-liner PowerShell commands. (I typically use <code class="highlighter-rouge">C:\Users\pjorg\Documents\Scripts\PowerShell</code> as my working directory, thus PowerShell is shown above.)</p>
<p>The key to achieving this is in the global:prompt function in PowerShell. This function is called every time the prompt is drawn, so overwriting the default with your own code allows you to modify what is printed. Here’s mine:</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="c1"># Modify the prompt function to change the console prompt.</span>
<span class="c1"># Save the previous function, to allow restoring it back.</span>
<span class="nv">$originalPromptFunction</span> <span class="o">=</span> <span class="nv">$function</span>:prompt
<span class="k">function </span>global:prompt<span class="o">{</span>
<span class="c1"># change prompt text</span>
<span class="k">if</span><span class="o">(</span><span class="nb">Test-Path </span>variable:global:defaultviservers<span class="o">)</span> <span class="o">{</span>
<span class="nv">$global</span>:DefaultVIServers | %<span class="o">{</span> <span class="nb">Write-Host</span> <span class="nv">$_</span>.Name.Split<span class="o">(</span><span class="s1">'.'</span><span class="o">)[</span>0].Trim<span class="o">()</span> -NoNewLine -foregroundcolor green;Write-Host <span class="s1">' '</span> -NoNewLine <span class="o">}</span>
<span class="o">}</span>
<span class="nb">Write-Host</span> <span class="o">(</span><span class="nb">Get-Location</span><span class="o">)</span>.Path.Split<span class="o">(</span><span class="s2">"\"</span><span class="o">)[</span>-1] -NoNewLine
<span class="k">return</span> <span class="s2">"> "</span>
<span class="o">}</span></code></pre></figure>
<p>Pretty simple, really. To break it apart:</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="c1"># Modify the prompt function to change the console prompt.</span>
<span class="c1"># Save the previous function, to allow restoring it back.</span>
<span class="nv">$originalPromptFunction</span> <span class="o">=</span> <span class="nv">$function</span>:prompt</code></pre></figure>
<p>Some initial comments to explain what we’re doing, and save the preexisting prompt function into a variable for safekeeping.</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="c1"># change prompt text</span>
<span class="k">if</span><span class="o">(</span><span class="nb">Test-Path </span>variable:global:defaultviservers<span class="o">)</span> <span class="o">{</span>
<span class="nv">$global</span>:DefaultVIServers | %<span class="o">{</span> <span class="nb">Write-Host</span> <span class="nv">$_</span>.Name.Split<span class="o">(</span><span class="s1">'.'</span><span class="o">)[</span>0].Trim<span class="o">()</span> -NoNewLine -foregroundcolor green;Write-Host <span class="s1">' '</span> -NoNewLine <span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>Check if the <code class="highlighter-rouge">$global:DefaultVIServers</code> variable is set (this contains all connected vSphere endpoints). If it is, then iterate through each member of the array and print the part of the hostname prior to the first dot in green, then add a space afterward. Don’t append a newline character to the end of the string (by default, <code class="highlighter-rouge">Write-Host</code> does).</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="nb">Write-Host</span> <span class="o">(</span><span class="nb">Get-Location</span><span class="o">)</span>.Path.Split<span class="o">(</span><span class="s2">"\"</span><span class="o">)[</span>-1] -NoNewLine
<span class="k">return</span> <span class="s2">"> "</span>
<span class="o">}</span></code></pre></figure>
<p>Finally, print the portion of the current working directory path that is after the last backslash to the screen. Again, no newline. Finish up with the greater-than symbol, for tradition.</p>
<p>You can run this ad-hoc to overwrite the default prompt every time you open a PowerCLI session, but a better way might be to drop it into <code class="highlighter-rouge">C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-PowerCLIEnvironment_Custom.ps1</code>, which the PowerCLI session setup process runs every time you start it.</p>
<p>Now, the list of vCenter Servers that you are connected to will always be clearly visible. Still, don’t forget that <code class="highlighter-rouge">-Server</code> argument…</p>pjorgSeveral people that I work with have custom bash prompts that they use to visually indicate which Git branch they are in, and the state of that branch. It occurred to me that this would be very useful in PowerCLI; I often avoid connecting to more than one vSphere endpoint at a time because I’m afraid of the consequences of a forgotten -Server argument.PowerCLI One-Liner: Rename Default “datastore1” Datastores in Bulk2015-10-08T18:30:46-04:002015-10-08T18:30:46-04:00https://pjorg.com/rename-default-datastores-in-bulk<p>The ESXi installer, by default, creates a datastore on the local disk called “datastore1” on all new hosts. When you add a bunch of new hosts to vCenter, it will resolve the duplicate names by appending a sequence number in parenthesis, like so:</p>
<pre>datastore1
datastore1 (1)
datastore1 (2)
datastore1 (3)</pre>
<p>That’s pretty ugly. In my environments, we rename these and use the hostname of the owning host to make the names unique. Our format is a prefix consisting of <code class="highlighter-rouge">z-</code>, the unqualified hostname, and then a suffix <code class="highlighter-rouge">_boot</code>. So, if a host is named <code class="highlighter-rouge">ny-esxi-001.mycompany.com</code>, the datastore name would be:</p>
<pre>z-ny-esxi-001_boot</pre>
<p>The prefix ensures that these datastores sort to the bottom of the list, which is nice.</p>
<p>Problem is, if you are adding more than just one or two hosts, it can be time consuming to go rename each datastore individually. Enter PowerCLI!</p>
<pre>Get-Datastore -Name datastore1* | %{ $n = 'z-' + (Get-VMHost -Id $_.ExtensionData.Host[0].Key[0]).Name.Split('.')[0] + '_boot';Set-Datastore -Datastore $_ -Name $n }</pre>
<p>This could definitely be broken into multiple lines for clarity, but I’m a fan of one-liners. Here’s what’s going on there:</p>
<ol>
<li>Get an array of datastores with names starting with “datastore1”</li>
<li>For each datastore in that array:
<ul>
<li>Set a variable <code class="highlighter-rouge">$n</code> that takes the part of the owning-host’s FQDN before the first dot and attaches the prefix and suffix to it, and</li>
<li>Rename the datastore with the <code class="highlighter-rouge">Set-Datastore</code> cmdlet</li>
</ul>
</li>
</ol>pjorgThe ESXi installer, by default, creates a datastore on the local disk called “datastore1” on all new hosts. When you add a bunch of new hosts to vCenter, it will resolve the duplicate names by appending a sequence number in parenthesis, like so:vRealize Orchestrator: Get the datastore containing a VM’s configuration files2015-09-12T20:40:39-04:002015-09-12T20:40:39-04:00https://pjorg.com/vro-get-datastore-containing-vmx-file<p>I’ve recently had reason to make use of the built-in “Add disk” workflow in vRealize Orchestrator (formerly vCenter Orchestrator) as part of an overall VM provisioning workflow. One of the inputs that “Add disk” requires is a <code class="highlighter-rouge">VC:Datastore</code> object where the disk should be placed.</p>
<p>Typically (in my experience), you want new disks to be placed on the same datastore as the VM configuration files. However, there don’t appear to be any out-of-the-box solutions to this particular problem—you have to figure out which datastore is hosting a VM’s config files yourself.</p>
<p>Fortunately, the vSphere API makes this information available in the <code class="highlighter-rouge">VirtualMachineFileInfo</code> data object of every VM.</p>
<p>Let’s take a look inside this object with PowerCLI:</p>
<pre>PowerCLI> $vm = Get-VM MyVM
PowerCLI> $vm.ExtensionData.Config.Files
VmPathName : [my_datastore] MyVM/MyVM.vmx
SnapshotDirectory : [my_datastore] MyVM/
SuspendDirectory : [my_datastore] MyVM/
LogDirectory : [my_datastore] MyVM/
FtMetadataDirectory :</pre>
<p>We can see that the full path to the VMX file is stored in <code class="highlighter-rouge">VmPathName</code>. Particularly, the information we need (the name of the datastore) is inside the two square brackets. All that is necessary is to extract the string between those two brackets, and then search inventory for a datastore with that name.</p>
<p>From here, it’s relatively straightforward to create a vRO Action that will accept a <code class="highlighter-rouge">VC:VirtualMachine</code> as an input, and return a <code class="highlighter-rouge">VC:Datastore</code>.</p>
<pre>// INPUT: VC:VirtualMachine in variable 'vm'
var vmxPath = vm.config.files.vmPathName;
var startOfDatastoreName = vmxPath.indexOf('[')+1;
var endOfDatastoreName = vmxPath.indexOf(']');
var vmxDatastore = vmxPath.substring(startOfDatastoreName,endOfDatastoreName);
var xpath = "xpath:name='" + vmxDatastore + "'";
var datastores = VcPlugin.getAllDatastores(null, xpath);
if(datastores.length === 0) {
throw "Unable to find a datastore named '" + vmxDatastore + "'";
}
else if (datastores.length > 1) {
throw "Found more than one datastore named '" + vmxDatastore + "'";
}
return datastores[0];
</pre>pjorgI’ve recently had reason to make use of the built-in “Add disk” workflow in vRealize Orchestrator (formerly vCenter Orchestrator) as part of an overall VM provisioning workflow. One of the inputs that “Add disk” requires is a VC:Datastore object where the disk should be placed.ESXi Host Serial Number with PowerCLI (v5.0 or later)2015-01-30T16:55:25-05:002015-01-30T16:55:25-05:00https://pjorg.com/esxi-serial-number-w-powercli-v5<p>I was recently asked to pull serial numbers for each of the ESXi hosts in my environment. I initially assumed that this would be a property in the VMHost object in PowerCLI, and would thus be a trivial task.</p>
<p>Guess what? It isn’t.<!--more--></p>
<pre>PowerCLI> Get-VMHost | Get-Member -MemberType Property | select Name
Name
----
Parent
NumCpu
PowerState
ParentId
Model
MemoryUsageMB
NetworkInfo
Name
ProcessorType
VMSwapfileDatastore
Version
VMSwapfilePolicy
VMSwapfileDatastoreId
StorageInfo
State
Uid
TimeZone
MemoryUsageGB
CustomFields
CpuUsageMhz
ExtensionData
DiagnosticPartition
Build
ApiVersion
CpuTotalMhz
ConnectionState
FirewallDefaultPolicy
MaxEVCMode
Manufacturer
MemoryTotalMB
MemoryTotalGB
Id
HyperthreadingActive
LicenseKey
IsStandalone</pre>
<p>Hardware serial number is one of those “I could have sworn I saw it here once” properties that isn’t actually available.</p>
<p>Some Google searching yielded a few different approaches to solve for this question. From what I found, they seem to either pull the serial number from a vSphere View of each host, or from some out-of-band management system (such as HP iLO). Unfortunately, for reasons that are not important, neither of these methods worked for me.</p>
<p>It turns out, though, that there is a way to get the serial number using esxcli:</p>
<p><code class="highlighter-rouge">esxcli hardware platform</code></p>
<p>Since PowerCLI 5.0, you can make calls to esxcli using the <code class="highlighter-rouge">Get-EsxCli</code> cmdlet. If you combine this with the very powerful <code class="highlighter-rouge">New-VIProperty</code> cmdlet, you can actually add the serial number to the VMHost object, and thus to the output of <code class="highlighter-rouge">Get-VMHost</code>.</p>
<pre>New-VIProperty -ObjectType VMHost -Name SerialNumber -Value `
{ (Get-EsxCli -VMHost $Args[0]).hardware.platform.get().SerialNumber }
Get-VMHost | Select Name,SerialNumber</pre>
<p>Not exactly a high-performance solution, but as long as your hosts are all at ESXi 5.0 or later, it works!</p>
<p>Special thanks to Luc Dekens for his valuable <a href="http://www.lucd.info/2010/07/13/powercli-4-1-brings-the-new-viproperty-cmdlet/"><code class="highlighter-rouge">New-VIProperty</code> insights</a>.</p>pjorgI was recently asked to pull serial numbers for each of the ESXi hosts in my environment. I initially assumed that this would be a property in the VMHost object in PowerCLI, and would thus be a trivial task.FiOS “Three-Router” with VyOS and ESXi, Part 11: Wrap-Up2015-01-15T07:10:52-05:002015-01-15T07:10:52-05:00https://pjorg.com/fios-3-router-w-virtualized-vyos-11-wrap-up<p>This configuration, as described in the prior 10 parts, has been serving me well for at least the last two years, so it is a proven design. There are, however, some things to keep in mind.<!--more--></p>
<p><strong>This is not supported by Verizon.</strong> Or me, for that matter. If you ever have an issue that requires you to call Verizon for support, you may have to revert to a configuration that they support for them to assist you. (Basically, this means plugging your Ethernet drop directly into the WAN port on the back of the VZ router.)</p>
<p><strong>Be cognizant of changes to your WAN IP address.</strong> If the WAN IP address changes for any reason, you will need to go into your secondary router and reconfigure the various items that refer to this IP directly to use the newly assigned IP. Even in such a situation, the primary router should pick up a new DHCP lease and your home network clients, at least, should work fine.[footnote]</p>
<p>Eventually, the DHCP lease for the WAN address will expire and the primary router will try to get a new one. When that happens, if the DHCP server issues a new address then the router will start using it and network access will resume.</p>
<p>I have had Verizon change my DHCP assignment once (due to a change in the IP address block assigned to my area) and I had to manually reacquire a DHCP lease at the primary router because the old lease hadn’t expired, but the address space had changed nonetheless. I speculate that Verizon probably has a way to force their own hardware routers to release/renew leases on demand, which they would naturally do right before making a network change like that.[/footnote]</p>
<p>As a practical matter, WAN IP changes are unlikely to happen very often unless you routinely have protracted periods of time when your network gear is powered off and/or not connected to the FiOS ISP network. It should be possible to configure some kind of automated process that will fix all these bits for you across the network when a WAN IP change is detected, but I haven’t managed to put that part together yet.</p>
<p>I sincerely hope this is helpful to someone!</p>pjorgThis configuration, as described in the prior 10 parts, has been serving me well for at least the last two years, so it is a proven design. There are, however, some things to keep in mind.FiOS “Three-Router” with VyOS and ESXi, Part 10: Port Forwarding2015-01-15T07:09:44-05:002015-01-15T07:09:44-05:00https://pjorg.com/fios-3-router-w-virtualized-vyos-10-port-forwarding<p>At this point the network should be mostly functional. Hosts on the internal home network (<samp>10.0.0.0/24</samp>) should have Internet access, and the Verizon router should be also be accessing the FiOS ISP network via secondary router impersonation scheme.</p>
<p>Some FiOS features still won’t be working, though. These include remote web access to your DVR, on-screen caller ID, and possibly others. The reason these features still don’t work is that they depend on Verizon’s ability to initiate connections to the Verizon hardware router from the WAN. As it stands right now, all traffic that is not initiated from behind the primary router is dropped by the <samp>FROM-EXTERNAL</samp> firewall rule set.</p>
<p>The solution to this is to create rules allowing the correct ports to pass through the <samp>FROM-EXTERNAL</samp> firewall, and also to create NAT rules that will correctly map traffic arriving on those ports to the Verizon router.<!--more--></p>
<p>The required ports are:</p>
<ul>
<li>TCP 4567</li>
<li>UDP 63145 (incrementing 1 for each STB you have)</li>
<li>TCP 35000 (incrementing 1 for each STB you have)</li>
<li>UDP 3500</li>
</ul>
<p>Thus, if you have 3 set-top boxes in your home, you need to forward UDP 63145, UDP 63146, UDP 63147, TCP 35000, TCP 35001 and TCP 35002.[footnote]These ports and protocols are correct according to dslreports.com, but to avoid issues I just forwarded all of these ports for both TCP and UDP to the VZ router. This is arguably a weak security decision, but the VZ router would normally be facing the Internet anyway, so it’s not something that causes me to lose sleep.[/footnote]</p>
<p>The destination of these forwards should be the proxy address that we set up for the Verizon router on the internal network (<samp>10.0.0.3</samp>). Recall that, via 1:1 NAT, traffic arriving on all ports on that proxy address on the secondary router is forwarded to the same port on the VZ router.</p>
<p>To implement these rules, run the following commands on the primary router (this assumes 5 STBs, which is probably sufficient for most households):</p>
<pre>set nat destination rule 10 destination port 3500,35000-35004,63145-63149
set nat destination rule 10 inbound-interface eth0
set nat destination rule 10 protocol udp
set nat destination rule 10 translation 10.0.0.3
set nat destination rule 12 destination port 4567
set nat destination rule 12 inbound-interface eth0
set nat destination rule 12 protocol tcp
set nat destination rule 12 translation 10.0.0.3
compare
commit
save
</pre>
<p>Traffic arriving on these ports on <samp>primary-eth0</samp> should now be forwarded to <samp>10.0.0.3</samp>. Since <samp>10.0.0.3</samp> is a proxy address, the secondary router will forward all traffic received there to the Verizon hardware router. You can now test that all FiOS features are working (they should be).</p>pjorgAt this point the network should be mostly functional. Hosts on the internal home network (10.0.0.0/24) should have Internet access, and the Verizon router should be also be accessing the FiOS ISP network via secondary router impersonation scheme.FiOS “Three-Router” with VyOS and ESXi, Part 9: Secondary Router Configuration2015-01-15T07:08:16-05:002015-01-15T07:08:16-05:00https://pjorg.com/fios-3-router-w-virtualized-vyos-9-secondary-router-config<p>The secondary router requires much less configuration to get working, though there is a slight twist. Let’s briefly rediscuss what the role of the secondary router is.</p>
<p>In essence, the secondary router’s job is to sit between the Verizon hardware router and the primary router, allowing the VZ hardware to believe it is connected directly to the FiOS ISP network. It does this by assigning (via DHCP) the VZ router the same public IP that the FiOS ISP network has assigned the primary router for actual Internet connectivity, and then performing Network Address Translation to convert packets originating from the VZ router to correct IP segment for the home network. They can then be handled by the primary router just like any other internal host’s traffic.<!--more--></p>
<p>Recall that for our secondary router, the first vNIC is placed on the Home Network port group on the ESXi virtual switch, and the second vNIC is placed on the Verizon Router Network. Let’s log into our secondary router’s console and run some commands to confirm that we see two interfaces.</p>
<pre>show interfaces
</pre>
<p>The interface called <samp>eth0</samp> corresponds to the adapter on the Home Network port group, and the <samp>eth1</samp> interface corresponds to the adapter on the Verizon Router Network. Let’s configure the <samp>eth0</samp> interface first.</p>
<pre>configure
set interfaces ethernet eth0 address 10.0.0.2/24
set interfaces ethernet eth0 address 10.0.0.3/24
set interfaces ethernet eth0 duplex auto
set interfaces ethernet eth0 speed auto
</pre>
<p>The <samp>eth0</samp> interface will have two static addresses assigned to it on the internal network. <samp>10.0.0.2</samp> will be the address for the secondary router itself, and we will configure <samp>10.0.0.3</samp> to be the address that the VZ router will use via 1:1 NAT.</p>
<p>Next we need to configure <samp>eth1</samp>, which will be the secondary router’s interface to the Verizon Router Network and the Verizon router itself. In order to imitate the external network for the VZ hardware router, we need to review the settings for our primary router and pull them over.</p>
<p>On the primary router:</p>
<pre>vyos@primary-router:~$ show interfaces ethernet
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 108.0.0.123/24 u/u FiOS Public Internet
eth1 10.0.0.1/24 u/u Home Network
eth2 10.10.10.1/24 u/u Lab Network
</pre>
<p>We can see that the DHCP address that the FiOS ISP network has assigned to our primary router is <samp>108.0.0.123</samp>. Additionally, this address is on a <samp>/24</samp> network, which has 256 addresses. This is an example only; your address (and possibly network size) will be different. We need to recreate the network presented to <samp>primary-eth0</samp> for the VZ router, using <samp>secondary-eth1</samp> to do so.</p>
<p>On the secondary router:</p>
<pre>configure
set interfaces ethernet eth1 address 108.0.0.1/24
set interfaces ethernet eth1 duplex auto
set interfaces ethernet eth1 speed auto
</pre>
<p>Basically, we are setting the <samp>eth1</samp> address to be in the same IP space as what we pulled from <samp>primary-eth0</samp>. Assuming a <samp>/24</samp> network, you can just take the first three octets of that address and add a <samp>.1</samp> to the end.</p>
<p>Next we need to set up DHCP to statically assign the correct address to to hardware router. To do this we will need the WAN MAC address of the VZ hardware router once more. You can read it from the label on the router again, or run the following command on the primary router to pull it up:</p>
<pre>show interfaces ethernet eth0 | grep -i link/ether | gawk -F " " '{print $2}'
</pre>
<p>Let’s start our DHCP configuration with some basic configs:</p>
<pre>set service dhcp disabled false
set service dhcp shared-network-name VZ-RouterNet
set service dhcp shared-network-name VZ-RouterNet authoritative enable
</pre>
<p>Now we need to configure the correct subnet. This will be the net that we are impersonating from <samp>primary-eth0</samp>; continuing from our example earlier it is <samp>108.0.0.0/24</samp>.</p>
<pre>set service dhcp shared-network-name VZ-RouterNet subnet 108.0.0.0/24
</pre>
<p>Let’s set a short 60 second lease time, to allow us to quickly make changes if the upstream IP at <samp>primary-eth0</samp> ever changes.</p>
<pre>set service dhcp shared-network-name VZ-RouterNet subnet 108.0.0.0/24 lease 60
</pre>
<p>The default router for this net should be the secondary router, which listens on this net at <samp>108.0.0.1</samp>.</p>
<pre>set service dhcp shared-network-name VZ-RouterNet subnet 108.0.0.0/24 default-router 108.0.0.1
</pre>
<p>Since this VZ router is used to run the Verizon services (such as set-top boxes) it makes sense to use the Verizon DNS servers.</p>
<pre>set service dhcp shared-network-name VZ-RouterNet subnet 108.0.0.0/24 dns-server 68.237.161.14
</pre>
<p>The network in our example is a <samp>/24</samp>, which has 256 addresses. We’re going to make our DHCP pool start after our router (<samp>.1</samp>) and stop before the broadcast address (<samp>.255</samp>). This shouldn’t really matter in any case, since we are going to statically-assign the appropriate address in the next step.</p>
<pre>set service dhcp shared-network-name VZ-RouterNet subnet 108.0.0.0/24 start 108.0.0.2 stop 108.0.0.254
</pre>
<p>We always want the hardware router to get the same address that <samp>primary-eth0</samp> has, so we’re going to statically assign it based on the WAN MAC of the hardware router. Make sure to substitute the correct MAC from your hardware here, as well as the correct IP from <samp>primary-eth0</samp>.</p>
<pre>set service dhcp shared-network-name VZ-RouterNet subnet 108.0.0.0/24 static-mapping vz-router mac-address 00:11:22:33:44:55
set service dhcp shared-network-anme VZ-RouterNet subnet 108.0.0.0/24 static-mapping vz-router ip-address 108.0.0.123
</pre>
<p>Lastly, we need to set up 1:1 NAT so that traffic can pass through the secondary router appropriately. Run these commands to set up the NAT rules, being sure to substitute the correct public IP where appropriate.</p>
<pre>set nat destination rule 10 destination address 10.0.0.3
set nat destination rule 10 inbound-interface eth0
set nat destination rule 10 protocol all
set nat destination rule 10 translation address 108.0.0.123
set nat source rule 10 outbound-interface eth0
set nat source rule 10 protocol all
set nat source rule 10 source address 108.0.0.123
set nat source rule 10 translation address 10.0.0.3
</pre>
<p>That pretty much does it for secondary router configs. Review, commit, save.</p>
<pre>compare
commit
save
</pre>
<p>You can now plug your Verizon router into the appropriate port on your switch (with VLAN 102) and power it up. If all is well the hardware router will come online and start functioning. You can test this by going to http://10.0.0.3/ with a web browser from any host on the <samp>10.0.0.0/24</samp> internal network. If you are greeted with the Verizon router login page, things are working.</p>pjorgThe secondary router requires much less configuration to get working, though there is a slight twist. Let’s briefly rediscuss what the role of the secondary router is.FiOS “Three-Router” with VyOS and ESXi, Part 8: Network Address Translation2015-01-15T07:07:47-05:002015-01-15T07:07:47-05:00https://pjorg.com/fios-3-router-w-virtualized-vyos-8-nat<p>At this point, our primary router has connectivity to the Internet, and we have a functioning home network with DHCP and DNS. Now we need to implement Network Address Translation (NAT) so that all of the hosts on the Home Network can share Internet access through the primary router’s one public IP.</p>
<p>Sharing one public IP with many clients to is called <strong>source NAT</strong>. This is because the translation is occurring at the source of the traffic—our home network. The other type, <strong>destination NAT</strong>, is better known in the consumer space as port forwarding. We’ll discuss that later.</p>
<p>The behavior we are looking for is simple. We want traffic that is bound for the Internet to be received by the router, have its source address changed to that of the router itself (instead of the private <samp>10.0.0.0/24</samp> address), and then forwarded on to its destination. When a response is received, the opposite should happen so that the response makes its way to the host on the <samp>10.0.0.0/24</samp> net that originally requested it.</p>
<p>We need only one source NAT rule for this.<!--more--></p>
<pre>vyos@primary-router:~# set nat source rule 10
vyos@primary-router:~# set nat source rule 10 outbound-interface eth0
vyos@primary-router:~# set nat source rule 10 protocol all
vyos@primary-router:~# set nat source rule 10 source address 10.0.0.0/24
vyos@primary-router:~# set nat source rule 10 translation address masquerade
</pre>
<p>The router will listen for traffic from <samp>10.0.0.0/24</samp>, and forward it out of <samp>eth0</samp> while “masquerading” as the original source.</p>
<pre>vyos@primary-router:~# compare
vyos@primary-router:~# commit
vyos@primary-router:~# save
</pre>
<p>At this point, hosts on your <samp>10.0.0.0/24</samp> internal network should have Internet access. You can connect all your home devices to this network and use them normally; wireless devices will require you to connect a <a href="http://en.wikipedia.org/wiki/Wireless_bridge">wireless bridge</a> to your switch. Many consumer-grade wireless routers can be configured to operate in bridge mode to serve in this role.</p>pjorgAt this point, our primary router has connectivity to the Internet, and we have a functioning home network with DHCP and DNS. Now we need to implement Network Address Translation (NAT) so that all of the hosts on the Home Network can share Internet access through the primary router’s one public IP.