DataYard’s Speed Test Going Offline

DataYard’s Speed Test Going Offline

Beginning January 21st, 2015, we are retiring the Internet speed test that we have offered at  There are numerous other speed test sites available, the most well-known being Although we know a lot of our customers have used this service over the years, we believe the functionality it provided can be better served by other services out there on the Internet.  

If you need help finding a new way to perform Internet speed tests feel free to give our support team a call at 937-226-6896 or send us an email at [email protected].  We are always here to help.

Ready Backup Scheduled Maintenance – October 06, 2014

The DataYard Technical Operations team will be performing an upgrade to our Ready Backup infrastructure beginning at 10:30 AM EDT on Monday October 06, 2014 with an anticipated completion time of 4:30 PM EDT.

During this maintenance, access to backup and restore capabilities will be limited or unavailable. No interruption or delay to regularly scheduled backup windows is anticipated. If you require an urgent restore operation during this maintenance or have any questions, please contact our Customer Service team at (800) 982-4539 or [email protected].

Please remember to follow us on Twitter (@datayard) for maintenance updates.

Layer 2 Networking Pseudo Redundancies

Layer 2 Networking Pseudo Redundancies


I’ve seen many production networks that rely on layer 2 pseudo-redundancies through the core, and I’m here to explain why such designs are unnecessary, often dangerous, and how they can be improved. This article assumes an understanding of the OSI model, the most common protocols at layers 2 and 3, and the common “Cisco campus” Core/Distribution/Access network model.

This article aims to cover the following:

  • Collapsed core and L2 core/distribution networks
  • Limitations of common L2 HA tools – VSS, STP
  • L3 alternatives

Layer 2 HA Scenarios

Consider the following common collapsed core environment, and the protocols at play to provide redundancy.


  • Core uses Cisco VSS, StackWise or similar shared control plane redundancy mechanism
  • Some or all VLANs span all access switches through L3 switch core
  • Spanning-tree (Classic STP, Rapid-STP, or MST) handling failover between physical access-core links
  • Little or no dynamic routing use
  • “Distribution” layer may be created to support campus expansion

Let’s touch on a few points that may seem like advantages to an engineer jumping into such a network.

The single shared Layer 2 domain allows for an “any user, any subnet, anywhere” approach to provisioning new ports and access for your users and systems. A user moving between physical points, departments, or switches in your campus or network does not mean re-addressing or re-configuring access rules, software, etc.

Having minimal routing protocol usage can be attractive to some – as all of the tunable knobs of OSPF, EIGRP, and BGP can be a little overwhelming. When a new port or subnet is needed, an SVI in the Layer 3 core can easily be configured, and a new VLAN trunked where needed.

VSS and other shared control plane solutions create a single logical switch. This has the apparent benefit of reducing administrative burden by having only a “single switch” to manage, and gaining physical hardware redundancy without using more complicated (administratively, anyway) protocols such as HSRP, GLBP, and dynamic routing.

Limitations of Common Designs

When you look beyond some of the key points above, you find some serious issues with collapsed core and shared control plane technologies in the core. Consider the diagram above with a single VSS pair in the collapsed core. Although the core switch and it’s connected infrastructure can survive a single physical hardware failure (up to a full VSS chassis), assuming all connected devices are dual homed using port channels, the entire network can be interrupted by a single “logical” failure, human error, or software bug. While VSS pairs can make an excellent edge or access layer switch, you should carefully consider the risks involved before deploying them in a core setting without other redundancy mechanisms.

In addition to increasing the risk of large-scale failure, solely relying on shared control plane redundancies also reduces flexibility in routing and traffic engineering. Switches configured in VSS or StackWise pairs maintain a shared routing table, routing processes, and make the same routing decisions.

Wasted bandwidth can be another issue when relying on Layer 2 transport and Spannning Tree failover from core to distribution or access network segments. When relying on any STP protocol (Multiple Spanning Tree with properly configured instances being the exception) for link redundancy, one or more links in a bridge must operate in a blocked state by design – reducing total available bandwidth of the topology.

Finally, STP (and VTP, if you choose to run it) is often classified as a dangerous and slow protocol. By design, every STP speaking bridge in a tree participates in Root Bridge election and exchanges bridge protocol data units (BPDUs) to determine who the root bridge is, where it’s located, and what paths need to be blocked. All it takes is one misconfigured switch inserted into the network to cause wide-sweeping failures or performance issues, which are often cumbersome to troubleshoot. If you’ve never had to troubleshoot spanning tree issues during a network failure, count your blessings!


Layer 3 Alternatives

Route everything – many networks and network engineers would benefit from abandoning the idea that any modern application actually requires layer 2 adjacency. Almost any broadcast domain spanning more than one device hop, or two network hosts total, can be eliminated and routed over without negative impact to the applications traversing the path. The most common exception to this rule is virtual machine migration traffic such as VMWare vMotion; technologies such as Cisco Overlay Transport Virtualization exist for remedying these exceptions.


Moving to a layer 3, route-centric design, also provides more options for protocol selection in how you transport your traffic from one network segment to another. Most common small and medium size networks will be sufficiently supported with basic OSPF or EIGRP implementations while larger enterprise and service provider networks will often rely more heavily on BGP and IS-IS. Your choice of protocols should be influenced by administrative experience, networking vendors involved, and your requirements for scale.

Proper use of first hop redundancy protocols such as HSRP, VRRP, and GLBP can also provide hardware redundancy at the first hop gateway much like VSS and StackWise – without the inherent risks of putting so many eggs in one proprietary basket.

Finally, depending on your route design and configuration, relying on layer 3 protocols for redundancy and transport allows for load balancing traffic across multiple paths. Equal cost and unequal cost load balancing can allow for full utilization of all network links without sacrificing ports and hardware for failover/blocked paths.


The topologies, protocols, and designs mentioned above are all only examples of tools that should be readily available in every networkers’ toolbox. Hopefully, this article has sparked some ideas that will lead to building a more redundant, better-performing network.

Routing Around Internet Congestion

Around 9:30AM EST, we started experiencing some choppy and dropped phone calls through our own internal SIP based phone system, and shortly after began receiving customer reports of slow and limited access to internet resources. After some investigation (20-30 minutes of troubleshooting) we determined that the access problems were isolated to one of our three major internet peers, Level3 communications. We immediately disconnected from Level3 to minimize the impact to us and our and customers, and this resolved most connectivity problems over the next few minutes while other networks began to make routing changes. Fortunately, our other ample upstream paths carried our full traffic load without issue.

Despite the change we made, Level3 is one of the largest and most utilized carriers in the world. As a result, many external networks still rely on Level3 as a transit or intermediary path to reach one of our remaining peers (Cogent, TWTelecom) which results in poor performance for some users. 

At this time, Level3 is still showing signs of loss in the Midwest and we are still de-peered. We have had a ticket open with Level3 since early this morning, and we are poking our contacts for updates as they become available. We will be monitoring throughout the evening, and re-peering when we have confirmation that things are stable.

So while it’s common to have multiple paths to reaching the rest of the internet, does your IT partner really have the ability to respond quickly and efficiently to a critical failure? If you’re not confident in the answer to that question, give us a call and you will be. 

Mail Routing Fundamentals

Mail Routing Fundamentals

E-mail can be accessed multiple ways from the e-mail program installed on your desktop to the web application you use when you are traveling.  Even though these methods of accessing e-mail have a different interface their method for sending e-mail is very similar. The process of directing a message to the recipient’s host is called routing. Apart from finding a path from the sending site to the destination, it involves error checking as well as potentially virus and spam filtering. There are many processes, systems and protocols involved in the successful delivery of one e-mail message. Sometimes these messages arrive at their destination almost immediately, but sometimes they may be delayed or even rejected somewhere in the process. We are going to discuss the basic process for e-mail delivery and look at a few reasons that messages can be delayed or rejected along the way.

When you click the “Send” button your e-mail client initiates a two-way conversation with your e-mail provider’s SMTP (Simple E-mail Transfer Protocol) servers over standard SMTP ports – TCP ports 25 and 587.  SMTP servers only understand very basic commands, so your mail client must initiate this connection using only commands that the SMTP server can interpret. The most common SMTP commands are:

SMTP Command, Description
HELO / EHLO, This command is used to identify the sender (client) to the SMTP server. EHELO is a newer command than HELO and includes more functionality\, however most servers still accept either command as an introduction.
MAIL FROM, Specifies the e-mail address of the sender.
RCPT TO, Specifies the e-mail address of the recipient.
DATA, Starts the transfer of the actual data (body text\, attachments etc).
RSET (RESET), Specifies that the current mail transaction will be aborted.
VRFY (VERIFY), Asks the receiver to confirm that the argument identifies a user or mailbox.
HELP, This command causes the server to send helpful information to the client.
QUIT, Quits the session.
AUTH LOGIN, Initiates SMTP Authentication when sending

The e-mail client sends these commands to the mail server and the server replies with numerical codes and additional information after the numerical code. These reply codes tell the client if the previously sent command succeeded or failed.  Commands and replies are composed of characters from the ASCII character set. EHELO is a newer command than HELO and includes more functionality, however most servers still accept either command as an introduction.

Once the server connection is initiated it will then authenticate your username and password. SMTP authentication is initiated with the AUTH LOGIN command. The server will then responds requesting your username, 334 VXNlcm5hbWU6, which is the phrase Username: in base64 encoding. You then send your base64 encoded username, the server then responds with 334 UGFzc3dvcmQ6 which is Password: in base64 encoding. You then send your base64 encoded password and should receive a 235 Authentication Succeeded message. If the e-mail server accepts your message it will proceed to the MAIL FROM command. This process is done inside your mail client, however with some know-how and command prompt this can be done manually as well.

The output of a telnet session will look something like this:

EHLO Hello []
250-SIZE 52428800
250 HELP
334 VXNlcm5hbWU6dXNlcm5hbWUuY29t
334 UGFzc3dvcmQ6bXlwYXNzd29yZA==

235 Authentication succeeded

You will know if your message is rejected by your SMTP server and why by looking at the undeliverable message that you receive back at the e-mail account you sent the message from. An undeliverable message will look something like this though the information and formatting will vary based on the provider. It is also possible that the message will be rejected before ever reaching your SMTP server, which will appear in your browser as a pop-up notification showing the error. (see error section below)

If you are using an outside company as a smart host the rejections that you will see should be checked for the rejecting server which could now be your own internal mail server, the smart host’s server or the recipient’s servers.

Every domain that is used for e-mail, for example, needs to have an MX-record (Mail Exchanger Record) in order to send and receive e-mail. An MX-record is a DNS-entry that tells the sending SMTP server where to deliver the e-mail for a domain. Your MX record will be determined by where your mail is hosted, each company has a different MX record, larger companies will have several.  An MX record looks like this:



In our example the hostname tells querying SMTP servers that e-mail addressed to someone should be delivered to the server ‘’.  This requires that ‘’ exists (either as a CNAME or an A-record) in the authoritative zone.

There are two numeric values in the above record. The first one is 3600, which is the TTL (Time to Live). This record tells other DNS-servers that it is OK to cache the above record for up to 3600 seconds (or one hour). The second numerical value is ’10’ and is the MX-record priority. In this example, it doesn’t matter, as we only have one record, but if we were to have multiple MX records, it would determine the priority order of the servers. If the first one is unresponsive, the second one will be used in order of priority with the lowest priority being used first (example: a record with the priority of 10 would be used before the record with a priority of 30). In a server with multiple MX records you would see this:


As you can see, there are multiple servers listed for the same domain ( They all have the same TTL, 14400 seconds (four hours). The difference is the priority and the server. The sending server would try to deliver e-mails to the servers in the order from lowest priority value to highest until it finds one server who accepts the connection. This means if the server ‘’ goes down or is too busy to accept the connection the message will then try to deliver to ‘’ and ‘’ since these two servers have the same priority they will be chosen at random. In the case of large providers with multiple domains relaying on their servers for SMTP they are likely using a clustered mail environment which means many servers with different IP addresses can send mail from the same host, such as ‘’.

If you want to know how your mail server might determine the DNS record read my article on DNS resolution here:

Once your SMTP server has determined the server for the mail to be sent to it will then attempt to make a connection with that server using the same basic commands as your mail client used to connect to your SMTP server. The message will first pass through the mail provider’s content filtering servers before even making it to their SMTP server where it may be rejected. There are a few reasons that mail can be rejected or even delayed by a provider’s content filters. First, if a content filtering server deems your mail content to be junk mail it will likely tag the message as junk and deliver it according to the domains preferred settings, most domains choose to have spam messages delivered to a specific “Junk Mail” folder.

If you are on a blockhole list you may receive undeliverable email notifications stating that the mail server you use is on a blackhole list. The first course of action to get removed from the blackhole list is to resolve any outstanding issues with your mail server. If this mail server is provided by an outside company or managed by your IT department you want to send them the bounceback you received so they can address the issue.  If you manage this mail server you can check some of the major blacklists for more information and to see if your server is listed in their database.

Once the server has determined that your mail is authorized and not on a blackhole list it will attempt to deliver this message to the recipients’ mailbox. The mail server will check to make sure that the recipient address specified does exist as a mailbox on the server. As long it does exist, the message will be delivered to the recipient’s mailbox for their mail client to retrieve. However, the message can again be rejected at this point. For example if the recipient email address is spelled incorrectly, has been deleted or even if the users’ mailbox is full you may receive a bounceback stating why your message was rejected.

Alternate delivery methods and setting up your own mail server

A smart host is an outside mail server that accepts inbound SMTP and routes it out again however the point of sending your mail to one of these is that you may offload the task of routing and retrying mail to another location. Directing your outbound traffic to an SMTP host at your ISP will allow your outgoing internet connection to free up much quicker than sending through your own network. A relay SMTP server is usually the target of a DNS MX record that designates it, rather than the final delivery system.  The relay server may accept or reject the task of relaying the mail in the same way it accepts or rejects mail for a local user.  If it accepts the task, it then becomes an SMTP client, establishes a transmission channel to the next SMTP server specified in the DNS and sends it the mail.  If it declines to relay the message a 550 error should be returned to the sender.

Common Rejections

From: Mail Delivery System <[email protected]>
To: <[email protected]>
Subject: Undelivered Mail Returned to Sender
file-2 , 416 Bytes
file-3 , 5.9 KBytes
file-4 , 16.2 KBytes
This is the mail system at host

I’m sorry to have to inform you that your message could not
be delivered to one or more recipients. It’s attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<[email protected]>: host[] said: 554
5.7.1 <>: Relay access denied (in reply to RCPT TO

From this delivery failure notice you can find the following information

1. The name of the server that is rejecting your message (this message can either be rejected by your server or by the recipients’ mail server):  This is the mail system at host, in this case the message was rejected by

2. The cause of the rejection: said: 554 5.7.1 <[email protected]>: Relay access denied (in reply to RCPT TO command, this particular rejection is rejecting relay access which means you do not have permission to send your mail through this mail server.

SMTP errors from a Mail client

No Relaying Allowed: This rejection message tells you that the credentials (username and password) that you are sending to the e-mail server are not authenticating. You should check the settings in your e-mail client to ensure your username and password fields are not blank, the correct credentials are entered and that you have outgoing server authentication enabled.

<[email protected]>: host[] said: 554

    5.7.1 <>: Relay access denied (in reply to RCPT TO


Could not connect to SMTP server: This rejection tells you that the connection from your e-mail client to your e-mail providers’ server failed. You should check the SMTP server name and ensure that your SSL types match the port numbers specified and that you have no firewall or antivirus program that is blocking outbound connections.


Greylisting is a way mail providers attempt to limit the amount of spam that gets through. Grey listing checks the legitimacy of an e-mail message by temporarily rejecting the incoming message with a 451 error.  The sending SMTP server will recognize this as a temporary rejection and attempt to re-send the message. If there is a prior relationship between the sending server and the recipient server, the message will get delivered without any issue. At the same time the 451 error is sent a triplet is recorded in the Greylisting database as an unconfirmed triplet.

A triplet consists of:

  • IP address of the sending server
  • Sender e-mail address
  • Recipient e-mail address

This is where delays come in, this response can take up to 24 hours depending on the settings of the server and how it is set to respond. If you have a small one server mail setup then the greylisting process generally doesn’t cause a significant delay. However, in a clustered mail environment where many different servers, with different IP addresses, can be used to send and receive mail the delays can be significant. If the rejected message is re-sent and all information matches the prior unconfirmed triplet then the message will pass through greylisting and be delivered. If any of the three items do not match then it will be seen as a new message and will begin the greylisting process again.

RBLs (Realtime Black Hole Lists)

There are specific companies who run blackhole lists and those lists are used by mail providers to determine when a server’s traffic has become unfavorable as a way to protect them from accepting unfavorable traffic. It’s common for a server’s mail IP address to temporarily end up on a public blackhole list. It could be because of the overall volume of mail that is coming from that server, messages that seem to have characteristics of junk in them, or that list has received messages to spam traps it operates.

How can you identify if your mail server is blackholed?

Spamhaus (

MXtoolBox (

Sorbs (

Remote Server Administration with PowerShell

Remote Server Administration with PowerShell

Your RDP Client is going to get straight up jealous


In the past remote Window’s administration has been limited to RDP, Citrix, DRAC/KVM, VNC, or the command line using PsExec. Developing scripts to remotely connect to hosts and gather data utilizing these legacy technologies became very cumbersome. One of the best things about PowerShell is the ability to manage hosts remotely quickly and efficiently. In a previous post I wrote about Local Server Administration with PowerShell which included quick glances at cmdlets for managing Windows features, firewall administration, network connectivity diagnostic tools, HTTP and HTTPS diagnostic cmdlets, and scheduled job management. The same cmdlets are available remotely; allowing us to configure servers without the need to RDP and executing PowerShell commands locally.

First we need to understand Windows Remote Management (WinRM) as PowerShell uses this technology to remotely connect to servers and other infrastructure. WinRM provides a common way for systems to access and exchange management information across an IT infrastructure. WinRM was introduced with the Windows Vista/Server 2008 kernel and can retroactively be installed in earlier versions by installing Windows Management Framework 2.0.

Security Implications with WinRM

Before implementing PowerShell Remoting it is important to understand the security implications. WinRM, by default, will use the HTTP protocol on port 5985 to communicate with a remote host. Microsoft Negotiate Authentication will decide to authenticate using either Kerberos and NTLM. The preference will be Kerberos, which will be used in the same AD domains or domains with trust relationships.

If you analyze packet captures during the WinRM process over HTTP, and follow the entire TCP stream, you will see that the content is encrypted with the content type of HTTP-SPNEGO-session-encrypted for NTLM v2 and HTTP-Kerberos-session-encrypted for Kerberos. Here is an example of a WinRM session using Kerberos as the local and remote computers were on the same domain:


The actual password is never sent over the wire and the only way that a password could be decoded is if the Security Accounts Manager (SAM) on the remote machine had already been compromised. The hash that is stored in the Security Accounts Manager is never actually sent and cannot be decoded. Even if there was someone snooping packets on the network they wouldn’t see anything in clear text and therefore an exploit is extremely unlikely. With this data we conclude that WinRM is encrypted by default and that configuring WinRM for HTTPS would be an added layer of security.

Configuring PowerShell Remoting

Configuring PowerShell Remoting was originally difficult and time consuming because administrators had to manually create the WinRM listener, set the service to automatically start, enable PowerShell session configurations, and troubleshooting authentication. Now, all of this is configured out of the box; primarily because the majority of new management tools application require this to properly function. Additionally, several cmdlets have a ComputerName parameter that is supported to identify which computer you want to run the command on. Here is how you can enable PowerShell remoting on previous versions of Windows.

Configure PowerShell Remoting with HTTP

Starting in PowerShell 2.0 you can enable PowerShell remoting with a single command:


The Enable-PSRemoting cmdlet runs the Set-WSManQuickConfig cmdlet, which performs the following tasks:

  • Starts the WinRM service.
  • Sets the startup type on the WinRM service to Automatic.
  • Creates a listener to accept requests on any IP address.
  • Enables a firewall exception for WS-Management communications.
  • Registers the Microsoft.PowerShell and Microsoft.PowerShell.Workflow session configurations, if it they are not already registered.
  • Registers the Microsoft.PowerShell32 session configuration on 64-bit computers, if it is not already registered.
  • Enables all session configurations.
  • Changes the security descriptor of all session configurations to allow remote access.
  • Restarts the WinRM service to make the preceding changes effective.

Configuring PowerShell Remoting with HTTPS

If you would like this additional layer of security, the process is slightly more complex than the Enable-PSRemoting command. The first thing you need to do is obtain an SSL cert for the remote server or workstation. You can script out the issuing of certificates by using Certificate Enrollment Web Service and Certificate Enrollment Policy Web Service in Windows Server 2012 R2. I would also recommend deploying a duplicate of the Web Server certificate template and modifying the settings to meet your organization’s needs. Here is an example of how you can obtain these certificates using PowerShell.

PS C:\> $uri = “https://yourca.domain.tld/Policy/service.svc”

PS C:\> $fqdn = “remotehost.domain.tld”

PS C:\> $subject = “[email protected], C=US, S=Ohio, L=Dayton, O=DataYard, OU=TechOps,CN=$fqdn”

PS C:\> $netbios = “remotehost”

PS C:\> $creds = Get-Credential

PS C:\> Get-Certificate -Template “WinRM” -Url $uri -DNSName $fqdn,$netbios -CertStoreLocation cert:\LocalMachine\My `

-SubjectName $fqdn `

-Credential $creds

You can issue these commands from the remote host in the same domain. If the certificate request is issued, then the returned certificate is installed in the machine’s personal certificate store. If the request is made pending, then the request is installed in the machine’s request store. Once you have the certificate installed you will need to obtain the certificate thumbprint for the issued certificate and copy it to your clipboard:

PS C:\> $thumbprint = (gci cert:\LocalMachine\My | Where {$_.Subject -eq $subject}).Thumbprint

PS C:\> $thumbprint | clip

In order to create the HTTPS WinRM listener you will have to use a command prompt and not PowerShell as you will receive an error “Error: Invalid use of command line. Type “winrm -?” for help”. In the following example we also delete the default HTTP listener:

C:\> winrm create winrm/config/Listener?Address=*+Transport=HTTPS  @{Hostname=”remotehost.domain.tld”;CertificateThumbprint=”E1ED1974E00A9789DFDD417A690F2C1480FD224A”}

C:\> winrm delete winrm/config/Listener?Address=*+Transport=HTTP

Now that WinRM is actively listening on 5986 we need to modify the firewall rules in PowerShell to reflect our changes. We strongly suggest restricting the remote addresses able to access this service for an additional layer of security:

PS C:\> Get-NetFirewallRule | Where {$_.Name -match “WINRM-HTTP-In-TCP”} | Disable-NetFirewallRule

PS C:\> New-NetFirewallRule -Name “WINRM-HTTPS-In-TCP” -DisplayName “Windows Remote Management (HTTPS-In)” `

-Direction Inbound -Action Allow -Protocol TCP -LocalPort 5986 -RemoteAddress

Utilizing PowerShell Remoting

Now that we have PowerShell Remoting configured there are several options to remotely connect. These options include the ComputerName parameter, Invoke-Command, PSSession capabilities, and implicit remoting.

ComputerName Parameter

You can run commands on a remote computer by using the ComputerName parameter. Although many cmdlets now offer a ComputerName parameter, there are a limited number of cmdlets with this parameter available. When you use this parameter Windows PowerShell creates a temporary connection that is used for the command and is then closed. Here is an example regarding memory consumption on a remote host:

PS C:\> Get-Process -ComputerName remotehost.domain.tld | sort WorkingSet | select -last 5

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName

——-  ——    —–      —– —–   ——     — ———–

   4658      93   100208     185996   669            3612 OUTLOOK

   1358     130   178744     214380   631           11912 VpxClient

   5917     104   136148     220148   772            5480 OUTLOOK

    463      32   268112     231492   780           10780 javaw

   7114     108   175692     269376   878            3528 OUTLOOK

Here are all of the commands with the ComputerName parameter available:

PS C:\> Get-Command | ForEach { if ($_.Parameters.Keys -Contains “ComputerName”){$_}} | select name

































I primarily utilize the ComputerName parameter for quickly identifying issues with servers in our local domain or dev environments. Since invoking a cmdlet only creates a temporary connection, any time you need to perform several action items on a remote host I would recommend to utilize PowerShell Sessions (PSSessions).


Invoke-Command is perhaps the most versatile way to remotely connect to hosts with PowerShell. You can use this command to execute scriptblocks locally, remotely, through a remote session with or without arguments, or through a connection endpoint URI. Invoke-command can also be used to invoke the same scriptblock on several different servers; we can also identify a filepath to predefined scripts to run remotely. You can also use Invoke-Command on a local computer to evaluate or run a string in a script block as a command. Windows PowerShell converts the script block to a command and runs the command immediately in the current scope, instead of just echoing the string at the command line.The Invoke-Command cmdlet runs commands on a local or remote computer and returns all output from the commands, including errors.

Here is an example of grabbing the configured DNS servers on all interfaces for several servers in your domain:

PS C:\> $server1 = “server1.domain.tld”

PS C:\> $server2 = “server2.domain.tld”

PS C:\> $server3 = “server3.domain.tld”

PS C:\> Invoke-Command -ScriptBlock { Get-DNSClientServerAddress } `

>> -ComputerName $server1,$server2,$server3


InterfaceAlias               Interface Address ServerAddresses                                PSComputerName

                             Index     Family

————–               ——— ——- —————                                ————–

Ethernet                            12 IPv4    {,}                 server1.domain.tld

Ethernet                            12 IPv4    {,}                 server2.domain.tld

Ethernet                            12 IPv4    {,}                 server3.domain.tld

PSSession Capabilities

A PSSession is a persistent PowerShell Session on a local or remote computer. Use a PSSession to run multiple commands that share data, such as a function or the value of a variable. Commands are executed inside of a PSSession using the Invoke-Command. Through PowerShell session options and New-PSSession parameters we can connect to a remote machine to perform the vast majority of our actions, and we can develop very elaborate scripts by connecting to and storing persistent PowerShell sessions.

Invoke-Command and PSSessions

Let’s start with the basics of creating a new session, storing it in a variable, and then issuing a command:

PS C:\> $session = New-PSSession remotehost.domain.tld

PS C:\> Invoke-Command -Session $session -ScriptBlock { (Get-NetAdapter).Name }

vEthernet (Internal Ethernet Port Windows Phone Emulator Internal Switch)

vEthernet (Realtek PCIe GBE Family Controller – Virtual Switch)

Production Interface (95)

That is a very basic example of capturing data from a remote host. Here is a little bit more of an advanced example of how we can capture all IPs from a remote host, and create A records on our domain controller (a different remote host) for all the server IPs:

PS C:\> $creds = Get-Credential

PS C:\> $remote_fqdn = “remotehost.domain.tld”

PS C:\> $dc = “domaincontroller.domain.tld”

PS C:\> $session1 = New-PSSession $remote_fqdn -Credential $creds

PS C:\> $session2 = New-PSSession $dc -Credential $creds

PS C:\> $ips = Invoke-Command -Session $session1 -ScriptBlock {

(Get-NetIPAddress | Where {$_.InterfaceAlias -notmatch “Loopback”}).IPAddress


PS C:\> $ips

PS C:\> Invoke-Command -session $session2 -ArgumentList $remote_fqdn,$ips -scriptblock {

param ($remote_fqdn, $ips)

$name = $remote_fqdn.split(“.”)[0]

ForEach ($ip in $ips){

#Add-DnsServerResourceRecordA -ZoneName domain.tld. -Name $name -IPv4Address $ip





Persistent PowerShell sessions are great for scripting purposes, but troubleshooting on the fly and utilizing Invoke-Command and ScriptBlocks is not a practical solution. However, with the Enter-PSSession command you can connect interactively to a PowerShell session on any properly configured remote system. The Enter-PSSession cmdlet starts an interactive session with a remote computer. During the session, the commands that you execute run on the remote computer, just as though you had a local PowerShell sesssion on it (one thing to note is that you can have only one interactive session at a time). Here is an example of connecting to a remote computer:

PS C:\> Enter-PSSession remotehost.domain.tld -Credential (Get-Credential)

In the past we had to launch an RDP client, enter in credentials, wait on the user profile and startup programs to load, and launch a PowerShell session. These actions took much longer than Enter-PSSession. The Enter-PSSession cmdlet is my personal favorite for PowerShell remoting because it is a subsitute for remote desktop and decreases the administrative time to connect to remote hosts.

Implicit Remoting

PowerShell remoting gets even better as you can take it one step further and create PowerShell modules from your PowerShell sessions you create. For example, I created modules for managing our corporate exchange servers (Exchange 2013):

PS C:\> $creds = Get-Credential

PS C:\> $uri = “https://exchange.domain.tld/PowerShell”

PS C:\> $exchange = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionURI $uri -Authentication Basic `

-Credential $creds -SessionOption (New-PSSessionOption -NoMachineProfile -SkipCACheck -SkipCNCheck -SkipRevocationCheck)

PS C:\> Export-PSSession -Session $exchange -CommandName * -OutputModule DataYardExchange -AllowClobber

Managing remote infrastructure becomes significantly easier as we build custom modules for our environment. Implicit remoting and PowerShell modules will not store your password(s) and will prompt you accordingly upon importing the module. Here is an example of modifying DataYard’s corporate exchange environment with the module we recently created:

PS C:\> Import-Module DataYardExchange

PS C:\> Set-Mailbox eric.wright -EmailAddresses @{add=”[email protected]”}

PS C:\> (Get-Mailbox eric.wright).EmailAddresses

smtp:[email protected]

smtp:[email protected]

Another way to take advantage of implicit remoting is by utilizing the cmdlet Import-PSSession. Rather than creating a module we can create a PSSession and then use the Import-PSSession cmdlet to implicitly remote to the server:

PS C:\> $creds = Get-Credential

PS C:\> $uri = “https://exchange.domain.tld/PowerShell”

PS C:\> $exchange = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionURI $uri -Authentication Basic `

-Credential $creds -SessionOption (New-PSSessionOption -NoMachineProfile -SkipCACheck -SkipCNCheck -SkipRevocationCheck)

PS C:\> Import-PSSession $exchange

PowerShell Web Access

The last feature to discuss is PowerShell Web Access (WindowsPowerShellWebAccess). This allows you to execute PowerShell commands through an encrypted website from any web browser. PowerShell Web Access can be configured using the following commands:

PS C:\> Install-WindowsFeature -Name WindowsPowerShellWebAccess -IncludeManagementTools

PS C:\> Install-PswaWebApplication -webApplicationName pswa -UseTestCertificate

PS C:\> Add-PswaAuthorizationRule -UserName domain\user -ComputerName localhost -ConfigurationName Microsoft.PowerShell

PS C:\> start iexplore.exe https://localhost/pswa

In your favorite web browser (Chrome, Firefox, Safari, or IE) you can utilize PowerShell Web Access from anywhere in the world. This is a screenshot of PowerShell Web Access from Safari on an iPhone:


I strongly suggest to implement additional layers of security by restricting the IP space that is able to access this feature as well as utilizing additional authentication schemes such as Windows Authentication, Basic Authentication, and/or client certificate verification at the IIS level.


There are many ways to connect to and manage remote systems, PowerShell becoming the most consistent way to manage Window’s infrastructure. As you can see the functionality of persistent PSSessions and modules can become very elaborate. Remote management techniques for Windows’ infrastructure has been constantly improving and I look forward to the incremental improvements.

BGP, ASICs, and IPv4 at scale – Why the Internet is (or was) Broken

BGP, ASICs, and IPv4 at scale – Why the Internet is (or was) Broken

On Tuesday August 12, 2014, the global IPv4 Internet routing table exceeded 512,000 routes. This is an important number for several very common routing platforms across multiple networking vendors, and as a result many networks across the globe experienced interruptions and performance issues. While some articles have reported on the scope of impacted carriers and services, I’ll discuss the technologies and limitations that led to this failure.

To understand the source of this failure, there are two critical pieces of Internet technology to grasp. First is BGP, the way routers around the Internet exchange information about what networks they know about, and how to reach them. Second, and the more immediate cause of the recent failure, are the mechanics or implementation of routing and destination lookups in high performance, carrier grade routers and switches.

Border Gateway Protocol

The common language that all carrier and ISP routers speak throughout the Internet is crucial to the delivery of the interconnected, distributed network that we all use today. BGP dates back to 1995 and even earlier in older iterations, and is still in use today, albeit with many adjustments and new features. At the core of BGP is a common method for routers and network devices to tell each other about networks, and their relationship to those networks. A critical component to BGP operation is the Autonomous System or ASN, which defines a network and its associated prefixes (known routes) as viewed from the perspective of the BGP protocol.


BGP speaking routers across autonomous systems peer with each other in a variety of arrangements such as dedicated peering points or direct connections, and exchange their known routes or a subset of known routes with each other. This exchange also includes a large number of attributes (Origin, As_Path, Next_Hop, Local_Pref, communities…) associated with each prefix. As you might expect, at a global scale this table is quite large – and each time a new autonomous system (typically an ISP or larger enterprise) pops up, or an existing autonomous system further distributes its network, this table continues to grow.



Routers in larger scale networks need to store, reference, and update this large global BGP routing table quickly and frequently. So where does all of this information get stored? The answer isn’t simple, and varies wildly by vendor and hardware, but in large scale network hardware, the general implementation looks like this.


Routers handling BGP (and other routing protocols) maintain routing tables and store known routes in a Routing Information Base or RIB specific to that protocol. The best known route for a destination is then inserted into the Forwarding Information Base or FIB, which the router will use for forwarding lookups on packets due for processing. While the RIB is typically stored in general system memory, of which space is often readily available, the FIB is stored on TCAM residing on ASICs. Application Specific Integrated Circuits are the secret sauce to high performing networking appliances, enabling fast routing and switching above and beyond what traditional x86 based processing and memory can achieve. The ASICs on modern routers and switches contain a fixed amount of TCAM, enabling the low latency lookups and updates required for routing and switching.

So what happened and what does it have to do with TCAM? Going back to the original issue with our picture of routing tables and TCAMs in mind, you can see how the growing IPv4 Internet might be a problem for platforms with a fixed amount of space available. As it so happens, some of the most commonly deployed BGP speaking routers and switches over the last decade, including the Cisco 6500 and 7600 platform, have had default TCAM allocations allowing for a maximum of 512,000 IPv4 entries. On many platforms, as soon as TCAM space reaches its limit, failures can occur ranging from reduced performance to catastrophic failure. On Tuesday that 512,000 IPv4 limit was reached and the straw that broke the camel’s back, so to speak, was Verizon owned ASN 701 & 705. While the change, the apparently inadvertent advertisement of thousands of new routes, was quickly reversed, routers across the Internet saw performance issues and failures for hours due to the nature of BGP propagation.


Even though we are again under the 512k IPv4 route marker, it’s only a matter of time until we cross that barrier and beyond. This expansion of IPv4 tables is ultimately at the fault of no one, and should only be seen as the expected progression and growth of the “Internet of Things”. It’s hard to predict what type of scale future networks will need to adapt to, and in a world of fast expanding virtualization and cloud services it can be easy to forget that the underlying network transport behind these services must scale to cope with demand. Hold on to your hats over the next few weeks as we inevitably cross the 512k barrier again, and see who accepted this as a wake-up call to upgrade where necessary, and who didn’t.

15th Annual SysAdmin Appreciation Day

15th Annual SysAdmin Appreciation Day

Today — Friday, July 25, 2014 — is the 15th Annual System Administrator Appreciation Day!

Please join us in thanking the server, network, and data center people who keep all our technical ships running smoothly — no matter what time it is, day or night, weekday, weekend, or holiday, 365 days a year.

We designed and printed a custom 11″ x 14″ letterpress poster to commemorate the day. If you’d like a copy just let us know via email ([email protected]) and we’ll get it to you. This was a limited production run poster so we may run out soon.

P.S. There is a movie reference hidden somewhere on the poster. Can you identify the reference and the movie?

15th Annual SysAdmin Appreciation Day

Your Data is Telling You Something


I found an article this morning through my daily cruise through tech news. I mentioned the company mentioned in the article, Codespaces, yesterday when I was talking with our management team. Codespaces was wiped out of existence by an attack that ultimately led to the near total destruction of their data. The timing is serendipitous because this article touches on some of the points I was trying to make about looking for anomalous data in the many streams of data we already have at our disposal, or could easily add.

From the article: “Every attack is a sequence of events.”

I’ll extend that: “Every system failure or security breach is a sequence of events.”

Failures are generally not atomic events. Systems aren’t 100% functional one instant, and down to 0% the next. Anomalous data is a bellwether that something different is happening, that something worth taking a closer look at is in progress. Not that something necessarily wrong is happening. You don’t know that yet. All you know is that something different is happening, a possible indicator of trouble on the horizon. You’re begging for trouble if you know you have an anomaly in progress and you ignore it.

A simple example: interpreting car dashboards in context. I’ve driven two Volkswagen Beetles in recent years. One Beetle had a faulty coolant sensor that would indicate an over temperature condition no matter what, even when the coolant level was fine and the car was cooling properly. The other Beetle, driven by my daughter, had a coolant sensor that was OK. Last week it lit up for the first time. She mentioned it to me, and I did what I could do at 10:30 at night in the dark: checked the coolant with flashlight in mouth, and topped it off. I told her to keep an eye on it. The next day the sensor went off again, but she kept driving the car. A short while after that, when my wife was driving the car, billows of white smoke poured out of the car because a cheap plastic connector had broken.

A problem started to unfold with my car, and an indicator went off. I responded (added coolant to a leaky system), but that didn’t resolve the core problem. The indicator went off again, was ignored, and the problem got worse. If my wife hadn’t stopped the car immediately with the third indicator (smoke) the over temp could have warped the rods and destroyed the engine. Furthermore, the status of the indicator had an entirely different meaning based on the historical trend data of the particular Beetle.

Changing data is telling you something. Listen to it.

Simple Desktop Virtualization: Building a Robust Environment Using Microsoft Remote Desktop Services

There have been a number of new features introduced to Remote Desktop Services in Windows Server 2012 that make the deployment of desktop virtualization an attractive option for businesses looking to enhance the experience of their desktop users.  This post will briefly review the concept of desktop virtualization and the two primary deployment models but will focus primarily on session-based desktop virtualization architecture.  This post is intended for administrators or engineers considering desktop virtualization deployment options.

Desktop virtualization allows you to abstract the desktop and application environment from the endpoint device, centrally manage and deploy applications and offer flexibility to your end-users. There are two primary deployment models: session-based and virtual machine-based desktop virtualization. Regardless of which deployment model (or combination) you choose, you can realize a number of benefits. Users can securely access their corporate desktop and applications from a thin-client, from a tablet in a coffee shop or from their computer at home.

Instead of deploying full desktop computers which require you to maintain support agreements and pay for OS and application licenses, you can simply deploy thin client devices with no moving parts to each desk. By providing a standardized desktop environment and removing hardware support from the equation, you also reduce support overhead. Centralized OS and application management improves security and reduces the risk of malware infections or other compromises due to misconfiguration.

Provisioning a new desktop environment becomes a simple task of setting up a new user and placing a thin client device on their desk. Centralizing the desktop environment in the data center, or the cloud, also allows you to reduce licensing costs as well as deploy or remove resources based on user-demand.

RDS Role Services

The Remote Desktop Services server role in Windows Server 2012 is comprised of multiple role services that can be deployed together for small deployments or spread across multiple servers for fault-tolerance and scalability.  The six available roles services are described below (some are supplemental or not required for the deployment of session-based desktop virtualization):

Session Host: This is one of the core role services and allows a server to host session-based virtual desktops or RemoteApp applications (more on this later) – this role service was formerly known as terminal services in legacy versions of Windows Server.

Virtualization Host: This is another core role service that allows a server to host virtual machine-based virtual desktops by integrating Remote Desktop Services with the Hyper-V server role.  This is the counterpart role service to the session host role service and is required when you wish to support user connections to either pooled virtual desktops or personal virtual desktops.

Licensing: This is a core role service that is required in all deployments and allows administrators to centrally manage the licenses required for users to connect to session host or virtualization host servers.  This role can often be placed on a host running additional Remote Desktop Services role services due to its lightweight nature.

Connection Broker: This role service is critical to enterprise deployments because it allows administrators to build high-availability by load-balancing incoming connection requests across available session host or virtualization host servers. The connection broker role service keeps track of sessions and routes connection requests to available hosts based on host load – in the case of existing connections, the connection broker will re-connect the user to their existing session. In Windows Server 2012, this role service now supports Active/Active operation allowing you to meet scalability requirements in addition to fault-tolerance in the connection broker layer of the infrastructure.

Gateway: This is a supplemental role service that allows users to securely connect to their session-based desktops, virtual machine-based desktops or RemoteApp published applications on an internal corporate network from any device.

Web Access: Another supplemental role service that allows users to securely connect to their session-based desktops, virtual machine-based desktops or RemoteApp published applications through a web browser.

VDI Enabling RDS Features

RemoteFX: RemoteFX provides end-user experience enhancements to the Remote Desktop Protocol that allow RDP to operate as a truly viable desktop virtualization platform.  A few of the RemoteFX components that provide critical capabilities when it comes to desktop virtualization are full USB redirection support for all desktop virtualization workloads, vGPU hardware acceleration (the ability to use a physical GPU in Hyper-V hosts to accelerate host-side rendering of graphics intensive content), intelligent transport enhancements that introduce UDP support to allow fuller/smoother desktop experiences over low bandwidth/high-latency links and finally multi-touch capabilities to support gestures (e.g. pinch, zoom, swipe).

User Profile Disks: The introduction of User Profile Disks greatly enhances the ability for users to personalize pooled virtual machines removing the necessity to implement more complex roaming profile or folder redirection methodologies. This technology is really only applicable to virtual machine-based deployments but is worth mentioning as it makes deployment of virtual machine-based VDI much simpler and more robust.

Fairshare Resource Management: New to Windows Server 2012, the session host role service now introduces “fair share” resource management for network I/O, storage I/O and CPU.  This resource management ensures that no single user can adversely impact users on the same host by ensuring each session is allocated an equal share of the resources described above.  This simple enhancement makes the user experience in session-based deployments much smoother and consistent and eases the burden on administrators when deploying additional RD Session Host servers to the session collection.

Deployment Models

Now that we’ve gone through a brief review of Remote Desktop Services and its various role services, let’s focus on examining the session-based deployment model. First and foremost, let’s address why the session-based deployment model is so attractive when compared to the virtual machine-based deployment model.

The “traditional” approach to VDI is to utilize virtual machine-based desktop virtualization – when using Microsoft Remote Desktop Services this generally means one or more RD Session Broker instances and a number of dedicated Hyper-V hosts running the RD Virtualization Host RDS role service.  In this model, users connect to a virtual machine either from a pool of identically configured virtual machines (called pooled virtual desktops) or a personal virtual desktop that is dedicated to a specific user. The drawback is that this model requires standing up dedicated virtual machines per user which requires significantly greater resources than a session-based deployment.  In addition to the increased resource demands, administrators must ensure there are adequate pooled virtual desktops or personal virtual desktops in place to meet user load requirements. You should consider virtual machine-based deployment in those cases where users require significant control over their desktop environment (e.g. administrative rights, installing applications, etc.).

In the session-based model, we continue to rely on session broker instances to load-balance incoming connections but replace the dedicated Hyper-V hosts running the RD Virtualization Host role service with servers running the RD Session Host role service. In this model, users connect to a centralized installation of a desktop running on one or more session host servers (when session hosts are configured in a farm, it is referred to as a session collection) only receiving the user interface from the session host server and sending input commands to the server. When the RD Session Host role service is installed, fairshare resource management ensures storage I/O, network I/O and CPU are evenly distributed to all users; in addition, Windows automatically configures processor scheduling to prioritize performance of user applications over background processes. This is a good reason to avoid collocating this role service on a server with “traditional” background type services (e.g. IIS, Exchange, etc.). In addition to accessing full virtual desktop environments, applications can be published as RemoteApp programs from the session host servers allowing users to connect to specific applications over Remote Desktop – these applications are rendered in a local window on the client and from the end-user perspective, behave almost identically to local applications.

Sample Session-Based Deployment Architectures

For testing or starting a small session-based VDI deployment, you can install the RD Session Host, RD Connection Broker and RD Licensing role services on a single server through the Quick Start deployment option.  For an enterprise deployment, we would need to design the infrastructure to deal with availability and scaling concerns.  As with any environment, the complexity increases with the availability requirements – a session-based VDI deployment that requires five¬¬¬ nines availability and needs to support a large concurrent user base will be significantly larger and more complex than a solution for a small business with reduced availability requirements. Let’s examine three sample deployment architectures designed to meet the requirements of organizations of varying size and complexity.

Small Business Deployments

For the small business environment that doesn’t have strict availability requirements and only a small number of thin-client devices and remote users, we can deploy a simple infrastructure. For these types of environments, you can deploy the following RDS role services on a single host by using the Quick Start deployment option: RD Session Broker, RD Session Host and RD Licensing.

While a detailed discussion about sizing requirements is outside the scope of this article, this type of environment is generally suitable for 45-50 simultaneous user connections and using standard line of business applications (e.g. Office, Web Browsers, Thick Client Applications) that are not graphics intensive.

Figure 1. – Basic Small Business Session-Based Architecture

Small Business Deployments with Increased Availability

For the small business environment that has increased availability requirements, the deployment becomes only slightly more complex. Windows Server 2012 introduces the Active/Active Broker feature that allows you to provide fault-tolerance and scalability to the connection broker role service without the requirement to implement complex failover-clustering. Instead, you simply combine both connection broker servers under a single DNS entry.

In this architecture, we deploy both the session host and connection broker role services on two identically configured servers, configure Active/Active brokering and register both session hosts in a new session collection. A SQL Server instance is required to allow the connection brokers to synchronize and track sessions. User connections will be evenly distributed across the session host servers – if a single RDS server fails, users will still be able to access their virtual desktop environment.

If this type of environment needs to support the loss of a single RDS server without degradation in the end-user experience, then the previous recommendation of 45-50 simultaneous users should be followed. If some amount of degradation in the end-user experience is acceptable or the business accepts that a reduced number of users will be able to connect in the event of a server loss, this number could be increased to 75-100 simultaneous users.

Figure 2. – Small Business Session-Based Architecture with Increased Availability

Enterprise Scale Deployments

For enterprise deployments that have strict availability requirements, need to support a large number of thin-clients and remote users and also require the ability to continue scaling out the environment as the number of users increases, the architecture becomes much larger and more complex.

In this architecture, the connection broker role service is deployed on two dedicated servers in Active/Active Broker mode using round-robin load balancing. For connection broker synchronization and user session tracking, SQL Server is deployed as a clustered failover instance on two dedicated servers to provide fault-tolerance. Microsoft testing shows the connection broker role service operating in Active/Active Broker mode on two servers can maintain reasonable endpoint connection response times (< 2.5 seconds) for up to 2,000 simultaneous connections before end-user observable degradation is encountered.

Fault tolerance and scalability is further accomplished by deploying the session host role service on three dedicated servers registered in a session collection with the Active/Active connection broker farm. User connections will be evenly distributed across the session host servers – if a single RDS server fails, users will still be able to access their virtual desktop environment. To meet scalability requirements, the enterprise can simply add additional session host servers to the session collection to support the increasing number of users.

This architecture also provides fault tolerance in Active Directory Domain Services with two domain controllers deployed on dedicated servers.

If this environment needs to support the loss of a single RDS server without degradation in the end-user experience, then the session host servers should not exceed 66% of their total capacity or about 34 simultaneous connections per session host server. If some amount of degradation in the end-user experience is acceptable or the business accepts that a reduced number of users will be able to connect in the event of a server loss, this number could be increased to 125-150 simultaneous users. As stated before, this environment can continue to scale out to meet user load requirements by simply adding additional session host servers to the session collection and if exceeding 2,000 simultaneous connections, adding additional connection broker servers.

Figure 3. – Enterprise Scale Session-Based Architecture


As was pointed out in some of the example architectures, there are a number of factors that can affect the deployment model and architecture you choose and the sizing requirements of the individual role services. You should carefully consider things like application memory requirements, graphic intensive workloads, availability requirements and budget constraints when deciding if desktop virtualization is right for your business.

Every environment is different and not every business will be able to realize cost savings but every business can certainly gain flexibility and enhanced user experience by implementing desktop virtualization. Hopefully this article will help you get you started down the path to providing true flexibility and simplicity to your end-user computing environment.