EMERGENCY MAINTENANCE WINDOW 9-30-2015 T1 Customers only

Wednesday September 30th at 5PM EST, the DataYard Technical Operations Team will be performing a emergency software update on Core Networking Equipment. This emergency update will be updating core infrastructure for our T1 customers only. We do not anticipate any downtime, the estimated time is < 20 minutes for completion.  Affected clients will receive a notification via email.

If you have any questions regarding this maintenance please contact our technical support team at 1.800.982.4539 or [email protected]. Remember to follow us on Twitter (@datayardtechops) for the latest service and maintenance status updates at DataYard.

Emergency Maintenance Window 7-20-2015

This evening, Monday July 20th at 10PM EST, the DataYard Technical Operations Team will be performing emergency software updates on Windows VIPs, Fusion and Connect Exchange systems.  These emergency updates are in response to a remote security vulnerability that Microsoft has announced (https://technet.microsoft.com/en-us/library/security/ms15-078.aspx).   Patching the vulnerability quickly will help ensure the stability and security of your environment.  Affected clients will receive notification via email that their systems will be updated.

If you have any questions regarding this maintenance please contact our technical support team at 1.800.982.4539 or [email protected]. Remember to follow us on Twitter (@datayardtechops) for the latest service and maintenance status updates at DataYard.

Industry “Best Practices”?? I Don’t Care!

A staple of the information technology industry is the notion of a “best practice”. I’m not very fond of it.

Best practices are supposed to be the the golden standards by which technological solutions are implemented and problems resolved in the surest way possible. They’re often researched and developed by the vendor who produced the technology, and so they come with a certain weightiness. For example, if Microsoft says that an Exchange environment needs to be built a certain way for a certain number of users to work well, the conventional wisdom asks, “who am I — a lowly user of their technology — to disagree?”

There’s an irony in this. For years, I have heard technology experts complain that Cisco certification exams reflect perfect-world environments that don’t exist. They’ve said that they were only able to pass their exams by answering questions the way Cisco says you should do things, not the way they would actually solve networking problems in real life. This is just one example with one vendor, but it brings to the surface the familiar conflict between book learning and hands-on experience. People in the business know that the two solution paths are often different.

Therein lies the paradox. When it comes to certification exams, our experience tells us that the textbook solution, the “best practice”, may not genuinely be the best way to address a problem. It bugs us when these disconnects happen, but we play along with the vendor in order to get the certification and their stamp of approval. Yet, when we face a real world problem in search of a solution, we tend to seek out the industry whitepaper on the best practice and give it special reverence.

Whitepapers serve their purpose, but they’re crippled from the start as guides to perfection. First of all, who gets the special privilege of defining the criteria for all that is required to be “best”? Is there any room in the determination of what is best for a customer’s hard budget constraints, deployment timelines, and flexibility? Or is an industry best practice limited from the very beginning because it starts with a rigid problem specification that doesn’t match a real technical challenge, assumes unlimited access to resources and time, and assumes a pristine lab environment in which to operate?

Experience has taught me that best practices are merely templates to start from and nothing more. They are just tools that give us a benchmark to work from, and maybe they establish some realistic performance expectations. However, a person’s real-world experience deploying and understanding technology is always infinitely more valuable to me. Over-reliance on vendor best practices can be seen as forever leaning on a technological crutch. Saying that “vendor’s so-and-so’s best practice in this situation is to…” may appear to add credibility to a course of action, but it can also stifle experimentation during problem solving, innovation, and independent thought. We need more than just the ability to read and regurgitate our Google search results.

My advice? Read the whitepaper, read the best practice. Bask in the information presented, and then put it aside and be a critical thinker. Technology professionals are fantastically innovative, and they need to trust their own experience and imagination to solve their own unique problems, perhaps in even new ways that make sense to them, their customers, and their employers. No new thinking ever came out of blindly following a vendor best practice.

DataYard’s Speed Test Going Offline

DataYard’s Speed Test Going Offline

Beginning January 21st, 2015, we are retiring the Internet speed test that we have offered at http://speedtest.donet.com/.  There are numerous other speed test sites available, the most well-known being http://speedtest.net/. Although we know a lot of our customers have used this service over the years, we believe the functionality it provided can be better served by other services out there on the Internet.  

If you need help finding a new way to perform Internet speed tests feel free to give our support team a call at 937-226-6896 or send us an email at [email protected].  We are always here to help.

Ready Backup Scheduled Maintenance – October 06, 2014

The DataYard Technical Operations team will be performing an upgrade to our Ready Backup infrastructure beginning at 10:30 AM EDT on Monday October 06, 2014 with an anticipated completion time of 4:30 PM EDT.

During this maintenance, access to backup and restore capabilities will be limited or unavailable. No interruption or delay to regularly scheduled backup windows is anticipated. If you require an urgent restore operation during this maintenance or have any questions, please contact our Customer Service team at (800) 982-4539 or [email protected].

Please remember to follow us on Twitter (@datayard) for maintenance updates.

Layer 2 Networking Pseudo Redundancies

Layer 2 Networking Pseudo Redundancies

Premise

I’ve seen many production networks that rely on layer 2 pseudo-redundancies through the core, and I’m here to explain why such designs are unnecessary, often dangerous, and how they can be improved. This article assumes an understanding of the OSI model, the most common protocols at layers 2 and 3, and the common “Cisco campus” Core/Distribution/Access network model.

This article aims to cover the following:

  • Collapsed core and L2 core/distribution networks
  • Limitations of common L2 HA tools – VSS, STP
  • L3 alternatives

Layer 2 HA Scenarios

Consider the following common collapsed core environment, and the protocols at play to provide redundancy.

collapsed-core

  • Core uses Cisco VSS, StackWise or similar shared control plane redundancy mechanism
  • Some or all VLANs span all access switches through L3 switch core
  • Spanning-tree (Classic STP, Rapid-STP, or MST) handling failover between physical access-core links
  • Little or no dynamic routing use
  • “Distribution” layer may be created to support campus expansion

Let’s touch on a few points that may seem like advantages to an engineer jumping into such a network.

The single shared Layer 2 domain allows for an “any user, any subnet, anywhere” approach to provisioning new ports and access for your users and systems. A user moving between physical points, departments, or switches in your campus or network does not mean re-addressing or re-configuring access rules, software, etc.

Having minimal routing protocol usage can be attractive to some – as all of the tunable knobs of OSPF, EIGRP, and BGP can be a little overwhelming. When a new port or subnet is needed, an SVI in the Layer 3 core can easily be configured, and a new VLAN trunked where needed.

VSS and other shared control plane solutions create a single logical switch. This has the apparent benefit of reducing administrative burden by having only a “single switch” to manage, and gaining physical hardware redundancy without using more complicated (administratively, anyway) protocols such as HSRP, GLBP, and dynamic routing.

Limitations of Common Designs

When you look beyond some of the key points above, you find some serious issues with collapsed core and shared control plane technologies in the core. Consider the diagram above with a single VSS pair in the collapsed core. Although the core switch and it’s connected infrastructure can survive a single physical hardware failure (up to a full VSS chassis), assuming all connected devices are dual homed using port channels, the entire network can be interrupted by a single “logical” failure, human error, or software bug. While VSS pairs can make an excellent edge or access layer switch, you should carefully consider the risks involved before deploying them in a core setting without other redundancy mechanisms.

In addition to increasing the risk of large-scale failure, solely relying on shared control plane redundancies also reduces flexibility in routing and traffic engineering. Switches configured in VSS or StackWise pairs maintain a shared routing table, routing processes, and make the same routing decisions.

Wasted bandwidth can be another issue when relying on Layer 2 transport and Spannning Tree failover from core to distribution or access network segments. When relying on any STP protocol (Multiple Spanning Tree with properly configured instances being the exception) for link redundancy, one or more links in a bridge must operate in a blocked state by design – reducing total available bandwidth of the topology.

Finally, STP (and VTP, if you choose to run it) is often classified as a dangerous and slow protocol. By design, every STP speaking bridge in a tree participates in Root Bridge election and exchanges bridge protocol data units (BPDUs) to determine who the root bridge is, where it’s located, and what paths need to be blocked. All it takes is one misconfigured switch inserted into the network to cause wide-sweeping failures or performance issues, which are often cumbersome to troubleshoot. If you’ve never had to troubleshoot spanning tree issues during a network failure, count your blessings!

STP_block

Layer 3 Alternatives

Route everything – many networks and network engineers would benefit from abandoning the idea that any modern application actually requires layer 2 adjacency. Almost any broadcast domain spanning more than one device hop, or two network hosts total, can be eliminated and routed over without negative impact to the applications traversing the path. The most common exception to this rule is virtual machine migration traffic such as VMWare vMotion; technologies such as Cisco Overlay Transport Virtualization exist for remedying these exceptions.

l3-core

Moving to a layer 3, route-centric design, also provides more options for protocol selection in how you transport your traffic from one network segment to another. Most common small and medium size networks will be sufficiently supported with basic OSPF or EIGRP implementations while larger enterprise and service provider networks will often rely more heavily on BGP and IS-IS. Your choice of protocols should be influenced by administrative experience, networking vendors involved, and your requirements for scale.

Proper use of first hop redundancy protocols such as HSRP, VRRP, and GLBP can also provide hardware redundancy at the first hop gateway much like VSS and StackWise – without the inherent risks of putting so many eggs in one proprietary basket.

Finally, depending on your route design and configuration, relying on layer 3 protocols for redundancy and transport allows for load balancing traffic across multiple paths. Equal cost and unequal cost load balancing can allow for full utilization of all network links without sacrificing ports and hardware for failover/blocked paths.

Summary

The topologies, protocols, and designs mentioned above are all only examples of tools that should be readily available in every networkers’ toolbox. Hopefully, this article has sparked some ideas that will lead to building a more redundant, better-performing network.

Routing Around Internet Congestion

Around 9:30AM EST, we started experiencing some choppy and dropped phone calls through our own internal SIP based phone system, and shortly after began receiving customer reports of slow and limited access to internet resources. After some investigation (20-30 minutes of troubleshooting) we determined that the access problems were isolated to one of our three major internet peers, Level3 communications. We immediately disconnected from Level3 to minimize the impact to us and our and customers, and this resolved most connectivity problems over the next few minutes while other networks began to make routing changes. Fortunately, our other ample upstream paths carried our full traffic load without issue.

Despite the change we made, Level3 is one of the largest and most utilized carriers in the world. As a result, many external networks still rely on Level3 as a transit or intermediary path to reach one of our remaining peers (Cogent, TWTelecom) which results in poor performance for some users. 

At this time, Level3 is still showing signs of loss in the Midwest and we are still de-peered. We have had a ticket open with Level3 since early this morning, and we are poking our contacts for updates as they become available. We will be monitoring throughout the evening, and re-peering when we have confirmation that things are stable.

So while it’s common to have multiple paths to reaching the rest of the internet, does your IT partner really have the ability to respond quickly and efficiently to a critical failure? If you’re not confident in the answer to that question, give us a call and you will be. 

Mail Routing Fundamentals

Mail Routing Fundamentals

E-mail can be accessed multiple ways from the e-mail program installed on your desktop to the web application you use when you are traveling.  Even though these methods of accessing e-mail have a different interface their method for sending e-mail is very similar. The process of directing a message to the recipient’s host is called routing. Apart from finding a path from the sending site to the destination, it involves error checking as well as potentially virus and spam filtering. There are many processes, systems and protocols involved in the successful delivery of one e-mail message. Sometimes these messages arrive at their destination almost immediately, but sometimes they may be delayed or even rejected somewhere in the process. We are going to discuss the basic process for e-mail delivery and look at a few reasons that messages can be delayed or rejected along the way.

When you click the “Send” button your e-mail client initiates a two-way conversation with your e-mail provider’s SMTP (Simple E-mail Transfer Protocol) servers over standard SMTP ports – TCP ports 25 and 587.  SMTP servers only understand very basic commands, so your mail client must initiate this connection using only commands that the SMTP server can interpret. The most common SMTP commands are:

[table]
SMTP Command, Description
HELO / EHLO, This command is used to identify the sender (client) to the SMTP server. EHELO is a newer command than HELO and includes more functionality\, however most servers still accept either command as an introduction.
MAIL FROM, Specifies the e-mail address of the sender.
RCPT TO, Specifies the e-mail address of the recipient.
DATA, Starts the transfer of the actual data (body text\, attachments etc).
RSET (RESET), Specifies that the current mail transaction will be aborted.
VRFY (VERIFY), Asks the receiver to confirm that the argument identifies a user or mailbox.
HELP, This command causes the server to send helpful information to the client.
QUIT, Quits the session.
AUTH LOGIN, Initiates SMTP Authentication when sending
[/table]

The e-mail client sends these commands to the mail server and the server replies with numerical codes and additional information after the numerical code. These reply codes tell the client if the previously sent command succeeded or failed.  Commands and replies are composed of characters from the ASCII character set. EHELO is a newer command than HELO and includes more functionality, however most servers still accept either command as an introduction.

Once the server connection is initiated it will then authenticate your username and password. SMTP authentication is initiated with the AUTH LOGIN command. The server will then responds requesting your username, 334 VXNlcm5hbWU6, which is the phrase Username: in base64 encoding. You then send your base64 encoded username, the server then responds with 334 UGFzc3dvcmQ6 which is Password: in base64 encoding. You then send your base64 encoded password and should receive a 235 Authentication Succeeded message. If the e-mail server accepts your message it will proceed to the MAIL FROM command. This process is done inside your mail client, however with some know-how and command prompt this can be done manually as well.

The output of a telnet session will look something like this:

EHLO example.mailserver.com
250-server1.exampledomain.com Hello [1.1.1.2]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250-STARTTLS
250 HELP
AUTH LOGIN
334 VXNlcm5hbWU6dXNlcm5hbWUuY29t
334 UGFzc3dvcmQ6bXlwYXNzd29yZA==

235 Authentication succeeded

You will know if your message is rejected by your SMTP server and why by looking at the undeliverable message that you receive back at the e-mail account you sent the message from. An undeliverable message will look something like this though the information and formatting will vary based on the provider. It is also possible that the message will be rejected before ever reaching your SMTP server, which will appear in your browser as a pop-up notification showing the error. (see error section below)

If you are using an outside company as a smart host the rejections that you will see should be checked for the rejecting server which could now be your own internal mail server, the smart host’s server or the recipient’s servers.

Every domain that is used for e-mail, datayardworks.com for example, needs to have an MX-record (Mail Exchanger Record) in order to send and receive e-mail. An MX-record is a DNS-entry that tells the sending SMTP server where to deliver the e-mail for a domain. Your MX record will be determined by where your mail is hosted, each company has a different MX record, larger companies will have several.  An MX record looks like this:

[table]

HOSTNAME, TTL, TYPE, PRIORITY, Host
datayardworks.com, 3600, MX, 10, mail.donet.com.
[/table]

In our example the hostname tells querying SMTP servers that e-mail addressed to someone @datayardworks.com should be delivered to the server ‘mail.donet.com’.  This requires that ‘mail.donet.com’ exists (either as a CNAME or an A-record) in the authoritative zone.

There are two numeric values in the above record. The first one is 3600, which is the TTL (Time to Live). This record tells other DNS-servers that it is OK to cache the above record for up to 3600 seconds (or one hour). The second numerical value is ’10’ and is the MX-record priority. In this example, it doesn’t matter, as we only have one record, but if we were to have multiple MX records, it would determine the priority order of the servers. If the first one is unresponsive, the second one will be used in order of priority with the lowest priority being used first (example: a record with the priority of 10 would be used before the record with a priority of 30). In a server with multiple MX records you would see this:

[table]
HOSTNAME, TTL, TYPE, PRIORITY, Host
google.com., 14400, MX, 20, ALT1.ASPMX.L.GOOGLE.COM.
google.com., 14400, MX, 30, ASPMX4.GOOGLE-MAIL.COM.
google.com., 14400, MX, 30, ASPMX5.GOOGLE-MAIL.COM.
google.com., 14400, MX, 30, ASPMX2.GOOGLE-MAIL.COM.
google.com., 14400, MX, 30, ASPMX3.GOOGLE-MAIL.COM.
google.com., 14400, MX, 20, ALT2.ASPMX.L.GOOGLE.COM.
google.com., 14400, MX, 10, ASPMX.L.GOOGLE.COM.
[/table]

As you can see, there are multiple servers listed for the same domain (google.com). They all have the same TTL, 14400 seconds (four hours). The difference is the priority and the server. The sending server would try to deliver e-mails to the servers in the order from lowest priority value to highest until it finds one server who accepts the connection. This means if the server ‘aspmx.l.google.com’ goes down or is too busy to accept the connection the message will then try to deliver to ‘alt2.aspmx.l.google.com’ and ‘alt1.aspmx.l.google.com’ since these two servers have the same priority they will be chosen at random. In the case of large providers with multiple domains relaying on their servers for SMTP they are likely using a clustered mail environment which means many servers with different IP addresses can send mail from the same host, such as ‘mail.donet.com’.

If you want to know how your mail server might determine the DNS record read my article on DNS resolution here: https://www.datayard.us/blog/article/domain-name-resolution-introduction

Once your SMTP server has determined the server for the mail to be sent to it will then attempt to make a connection with that server using the same basic commands as your mail client used to connect to your SMTP server. The message will first pass through the mail provider’s content filtering servers before even making it to their SMTP server where it may be rejected. There are a few reasons that mail can be rejected or even delayed by a provider’s content filters. First, if a content filtering server deems your mail content to be junk mail it will likely tag the message as junk and deliver it according to the domains preferred settings, most domains choose to have spam messages delivered to a specific “Junk Mail” folder.

If you are on a blockhole list you may receive undeliverable email notifications stating that the mail server you use is on a blackhole list. The first course of action to get removed from the blackhole list is to resolve any outstanding issues with your mail server. If this mail server is provided by an outside company or managed by your IT department you want to send them the bounceback you received so they can address the issue.  If you manage this mail server you can check some of the major blacklists for more information and to see if your server is listed in their database.

Once the server has determined that your mail is authorized and not on a blackhole list it will attempt to deliver this message to the recipients’ mailbox. The mail server will check to make sure that the recipient address specified does exist as a mailbox on the server. As long it does exist, the message will be delivered to the recipient’s mailbox for their mail client to retrieve. However, the message can again be rejected at this point. For example if the recipient email address is spelled incorrectly, has been deleted or even if the users’ mailbox is full you may receive a bounceback stating why your message was rejected.

Alternate delivery methods and setting up your own mail server

A smart host is an outside mail server that accepts inbound SMTP and routes it out again however the point of sending your mail to one of these is that you may offload the task of routing and retrying mail to another location. Directing your outbound traffic to an SMTP host at your ISP will allow your outgoing internet connection to free up much quicker than sending through your own network. A relay SMTP server is usually the target of a DNS MX record that designates it, rather than the final delivery system.  The relay server may accept or reject the task of relaying the mail in the same way it accepts or rejects mail for a local user.  If it accepts the task, it then becomes an SMTP client, establishes a transmission channel to the next SMTP server specified in the DNS and sends it the mail.  If it declines to relay the message a 550 error should be returned to the sender.

Common Rejections

Bounceback
From: Mail Delivery System <[email protected]>
To: <[email protected]>
Subject: Undelivered Mail Returned to Sender
Attachment:
file-2 , 416 Bytes
file-3 , 5.9 KBytes
file-4 , 16.2 KBytes
This is the mail system at host smtp.donet.com.

I’m sorry to have to inform you that your message could not
be delivered to one or more recipients. It’s attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<[email protected]>: host mail.donet.com[64.56.96.133] said: 554
5.7.1 <[email protected]>: Relay access denied (in reply to RCPT TO
command)

From this delivery failure notice you can find the following information

1. The name of the server that is rejecting your message (this message can either be rejected by your server or by the recipients’ mail server):  This is the mail system at host smtp.donet.com, in this case the message was rejected by smtp.donet.com.

2. The cause of the rejection: said: 554 5.7.1 <[email protected]>: Relay access denied (in reply to RCPT TO command, this particular rejection is rejecting relay access which means you do not have permission to send your mail through this mail server.

SMTP errors from a Mail client

No Relaying Allowed: This rejection message tells you that the credentials (username and password) that you are sending to the e-mail server are not authenticating. You should check the settings in your e-mail client to ensure your username and password fields are not blank, the correct credentials are entered and that you have outgoing server authentication enabled.

<[email protected]>: host mail.donet.com[64.56.119.17] said: 554

    5.7.1 <[email protected]>: Relay access denied (in reply to RCPT TO

    command)

Could not connect to SMTP server: This rejection tells you that the connection from your e-mail client to your e-mail providers’ server failed. You should check the SMTP server name and ensure that your SSL types match the port numbers specified and that you have no firewall or antivirus program that is blocking outbound connections.

Greylisting

Greylisting is a way mail providers attempt to limit the amount of spam that gets through. Grey listing checks the legitimacy of an e-mail message by temporarily rejecting the incoming message with a 451 error.  The sending SMTP server will recognize this as a temporary rejection and attempt to re-send the message. If there is a prior relationship between the sending server and the recipient server, the message will get delivered without any issue. At the same time the 451 error is sent a triplet is recorded in the Greylisting database as an unconfirmed triplet.

A triplet consists of:

  • IP address of the sending server
  • Sender e-mail address
  • Recipient e-mail address

This is where delays come in, this response can take up to 24 hours depending on the settings of the server and how it is set to respond. If you have a small one server mail setup then the greylisting process generally doesn’t cause a significant delay. However, in a clustered mail environment where many different servers, with different IP addresses, can be used to send and receive mail the delays can be significant. If the rejected message is re-sent and all information matches the prior unconfirmed triplet then the message will pass through greylisting and be delivered. If any of the three items do not match then it will be seen as a new message and will begin the greylisting process again.

RBLs (Realtime Black Hole Lists)

There are specific companies who run blackhole lists and those lists are used by mail providers to determine when a server’s traffic has become unfavorable as a way to protect them from accepting unfavorable traffic. It’s common for a server’s mail IP address to temporarily end up on a public blackhole list. It could be because of the overall volume of mail that is coming from that server, messages that seem to have characteristics of junk in them, or that list has received messages to spam traps it operates.

How can you identify if your mail server is blackholed?

Spamhaus (http://www.spamhaus.org/)

MXtoolBox (http://www.mxtoolbox.com/blacklists.aspx)

Sorbs (http://www.sorbs.net/

http://multirbl.valli.org/lookup/

Remote Server Administration with PowerShell

Remote Server Administration with PowerShell

Your RDP Client is going to get straight up jealous

Intro

In the past remote Window’s administration has been limited to RDP, Citrix, DRAC/KVM, VNC, or the command line using PsExec. Developing scripts to remotely connect to hosts and gather data utilizing these legacy technologies became very cumbersome. One of the best things about PowerShell is the ability to manage hosts remotely quickly and efficiently. In a previous post I wrote about Local Server Administration with PowerShell which included quick glances at cmdlets for managing Windows features, firewall administration, network connectivity diagnostic tools, HTTP and HTTPS diagnostic cmdlets, and scheduled job management. The same cmdlets are available remotely; allowing us to configure servers without the need to RDP and executing PowerShell commands locally.

First we need to understand Windows Remote Management (WinRM) as PowerShell uses this technology to remotely connect to servers and other infrastructure. WinRM provides a common way for systems to access and exchange management information across an IT infrastructure. WinRM was introduced with the Windows Vista/Server 2008 kernel and can retroactively be installed in earlier versions by installing Windows Management Framework 2.0.

Security Implications with WinRM

Before implementing PowerShell Remoting it is important to understand the security implications. WinRM, by default, will use the HTTP protocol on port 5985 to communicate with a remote host. Microsoft Negotiate Authentication will decide to authenticate using either Kerberos and NTLM. The preference will be Kerberos, which will be used in the same AD domains or domains with trust relationships.

If you analyze packet captures during the WinRM process over HTTP, and follow the entire TCP stream, you will see that the content is encrypted with the content type of HTTP-SPNEGO-session-encrypted for NTLM v2 and HTTP-Kerberos-session-encrypted for Kerberos. Here is an example of a WinRM session using Kerberos as the local and remote computers were on the same domain:

WinRM-Kerberos-Capture

The actual password is never sent over the wire and the only way that a password could be decoded is if the Security Accounts Manager (SAM) on the remote machine had already been compromised. The hash that is stored in the Security Accounts Manager is never actually sent and cannot be decoded. Even if there was someone snooping packets on the network they wouldn’t see anything in clear text and therefore an exploit is extremely unlikely. With this data we conclude that WinRM is encrypted by default and that configuring WinRM for HTTPS would be an added layer of security.

Configuring PowerShell Remoting

Configuring PowerShell Remoting was originally difficult and time consuming because administrators had to manually create the WinRM listener, set the service to automatically start, enable PowerShell session configurations, and troubleshooting authentication. Now, all of this is configured out of the box; primarily because the majority of new management tools application require this to properly function. Additionally, several cmdlets have a ComputerName parameter that is supported to identify which computer you want to run the command on. Here is how you can enable PowerShell remoting on previous versions of Windows.

Configure PowerShell Remoting with HTTP

Starting in PowerShell 2.0 you can enable PowerShell remoting with a single command:

Enable-PSRemoting

The Enable-PSRemoting cmdlet runs the Set-WSManQuickConfig cmdlet, which performs the following tasks:

  • Starts the WinRM service.
  • Sets the startup type on the WinRM service to Automatic.
  • Creates a listener to accept requests on any IP address.
  • Enables a firewall exception for WS-Management communications.
  • Registers the Microsoft.PowerShell and Microsoft.PowerShell.Workflow session configurations, if it they are not already registered.
  • Registers the Microsoft.PowerShell32 session configuration on 64-bit computers, if it is not already registered.
  • Enables all session configurations.
  • Changes the security descriptor of all session configurations to allow remote access.
  • Restarts the WinRM service to make the preceding changes effective.

Configuring PowerShell Remoting with HTTPS

If you would like this additional layer of security, the process is slightly more complex than the Enable-PSRemoting command. The first thing you need to do is obtain an SSL cert for the remote server or workstation. You can script out the issuing of certificates by using Certificate Enrollment Web Service and Certificate Enrollment Policy Web Service in Windows Server 2012 R2. I would also recommend deploying a duplicate of the Web Server certificate template and modifying the settings to meet your organization’s needs. Here is an example of how you can obtain these certificates using PowerShell.

PS C:\> $uri = “https://yourca.domain.tld/Policy/service.svc”

PS C:\> $fqdn = “remotehost.domain.tld”

PS C:\> $subject = “[email protected], C=US, S=Ohio, L=Dayton, O=DataYard, OU=TechOps,CN=$fqdn”

PS C:\> $netbios = “remotehost”

PS C:\> $creds = Get-Credential

PS C:\> Get-Certificate -Template “WinRM” -Url $uri -DNSName $fqdn,$netbios -CertStoreLocation cert:\LocalMachine\My `

-SubjectName $fqdn `

-Credential $creds

You can issue these commands from the remote host in the same domain. If the certificate request is issued, then the returned certificate is installed in the machine’s personal certificate store. If the request is made pending, then the request is installed in the machine’s request store. Once you have the certificate installed you will need to obtain the certificate thumbprint for the issued certificate and copy it to your clipboard:

PS C:\> $thumbprint = (gci cert:\LocalMachine\My | Where {$_.Subject -eq $subject}).Thumbprint

PS C:\> $thumbprint | clip

In order to create the HTTPS WinRM listener you will have to use a command prompt and not PowerShell as you will receive an error “Error: Invalid use of command line. Type “winrm -?” for help”. In the following example we also delete the default HTTP listener:

C:\> winrm create winrm/config/Listener?Address=*+Transport=HTTPS  @{Hostname=”remotehost.domain.tld”;CertificateThumbprint=”E1ED1974E00A9789DFDD417A690F2C1480FD224A”}

C:\> winrm delete winrm/config/Listener?Address=*+Transport=HTTP

Now that WinRM is actively listening on 5986 we need to modify the firewall rules in PowerShell to reflect our changes. We strongly suggest restricting the remote addresses able to access this service for an additional layer of security:

PS C:\> Get-NetFirewallRule | Where {$_.Name -match “WINRM-HTTP-In-TCP”} | Disable-NetFirewallRule

PS C:\> New-NetFirewallRule -Name “WINRM-HTTPS-In-TCP” -DisplayName “Windows Remote Management (HTTPS-In)” `

-Direction Inbound -Action Allow -Protocol TCP -LocalPort 5986 -RemoteAddress 10.0.0.0/24

Utilizing PowerShell Remoting

Now that we have PowerShell Remoting configured there are several options to remotely connect. These options include the ComputerName parameter, Invoke-Command, PSSession capabilities, and implicit remoting.

ComputerName Parameter

You can run commands on a remote computer by using the ComputerName parameter. Although many cmdlets now offer a ComputerName parameter, there are a limited number of cmdlets with this parameter available. When you use this parameter Windows PowerShell creates a temporary connection that is used for the command and is then closed. Here is an example regarding memory consumption on a remote host:

PS C:\> Get-Process -ComputerName remotehost.domain.tld | sort WorkingSet | select -last 5

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName

——-  ——    —–      —– —–   ——     — ———–

   4658      93   100208     185996   669            3612 OUTLOOK

   1358     130   178744     214380   631           11912 VpxClient

   5917     104   136148     220148   772            5480 OUTLOOK

    463      32   268112     231492   780           10780 javaw

   7114     108   175692     269376   878            3528 OUTLOOK

Here are all of the commands with the ComputerName parameter available:

PS C:\> Get-Command | ForEach { if ($_.Parameters.Keys -Contains “ComputerName”){$_}} | select name

Name

—-

Add-Computer

Clear-EventLog

Connect-PSSession

Enter-PSSession

Get-EventLog

Get-HotFix

Get-Process

Get-PSSession

Get-Service

Get-WmiObject

Invoke-Command

Invoke-WmiMethod

Limit-EventLog

New-EventLog

New-PSSession

Receive-Job

Receive-PSSession

Register-WmiEvent

Remove-Computer

Remove-EventLog

Remove-PSSession

Remove-WmiObject

Rename-Computer

Restart-Computer

Set-Service

Set-WmiInstance

Show-EventLog

Stop-Computer

Test-Connection

Write-EventLog

I primarily utilize the ComputerName parameter for quickly identifying issues with servers in our local domain or dev environments. Since invoking a cmdlet only creates a temporary connection, any time you need to perform several action items on a remote host I would recommend to utilize PowerShell Sessions (PSSessions).

Invoke-Command

Invoke-Command is perhaps the most versatile way to remotely connect to hosts with PowerShell. You can use this command to execute scriptblocks locally, remotely, through a remote session with or without arguments, or through a connection endpoint URI. Invoke-command can also be used to invoke the same scriptblock on several different servers; we can also identify a filepath to predefined scripts to run remotely. You can also use Invoke-Command on a local computer to evaluate or run a string in a script block as a command. Windows PowerShell converts the script block to a command and runs the command immediately in the current scope, instead of just echoing the string at the command line.The Invoke-Command cmdlet runs commands on a local or remote computer and returns all output from the commands, including errors.

Here is an example of grabbing the configured DNS servers on all interfaces for several servers in your domain:

PS C:\> $server1 = “server1.domain.tld”

PS C:\> $server2 = “server2.domain.tld”

PS C:\> $server3 = “server3.domain.tld”

PS C:\> Invoke-Command -ScriptBlock { Get-DNSClientServerAddress } `

>> -ComputerName $server1,$server2,$server3

>>

InterfaceAlias               Interface Address ServerAddresses                                PSComputerName

                             Index     Family

————–               ——— ——- —————                                ————–

Ethernet                            12 IPv4    {192.168.1.101, 192.168.1.102}                 server1.domain.tld

Ethernet                            12 IPv4    {192.168.1.101, 192.168.1.102}                 server2.domain.tld

Ethernet                            12 IPv4    {192.168.1.101, 192.168.1.102}                 server3.domain.tld

PSSession Capabilities

A PSSession is a persistent PowerShell Session on a local or remote computer. Use a PSSession to run multiple commands that share data, such as a function or the value of a variable. Commands are executed inside of a PSSession using the Invoke-Command. Through PowerShell session options and New-PSSession parameters we can connect to a remote machine to perform the vast majority of our actions, and we can develop very elaborate scripts by connecting to and storing persistent PowerShell sessions.

Invoke-Command and PSSessions

Let’s start with the basics of creating a new session, storing it in a variable, and then issuing a command:

PS C:\> $session = New-PSSession remotehost.domain.tld

PS C:\> Invoke-Command -Session $session -ScriptBlock { (Get-NetAdapter).Name }

vEthernet (Internal Ethernet Port Windows Phone Emulator Internal Switch)

vEthernet (Realtek PCIe GBE Family Controller – Virtual Switch)

Production Interface (95)

That is a very basic example of capturing data from a remote host. Here is a little bit more of an advanced example of how we can capture all IPs from a remote host, and create A records on our domain controller (a different remote host) for all the server IPs:

PS C:\> $creds = Get-Credential

PS C:\> $remote_fqdn = “remotehost.domain.tld”

PS C:\> $dc = “domaincontroller.domain.tld”

PS C:\> $session1 = New-PSSession $remote_fqdn -Credential $creds

PS C:\> $session2 = New-PSSession $dc -Credential $creds

PS C:\> $ips = Invoke-Command -Session $session1 -ScriptBlock {

(Get-NetIPAddress | Where {$_.InterfaceAlias -notmatch “Loopback”}).IPAddress

}

PS C:\> $ips

10.0.1.42

10.0.0.42

172.16.2.1

192.168.1.108

192.168.1.105

PS C:\> Invoke-Command -session $session2 -ArgumentList $remote_fqdn,$ips -scriptblock {

param ($remote_fqdn, $ips)

$name = $remote_fqdn.split(“.”)[0]

ForEach ($ip in $ips){

#Add-DnsServerResourceRecordA -ZoneName domain.tld. -Name $name -IPv4Address $ip

$ip

}

}

Enter-PSSession

Persistent PowerShell sessions are great for scripting purposes, but troubleshooting on the fly and utilizing Invoke-Command and ScriptBlocks is not a practical solution. However, with the Enter-PSSession command you can connect interactively to a PowerShell session on any properly configured remote system. The Enter-PSSession cmdlet starts an interactive session with a remote computer. During the session, the commands that you execute run on the remote computer, just as though you had a local PowerShell sesssion on it (one thing to note is that you can have only one interactive session at a time). Here is an example of connecting to a remote computer:

PS C:\> Enter-PSSession remotehost.domain.tld -Credential (Get-Credential)

In the past we had to launch an RDP client, enter in credentials, wait on the user profile and startup programs to load, and launch a PowerShell session. These actions took much longer than Enter-PSSession. The Enter-PSSession cmdlet is my personal favorite for PowerShell remoting because it is a subsitute for remote desktop and decreases the administrative time to connect to remote hosts.

Implicit Remoting

PowerShell remoting gets even better as you can take it one step further and create PowerShell modules from your PowerShell sessions you create. For example, I created modules for managing our corporate exchange servers (Exchange 2013):

PS C:\> $creds = Get-Credential

PS C:\> $uri = “https://exchange.domain.tld/PowerShell”

PS C:\> $exchange = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionURI $uri -Authentication Basic `

-Credential $creds -SessionOption (New-PSSessionOption -NoMachineProfile -SkipCACheck -SkipCNCheck -SkipRevocationCheck)

PS C:\> Export-PSSession -Session $exchange -CommandName * -OutputModule DataYardExchange -AllowClobber

Managing remote infrastructure becomes significantly easier as we build custom modules for our environment. Implicit remoting and PowerShell modules will not store your password(s) and will prompt you accordingly upon importing the module. Here is an example of modifying DataYard’s corporate exchange environment with the module we recently created:

PS C:\> Import-Module DataYardExchange

PS C:\> Set-Mailbox eric.wright -EmailAddresses @{add=”[email protected]”}

PS C:\> (Get-Mailbox eric.wright).EmailAddresses

smtp:[email protected]

smtp:[email protected]

Another way to take advantage of implicit remoting is by utilizing the cmdlet Import-PSSession. Rather than creating a module we can create a PSSession and then use the Import-PSSession cmdlet to implicitly remote to the server:

PS C:\> $creds = Get-Credential

PS C:\> $uri = “https://exchange.domain.tld/PowerShell”

PS C:\> $exchange = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionURI $uri -Authentication Basic `

-Credential $creds -SessionOption (New-PSSessionOption -NoMachineProfile -SkipCACheck -SkipCNCheck -SkipRevocationCheck)

PS C:\> Import-PSSession $exchange

PowerShell Web Access

The last feature to discuss is PowerShell Web Access (WindowsPowerShellWebAccess). This allows you to execute PowerShell commands through an encrypted website from any web browser. PowerShell Web Access can be configured using the following commands:

PS C:\> Install-WindowsFeature -Name WindowsPowerShellWebAccess -IncludeManagementTools

PS C:\> Install-PswaWebApplication -webApplicationName pswa -UseTestCertificate

PS C:\> Add-PswaAuthorizationRule -UserName domain\user -ComputerName localhost -ConfigurationName Microsoft.PowerShell

PS C:\> start iexplore.exe https://localhost/pswa

In your favorite web browser (Chrome, Firefox, Safari, or IE) you can utilize PowerShell Web Access from anywhere in the world. This is a screenshot of PowerShell Web Access from Safari on an iPhone:

 IMG_6112

I strongly suggest to implement additional layers of security by restricting the IP space that is able to access this feature as well as utilizing additional authentication schemes such as Windows Authentication, Basic Authentication, and/or client certificate verification at the IIS level.

Conclusion

There are many ways to connect to and manage remote systems, PowerShell becoming the most consistent way to manage Window’s infrastructure. As you can see the functionality of persistent PSSessions and modules can become very elaborate. Remote management techniques for Windows’ infrastructure has been constantly improving and I look forward to the incremental improvements.

BGP, ASICs, and IPv4 at scale – Why the Internet is (or was) Broken

BGP, ASICs, and IPv4 at scale – Why the Internet is (or was) Broken

On Tuesday August 12, 2014, the global IPv4 Internet routing table exceeded 512,000 routes. This is an important number for several very common routing platforms across multiple networking vendors, and as a result many networks across the globe experienced interruptions and performance issues. While some articles have reported on the scope of impacted carriers and services, I’ll discuss the technologies and limitations that led to this failure.

To understand the source of this failure, there are two critical pieces of Internet technology to grasp. First is BGP, the way routers around the Internet exchange information about what networks they know about, and how to reach them. Second, and the more immediate cause of the recent failure, are the mechanics or implementation of routing and destination lookups in high performance, carrier grade routers and switches.

Border Gateway Protocol

The common language that all carrier and ISP routers speak throughout the Internet is crucial to the delivery of the interconnected, distributed network that we all use today. BGP dates back to 1995 and even earlier in older iterations, and is still in use today, albeit with many adjustments and new features. At the core of BGP is a common method for routers and network devices to tell each other about networks, and their relationship to those networks. A critical component to BGP operation is the Autonomous System or ASN, which defines a network and its associated prefixes (known routes) as viewed from the perspective of the BGP protocol.

as14131

BGP speaking routers across autonomous systems peer with each other in a variety of arrangements such as dedicated peering points or direct connections, and exchange their known routes or a subset of known routes with each other. This exchange also includes a large number of attributes (Origin, As_Path, Next_Hop, Local_Pref, communities…) associated with each prefix. As you might expect, at a global scale this table is quite large – and each time a new autonomous system (typically an ISP or larger enterprise) pops up, or an existing autonomous system further distributes its network, this table continues to grow.

multi-as

TCAM, FIB, ASIC what?

Routers in larger scale networks need to store, reference, and update this large global BGP routing table quickly and frequently. So where does all of this information get stored? The answer isn’t simple, and varies wildly by vendor and hardware, but in large scale network hardware, the general implementation looks like this.

ribfib

Routers handling BGP (and other routing protocols) maintain routing tables and store known routes in a Routing Information Base or RIB specific to that protocol. The best known route for a destination is then inserted into the Forwarding Information Base or FIB, which the router will use for forwarding lookups on packets due for processing. While the RIB is typically stored in general system memory, of which space is often readily available, the FIB is stored on TCAM residing on ASICs. Application Specific Integrated Circuits are the secret sauce to high performing networking appliances, enabling fast routing and switching above and beyond what traditional x86 based processing and memory can achieve. The ASICs on modern routers and switches contain a fixed amount of TCAM, enabling the low latency lookups and updates required for routing and switching.

So what happened and what does it have to do with TCAM? Going back to the original issue with our picture of routing tables and TCAMs in mind, you can see how the growing IPv4 Internet might be a problem for platforms with a fixed amount of space available. As it so happens, some of the most commonly deployed BGP speaking routers and switches over the last decade, including the Cisco 6500 and 7600 platform, have had default TCAM allocations allowing for a maximum of 512,000 IPv4 entries. On many platforms, as soon as TCAM space reaches its limit, failures can occur ranging from reduced performance to catastrophic failure. On Tuesday that 512,000 IPv4 limit was reached and the straw that broke the camel’s back, so to speak, was Verizon owned ASN 701 & 705. While the change, the apparently inadvertent advertisement of thousands of new routes, was quickly reversed, routers across the Internet saw performance issues and failures for hours due to the nature of BGP propagation.

 

Even though we are again under the 512k IPv4 route marker, it’s only a matter of time until we cross that barrier and beyond. This expansion of IPv4 tables is ultimately at the fault of no one, and should only be seen as the expected progression and growth of the “Internet of Things”. It’s hard to predict what type of scale future networks will need to adapt to, and in a world of fast expanding virtualization and cloud services it can be easy to forget that the underlying network transport behind these services must scale to cope with demand. Hold on to your hats over the next few weeks as we inevitably cross the 512k barrier again, and see who accepted this as a wake-up call to upgrade where necessary, and who didn’t.