Today — Friday, July 25, 2014 — is the 15th Annual System Administrator Appreciation Day!

Please join us in thanking the server, network, and data center people who keep all our technical ships running smoothly — no matter what time it is, day or night, weekday, weekend, or holiday, 365 days a year.

We designed and printed a custom 11″ x 14″ letterpress poster to commemorate the day. If you’d like a copy just let us know via email ([email protected]) and we’ll get it to you. This was a limited production run poster so we may run out soon.

P.S. There is a movie reference hidden somewhere on the poster. Can you identify the reference and the movie?

There have been a number of new features introduced to Remote Desktop Services in Windows Server 2012 that make the deployment of desktop virtualization an attractive option for businesses looking to enhance the experience of their desktop users.  This post will briefly review the concept of desktop virtualization and the two primary deployment models but will focus primarily on session-based desktop virtualization architecture.  This post is intended for administrators or engineers considering desktop virtualization deployment options.

Desktop virtualization allows you to abstract the desktop and application environment from the endpoint device, centrally manage and deploy applications and offer flexibility to your end-users. There are two primary deployment models: session-based and virtual machine-based desktop virtualization. Regardless of which deployment model (or combination) you choose, you can realize a number of benefits. Users can securely access their corporate desktop and applications from a thin-client, from a tablet in a coffee shop or from their computer at home.

Instead of deploying full desktop computers which require you to maintain support agreements and pay for OS and application licenses, you can simply deploy thin client devices with no moving parts to each desk. By providing a standardized desktop environment and removing hardware support from the equation, you also reduce support overhead. Centralized OS and application management improves security and reduces the risk of malware infections or other compromises due to misconfiguration.

Provisioning a new desktop environment becomes a simple task of setting up a new user and placing a thin client device on their desk. Centralizing the desktop environment in the data center, or the cloud, also allows you to reduce licensing costs as well as deploy or remove resources based on user-demand.

RDS Role Services

The Remote Desktop Services server role in Windows Server 2012 is comprised of multiple role services that can be deployed together for small deployments or spread across multiple servers for fault-tolerance and scalability.  The six available roles services are described below (some are supplemental or not required for the deployment of session-based desktop virtualization):

Session Host: This is one of the core role services and allows a server to host session-based virtual desktops or RemoteApp applications (more on this later) – this role service was formerly known as terminal services in legacy versions of Windows Server.

Virtualization Host: This is another core role service that allows a server to host virtual machine-based virtual desktops by integrating Remote Desktop Services with the Hyper-V server role.  This is the counterpart role service to the session host role service and is required when you wish to support user connections to either pooled virtual desktops or personal virtual desktops.

Licensing: This is a core role service that is required in all deployments and allows administrators to centrally manage the licenses required for users to connect to session host or virtualization host servers.  This role can often be placed on a host running additional Remote Desktop Services role services due to its lightweight nature.

Connection Broker: This role service is critical to enterprise deployments because it allows administrators to build high-availability by load-balancing incoming connection requests across available session host or virtualization host servers. The connection broker role service keeps track of sessions and routes connection requests to available hosts based on host load – in the case of existing connections, the connection broker will re-connect the user to their existing session. In Windows Server 2012, this role service now supports Active/Active operation allowing you to meet scalability requirements in addition to fault-tolerance in the connection broker layer of the infrastructure.

Gateway: This is a supplemental role service that allows users to securely connect to their session-based desktops, virtual machine-based desktops or RemoteApp published applications on an internal corporate network from any device.

Web Access: Another supplemental role service that allows users to securely connect to their session-based desktops, virtual machine-based desktops or RemoteApp published applications through a web browser.

VDI Enabling RDS Features

RemoteFX: RemoteFX provides end-user experience enhancements to the Remote Desktop Protocol that allow RDP to operate as a truly viable desktop virtualization platform.  A few of the RemoteFX components that provide critical capabilities when it comes to desktop virtualization are full USB redirection support for all desktop virtualization workloads, vGPU hardware acceleration (the ability to use a physical GPU in Hyper-V hosts to accelerate host-side rendering of graphics intensive content), intelligent transport enhancements that introduce UDP support to allow fuller/smoother desktop experiences over low bandwidth/high-latency links and finally multi-touch capabilities to support gestures (e.g. pinch, zoom, swipe).

User Profile Disks: The introduction of User Profile Disks greatly enhances the ability for users to personalize pooled virtual machines removing the necessity to implement more complex roaming profile or folder redirection methodologies. This technology is really only applicable to virtual machine-based deployments but is worth mentioning as it makes deployment of virtual machine-based VDI much simpler and more robust.

Fairshare Resource Management: New to Windows Server 2012, the session host role service now introduces “fair share” resource management for network I/O, storage I/O and CPU.  This resource management ensures that no single user can adversely impact users on the same host by ensuring each session is allocated an equal share of the resources described above.  This simple enhancement makes the user experience in session-based deployments much smoother and consistent and eases the burden on administrators when deploying additional RD Session Host servers to the session collection.

Deployment Models

Now that we’ve gone through a brief review of Remote Desktop Services and its various role services, let’s focus on examining the session-based deployment model. First and foremost, let’s address why the session-based deployment model is so attractive when compared to the virtual machine-based deployment model.

The “traditional” approach to VDI is to utilize virtual machine-based desktop virtualization – when using Microsoft Remote Desktop Services this generally means one or more RD Session Broker instances and a number of dedicated Hyper-V hosts running the RD Virtualization Host RDS role service.  In this model, users connect to a virtual machine either from a pool of identically configured virtual machines (called pooled virtual desktops) or a personal virtual desktop that is dedicated to a specific user. The drawback is that this model requires standing up dedicated virtual machines per user which requires significantly greater resources than a session-based deployment.  In addition to the increased resource demands, administrators must ensure there are adequate pooled virtual desktops or personal virtual desktops in place to meet user load requirements. You should consider virtual machine-based deployment in those cases where users require significant control over their desktop environment (e.g. administrative rights, installing applications, etc.).

In the session-based model, we continue to rely on session broker instances to load-balance incoming connections but replace the dedicated Hyper-V hosts running the RD Virtualization Host role service with servers running the RD Session Host role service. In this model, users connect to a centralized installation of a desktop running on one or more session host servers (when session hosts are configured in a farm, it is referred to as a session collection) only receiving the user interface from the session host server and sending input commands to the server. When the RD Session Host role service is installed, fairshare resource management ensures storage I/O, network I/O and CPU are evenly distributed to all users; in addition, Windows automatically configures processor scheduling to prioritize performance of user applications over background processes. This is a good reason to avoid collocating this role service on a server with “traditional” background type services (e.g. IIS, Exchange, etc.). In addition to accessing full virtual desktop environments, applications can be published as RemoteApp programs from the session host servers allowing users to connect to specific applications over Remote Desktop – these applications are rendered in a local window on the client and from the end-user perspective, behave almost identically to local applications.

Sample Session-Based Deployment Architectures

For testing or starting a small session-based VDI deployment, you can install the RD Session Host, RD Connection Broker and RD Licensing role services on a single server through the Quick Start deployment option.  For an enterprise deployment, we would need to design the infrastructure to deal with availability and scaling concerns.  As with any environment, the complexity increases with the availability requirements – a session-based VDI deployment that requires five¬¬¬ nines availability and needs to support a large concurrent user base will be significantly larger and more complex than a solution for a small business with reduced availability requirements. Let’s examine three sample deployment architectures designed to meet the requirements of organizations of varying size and complexity.

Small Business Deployments

For the small business environment that doesn’t have strict availability requirements and only a small number of thin-client devices and remote users, we can deploy a simple infrastructure. For these types of environments, you can deploy the following RDS role services on a single host by using the Quick Start deployment option: RD Session Broker, RD Session Host and RD Licensing.

While a detailed discussion about sizing requirements is outside the scope of this article, this type of environment is generally suitable for 45-50 simultaneous user connections and using standard line of business applications (e.g. Office, Web Browsers, Thick Client Applications) that are not graphics intensive.

Small Business Deployments with Increased Availability

For the small business environment that has increased availability requirements, the deployment becomes only slightly more complex. Windows Server 2012 introduces the Active/Active Broker feature that allows you to provide fault-tolerance and scalability to the connection broker role service without the requirement to implement complex failover-clustering. Instead, you simply combine both connection broker servers under a single DNS entry.

In this architecture, we deploy both the session host and connection broker role services on two identically configured servers, configure Active/Active brokering and register both session hosts in a new session collection. A SQL Server instance is required to allow the connection brokers to synchronize and track sessions. User connections will be evenly distributed across the session host servers – if a single RDS server fails, users will still be able to access their virtual desktop environment.

If this type of environment needs to support the loss of a single RDS server without degradation in the end-user experience, then the previous recommendation of 45-50 simultaneous users should be followed. If some amount of degradation in the end-user experience is acceptable or the business accepts that a reduced number of users will be able to connect in the event of a server loss, this number could be increased to 75-100 simultaneous users.

Enterprise Scale Deployments

For enterprise deployments that have strict availability requirements, need to support a large number of thin-clients and remote users and also require the ability to continue scaling out the environment as the number of users increases, the architecture becomes much larger and more complex.

In this architecture, the connection broker role service is deployed on two dedicated servers in Active/Active Broker mode using round-robin load balancing. For connection broker synchronization and user session tracking, SQL Server is deployed as a clustered failover instance on two dedicated servers to provide fault-tolerance. Microsoft testing shows the connection broker role service operating in Active/Active Broker mode on two servers can maintain reasonable endpoint connection response times (< 2.5 seconds) for up to 2,000 simultaneous connections before end-user observable degradation is encountered.

Fault tolerance and scalability is further accomplished by deploying the session host role service on three dedicated servers registered in a session collection with the Active/Active connection broker farm. User connections will be evenly distributed across the session host servers – if a single RDS server fails, users will still be able to access their virtual desktop environment. To meet scalability requirements, the enterprise can simply add additional session host servers to the session collection to support the increasing number of users.

This architecture also provides fault tolerance in Active Directory Domain Services with two domain controllers deployed on dedicated servers.

If this environment needs to support the loss of a single RDS server without degradation in the end-user experience, then the session host servers should not exceed 66% of their total capacity or about 34 simultaneous connections per session host server. If some amount of degradation in the end-user experience is acceptable or the business accepts that a reduced number of users will be able to connect in the event of a server loss, this number could be increased to 125-150 simultaneous users. As stated before, this environment can continue to scale out to meet user load requirements by simply adding additional session host servers to the session collection and if exceeding 2,000 simultaneous connections, adding additional connection broker servers.

Conclusion

As was pointed out in some of the example architectures, there are a number of factors that can affect the deployment model and architecture you choose and the sizing requirements of the individual role services. You should carefully consider things like application memory requirements, graphic intensive workloads, availability requirements and budget constraints when deciding if desktop virtualization is right for your business.

Every environment is different and not every business will be able to realize cost savings but every business can certainly gain flexibility and enhanced user experience by implementing desktop virtualization. Hopefully this article will help you get you started down the path to providing true flexibility and simplicity to your end-user computing environment.

Last week I had the great opportunity to present at and participate in a panel discussion at FISPA’s Atlanta 2014 conference.  The panel and presentations were focused on the lessons that the presenters and panelists had learned from providing hosted solutions and services to their customers.  This topic suited DataYard very well, as we have been providing hosted services since 1995.  The presentation time was brief, less than five minutes, to allow for questions from the moderator and crowd.  Given the small presentation window, I thought I would elaborate further on some of the points I briefly touched on.

Lesson #1

When I was preparing for this presentation and the corresponding panel discussion, a quote crept into my mind,

“The man who promises everything is sure to fulfill nothing, and everyone who promises too much is in danger of using evil means in order to carry out his promises, and is already on the road to perdition.” – Carl Jung

This quote, albeit a bit theatric, catches your attention and makes you start thinking about the first lesson, Focus…

Hosted services, like e-mail, web hosting, etc., are primarily software driven services.  The installation and configuration of them is something many IT professionals can do quite easily and here is where the danger lies.  After, or even during, the initial deployment of a service it is easy to lose focus on the core service it is intended to provide.  Unlike a physical product, features and core functionality can easily be changed with modifications to software configuration.  This can easily lead to a mutation of a hosted service that was originally intended to provide one thing but now does something very different.

A hosted service must be flexible enough to accommodate new customer needs, but must still provide it’s core functionality.  When considering modifications to a hosted service, it is imperative to consider how it adjusts the core functionality of the service.  Does the adjustment add value to the service or take the service in a different direction?  If it takes the service in a different direction, should a new service be created to provide this functionality?  Is there enough demand for this as a new service?

These are difficult questions to answer, but retaining the “Focus” of the service while answering these questions will help guide it in the proper direction. Don’t lose site of the core focus of the hosted service, when dealing with add-ons and service tweaks.

Lesson #2

Building a hosted service can be both an easy and daunting task all at once.  What type of infrastructure should be deployed for this service? Use an open-source project or a white-box solution?  How much more support will this generate?  How will growth be handled? How big should the infrastructure be? 

I have had that conversation many times and there is a two part rule that I have found works best. It does not answer all of the questions but provides a guideline to help build an excellent hosted service.  The first is Build for Quality.  When I bring this up as a guideline for building new hosted services, there are generally two response, “That sounds expensive” and “Doesn’t everyone do that”. 

The answer to the first objection is not really; quality doesn’t always mean drastically increased costs.  Consider the fact that quality will be perceived by the end user of the service.  Good quality of a hosted service may mean providing accurate details and instructions on how to use the service.  Good quality might also mean getting the exact same service every time a customer requests it.  There is the potential for increased expenses if you need to invest in new infrastructure or software, but this isn’t the only way to build for quality.

The second response, the belief that everyone builds hosted services for quality is misguided.  Due to the ease of deployment for hosted services many providers play the “Me Too!” game.  If someone else is providing a hosted service they feel the necessity to “stay competitive” by quickly rolling out the same service.  This can lead to a lower quality service or one the provider may not be equipped to support.  We all know quality products once we use them and quite frequently when they aren’t quality products we like to let everyone else know about it.

Building for quality has several advantages.  First, it allows you to be competitive on the service and not just on the price.  If the service provided is of a higher caliber you can use that to differentiate yourself from the crowd.  Second, a hosted service built for quality will hopefully have a lower support cost.  A hosted service with good documentation, that is always delivered in the same manner and is generally reliable should mean less customer requests.  This can help reduce and potentially make-up the costs you had to invest to build for quality.

The second half to this lesson is Plan for Quantity.

When building a hosted service the initial build scale should be small, but the infrastructure and service should be designed with growth in mind.  It is easy to think that a new hosted service will be the next big thing, but the cost to build infrastructure to support it at scale is probably not practical. However, if the infrastructure is designed with the intention to grow it, when it does become a big hit you can expand it easily.  Then, hopefully costs are less of an issue as revenue is being generated by the service.

 

Wrap Up

These are only a sub-set of lessons that I have learned from providing hosted services, but they provide some foundational guidelines I have found for success.  Of course your approach to hosted services may be very different from mine and if so I would rather enjoy hearing your lessons learned.

How often have you endured complaints from users about server or service performance because of a small subset of users who are taxing the system, utilizing the majority of the available resources? How do you manage resource sharing on a multi-purpose server that may have a database server and web server running together? We have experienced these situations many times, but rather than write exotic process management scripts, maintain heavily modified init scripts or complicated user environments; we have switched to using cgroups.

What are Cgroups?

Cgroups or container groups started in September 2006 as a patch for the Linux kernel initially developed by a few Google engineers. Process containers, as they were initially called, allowed administrators to restrict or prioritize processes and their access to server subsystems. In the 2.6.24 kernel release process containers was merged into the mainline kernel and were renamed to cgroups.

Cgroups provide hooks into various kernel subsystems which allow an administrator to limit the resource utilization of a single process or group of processes on one or more subsystems on the server. Cgroups are currently able to limit CPU, memory, network, and disk I/O. They are also able to perform resource accounting and have the ability to pause and restart processes. As cgroups have grown and developed they have become part of LXC, an OS level virtual server implementation similar to Linux-Vserver or OpenVZ.

Take the following example as an example of how cgroups can be used to balance resource utilization on a server. Company XYZ, Inc has a multi-purpose application server with three departments utilizing it, Accounting, HR, and Sales. This server is also providing XYZ, Inc’s intranet site. Administrators frequently receive complaints from internal users about the performance of the corporate intranet site when HR is running their payroll report. dditionally, Accounting and HR are unable to run their reports at the same time due to the degradation of performance. By using cgroups on this application server it is possible to improve the end users experience for all departments.

Since cgroups work in a hierarchical fashion and allows for multiple hierarchal groups to co-exist we can setup multiple, nested cgroups to limit users or processes from consuming all the system resources. This will protect system resources for other processes ensuring they will always have access to system resources. For XYZ, Inc’s application server we can create two “top level” cgroups, one we would call ‘departments’ and the other ‘daemons’. Inside the root level of each hierarchy we can set the maximum amount of CPU time the whole group will have available to it. We can give our ‘departments’ group 80% of the CPU and memory and ‘daemons’ group 20% of the CPU and memory. The application accounting uses is CPU and memory intensive and we want to prevent those applications from negatively impacting the intranet site and other department’s applications. Within our ‘departments’ hierarchy we can create three additional cgroups, one for each department, giving each a percentage of the 80% of the CPU and memory that has been allocated to the ‘departments’ cgroup. This now limits the memory and cpu that can be consumed by each department and allows a more equitable sharing of the resources. We could continue on adding additional sub-cgroups in ‘departments’ limiting or guaranteeing the amount of block I/O or network throughput for certain departments, applications or users.

Utilizing Cgroups

While the explanation provided is fairly straight forward, when it comes to configuring cgroups, it can be rather daunting as the process isn’t well documented. Cgroups utilizes a pseudo file system for configuration and reporting very similar to the proc or sys files systems. There are two ways to setup cgroups. The first is to write scripts that will mount the pseudo file system which is composed of the subsystems that will be used, create the hierarchy that will be used and and finally set the values for the cgroups. The second way uses the scripts and daemons that come in Debian’s cgroup-bin package or Red Hat’s libcgroup package. With the second method only two configuration files need to be modified.

In Debian based distributions those files are /etc/cgconfig.conf and /etc/cgrules.conf. The basic tools to configure and manage cgroups comes in packages for Debian and Redhat, cgroup-bin and libcgroup which provide the cgconf init script for setting up cgroups and the cgred daemon which places processes in cgroups based on user defined rules.

cgconf

The first init script is cgconf. This script will setup cgroups based on the /etc/cgconfig.conf configuration file. In this file you are able to set the subsystems that you wish to be able to control and then set the values for the various subsystems. Currently cgroups supports the following subsystems: blkio (block device I/O), cpu, cpuacct (CPU accounting), cpuset (CPU core affinity and NUMA access), devices (allows and denies access to block and character devices), memory, freezer (suspend and resume of processes), net_cls (allows for tagging and classification of packets), net_prio (allows for setting the priority of the outbound traffic on a specific interface), ns (limits access to a specific namespace), perf_event (allows for monitoring of processes via the perf tool).

An example from the Red Hat documentation of a cgconf configuration file.

mount {   

    cpu     = /cgroup/cpu_and_mem;

    cpuacct = /cgroup/cpu_and_mem;

    memory  = /cgroup/cpu_and_mem;

}

group finance {

        cpu {

                cpu.shares=”250″;

        }

        cpuacct {

                cpuacct.usage=”0″;

        }

        memory {

                memory.limit_in_bytes=”2G”;

                memory.memsw.limit_in_bytes=”3G”;

        }

}

group sales {

        cpu {

                cpu.shares=”250″;

        }

        cpuacct {

                cpuacct.usage=”0″;

        }

        memory {

                memory.limit_in_bytes=”4G”;

                memory.memsw.limit_in_bytes=”6G”;

        }

}

group engineering {

        cpu {

                cpu.shares=”500″;

        }

        cpuacct {

                cpuacct.usage=”0″;

        }

        memory {

                memory.limit_in_bytes=”8G”;

                memory.memsw.limit_in_bytes=”16G”;

        }

}

When editing the configuration file for cgconf you will need to restart the /etc/init.d/cgconf init script. Restarting only unmounts and remounts the cgroups file system, but this will clear all current cgroups, their configuration directives and will also free all processes from cgroup restrictions. I have found that it is best for persistent daemons such as Apache or MySQL to be restarted after cgconf is restarted to ensure all processes are placed under cgroups control again.

cgrules

The second init script will start the cgred daemon which is the cgroup rules engine daemon. This daemon moves processes into the correct cgroup as the process is created. The daemon takes its configuration from /etc/cgrules.conf. You can configure processes to go into specific cgroups based on the user and/or group the process is running under and/or the process name.

Most documentation that I found while implementing this in some of our production environments was basic; either referencing or coming straight from the Linux kernel documentation of cgroups; one of the most helpful documents was from Red Hat. We are almost entirely a Debian based shop so there are a few difference in how Debian configures cgroups. One of the main differences is the location of the root cgroup. In Redhat based distributions this file system is mounted at /cgroups/. Debian based distributions mount this in /sys/fs/cgroups/. While you can change where this mounts, I would recommend keeping this at the default location as it makes it much easier with the default init scripts to work with the cgroups.

While deploying cgroups I encountered a known issue in Debian 7 “Wheezy” where the init scripts are broken. This was with libcgroup 0.38-1 which shipped with Wheezy. There was an attempt in Wheezy to fix a bug in the init scripts, which could cause processes to crash when loading a new configuration; however, the bug was not correctly fixed when Wheezy shipped. I had to use the init script for cgconf that shipped with Debian 6 “Squeeze” to get cgconf to behave as expected.

We have cgroups implemented on our shared Linux Fusion Web Hosting platform. Being able to restrict the amount of memory and cpu that processes like Apache or PHP can consume has helped to limit the impact of DoS attacks against our customer sites. Beyond this, we are implementing cgroups in other areas of our Linux infrastructure to provide better access to servers during times of high utilization. We have found cgroups as an effective way to manage and enforce resource sharing in our multi-user Linux environments.

Server Administration with PowerShell 4

TL;DR: PowerShell 4 is awesome, if you are a Windows Technician, Admin, Engineer, Architect, etc. you should be using it.

Intro

In 2007 Microsoft released PowerShell and since then the tool has grown to become one of the most, if not the most, powerful windows administration tool available for Microsoft operating systems and applications. Not only has Microsoft embraced PowerShell as the primary administration tool, but they have opened the door for vendors to develop modules to manage their own software and products. For example, we use Cisco PowerTool for managing their Unified Computing System (UCS) Platform and we utilize VMware PowerCLI for managing ESXi, vSphere, vCloud Director, and several other services and applications. PowerShell isn’t only for administrative purposes but also for personal use, you can even download a module for managing your Facebook account.

PowerShell provides cmdlets rather than command line executables. Cmdlets are instances of .NET framework classes that can perform CRUD actions on .NET framework objects. PowerShell’s runtime environment allows us to pipe results of cmdlets (.NET objects) to other cmdlets, providing a fluid way of administrating Microsoft Windows and applications. This post will focus on cmdlets that will streamline the administration of your server infrastructure.

Server Administration Cmdlets

Let’s geek out a little bit on some new cmdlets that have changed the way I work in regards to administrating a local server.

Add and Remove Windows Features

In legacy versions of windows, administrator’s hands were forced to use Server Manager or Control Panel to see what was installed or to install/uninstall window’s components. Starting with Server 2008 R2 the ServerManager module was introduced and it has never been easier to see what is installed, install additional features, uninstall features, and optionally remove them permanently.

Get-WindowsFeature – Gets information about Windows Server roles, role services, and features that are available for installation and/or are installed on a specified server. A few of the properties that we can filter these features by are DisplayName, InstallState, and Name. For example, in the following command we see what DHCP components are installed:

Get-WindowsFeature | Where Displayname -Match dhcp

            

Display Name                  Name      Install State

————                  —-      ————-

[ ] DHCP Server               DHCP          Available

        [X] DHCP Server Tools RSAT-DHCP     Installed

        

Install-WindowsFeature – This cmdlet installs one or more Windows Server roles, role services, or features on either the local or a specified remote server that is running Windows Server 2012 R2. This cmdlet replaces Add-WindowsFeature in previous versions. Here is how we can install all web server components on the local server:

Get-WindowsFeature Web* | Install-WindowsFeature

Success Restart Needed Exit Code      Feature Result

——- ————– ———      ————–

True    No             Success        {Remote Access, Application Initialization…

        

Uninstall-WindowsFeature – Uninstalls and optionally removes specified roles, role services, and features. We can use this cmdlet to actually get space back from the storage footprint of an installation of Windows. This cmdlet replaces Remove-WindowsFeature in previous versions. Let’s say we know we are never going to use SNMP, and you don’t want the additional installation files taking up space on your disk partition, you can remove it from the operating system to reclaim space:

Get-WindowsFeature *SNMP* | Uninstall-WindowsFeature -Remove -Confirm:$False

Success Restart Needed Exit Code      Feature Result

——- ————– ———      ————–

True    Yes            SuccessRest… {SNMP Tools, SNMP Service, SNMP WMI Provider}

WARNING: You must restart this server to finish the removal process.

        

Firewall Rules Administration

PowerShell 4, limited to Server 2012 or newer, introduces some game changers for managing host firewalls. The legacy command netsh advfirewall firewall was not only hard to remember, but very hard to automate firewall configuration. Let’s take a look at how we used to show all rules:

netsh advfirewall firewall show rule name=all

Once the rule name was found, we could modify it using even more complex syntax. Here is an example of how to change the remote addresses for WMI:

netsh advfirewall firewall set rule name=”Windows Management Instrumentation (WMI-In)” `

dir=in new remoteip=”192.168.1.0/24″

Remembering the syntax was difficult and if you wanted to do this from an object oriented perspective, you would need to use COM objects. This is how you could get a rule through COM:

$Rule = ((New-Object -ComObject HNetCfg.FwPolicy2).Rules) | `

Where-Object { $_.Name -eq “Windows Management Instrumentation (WMI-In)” }

This is how to change the remote IPs using COM:

$Rule.RemoteAddresses = “10.0.0.0/24”

This was easier, but remembering New-Object -ComObject HNetCfg.FwPolicy2 was still too complicated unless your primary job was firewall administration. To be honest, this resulted in me using the graphical Windows Firewall with Advanced Security Console the majority of the time. The only time I used PowerShell to adjust host firewalls was when I was forced to for scripting purposes. Starting with Server 2012 we have new cmdlets for host firewall administration which are included in the NetSecurity module:

Get-NetFirewallRule – Retrieves firewall rules from the target computer or server. This cmdlet allows the retrieval of all firewall rules or filtering by common things such as DisplayName, DisplayGroup, Enabled, and Action. The following snippet obtains information for WMI firewall rules, by grabbing all rules in the group:

Get-NetFirewallRule | Where DisplayGroup -Match “Windows Management Instrumentation” | `

select DisplayName,Name,Enabled,Action | Format-Table -AutoSize

DisplayName                                   Name                Enabled Action

———–                                   —-                ——- ——

Windows Management Instrumentation (DCOM-In)  WMI-RPCSS-In-TCP      False  Allow

Windows Management Instrumentation (WMI-In)   WMI-WINMGMT-In-TCP    False  Allow

Windows Management Instrumentation (WMI-Out)  WMI-WINMGMT-Out-TCP   False  Allow

Windows Management Instrumentation (ASync-In) WMI-ASYNC-In-TCP      False  Allow

Enable-NetFirewallRule – enables a previously disabled firewall rule to be enabled. Common parameters used by this cmdlet are a DisplayName, DisplayGroup, and Direction. These can be explicitly defined as a parameter of the cmdlet. This is how to enable a single rule:

Enable-NetFirewallRule -DisplayName “File and Printer Sharing (SMB-In)”

However, very often it is necessary to open up firewalls for purposes rather than single rules, for example enabling all WMI rules:

Enable-NetFirewallRule -DisplayGroup “Windows Management Instrumentation (WMI)” -PassThru | `

Select DisplayName,Enabled | ft -auto

DisplayName                                   Enabled

———–                                   ——-

Windows Management Instrumentation (WMI-In)      True

Windows Management Instrumentation (WMI-Out)     True

Windows Management Instrumentation (DCOM-In)     True

Windows Management Instrumentation (ASync-In)    True

The counteraction of this cmdlet is Disable-NetFirewallRule, which disables a previously enabled firewall rule.

Set-NetFirewallRule – modifies existing firewall rule properties. Common parameters used by this cmdlet are DisplayName and DisplayGroup. This is how to set the remote IP range able to access your host for File and Printer Sharing purposes:

Set-NetFirewallRule -DisplayGroup “File and Printer Sharing” -RemoteAddress 192.168.1.0/24

Get-NetFirewallRule -DisplayGroup “File and Printer Sharing” | `

Get-NetFirewallAddressFilter | Select RemoteAddress

RemoteAddress

————-

192.168.1.0/255.255.255.0

192.168.1.0/255.255.255.0

192.168.1.0/255.255.255.0

New-NetFirewallRule – Creates a new firewall rule. Common parameters used by this cmdlet are a DisplayName, Direction, Action, Protocol, and LocalPort. This is how to create a new firewall rule that requires a specific port to be open on the host:

New-NetFirewallRule -DisplayName “My Application” -Direction Inbound `

-Action Allow -EdgeTraversalPolicy Allow -Protocol TCP -LocalPort 28002 -RemoteAddress 192.168.1.0/24

Name                  : {487aedbf-0a0b-4965-baee-ec5a09024c73}

DisplayName           : My Application

Description           :

DisplayGroup          :

Group                 :

Enabled               : True

Profile               : Any

Platform              : {}

Direction             : Inbound

Action                : Allow

EdgeTraversalPolicy   : Allow

LooseSourceMapping    : False

LocalOnlyMapping      : False

Owner                 :

PrimaryStatus         : OK

Status                : The rule was parsed successfully from the store. (65536)

EnforcementStatus     : NotApplicable

PolicyStoreSource     : PersistentStore

PolicyStoreSourceType : Local

The counteraction of this cmdlet is Remove-NetFirewallRule.

Testing Network Connectivity

Troubleshooting network connectivity is a common task for administrators that is often more difficult than it has to be due to the poor tools available. Have you ever tried scripting out ping responses, telnet connections, or net view commands? Scripting out results of legacy tools was extremely hard and unreliable. Starting in Server 2012 R2 and Windows 8.1 there is a new cmdlet available that combines troubleshooting TCP connections, tracert, and ping and this cmdlet is part of the NetTCPIP module:

Test-NetConnection – cmdlet for testing network connectivity and common parameters include Port, Traceroute, and InformationLevel. Here is a very simple way to document which Active Directory computers have SMB enabled:

$smb = @{}

ForEach ($computer in Get-ADComputer -Filter *){

$test = Test-NetConnection -Computername $computer.DNSHostName -Port 445

$smb.Add($computer.DNSHostName, $test.TcpTestSucceeded)

}

$smb.count

30

Testing Web Sites and Web Requests

In PowerShell’s infancy, Windows admins had the hardest time trying to mimic common web tools such as wget on the Linux side. In the past I’ve scripted out starting IE through a COM object and attempting to script web requests. There are new cmdlets that save us significant amounts of time when we have to test HTTP responses from a variety of systems:

Invoke-WebRequest – sends HTTP and HTTPS requests to a web page or web service. It parses the response and returns collections of forms, links, images, and other significant HTML elements. Here’s a quick example of how we can test all of our sites using a CSV file as a data source, and store the results in a hash table:

$status = @{}

$sites = Import-Csv .\Hosted_Sites.csv

ForEach ($site in $sites){

    try {

        $result = (Invoke-WebRequest -Uri $site.Site).StatusCode

    }

    catch {

        $result = $_.ToString()

    }

    $status.Add($site.Site, $result)

}

Invoke-RestMethod – sends HTTP and HTTPS requests to Representational State Transfer (REST) web services that returns richly structured data. Windows PowerShell formats the response based to the data type. For an RSS or ATOM feed, Windows PowerShell returns the Item or Entry XML nodes. For JavaScript Object Notation (JSON) or XML, Windows PowerShell converts (or deserializes) the content into objects. With REST being a common web standard, this cmdlet expands the possibilities of PowerShell. We use Invoke-RestMethod to test our service APIs, but here is a quick example of using it to pull some map data from google map APIs:

Invoke-RestMethod -Uri “http://maps.googleapis.com/maps/api/geocode/xml?address=130+W+2nd+St+45402&sensor=false” -Method GET

Scheduled Jobs

PowerShell 4 introduces some improvements to some commands that have existed in previous versions. A great example of this is working with Scheduled Jobs (Scheduled Tasks). The legacy command line utility schtasks.exe was next to impossible to script and automate your work. Here is how we used to create a scheduled task:

schtasks /create /tn Update-File /tr ‘cmd /c ipconfig > c:\ip.txt’ /sc minute

A major problem is that you always have to identify a program that you want to run. Getting PowerShell to run as the program and execute a script was more difficult than it should have been. Now, we have PowerShell Scheduled Jobs, allowing us to utilize the .NET framework cmdlets to create and manage jobs:

Register-ScheduledJob – cmdlet for managing scheduled jobs on the local computer. A “scheduled job” is a Windows PowerShell background job that can be started automatically on a one-time or recurring schedule. Scheduled jobs are stored on disk and registered in Task Scheduler, so they can be managed in Task Scheduler or PowerShell. Here is an example of how to create a scheduled job to send an email every minute:

$minute = New-TimeSpan -Minute 1

$trigger = New-JobTrigger -Once -At 12am -RepetitionInterval $minute `

-RepeatIndefinitely

Register-ScheduledJob -Name “Send Email” -Trigger $Trigger -ScriptBlock {

    Send-MailMessage -SmtpServer localhost -To [email protected] -From [email protected] `

    -Subject “Minute Notice” -Body “Testing every minute emails!”

}

 

Conclusion

In 2009 I wrote the following in a blog post, “For those Windows Admins that haven’t tampered with Windows PowerShell I recommend you start… I am finding the more I dive in the less I want to come up for air.” The same holds true today and 3 PowerShell versions later. PowerShell is only becoming more powerful as time goes on, and DataYard is taking advantage on the ability to automate administrative tasks and make our jobs easier. Here are some quick PowerShell resources that will help you, no matter your familiarity with PowerShell:

• Management Framework 4.0 Download – download of PowerShell 4.0
• Script Center Repository – find anything from Active Directory management to SNMP modules
• PowerShell.org – Don Jones and company run a good blog discussing latest PowerShell features.

Stay tuned for future posts about PowerShell!

Introduction

DataYard has been providing internet access and dedicated WAN services for years, but did you know that as a CLEC, or competitive local exchange carrier, we are one of less than 100 such carriers in Ohio?

There’s a lot of mystery behind what a CLEC is and the relationship that exists between them and the ILECs such as AT&T, Verizon, and other incumbent local exchange carriers. The ILEC is a telephone company made up of one or more Regional Bell Operating Companies, or Baby Bells, that remain since the 1984 breakup of Bell Systems, and maintain and operate local wiring exchanges in their respective regions.

This post aims to demystify some of that relationship, and explain some of the technologies that the CLEC status allows us to deliver to our customers.

CLEC Landscape Summary

So what is a CLEC? CLECs are always in competition with the local incumbent carrier – they offer an alternative to ILECs through a different set of regulations. Where ILECs are required to provide a certain set of services to the public, CLECs have the right to compete in the same market using access to the same facilities and infrastructure.

Practically speaking, CLECs are given rights to place hardware in the local exchange facilities of their market, owned and operated by the incumbent carrier.  CLECs are also granted rights to access individual copper loops as an unbundled network element – that is, a single telecommunications network component, required to be offered by ILECs to competitors at near-cost rates.

The local exchange, or central office, is a facility in which all of the nearby residential and business copper infrastructure terminates. These facilities have stood for years delivering a variety of services (voice, DSL/T1, etc.) over copper infrastructure. For a more in depth look at the rights provided to CLECs, have a look at the Telecommunications Act of 1996 itself, which defined most of the regulation dictating the ILEC & CLEC relationship.

This agreement allows CLECs to offer many of the same services provided by ILECs to end users, using their own hardware, networks, and value add. Despite the competition enabled through the telecommunications act, many CLECs have crumbled or been acquired by the larger LECs over the last decade. Most that have survived have done so by focusing on business customers, and diversifying into cloud and managed services. DataYard has used its CLEC status to supplement and add value to its managed cloud offerings with a variety of xDSL (modern SHDSL, ADSL, and VDSL technologies), T1, and Ethernet services.

xDSL and Ethernet in the First Mile

While a market for TDM based T1, T3/DS3 & Sonet services still exists, in recent years the focus   has shifted heavily towards Ethernet based services delivered to the first mile. These Ethernet in the First Mile or EFM services are often delivered using a variety of xDSL flavors by CLECs (including DataYard) for flexible, reliable, and high bandwidth applications at small to medium sized business. Many users associate the term DSL with a standard residential ADSL modem, with marginal speeds and reliability, but the technology behind modern SHDSL, ADSL2+, and VDSL2 connections has made these Ethernet based services very attractive to businesses. Consider the following speeds achievable using a single copper pair (and more with circuit bonding) with current xDSL technologies when evaluating Ethernet and Metro Ethernet solutions against fiber and wireless solutions.

• SHDSL – 5.6Mbps download and upload
• ADSL2+ (Annex M) – 24Mbps download, 3mbps upload
• VDSL2 (Common 17a Profile) – 100Mbps download and upload

Like all implementations of DSL, these speeds are achieved with a method of modulating bits into a pattern of tones on the analog circuit, to be picked up on the remote end and converted back into bits. Because of that implementation, all xDSL speeds are very dependent on distance from the exchange where the CLEC or ILEC terminates its copper circuits.

While ILECs have been slow to adopt xDSL outside of the traditional ADSL model in the US, CLECs and much of Europe, Asia, and Australia make heavy use of high bandwidth DSL.

Conclusion

While fiber based solutions are going to become more and more common in coming years, copper based Ethernet services are going to continue to deliver a price competitive solution. With standards such as the new G.Fast ITU G.9701 delivering up to 1Gbps over existing copper, we should expect continued growth and offerings from CLECs and ILECs alike. DataYard has been delivering xDSL based, high bandwidth and highly available Ethernet services to businesses for years, standardizing on industry leading Adtran as our vendor of choice.

DNS RESOLUTION INTRODUCTION

DNS (Domain Name Service) is the process of resolving domain names, such as www.datayardworks.com to it’s IP address, An IP address is the numeric value that a computer connected to a network is assigned, which allows it to communicate with other systems, similar to a telephone number. DNS works as the “phone book” for the Internet by translating hostnames into IP addresses. Without DNS resolution, accessing content on the network would be much more difficult and require us to remember the IP address for many different systems and websites across a network.

What information can be obtained from a DNS lookup?

You can find name server information by running a ‘dig www.domain.tld’ command from a Linux terminal or using a browser based zone lookup.

; <<>> DiG 9.2.4 <<>> www.datayardworks.com

;; global options:  printcmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5193

;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:

;www.datayardworks.com.         IN      A

;; ANSWER SECTION:

www.datayardworks.com.  86400   IN      A       64.56.96.222

;; AUTHORITY SECTION:

datayardworks.com.      18320   IN      NS      dns2.donet.com.

datayardworks.com.      18320   IN      NS      dns1.donet.com.

;; ADDITIONAL SECTION:

dns1.donet.com.         3203    IN      A       64.56.119.67

dns2.donet.com.         668     IN      A       64.56.96.6

;; Query time: 1 msec

;; SERVER: 64.56.96.117#53(64.56.96.117)

;; WHEN: Tue Mar 25 12:55:38 2014

;; MSG SIZE  rcvd: 131

There are several record types associated with a domain. Some of the most common record types are A, MX, CNAME and TXT records. The majority of DNS records are A or MX records. These records are vital to successful domain name resolution as they each serve a different purpose but point the domain to its corresponding IP addresses.

ADDRESS Records (A) – Address records allow for you to point different sections of your domain to different IP addresses or servers. For example, this would be useful for having “www.datayarworks.com” point to your web server’s IP address and “mail.datayardworks.com” point to your mail server’s IP address. Each record includes a “Host Name” value, a corresponding IP Address and TTL (time to live) value which tells the system how long to cache the records before updating. You can find out the IP address that a domain name points to by running ‘nslookup www.domain.tld’ in command prompt.

Mail Exchanger (MX) – An important part of the email system are your domain’s “MX” records. MX records tell the world what server to send mail to for a particular domain name. These records include a “Host Name” value, a corresponding IP address and a TTL value. You can set priority on MX records to allow a server to serve as a backup in the case that your primary mail server is not responding.

Canonical name (CNAME) – These are usually referred to as alias records since they map an alias to its canonical name. When a name server looks up a name and finds a CNAME record, it replaces the name with the canonical name and looks up the new name. This allows you to point multiple DNS records to one IP without specifically assigning an A record to each host name. If your IP was ever to change you would only have to change one A record rather than many A records.

DNS queries are answered in a number of different ways. A client can answer a query locally using cached information obtained from a previous query. If the local system does not have cached information it may use an iterative query to find the needed information. An iterative name query is one in which a DNS client allows the server to return the best answer it can give based on its cache or zone data. If the queried DNS server does not have an exact match for the queried name, the best possible information it can return is a referral, which is a reference for another DNS server to check. The DNS client can then query the DNS server for which it obtained a referral. It continues this back and forth process until it locates a DNS server that is authoritative for the queried name, or a time out occurs. An authoritative name server provides actual answer to your DNS queries, this will come from the DNS server that hosts the records for the domain. You can find out the authoritative name servers for a domain by doing a WHOIS lookup on the domain, you can find out more information on WHOIS at http://whois.net/. A DNS server can also query or contact other DNS servers on behalf of the client to fully resolve the name, then send an answer back to the client. This process is known as recursion.

Domain Lookup Process

1. The web browser will check local cache on your computer to try and resolve the domain name. If it can get an answer directly, it proceeds no further. You can also override this lookup process by making changes to the hosts file on your local computer, this will allow your PC to override outside DNS information and look for the hostname at an IP address that you’ve specified.
2. If an answer cannot be obtained from your local cache your system will reach out to your ISP’s recursive DNS servers. You can find out your primary DNS servers by running ‘ipconfig /all’ from the command line and look for the IP address listed next to DNS servers.
3. These name servers will first search their own cache to see if this domain has been resolved recently or if the server is authoritative for the domain, if so it will return those results.
4. If this system does not have any cached information it will strip out the TLD (Top Level Domain) and will query a root name server, to find out what name server is responsible for that TLD. Once this information is obtained it will query authoritative server for that TLD for the IP for the domain you are trying to resolve.
5. The authoritative name server will tell you the absolute records for a domain name. It does not provide cached answers that were obtained from another name server, it does not query other servers it only returns answers to queries about domain names that are configured locally.
6. The authoritative name servers will respond with the IP address of the domain name you’ve looked up and will return this information to your system.

Breaking down a domain name

When a domain name needs to be resolved the DNS servers will first break the domain name down into pieces and start at the top level domain and follow a path to finding the authoritative name servers for the domain name that you are trying to resolve.

As you can see, the Domain Name System is essential in the use of the Internet. Without this complex system of servers working together you would not be able to simply type in names for websites, but would have to remember the numerical IP Addresses in order to get anywhere on the Internet.

Over the past few months the Internet has seen increased DDoS (distributed denial of service attack) activity which started with DNS amplification attacks and then moved onto NTP amplification attacks. For now the DDoS attacks have stopped, however it’s only a matter of time before the next DDoS attack method is discovered. It’s an ongoing effort for administrators to keep servers patched to prevent these type of exploits. The last one I found myself dealing with was the NTP monlist amplification attack, which used several customers NTP (Network Time Protocol) servers that were available to the public internet. One vulnerable NTP server was generating ~500 mpbs of outbound traffic before we shut its access down.

A DDoS is a distributed denial of service attack in which several computers are configured to flood data to a target. The target’s internet connection can become over saturated disrupting the target’s Internet connection. This in affect can take down any Internet server for the length of the attack. There are several types of DDoS attacks, one of them being a NTP amplification DDoS attack. The latest NTP DDoS attack method allows remote users to trick an Internet facing server running NTP to flood data to a target without having access to the NTP server. These attacks can be kicked off easily and are very hard to trace to the original source of the attack.

By using the NTP monlist command to query an exploitable NTP server’s last 600 associations, an attacker can send a small amount of data to the exploitable NTP server and get back a response back that is several times larger than the original request. When the NTP monlist command is used as intended, it will cause no harm, however when the source IP address is spoofed it can be used for denial of service attacks. If an attacker spoofs the source IP in there NTP monlist request, the response is sent to spoofed IP address. Using a script to repeat the NTP monlist command to an exploitable server, the attacker can generate a very large amount of traffic and target it where they want. Kicking off this type of DDoS requires a small amount of Internet bandwidth and could generate a flood of data hundreds of times larger than the original monlist request. If an attacker is able to identify an exploitable NTP server that has access to a lot of bandwidth, that would be a prime candidate of a large NTP amplification attack. An attacker could use several servers around the internet to query the NTP monlist against an exploitable NTP server with access to a large amount of bandwidth.

Basic NTP Amplification Attack flow

1. Attacker at IP 60.70.80.90 sends .5 mbps worth of NTP monlist requests, spoofs source IP as IP 123.123.123.1 to exploitable NTP Server.
2. The exploitable NTP Server responds with 15 mbps of response data to Target at IP 123.123.123.1 not the attacker who is at 60.70.80.90
3. The Targets ingress bandwidth is 10 mbps, they are not unable to use their internet connection due to over saturation from the NTP monlist flood running at 15 mbps

• Attacker is only using half of their available upload to completely take the target offline.
• This example would be a ntp amplification attack of 30 times the original data sent
• Repeat this process with several different client and NTP servers and you now have a NTP amplification DDoS attack.

I set up a lab example of an NTP server running with monlist enabled and used ntpdc to issue the monlist command. The request was 234 Bytes and the response generated back was 2676 Bytes, in this example it would be possible to amplify 11 times the original size of the request sent to the server. My test server had 33 NTP associations, while the max that the monlist will respond with is 600. With 600 associations, a much larger response with the same 234 Byte request is possible. Below is a screen shot of a wire shark packet capture that shows the monlist request and response data.

 

The response from the monlist request was 6 UDP packets, 5 that were 482 Bytes and the last one was 266 Bytes. If you add up the number of total bytes in those 6 packets the total is 2676 Bytes.

If an attacker spoofs the source address in the NTP monlist request, the returned data is sent to a different server. If the attacker continues to make this request with a spoofed IP address, the NTP server will keep sending the response to the victims’ computer. This amplification attack was recorded as CVE-2013-5211 in the middle of January 2014. A lot of these attacks could have been prevented if all internet providers implemented BCP38. BCP38 is also known as RFC 2827 was actually written up almost 14 years ago. It’s a best current Practice that recommends filtering traffic that should never been seen from a user. This would for example drop any traffic with a spoofed source IP address that is not in the valid range provided to the user. DataYard implements this using a combination Cisco ACLs and Cisco Unicast Reverse Path Forwarding.

How to see if your Server is vulnerable?

• There is an open project that was started to help identify vulnerable NTP servers, you could have your public address range scanned as long as it’s smaller than a /22 (1024 Block size).
• http://openntpproject.org/
• You can use ntpdc in linux and issue the monlist command to see if you get a response.

• NMAP has a useful script that can be used to see if a server is responding to the monlist ntp request as well.

nmap -sU -p U:123 -n -Pn –script ntp-monlist 192.168.1.0/24
|   Target is synchronised with xx.yy.61.67
|   Public Servers (1)
|       xx.yy.61.67
|   Public Clients (1)
|       xx.yy.177.51
|   Other Associations (13)
|       xx.yy.100.2 (You?) seen 5 times. last tx was unicast v2 mode 7
|       127.0.0.1 seen 78664 times. last tx was unicast v0 mode 0
|       xx.yy.177.108 seen 4 times. last tx was unicast v2 mode 7
|       xx.yy.129.66 seen 1 time. last tx was unicast v2 mode 7
|       xx.yy.203.115 seen 7 times. last tx was unicast v2 mode 7
|       xx.yy.95.174 seen 1 time. last tx was unicast v2 mode 7
|       xx.yy.253.2 seen 3 times. last tx was unicast v2 mode 7
|       xx.yy.54.31 seen 1 time. last tx was unicast v2 mode 7
|       xx.yy.177.66 seen 3 times. last tx was unicast v2 mode 7
|       xx.yy.81.113 seen 1 time. last tx was unicast v2 mode 7
|       xx.yy.54.103 seen 2 times. last tx was unicast v2 mode 7
|       xx.yy.244.49 seen 1 time. last tx was unicast v2 mode 7
|       xx.yy.230.75 seen 1 time. last tx was unicast v2 mode 7

Please Patch your NTP server!

There are still many unpatched NTP servers out on the internet that can be used in future DDoS. If your NTP server is responding to the ntp monlist command, you should upgrade to a later version of ntp. If you not able to upgrade your ntpd process there are several examples online that show how to lock down or even complete disable the monlist command.