15th Annual SysAdmin Appreciation Day

15th Annual SysAdmin Appreciation Day

Today — Friday, July 25, 2014 — is the 15th Annual System Administrator Appreciation Day!

Please join us in thanking the server, network, and data center people who keep all our technical ships running smoothly — no matter what time it is, day or night, weekday, weekend, or holiday, 365 days a year.

We designed and printed a custom 11″ x 14″ letterpress poster to commemorate the day. If you’d like a copy just let us know via email ([email protected]) and we’ll get it to you. This was a limited production run poster so we may run out soon.

P.S. There is a movie reference hidden somewhere on the poster. Can you identify the reference and the movie?

15th Annual SysAdmin Appreciation Day

Your Data is Telling You Something


I found an article this morning through my daily cruise through tech news. I mentioned the company mentioned in the article, Codespaces, yesterday when I was talking with our management team. Codespaces was wiped out of existence by an attack that ultimately led to the near total destruction of their data. The timing is serendipitous because this article touches on some of the points I was trying to make about looking for anomalous data in the many streams of data we already have at our disposal, or could easily add.


From the article: “Every attack is a sequence of events.”

I’ll extend that: “Every system failure or security breach is a sequence of events.”

Failures are generally not atomic events. Systems aren’t 100% functional one instant, and down to 0% the next. Anomalous data is a bellwether that something different is happening, that something worth taking a closer look at is in progress. Not that something necessarily wrong is happening. You don’t know that yet. All you know is that something different is happening, a possible indicator of trouble on the horizon. You’re begging for trouble if you know you have an anomaly in progress and you ignore it.

A simple example: interpreting car dashboards in context. I’ve driven two Volkswagen Beetles in recent years. One Beetle had a faulty coolant sensor that would indicate an over temperature condition no matter what, even when the coolant level was fine and the car was cooling properly. The other Beetle, driven by my daughter, had a coolant sensor that was OK. Last week it lit up for the first time. She mentioned it to me, and I did what I could do at 10:30 at night in the dark: checked the coolant with flashlight in mouth, and topped it off. I told her to keep an eye on it. The next day the sensor went off again, but she kept driving the car. A short while after that, when my wife was driving the car, billows of white smoke poured out of the car because a cheap plastic connector had broken.

A problem started to unfold with my car, and an indicator went off. I responded (added coolant to a leaky system), but that didn’t resolve the core problem. The indicator went off again, was ignored, and the problem got worse. If my wife hadn’t stopped the car immediately with the third indicator (smoke) the over temp could have warped the rods and destroyed the engine. Furthermore, the status of the indicator had an entirely different meaning based on the historical trend data of the particular Beetle.

Changing data is telling you something. Listen to it.

Leigh Sandy’s Fiber-to-the-Home Initiative

Leigh Sandy “surfing the net” — December 1995

After founding the company with me and David Hughes in 1995, Leigh Sandy has left DataYard to start working on a project that is near and dear to his heart: fiber to the home. Leigh has been wanting to do this for some time, since at least 2010 when he worked with the Dayton Development Coalition to try to bring Google Fiber to Dayton. Leigh is committed to developing the Dayton community and isn’t leaving town, he’s just going to focus his time and attention on a different set of services and customers. Leigh will continue as an owner and referral partner for future DataYard business. I suspect that someday we’ll be referring residential business his way.

It will definitely be strange not having him around, but everyone at DataYard wishes Leigh the best of luck in his future endeavors and hope for his success!

David Mezera

Masten Worley Joins DataYard


I am very excited to announce that, effective 3/21/2014, Masten Worley officially joined our team as DataYard’s Director of Sales and Marketing.

Before joining DataYard, Masten was the CEO of MacTown — the Premier Apple Specialist in Dayton, Ohio. There, Masten oversaw retail operations at two locations in the Miami Valley, as well as the training, network management, and system integration functions — a full assortment of consumer and business services designed to help users get the most out of their technology investment. Prior to that, Masten was the store leader at an Apple store in Tennessee.

At DataYard, Masten will be working to get the word out about all the great things we do for our access, hosting, and co-location customers. Masten understands that the strength of DataYard is not a one-size-fits-all technology services menu, but the personalized service and consultative approach DataYard uses to right-size solutions for customers who need our high-reliability services. He’ll be closely listening to our customers, looking for new business opportunities in our target markets. Masten will also be working to make sure that our sales and operational teams are in perfect synchronization for each new customer, ensuring that DataYard delivers the best possible service experience to our customers from the start.

Welcome to the DataYard team, Masten!

Forging a Team of Technical Professionals

Forging a Team of Technical Professionals

Over the last year, I’ve had the pleasure of watching DataYard’s Technical Operations team work under intense pressure. On the surface it may seem oddly sadistic for me to talk about it being my “pleasure” watching the team perform under these conditions, but I mean it with the deepest respect and admiration. The difficult conditions, the pressure, has done its work perfecting the diamond.


Over the last year I’ve had the pleasure of watching DataYard’s Technical Operations team work under intense pressure. On the surface it may seem oddly sadistic for me to talk about it being my “pleasure” watching the team perform under these conditions, but I mean it with the deepest respect and admiration. The difficult conditions, the pressure, has done its work perfecting the diamond.

In the years leading up to 2012 we talked and dreamed frequently about building a new facility. We’d been in our previous facility for nearly ten years, but we outgrew it at least three years earlier. Planning began early in 2012 for a new office space and data center, and the Technical Operations Group got involved in the process from the very beginning. They participated in design, oversaw construction of the data center, and began an aggressive test and checkout phase of the data center as soon as it became available.

We moved into our new facility at the end of October 2012, and began moving production server and network equipment into the new data center a few weeks later. Like all projects of this size and scope we encountered problems in the first phase of the migration, which lead to delays implementing our migration plan. An already compressed schedule became additionally compressed, and the increase in our collective stress level was palpable.

Earlier in the year, I gave the team the charge to complete the technical services migration before the end of the 2012. Two primary data centers running in tandem meant two sizable rent payments, two sizable electrical bills, and twice the facility maintenance burden — obviously, a situation that we didn’t want to continue a day longer than was absolutely necessary. As construction continued the likelihood of a schedule slip in the facility completion date became more apparent. It would have been easy for any other group facing a similar situation to lower expectations, to justify a schedule slip of their own lasting another month or two. Instead, this group pressed on.

The mettle of DataYard’s Technical Operations Group has been forged in fire over the last month. They’ve worked tirelessly to delicately migrate services over one by one — in the very early morning hours, in the very late evening hours, and over weekends — whatever it took to get the job done for the group with the least amount of impact to our customers. They’ve been careful, professional, and selfless, helping each other at every turn. I haven’t seen this kind of esprit de corps from a group this size since my days in the military.

DataYard’s Technical Operations Group is the finest team of technical professionals in the Dayton region, and we are honored to have them on our team.

Build Update 14: Real Ultimate Power

Build Update 14: Real Ultimate Power


It has, unfortunately, been a few weeks since my last Facility 2.0 Build Update. The activity level has been extraordinarily high as we work to get the facility operational and tested.

As you can see in the panoramic photo below, the data center is, structurally, very nearly completed.


Our main focus now is on the supporting electrical and cooling systems, and getting them online as soon as possible. Right now the key system to bring online is power; everything is dependent upon it. We must tie into the utility service, and distribute utility and UPS power wherever necessary to make the data center operational. Bringing a megawatt of electricity into the building and routing it as needed is a complicated, and potentially dangerous, task.


We plan on tying our data center infrastructure into 480V utility power this weekend. If all goes as planned, we’ll be able to bring the rest of our systems online for testing and checkout soon after.

Crews have been working in parallel to finish the chilled water piping installation and HVAC unit installation. Within the week, I hope to showcase some major progress on that environmental sub-system.