The Ferris Wheel of IT Troubleshooting

If you’re in IT, you’ve probably encountered a situation like this: A key component of your application stopped working. Your offshore development team assures you that it is not the code base. So, you call your CDN provider and ask them to investigate. They don’t see anything wrong and suggest you look further upstream. You then call your 3rd party WAF provider. They don’t see any errors or anything being blocked. OK, who’s left? Ah, you have an A/B testing vendor too! A call with them results in, you guessed it, nothing wrong on their end. Your next step is to get everyone on a call, where each SME can discuss why their layer is not causing the problem and your respective account managers start to fret your usage of the other vendors. All in all a toxic, finger pointing, situation. In the end you uncover that the issue was caused by a minor change made at the WAF that no one thought would be impactful, but ended up breaking your code base!

ferris wheel against a blue sky

The problem, as you can pretty quickly see, is that the team here focused too much on individual components without fully understanding the interactions between them.

Silos in the Cloud

As we move towards cloud based services, the work of IT management gets increasingly disjointed. More work happens through third-party vendors than it does internally, meaning that the silos aren’t just departmental anymore — they’re institutional too. Whereas 10 years ago you may have had John handling web operations, Jane handling the hardware, Julie in charge of the database, and Jared doing all of the software programming, now you have a CDN, a WAF provider, an A/B testing program managed by a vendor, a production environment hosted in the cloud, cloud services provided as part of that hosting, and maybe more. Further, each of these external services has its own teams that cover their slice of your operational pizza.

Troubleshooting Gone Wrong

Problems happen. It’s an unfortunate fact of IT, and your success is often measured by your speed in addressing the inevitable. So, as in the scenario above, what do you do when all the different pieces of your operations are not only not on your team, but not even in your office?

It starts a game of round robin. You call Vendor A and tell them what’s going on. Vendor A assures you it’s not them, but it may be Vendor B. Vendor B is sure it’s Vendor C. Vendor C thinks it must be Vendor A. The truth more likely has to do with how Vendor A, B, and C are interacting with each other, but no one oversees that dynamic. Now you’re stuck.

The Easy Resolution

It’s unlikely that you’ll be back in a situation where every part of IT operations are under one roof. There are so many vendors now who do their job so well that the benefits outweigh the cost — even to your own management operations. Fixing this can be as simple as changing our frame of mind. Rather than distinct spheres, each of these services must be viewed as part of a cohesive whole.

As you work with vendors and potentially even increase your reliance on vendors, make sure you understand very clearly how each interacts with the next. How do those interactions affect your domains? While we create and rely on more and more silos, it’s important that internal IT does the opposite. If you can be the string that ties everyone together, you can diagnose and solve problems a lot quicker, and lot more efficiently.

“Tal Talks” is an blog series for IT professionals from SiteSpect’s resident IT guru, Tal Cohen. For over 12 years Tal has been assisting SiteSpect customers with digital infrastructure issues and challenges in improving customer experience.