The data revolution and the IT department
As another year begins and the weather turns dreary, I find myself once again in a reflective frame of mind. I look back on my career as an IT consultant, specifically around Microsoft collaboration technologies, and can’t help but be amazed at the exponential rate of change that we’ve seen across this landscape.Back when I started working with Exchange 5.5 there was very little year-on-year change. Email was email, maybe some new features came out allowing people to add vomit-inducing background images and other HTML elements to their messages, but nothing moved the needle in terms of innovation.
It was a time when the IT department only spoke to the business when they upgraded an application version, preaching down that this was now “the right way” to perform a task – even though they had no idea what these employees did on a day-by-day basis or what challenges they actually faced. It was difficult for us (as inhabitants of the IT world) to understand that the employees of a company don’t care about High Availability, reduced IOPs or cluster quorums. The word “Redundancy” means totally different things to an employee in the IT department than to those working throughout the rest of the company.
Slowly this started to change as people became far more tech-savvy, informed and vocal. Product messaging changed from bragging about technical features to explaining how real business value could be derived from their use. To me, this was the start of a seismic shift in the way that technology companies developed and positioned their products. Microsoft themselves embraced this new world and realised that tighter integration of all these independent technologies allowed users to share information and data far more easily. I remember the first time I saw a green Office Communicator presence dot in Outlook 2003 – I was blown away. With each new version of Exchange, SharePoint or other Microsoft products there was tighter integration – further stretching the resources of IT departments everywhere. It was impossible to be just an Exchange administrator, I had to become an expert in every technology that interacted with Exchange.
IT departments were tasked with managing all these interconnected systems, and struggled to keep up in this new world. If the team failed to fulfil a business need fast enough, then it was all too easy for employees to go off and figure out their own solution using freely downloaded tools and other forms of Shadow IT.
Then, in 2011, something magical happened – the world began to embrace the cloud.
IT departments no longer had to manage the underlying infrastructure and interconnection of these systems – allowing them to take a much-needed breather. The users themselves no longer relied on a slow cadence of patches and updates to drive products forward. Development teams started pushing daily features and updates to products, and these were instantly available to all users who could then give immediate feedback directly to the development teams. IT departments could spend their newly acquired free time working alongside the business to understand how these new technologies could be used to leverage increases in efficiency and productivity. Technology itself came to play a different role in business – becoming central to productivity and problem solving. Suddenly, IT departments evolved into a core business function, they formed a crucial part of decision-making process, and were no longer stuck out on the periphery.
Anyone who has worked with Microsoft Office 365 over the past 4 years will have seen this in action for themselves. What started out as relatively independent cloud-based versions of SharePoint, Exchange and Lync have now rapidly evolved into a one-stop shop for productivity tools, including products like Teams and Groups which live across all these original workloads, but only in their cloud versions.
The interconnected nature of all these products, coupled with the effectively infinite storage systems that back them, creates a truly huge amount of data. Each day these systems analyse, collect and store trillions of data points. Data which could, if correctly managed and mined, drive a greater increase in efficiency and productivity than we’ve seen since the industrial revolution. We are at the dawn of the data revolution.
All major IT vendors know this revolution is here, and that it is impossible for humans to extract value out of this data without intelligent tools. These days, everyone provides some form of Business Intelligence solution, and every product now includes some implementation of machine learning.
The IT departments of today are no longer responsible for managing systems, but have instead become responsible for managing data. Our understanding of the term “Information Technology” itself has shifted in emphasis from the technology (the how), to the information (the what) it transmits, processes and stores.
As more and more data gets accumulated into these distributed, interconnected systems, IT departments are going to have to become specialists at securing this data to ensure that commercial and customer privacy is maintained. In the past, if incorrect permissions were set on a file it was only a problem if someone inadvertently stumbled upon it within a file share. With newer business intelligence technologies like Delve, there is a big chance that a sensitive, or confidential file with incorrect permissions could be pushed in front of everyone, creating the potential for data breaches – where previously, there was none.
The interrelationships between all these data pieces, known in the technology world as “the graph”, also makes moving data around incredible difficult. If we wish to migrate data from one technology stack to another it’s no longer as easy just moving files or email messages, we must ensure that the graph metadata remains intact during the migration or we lose most of the intrinsic value of the data.
The challenges facing IT departments over the future will become more complex as individual IT systems continue to converge into a single ecosystem. As a result, the companies that are the best at collecting, storing and disseminating this business intelligence data to the correct people, without breaching the trust of their customers will come out on top. If what we’re experiencing now is the dawn of the data revolution, only time will tell how technologies, industries, organisations and individuals will be react, respond, and leverage the deluge of data that they now have at their fingertips.