Big data is the practice of mining large datasets with the hope of realizing valuable business insights. Technology practitioners in IT industry believe the prospective value of Big Data is substantial, but many organizations are struggling to reap benefits out of it. According to recent survey Kelly, 2013, 46% of Big data investors (companies basically) report that they have only realized fractional value from their big data deployments. 2% of the investors have declared that they have failed to derive any value from Big Data investments. 55% of Big Data projects don’t get completed and many falls short of their objectives.
But the nature of these projects are so uncertain in the beginning that it is difficult to predict potential business benefits out of it. The technology is niche, the spending is more than traditional IT projects and it works different for different business use cases. That is, there is no standardized way of doing Big data projects in a given industry. It has certainly elevated the role of CIOs and have made them more accountable, visible and responsible to a business’ growth.
This paper deals about the challenges that CIOs are facing in running Big data projects. I will conclude with possible solutions to enhance benefits out of Big Data projects.
Every department in a business is trying to optimize the traditional ways and they believe that technology is the only solace. Nowadays, each department has a budget to spend on technology. Big data is the buzz word in the technology industry and hence there is exponential increase in spending on Big data projects. Therefore, in a given business (Ex. Retail) there are many departments operating many Big data projects. There are Marketing teams operating on Customer insights, enhancing customer experience etc., store management departments trying to optimize its space arrangement at stores using Big data, Operations and Supply Chain Management teams trying to increase operational efficiency etc., In most of the cases, the departments are bypassing IT Korzeniowski, 2015 to roll out new big data projects. This makes life difficult for CIO as he has to regulate, concentrate and realize multiple big data initiatives in a given period of time. There are two types of challenges that CIO generally face in Big data projects. They are Perennial and Practical Challenges.
Once a new technology becomes popular in an IT market, it gives rise to lots of new vendors who create off-the-shelf products to cater business needs. These vendors help In-house IT firms to fasten that technology implementation to their business. Of course, Big data gave rise to lot of Vendors. Vendors came up with new products time and again to cater business needs. But this time, it was not easy for IT teams to tie these technology products to the business. There were so many products and all of them were immature. Otherwise, they used to work well in pilot projects but were unable to cater business needs on a large scale. Also, there is a scarcity of technology practitioners (Data administrators, Data Scientist, Big data developers) to support large scale Big data deployments. Scarcity of standardized technological resources and qualified human resources are the practical challenges for the CIOs to implement successful Big data programs.
Use cases are very important in defining a technology’s usage in a business. A wrong use case means that there is no alignment between the business and the technology. Most failed companies in Big data projects have started with a wrong use case. It takes a long time for practitioners to first understand the mistake. By the time, the money, efforts and the time spent on these mistaken projects are too high. CIOs can’t afford so much time, money and resources on these projects and will be questionable by the executive members of the board even though the problem is out of his control. This is the perennial problem we are talking about.
Shadow IT is a term often used to describe information-technology systems and solutions built and used inside organizations without explicit organizational approval. As number of departments in a business started Big data projects, they started utilizing these kind of off-the-shelf products bypassing IT department. This led to increase in usage of auxiliary big data products and services (Cloud services) with no involvement from IT department. Traditionally IT department has to facilitate such services to other departments. But now, Lines of Business (LoB) has exponentially increased the utilization of public cloud services to get the work done. While accessing such services, these departments don’t care about the security considerations that they have to look into, leading to increased security risks, compliance concerns and hidden costs Earle, 2015.
Big data projects support the pervasiveness of Shadow IT into a business environment. Why? Because there are many public cloud services available in the internet that improves the agility of these projects. Hence it is very important for a CIO to devise compliance and security measures to counteract threats from this public data. Recently cisco made a study on CIOs awareness level on Shadow IT services that are utilized by employees and LoB of different companies Froehlich, 2015. CIOs and IT heads estimated a total of 56 services on an average. But actually that sample set was using around 730 cloud services (regardless of the industry). The sad news to CIOs is that, this trend is only going to increase over the period of time.
Why is this amount of shadow IT services usage very dangerous? Shadow IT leads to increase in organizational risk. CIOs should facilitate LoB and the employees by bringing these IT services out of their shadows. It is important for a CIO to devise a Hybrid model for cloud service utility and approve an expansive list of these services to legalize their usage. This way employees and LoB wont circumvent the IT department to use the cloud services. It also helps the CIO reduce riskiness in the business.
So far we have seen the implicit and explicit challenges that CIOs face in managing the Big data projects. Now we shall look into some of the best practices that are followed by CIOs in these projects.
CIOs interested in realizing Big Data to drive impactful business outcomes must be prepared to deal with practical challenges that we spoke about. (skilled Big Data practitioners and immature technology).
In order to overcome these obstacles and derive significant value from Big Data, CIOs should consider engaging professional organizations consulting on Big data and/or cloud services. The cost of engaging Big Data services will most often be more than offset by the increase in value attained by successful Big Data implementations. Professional consultants can also help CIOs in identifying initial Big Data use cases that can serve as proof-of-concepts for driving future projects.
Effective data management is an important strategy for success in Big data projects. There are two types of clouds. Public cloud from where LoB and employees try to consume data. This is unsecure cloud. Then there is private cloud, which are the militarized zones of company data that CIOs traditionally safeguard. The CIOs should device a hybrid cloud strategy, that embraces best of both the worlds. Cisco calls this strategy as CCC Earle, 2015, that is the Choice, Control and Compliance. This strategy effectively gives the CIOs to promise same levels of security and compliance in the public cloud as they showcase with their militarized data.
Statistics say that 41% Korzeniowski, 2015 of companies spend on Big data without knowing the return. Clearly, CIO has become very busy executive and he can’t be accountable for each and every department’s experimentation of big data on his companies. But, it is his responsibility to regulate departments and advise them to start the projects as pilots.
Many reports have confirmed on fact that Big data is not a hype. So it only makes sense for the CIO in developing In-house talent. CIOs can integrate with vendors (suppliers of Big Data products) to create Big data competence centers in the company. This will reduce cost of hiring talents, develops mature Big data practitioners over time and improve relationships with vendors.
Finally, CIOs have to make it clear to the executive board that ROI on Big data spending is with respect to a project. Not all projects will give same levels of payback in a given instance. Some might take time to mature, some might reap benefits immediately. If CIO believes that a project has lot of potential, He should always be in a position to convince the board, regardless of the time it takes to show the results. But at the same time, he should manage to set some expectations year on year to the board to continue the funding.