A miniature racing track with curves and slopes. 4 remote controlled speed cars by 4 different players. People cheering for the winner. After every lap, the players would swap the cars (thereby getting a chance to race on different track with different turning angles). Here we were, witnessing an interesting racing scenario. What was odd about it – It was not a gaming zone, it was a tech convention (ElasticON). And the purpose of this race was to demonstrate how one can collect and analyze real time data.
In fact, throughout the convention, there were many places where real time data was being collected constantly. Even at coffee stations, data was being collected and flashed on the display boards about the types of coffee and the number of cups being served. This was to highlight the popularity of the various types of coffee, using real time data.
We are living in a world where everything is being ‘datafied’. So many of our actions in daily life are being captured through sensors and converted into data in real time – right from the time we wake up till we hit bed at night. When human actions are being captured, there is no dearth of data being collected from the machines, computer systems and applications. But this is leading us to another challenge – massive amounts of data and making sense out of it. Debra Logan of Gartner has very aptly described this challenge as ‘abundance of data’ but ‘scarcity of insight’.
Today, decision making for businesses has become dependent on the availability of data. To the extent at times, the new data set being churned out is now forcing a change in business plans; and it is happening in near real time scenarios. The business planning for an enterprise has become a dynamic function rather than just once a year (or once in 3 years for some organizations) affair. This development has brought about a change in the expectations from CIOs and IT managers. They are no longer expected to just manage the technology of the organization but are now tasked to manage ‘data’. Many organizations have gone to the extent of creating a new role namely Chief Data Officer. Managing data is no more a purview of only IT organisations. Now, business units are directly empowering their teams to manage and analyze data in order to drive business decisions.
Data being generated is not just limited to the core operational systems that are under traditional IT organizations. With special sensors being deployed everywhere, data is being generated by so many sources – office doors, coffee machines, drinks dispensers, photocopying machines, video cameras and what not. On top of it, there are external data sources – news feeds, social media interactions, competitive information, buyer behaviors, buyer interactions with various promotions and purchase channels. Just so much data!
Thus, data generation and data capturing is happening in several systems which are often far away from the core operational systems.
In such scenarios, the older approach to data analysis wouldn’t work. In the traditional approach, data is brought to a central place, cleaned up (ETL) and selective data is fed to the DWH system. This process, though has been useful in the past, is no longer sufficient today. Also, this approach means all the data has to travel back to the central place for storing, analyzing and decision making. In many cases, this is even not necessary. Many times, local data needs to be analyzed in real time and local decisions need to be made immediately. Only a metadata is needed for further analysis in a batch mode. For example, some decisions regarding a bank’s branch operations need local data and local decisions.
Hence, a new approach is emerging fast. In this new method, analytics engines are embedded on clusters of data and insight given to the managers in near real time. The metadata then flows to the central system for further insights. In the IoT world, this is even more appropriate. The sensors are being deployed on remote locations such as street lamps, on windmills, airplane engines etc. For localized decision making from such devices, a light weight (almost embedded) analytics engine is being deployed that gives actionable insights to the relevant people at the location in real time. This analytics engine can combine past trends and other environmental data and provide some prescriptive analytics for timely action.
In a nutshell, what we are witnessing today is analytics is going to where the data is. Instead of the data flowing to the analytics systems. Welcome to the world of ‘Distributed Network of Analytics’.