The IT industry is notorious for creating buzzwords, a lot of noise, marketing material, research papers about predictions often times. Since I have been long enough in the industry, I can comfortably say that more trends or predictions have been created than actually delivered. At the outset, Big Data seemed to be one among that. The discussions seemed to touch an unusual high in 2013 and surprisingly continues to dominate the discourse, now in conjunction with ‘digital transformation’. I must admit, this is probably the most exciting time in the IT industry after the advent and proliferation of internet.
If one trend has managed to stay at the top of the discussion for this long, there has to be something in it, isn’t it? Despite the fact that you might have started feeling fatigued with daily news about Big Data, its virtues, size of the business etc., it is everywhere around you and you cannot escape from it, so let’s see why should one take it seriously.
By using the power of Big Data, many new businesses have come up, existing businesses have been challenged in a very short period of time and new ways of doing business have been created. There are many examples to illustrate this point – self driving cars, entire e-commerce businesses (recommendation engine in particular), Uber and its clones, payment solutions (mobile payments, wallets etc), Auto corrections in typing, voice based search and communication with mobile devices, use of sensors in heavy industry for predictive maintenance, DNA analysis, cancer research, drug formulation, fitness gadgets and the list can go on. Virtually, every aspect of life has been impacted by use of Big Data approach. So there is a huge credibility behind the Big Data buzzword and yet this is just the beginning. Hence, everyone in the IT industry should consider its impact on businesses and on personal growth.
What is that making such an impact?
There are many factors that define Big Data approach, but one thing is for sure that it is not a technology. It is about the approach and its application in the business. Unlike many other technologies, Big Data discussion has to start at the business level – how one can make a customer’s life better. To cite an example, a fitness equipment company should be thinking about how to help users use the equipment right way, how to help users monitor progress and give insight into impact on their bodies. This thinking led many equipment manufactures to combine data from sensors, camera and user’s health parameters. By mashing up this data on a continuous basis, users can get not only the feedback on correct usage of the equipment but also insight into impact on their bodies. It further gives them a predictive score. Take another example – of a refrigerator company. Consumers have one typical challenge – when you go to a grocery store, you tend to forget what is already stocked up in your refrigerator. You end up buying either a lot more or miss some crucial items. By attaching the camera to the doors, connecting them to home wi-fi network and making them available through mobile app, company has solved this common problem. Now the mobile app has the information about consumer’s buying habit.
So how are things like these a part of big data?
Essentially there are two major situations happening:
1) Datafication of Information: Information was always available. For example, in case of a fitness equipment manufacturer – it had the information about how long the equipment was used, how it was being used. But now recording that information in a continuous manner along with time of day, length of use, intervals, user info, user health data etc., it is ready to be analysed. Now this data can give much more insight than the random information available earlier. Another example of heavy engineering equipments, the manufacturers specified the periodicity of maintenance – annual, bi-annual etc. But after putting in the sensors and connecting them to the data gathering cloud, that information has been made much more granular. Combining that granularity with other parameters such as environmental temperature, heat emission from the equipment, pressure on crucial elements of the equipment etc. gives much more insight to it. It also enables creating alerts and predictability in maintenance time cycle. This is being made possible due to the data storing technologies, sensors, connectivity and cloud.
2) Machine learning: This is a very crucial aspect for truly leveraging the power of data. When a self-driving car is built, its computing engine is not fed with all the rules. Rather lot of data is fed to it and that computing engine is made to learn from it and then devise the course of action. It is made to learn when to stop, how to respond in case of closer contact etc. And this is done by feeding data from lots of sensors around the car and combined with environmental conditions. Based on the data and patterns, the computing engine starts learning the rules and applies it to the car. It makes mistakes but learns based on the experience and corrects itself. Take another example – that of translation between English and French. In the earliest attempts, the computing engine was fed with lot of rules and grammar of both the languages. The translation worked for maximum up-to 60% accuracy, it never gave results better than that. Then in a change of approach, they fed English books and their French translated version to the engine. Many such books were fed to the engine. The computing engine was then allowed to learn the rules, context and phrases. The result? We all are experiencing the power of Google translation and it works for so many languages. The key in these examples is, letting the computing engine learn the rules, understand the patterns, develop co-relation algorithm on its own. For some reasons, this aspect is being referred to as ‘machine learning’.
I can cite many more real life wonders of Big Data application, but I’ll stop this post here today. There is so much that is happening due the Big Data approach. Many technologies are emerging to address these requirements. The database technology itself is fast evolving.
My colleagues in Singapore are going to talk about some of these aspects as well as will bring some insights on to how to get ready for your next Big Data implementation. You can visit here for more information and also register yourself for the “Big Data Talk” session if you are in Singapore on 27^th Sept 2016.
Interesting times are ahead.