The Evolution of Big Data and IoT Systems in 2018
Brad Surak I Chief Product and Strategy Officer, Pentaho
Predictions season is once again upon us. And as uncertainty continues to loom above the world at large, we find ourselves looking to technology for answers about the future—for everything from overcoming societal changes to driving economic growth. With so many of our traditional institutions and global agreements in flux, we must maintain some level of stability in the enterprise.
Security threats are everywhere, with opportunists looking to capitalize on the current turbulent environment. But businesses can’t put their operations on hold and wait for things to settle; they must be vigilant and embrace what-if scenario planning. As new challenges continue to present themselves to the enterprise in 2018, IT and operational systems will need to be prepared.
To that end, here are a few predictions around how technology and innovation will influence big data and IoT systems for the better in 2018:
THE RISE OF AUTO MACHINE LEARNING
In 2018, we can expect to see more Artificial Intelligence (AI) techniques that push the boundaries between reinforcement learning, unsupervised learning and autogenerated training—especially in IoT environments where we’ll see more algorithms and bots enabling edge devices to learn on their own. With a shortage of talented data scientists with the necessary advanced skillsets to label training datasets from sensors and other systems for AI, machine learning or deep learning (a messy, expensive and time-consuming process in itself), a supervised learning model is completely impractical. Automated machine learning can reduce the still-expensive bandwidth required to submit data from the edge, while enabling programmed, unsupervised learning on real-world data. While more efficient, this does raise other, more ethical questions around how algorithms arrive at their decisions and who is responsible for those decisions and their consequences.
ALGORITHMIC TRANSPARENCY WILL START TO BECOME A COMPETITIVE DIFFERENTIATOR
As noted above, visibility into how algorithms make decisions will become increasingly important with the acceleration into auto data science. As algorithms start playing a larger role in areas affecting people’s lives and livelihood, more people will demand transparency on how the algorithms are created. Companies that market their algorithmic transparency in 2018 as a key, competitive differentiator will stand out from the masses.
BEWARE OF FAKE DATA SCIENTISTS
The big data skills shortage is creating a talent pool of “fake” data scientists—aka opportunists taking advantage of the fact that the role of the data scientist is poorly defined, but comes with a big pay check. Companies lacking resources should interview candidates extra carefully in order to spot those with real potential. The “holy trinity” of data science fundamental skills to look for include: strong problem-solving skills, an advanced degree in applied mathematics or statistics, familiarity with statistical analysis technologies like Python or R and familiarity with big data technologies like Hadoop and Spark.
EDGE DEVICES, OPEN DATA AND SECURITY WILL BRING IT AND OT CLOSER TOGETHER
As the IoT ecosystem continues to advance in 2018—with things like edge devices getting smarter—IT and operational technology (OT) systems will move closer together. Additionally, hardware manufacturers will begin to embrace open data policies. Driven by the need to move to new service models and optimize their hardware devices, they will begin to partially open up their systems, which will have an impact on security. As such, we’ll start to see priorities shifting from data protection and access denial to securing system operations. Organizations that embrace these developments will become more IoT-enabled as they move up the IoT maturity model.
Ultimately, despite these uncertain times, we may take solace in knowing that innovations in big data and IoT will continue to advance. Embracing these technologies will help to ensure your company is able to not only adapt to this new and ever-changing normal, but to thrive as well.
To get a more in-depth look at my full list of predictions for the big data and IoT market in 2018, check out my LinkedIn Pulse post from today, “Hitachi Vantara Predicts How Big Data and IoT Systems will Evolve in 2018.”
And from all of us at Hitachi Vantara—may you find prosperity and thrive in the New Year!
Brad S I Chief Product and Strategy Officer, Pentaho