Big Data Endeavors Across Manufacturing And Heavy Industries
To give you a better idea about how data streaming technology is transforming manufacturing and heavy industry today, I’ll provide a few examples below.
Using data streaming technology, you can stream processing technologies quickly (from milliseconds to minutes) and identify interdependencies in vast industrial and manufacturing systems. Stream processing at a medical warehouse could quickly warn an operator about a temperature rise, allowing them to react quickly and prevent costly damages.
Alternately, stream processing could help detect fraudulent behavior of a hacker in an industrial database and block the attacker before it damages the system. Production line monitoring, supply chain optimization, smart grid technology — these are all data stream processing applications for manufacturing and heavy industries. That list is far from exhaustive, but the value is immense.
Impressive, isn’t it? If that’s what we have today, imagine what will happen in even two years’ time. You can stay ahead of the curve today by starting small with simple big data endeavors.
Your First Big Data Endeavor
One of the simplest use cases for data streaming technologies is conveyor line monitoring. This suits the manufacturing companies operating numerous production lines. It can also be a useful solution for mining companies. For example, you could transport goods or materials from point A to point B. The conveyor chain may include up to dozens of conveyors. Therefore, each conveyor should be monitored to ensure flawless and fail-safe operation of the whole chain.
To achieve this, attach each of the following things to the conveyor from the chain:
• emergency stop switches
• sound alarm system
• current sensor (on all driving motors)
• temperature sensors (on the motor bearings)
• speed sensors (on the conveyor pulleys)
• conveyor belt feed rate sensors
• belt run-off sensors
• optical belt scales
All of these sensors can collect data and send it to the server for processing and analysis through a data streaming framework. This way, data streaming can help detect misalignment in the driving motors operation and motor bearings wear. It can also analyze overall conveyor utilization effectiveness and provide statistics that show how many goods or materials were delivered to the destination point.
What Is Behind The Magic Of Big Data?
As you can see, deriving value from big data is not magic. It’s all about technical solutions and how you apply them. Based on my experience, for big data processing I recommend the following stream processing frameworks: Apache Storm, Spark Streaming and Kafka Streams. Surely, each boasts its own incredible characteristics, and each also has certain operational limitations. Based on my experience, here’s an overview of each tool.
1. Apache Storm
• Low latency
• Mature (well-tested and widely applied)
• High throughput
• Ideal for simple streaming use cases
• Scalable with parallel calculations running across multiple clusters of machines
• Fault-tolerant (automatic nodes restart, and there is workload repositioning across nodes)
• Native streaming
2. Spark Streaming
• Combination of batch and streaming processing (Lambda architecture support)
• High throughput
• Mature
• Records are processed once even in case of failure
• The machine learning library provides a set of machine algorithms for classification, regression, collaborative filtering, etс.
• Fault-tolerant by default
3. Kafka Streams
• Incorporates the best from Kafka functionality and does not require a separate cluster
• Lightweight library
• State management
• Exactly once, end to end (as Kafka is both the source and the destination)
• Native streaming
• Combines distributed and traditional messaging systems, enabling storage and stream processing (this provides better measurability of streaming data pipeline, low latency and better storage capabilities, and ensures integration with offline systems, which comes in handy in case they collapse)
For the above use case, I recommend using Spark Streaming. It will fit the project because it boasts ready integration with the advanced analytics tools and allows you to perform stream processing through micro-batch processing, driven by discretized streams. In addition, Spark Streaming is part of the Hadoop and Azure HDInsight ecosystem, so if you already use Hadoop, it will integrate into your big data stack.
A Recap
Companies are now realizing how mighty big data can be. We are starting to see the massive potential it has in the world around us. Big data has applications from patient health monitoring to traffic flow and online shopping recommendations.
Manufacturing and heavy industry are also ready for big changes in the big data era. No doubt, a true transformation will take tons of time and resources. However, backed by the potential of modern technologies, the businesses of today will manage to build inspiring industries of tomorrow. Don’t miss a chance to be among the winners.
Originally published at Forbes.com.
