Dan Graham - Impact of IoT on Data Architectures

IoT has created a tidal wave that data savvy organizations can turn into profitable business solutions. Most IoT data comes from sensors, which are now attached to almost every device imaginable, from factory floor machines and agricultural fields to your cell phone and toothbrush. But IoT is forcing companies to rethink their data architectures to ingest, process, and analyze streaming data in real-time. 

To help us understand the impact of IoT on data architectures, we invited Dan Graham to our show for a second time. Dan is a former product marketing manager at both IBM and Teradata, renowned for combining deep technical knowledge with industry marketing savvy. During his tenure at those companies, he was responsible for MPP data management systems, data warehouses, and data lakes, and most recently, the Internet of Things. 

Key Takeaways:

• IoT is the biggest market expansion since the internet itself.
• Some analytics will move to the Edge and IoT platforms but the deep dive analytics will stay in the data warehouse and the data lake.
• You should only use edge systems to have real-time reactions because of the limitation of the memory. 
• The value of the sensor data increases exponentially when you add it to the data warehouse. 
• You can't store all sensor data into the data warehouse because of the challenge of volume so it's better to pick anomalies or aggravate data based on the use case.
• The database administrators and data science teams need to figure out the algorithms to deal with the format of IoT data. 
• If your industry is heavily regulated, you would need to keep the data for a longer period of time based on the industry. 
• To speed up queries, use compression by using the timestamp technique. 
• The IoT data should be put in the data lake so when needed, the company can perform deep-dive analytics. 
• The amount of code (50 million lines) today's autonomous car uses is only 10 million less than Facebook's code.
• Car manufacturers have reached the conclusion that AI hasn't made enough progress for them to take the step into autonomy. 

---