A workflow that moves data from one or more sources to one or more targets and often transforms it in the process. Typical pipelines include extract, transform, and load (ETL), extract, load, transform (ELT), and streaming.
Added Perspectives
Data pipelines are the means by which we move data through today’s complex analytics ecosystems.
- Dave Wells in The Complexities of Modern Data Pipelines
July 4, 2018 (Blog)
Data pipeline automation uses technology to gain efficiency and improve effectiveness for data pipeline development and operations. And it goes beyond simply automating the development process to encompass all aspects of pipeline engineering and operations including design, development, testing, deployment, orchestration, and change management.
- Dave Wells in Architecting and Automating Data Pipelines: A Guide to Efficient Data Engineering for BI and Data Science
October 16, 2020 (Report)
The purpose of a data pipeline is to move data from an origin to a destination. There are many different kinds of data pipelines: integrating data into a data warehouse, ingesting data into a data lake, flowing real-time data to a machine learning application, and many more. The variation in data pipelines depends on several factors that influence the shape of the solution.
- Dave Wells in Data Pipeline Design Patterns
May 6, 2020 (Blog)
Relevant Content
Sep 08, 2020 - Enterprises use both standard and custom data pipelines to process raw data into analytics-ready data sets. They must apply the right mix, creating...
Nov 13, 2020 - This blog, the first in a series, examines the evolution of enterprise data environments, rising pain of complexity, and resulting requirements for data...
Related Terms
Providing modern, comprehensive data solutions so you can turn data into your most powerful asset and stay ahead of the competition.
Learn how we can help your organization create actionable data strategies and highly tailored solutions.
© Datalere, LLC. All rights reserved
