Is a data pipeline a platform?

All organisations would benefit from a place where they can collect and analyse data from different parts of the business. Historically, this has often been met by a data platform, a centralised data store where useful data is collected and made available to approved people. But, whether they like it or not, most organisations are, in fact, a dynamic mesh of data connections which need to be continually maintained and updated.

Following a single platform pattern often leads to a central data engineering team tasked with implementing data flows. The complexities of meeting everyone’s needs and ensuring appropriate information governance, as well as a lack of self-service, often make it hard to ingest new data sources. This leads to backlog buildup, frustrated data users, and frustrated data engineers.

Thinking of these dataflows as pipelines changes the mindset away from monolithic solutions, to a more decentralised way of thinking - understanding what pipes and data stores you need and implementing them the right way for that case whilst reusing where appropriate.

Last updated

Was this helpful?