# [AG.DLM.3] Automate data processes for reliable collection, transformation, and storage using pipelines
<a name="ag.dlm.3-automate-data-processes-for-reliable-collection-transformation-and-storage-using-pipelines"></a>

 **Category:** FOUNDATIONAL 

 A data pipeline is a series of steps to systematically collect, transform, and store data from various sources. Data pipelines can follow different sequences, such as extract, transform, and load (ETL), or extract and load unstructured data directly into a data lake without transformations. 

 Consistent data collection and transformation fuels informed decision-making, proactive responses, and feedback loops. Data pipelines play a key role in enhancing data quality by performing operations like sorting, reformatting, deduplication, verification, and validation, making data more useful for analysis. 

 Just as DevOps principles are applied to software delivery, the same can be done with data management through pipelines using a methodology commonly referred to as DataOps. DataOps incorporates DevOps principles into data management, including the automation of testing and deployment processes for data pipelines. This approach improves monitoring, accelerates issue troubleshooting, and fosters collaboration between development and data operations teams. 

**Related information:**
+  [What Is A Data Pipeline?](https://aws.amazon.com/what-is/data-pipeline/) 
+  [AWS DataOps Development Kit](https://awslabs.github.io/aws-ddk/) 
+  [AWS Glue DataBrew](https://docs.aws.amazon.com/prescriptive-guidance/latest/serverless-etl-aws-glue/databrew.html) 
+  [AWS Glue ETL](https://docs.aws.amazon.com/prescriptive-guidance/latest/serverless-etl-aws-glue/aws-glue-etl.html) 
+  [AWS Step Functions](https://aws.amazon.com/step-functions/) 
+  [Data Matching Service – AWS Entity Resolution](https://aws.amazon.com/entity-resolution) 
+  [Blog: Build a DataOps platform to break silos between engineers and analysts](https://aws.amazon.com/blogs/big-data/build-a-dataops-platform-to-break-silos-between-engineers-and-analysts/) 
+  [DataOps](https://en.wikipedia.org/wiki/DataOps) 
+  [Using Amazon RDS Blue/Green Deployments for database updates](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html) 
+  [AWS Well-Architected Cost Optimization Pillar: COST11-BP01 Perform automations for operations](https://docs.aws.amazon.com/wellarchitected/latest/cost-optimization-pillar/cost_evaluate_cost_effort_automations_operations.html)