Azure Data Factory (ADF) is a data pipeline orchestrator as well as an ETL software that's an integral part of Microsoft Azure. Microsoft Azure cloud ecosystem. ADF can draw data from outside sources (FTP, Amazon S3, Oracle and numerous other sources) and transform it and filter it, improve it, and then move it to a different destination. For my job on the health-data initiative, we use ADF to control the flow of data from raw data ingestion to refined analysis that is ready for display.
There are numerous useful resources to get started with ADF which include the intro and an rapid-start. When I first started out using ADF but, I could not get any clear explanation of the fundamental concepts it's built on. If you're able get one or all of them, Azure Data Engineering certification that is designed around the tasks you do and the job you're doing will allow you to increase your professional profile and get recognition from the market.
In order to get ADF to perform real work for you is a matter of integrating the various layers of technology that are starting at the highest level of abstraction you work with all the way down to the application closest to your data.
- The pipeline is the graphic user interface that lets you insert widgets and trace data routes
- Activity is a graphic widget that can do something with your data
- Source and Sink are the components of an activity that define where the data is coming from and where it will go.
- Data Set A clearly defined set of data ADF can use to operate
- Linked Service, the connection information which permits ADF to connect to an external data resource
- Integration Runtime is the glue/gateway layer that allows ADF talk to other software independent of itself
Understanding the function of each layer and the way it contributes to the entire ADF solution is crucial to making the most of the tool. It is easier for me to comprehend ADF by looking at each layer in reverse, beginning at the bottom , close to the data.