Use Airflow Plugins, to connect to third-party tools/data sources.When you want to implement a task out of the box, create custom operators using the BaseOperator Class for re-using them.While migrating the airflow metadata db to a new database, use "airflow upgradedb" (For 2.0 airflow db upgrade) instead of "airflow initdb" (For 2.0 airflow db init).Use macros and templates to avoid hard coding values.(For data processing, Run the processing script in a batch compute or spark framework and invoke from airflow) Its focus is to orchestrate data flow jobs. Avoid creating complex data processing logic in Airflow, as airflow is not intended to work as a data processing engine.Once you have airflow installed, modify the configurations of your airflow instance in the master config file - airflow.cfg.There are other workflows in the market similar to Airflow : Whereas if you want to host your product in multi-cloud -Īirflow will be a better fit. For GCP, Google Cloud Composer will be the best fit.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |