This page contains the details of a technical writing project accepted for Season of Docs.
Project summary
- Open source organization:
- Apache Airflow
- Technical writer:
- kartik khare
- Project name:
- How to create a workflow
- Project length:
- Standard length (3 months)
Project description
I’ll be working on creating documentation for How to create new workflows easily and effectively. There are some of the steps involved in workflows which are -
- Read
- Pre processing
- Processing
- Post processing
- Save/Action
- Monitoring
Each step can involve multiple tasks and multitude of actions can be taken after each step such as aborting the job if 2 or more tasks fail in a stage or re run a task if it fails for at least 2 times.
Other part of the workflows include executing 2 or more jobs in parallel then utilising their combined result for the next stage.
Another aspect of a workflow is to alert the user in case anything goes wrong either through mail or slack or pager duty.
I also plan on including some non-trivial ways in which workflows can be used such as to run realtime streaming jobs on restart them on any missing data in downstream Kafka topics.
I’ll be working with mentors to make the scope of the project much more refined and then complete the tasks from there on.
Looking forward to amazing few months ahead.