Repetitive tasks as a template in your pipeline

benefits of having repetitive tasks as template in your pipeline :

I believe workflow automation is a key component of any business’s success.

Workflow automation saves time and boosts efficiency. It’s a series of automated actions that improve everyday processes and bring collaboration to a whole new level.

But What is workflow automation?

Workflow automation is the process of identifying tasks performed by employees and automating them with tools, apps and technology.

When CI/CD meets workflow automation ?!

Continuous integration (CI) and continuous delivery (CD) embody a culture, set of operating principles, and collection of practices that enable application development teams to deliver code changes more frequently and reliably. The implementation is also known as the CI/CD pipeline.

CI/CD is one of the best practices for DevOps teams to implement. It is also one of agile methodology best practices, as it enables software development teams to focus on meeting business requirements, code quality, and security because deployment steps are automated.

CI/CD practices enable development teams to do small code changes and deliver the code more frequently.

Which part of the software can be automated ?

When it comes to CI/CD and workflow automation any part of the software components can be automated , you could say components , modules , dependencies and any related steps , from testing to generating reports.

But what if you have more than one service that has the same pattern ?

To answer this question we need to look at the DevOps tool chains in the DevOps industry. From Jenkins to GCP cloud build and maybe Azure DevOps,

All these tools and services are providing you with different ways of creating pipelines and to reference to other resources while executing the flow.

For example Jenkins supports Jenkinsfile and Azure DevOps and GCP Cloud build support yaml file configuration that gives you ability to load configs from source control.

In the simple pattern you have to use a yaml configuration file and repeat this yaml file into multiple services that are having the same steps for build and deployment.

What will happen if you have 40 services and suddenly get to know that you have to fix 1 step within the pipeline ?

This is a horrible situation in that you might have services with complex and long running pipelines.

You have to first start fixing the pipeline in one service and after you make sure it’s done and stable then you need to spend some time to repeat the same steps for all other services. And since it’s a manual intervention which you have to copy paste or rewrite the process, there might be typo issues or other errors you would see during this process. So basically I can say you consumed a lot of energy and time.

So what is the best practice ?

In this kind of scenario where you have a services which is following the same patterns with very minor differences between, I would say the best practice is to create a reference template instead of a pipeline for a specific service itself.

For example in Azure DevOps you can create a template (Azure DevOps Yaml Template) and just reference your steps to an input from the user.

If we get back to those 40 services , instead of editing 40 services you just have to create a template once and use a reference to the template.

If for any reason you want to edit something in your pipeline , you don’t have to spend so much time working on 40 pipelines but you have to only work on one reference template and when you are done, all 40 services will be updated all together.

Conclusion

Understanding the pipeline and processes that you are going to make a workflow automation out of it is an important step in a designing the CI/CD pipeline and creating a good repetitive CI/CD that is repeatable in many services and can be edited and one place will save you and your team a lots of time and energy.

DevOps Engineer, Cloud Architect