How do you do your CI/CD pipeline? Sorry if this is a dumb question. Just wondering how the tests and deployment usually runs. Is it on a per team basis? Is it the whole release getting deployed to Test many times per day? What happens if too many automated tests fail or there is not enough coverage, does it abort the deployment? If so, how can every team get delayed by every issue - is that actually a good policy?

My pipeline is very slow and requires a team of 12 people working in shifts to complete it. I’m not an expert but I know it does a lot of steps and never completes without manual intervention. I would like to help but I’m not sure how bad it is.

A lot of it is specific to the CI -/ CD software. But in general, as a sort of "good recommendation"…

Always visualize a full CI - CD pipeline. Draw it, add information to the drawing, add facts (e.g. runtime, if it could run in parallel, resource consumption, costs, ...).

Most often this leads to finding out the culprits, but makes changes to the CI / CD pipeline also easier.

Especially parallelity can be great - as long as the CI cluster doesn't melt, it's fine :) ;)

Back to the pipeline itself:

I'm very fond of minimalistic units in pipelines. A unit is for me a "finished task".

As one example: Build a generic image without environment specific information. That's one unit.

For testing, you have to add the environment specific information (unit 2), run the test suite (unit 3).

A unit is the smallest step possible so to say.

Reason I recommend it if possible: It makes debugging way less painful - as you don't have a myriad of things a single job does, but instead one specific task (unix philosophy).

Plus it makes parallelization more manageable and resource consumption easier to track.

This philosophy goes all down to the programming.

If you have e.g. several different types of unit tests (e2e, unit, ...) or a lot of unit tests / large codebase... Use test suites. Make each group of tests independent, if possible. If you have thousands of test suites, chunk them into categories via test suites.

Running "more" commands might seem counterintuitive, but the start up time of an environment is usually negligible compared to the long runtime tests have...

Another nice thing when you create seperate units for tests: you have a fail fast approach. Most CI / CDs allow the process to continue running despite failures, but you'll get the output of the process for a specific test suite, thus one knows "earlier" that this build will not be successful. More failures could happen later, yes - but maybe you don't have to wait for them.

@IntrusionCM i have adopted a very similar approach to building Pipelines, and it works out quite well so far.
Build little tasks, make them fail fast, and try to make them parallel.

If a task changes slightly, you don't need to mess around with other stuff, only with the scripting of that specific task.

@thebiochemic exactly. plus you know what the task does (i tried to avoid words like task / job etc., hence unit - as it could be entirely different meaning depending on software)...

nothings sucks more than a 20 Mb plus logfile of "I did everything and failed somewhere".