Pipeline execution sequence - Azure Pipelines (2023)

Azure DevOps-Dienste | Azure DevOps Server 2022 – Azure DevOps Server 2019

Runs represent the execution of a pipeline. During a run, the pipeline is processed and agents process one or more jobs. A pipeline run includesJobs, steps and tasks. It runs on continuous integration (CI) and continuous delivery (CD) pipelines.

When you run a pipeline, a lot happens behind the scenes. While you often don't need to know anything about it, it occasionally helps to stay in control. At a high level, Azure Pipelines:

• process pipeline
• Request one or more agents to run jobs
• Send requests to agents and collect results

On the agent side, an agent for each job:

• Get ready for work
• Complete each job step
• Report results to Azure Pipelines

possibly jobssucceed, fail, or abort.There are also situations in which a jobcannot be completed. Understanding how this happens can help you troubleshoot.

Let's look at each action individually.

process pipeline

To convert a pipeline to an execution, Azure Pipelines follows several steps in this order:

1. first expandmodelsand evaluatemodel expressions.
2. Then evaluate the dependencies of thescenarioLevel to select the first levels to run.
3. For each phase selected for execution, two things happen:
   • All resources used in all jobs are logged and validatedpermissionrun.
   • To assessJob-level dependenciesto select the first jobs to run.
4. Expand for each job selected to runmultiple settings(Strategy: MatrixoStrategy: parallelin YAML) in various runtime jobs.
5. Evaluate for each runtime jobConditionsto decide whether this job can be run.
6. request an agentfor each eligible runtime task.

When runtime jobs complete, Azure Pipelines detects whether new jobs can run. If that's the case, repeat steps 4-6 with the new orders. Also, once phases are completed, steps 2 through 6 are repeated for all new phases.

This order helps answer a common question: why can't I use certain variables in my template parameters? Step 1, expanding the template, only affects the body of the YAML document. There are no runtime variables during this step. After step 1, the model parameters have been resolved and no longer exist.

It also answers another common problem: why can't I useVariablesresolve service connection/environment names? Resources are authorized before a stage can run, so the stage and task level variables are not available. Pipeline-level variables can be used, but only variables that are explicitly included in the pipeline. Variable groups themselves are a resource that requires authorization, their data is also not available when checking resource authorization.

request an agent

Whenever Azure Pipelines needs to run a job, it will ask for itPoolforAgent.(server jobsare an exception as they run on the Azure Pipelines server itself).Hosted by MicrosoftYself-hostedAgent groups work a little differently.

Microsoft Hosted Agent Pool Requirements

First, the service checks your company's side jobs. Sums up all jobs run on all Microsoft hosted agents and compares them to the number of parallel jobs purchased. If no parallel slots are available, the job must wait in a slot to be released.

As soon as a parallel slot is available, work is routed to the requested agent type. Conceptually, the pool hosted by Microsoft is a large global pool of machines. (That's actually many different physical groups, broken down by geography and operating system type.) Based onvmImage(in YAML) or group name (in the classic editor), an agent is selected.

All Microsoft group agents are new virtual machines with no pipelines running yet. When the job completes, the agent virtual machine is disposed of.

Self-hosted agent pool requests

similar toGroup hosted by Microsoft, the service first checks for parallel tasks in your organization. Sums all jobs run on all self-hosted agents and compares them to the number of parallel jobs purchased. If no parallel slots are available, the job must wait in a slot to be released.

As soon as a parallel slot becomes available, the self-hosted pool is searched for a compatible agent. Powered by self-hosted agentscapabilities, which are strings that indicate that specific software has been installed or settings have been configuredrequirementsThese are the skills needed to get the job done. If a free agent with skills matching the pipeline requirements cannot be found, the job will continue to wait. If there are no agents in the pool whose skills meet the requirements, the job will fail.

Self-hosted agents are typically reused from one run to the next. For self-hosted agents, a pipeline job can have side effects, for example, B. hot caches or already have most commits in the local repository.

Get ready to run a job

After an agent accepts an assignment, they must do some prep work. The agent downloads them all (and caches them for next time).Tasksnecessary to perform the job. Frees up disk space to store source code, artifacts and output used in execution. so it beginsexecution steps.

complete each step

The steps are performed one after the other. Before a step can be started, all previous steps must be completed (or skipped).

The steps are implemented byTasksThe tasks themselves are implemented as Node.js or PowerShell scripts. The task system passes input and output to support scripts. It also offers some general services like changing the system path and creating new ones.pipeline variables.

Each step runs in its own process, isolating it from the environment left by previous steps. Due to this step-by-step model, environment variables are not persisted across Agent steps:registry commands.When a task or script writes a logging command to standard output, the agent performs the requested action.

There is an agent command to create new pipeline variables. Pipe variables are automatically converted to environment variables in the next step to define a new variablemiVarValormeinWert, a script can do this:

echo '##vso[task.setVariable variable=myVar]myValue'

Write-Host "##vso[task.setVariable variable=myVar]myValue"

Report and collect results

Each step can report warnings, errors, and failures. Errors and warnings are reported on the pipeline summary page and the task is marked as "successful with problems". Errors are also reported on the summary page, but mark the task as "failed". .A step is an error if it explicitly reports an error (using a##vsocommand) or terminate the script with a non-zero exit code.

As the steps run, the agent continuously sends outbound lines to the service. That's why you can watch a live stream on the console. At the end of each step, the full step output is also uploaded as a log file. Logs can be downloaded once the pipeline is complete. Other items the agent can carry includeartifactsYTest resultsThey will also be available once the channeling is complete.

status and conditions

The agent tracks the success or failure of each step. If the problem steps succeed or fail, the task status is updated. The task always reflects the “worst” result of each of its steps: if a step fails, the task also fails.

Before executing a step, the agent checks that stepIllnessto determine whether to run it. By default, a step runs only when the task status is Succeeded or Succeeded with problems. Many jobs have cleanup steps that need to be done no matter what so they can specify the "always"() condition. Cleanup steps can also be configured to run only forcancellation.A subsequent cleanup step cannot prevent the job from failing; Jobs can never return to success after going into failure.

Wait times and disconnections

Each job has a time limit. If the job is not completed in the specified time, the server will cancel the job. Attempts to tell the agent to stop and marks the job as cancelled, which means canceling all remaining steps and loading all remaining steps.Results.

Jobs have a grace period known as the demolition lead time, in which all demolition work is completed. (Remember that steps can be marked for executionalso in case of cancellation.) If the agent has not reported that the job completed after the timeout plus the interrupt timeout, the server marks the job as an error.

As Azure Pipelines distributes work to agent machines, agents may stop responding to the server from time to time. This can happen when the agent's host computer fails (power failure, VM shutdown) or when a network error occurs. For detection Under these conditions, the agent sends a heartbeat message once a minute to notify the server that it is still operational. If the server does not receive a heartbeat for five consecutive minutes, it assumes that the agent will not return. The job is marked as an error telling the user to retry the pipeline.

az pipelines run list [--branch] [--org] [--pipeline-ids] [--project] [--query-order {FinishTimeAsc, FinishTimeDesc, QueueTimeAsc, QueueTimeDesc, StartTimeAsc, StartTimeDesc}] [--reason {all, batchedCI, buildCompletion, checkInShelveset, individualCI, manual, pullRequest, scheduled, activated, user-created, validShelveset}] [--requested-for] [--result {canceled, failed, none, partialSucceeded, succeeded }] [ -- status {all, canceling, completed, in progress, none, not started, deferred}] [--tags] [--top]

Pipeline run list az --status completed --successful result --top 3 --output table id number Run state result ID pipeline name source branch end time ------- - -- ----- -- - --------- --------- ---- --- ------ -------- --- ----- -- -------- ---------- ----- --- ---- - - -----125 20200124.1 completed successfully 12 Githubname.pipelines-java- master 2020-01-23 18:56:10.067588 56.633450 Manual122 2021 completed successfully 12 Githubname.pipelines-java master 2020-01-23 11:48:05.574742 manual

az pipes running show --id [--open] [--org] [--project]

az pipelines running show --id 122 --open --output table run id number status result pipeline id pipeline source name branch queue time reason -------- ---------- -- - - ----- --------- ------------- ------------------- -- - - --- --------------- ---------------------------- -- - --------- -- ---- --123 20200123.2 completed successfully 12 Githubname.pipelines-java master 2020-01-23 11:55:56.633450 manual

az-Pipelines executa a tag add --run-id --tags[--org][--project].

az-Pipelines performs adding tags --run-id 123 --tags YAML --output json[ "YAML"]

az-pipelines run lista de tags --run-id [--org] [--project]

az pipelines executam lista de tags --run-id 123 --output tableTags------YAML

az-Pipelines performs tag stripping --run-id --tag [--org] [--project].

az pipelines perform tag removal --run-id 123 --tag YAML