Play with Jenkins
Jenkins is one of the most popular tools in the automation world. Much flexibility, open-source and free availability, massive plugin support and integrations, easiness to start are some of the key factors which have been caused for the popularity of Jenkins.
You can start building pipelines with Jenkins by spending a couple of hours with Jenkins documentation. But to build some smart and nice pipelines you have to have a thorough understanding of Jenkins features and also you need to have some experience as well.
Here I’m sharing my experience on common use cases and issues which you will face while using Jenkins. I hope this will help you to polish your pipelines and make them better.
Note: Almost all the solutions here target multibranch declarative pipelines (pipelines with Jenkinsfiles in version control). But the same concepts can be applied to scripted and non-multibranch pipelines as well.
Agent Configurations
The way you want to assign agents to jobs will depend on your use case. Here I will explain different types of agent setups with their use cases.
Using a single agent for the complete pipeline
pipeline {
agent any
stages {
...
}
...
}This is the easiest way to assign an agent to your pipeline. This syntax will assign an agent at the beginning to be used for your entire pipeline and you don’t need to define agents for your stages thereafter. This will also guarantee that all the stages of the pipeline will be executed on the same agent. If your stages have some dependencies on previous stages (i.e. files created in the previous stages are used by latter stages) this configuration will help you.
Once the agent is assigned the executor will be consumed until the pipeline execution is complete even though you use the executor for stages or not. For example, let’s say you have 3 stages in the pipeline and you are using the agent assigned at the beginning for your first 2 stages. But for your last stage if you want to use another agent with some specific configurations still you can acquire that specific agent for the 3rd stage by defining the agent again in stage 3. But the agent assigned at the beginning will not be released until the pipeline execution is complete (all 3 stages are complete).
You cannot wait for user inputs without consuming an executor with this syntax. Check the following example.
pipeline {
agent any
stages {
stage('Deploy to QA') {
steps {
sh 'echo Run QA deployment here'
}
}
stage('Promote to Production?') {
agent none
input {
message "Do you want to deploy to Production?"
}
steps {
echo "Promoted to Production deployment"
}
}
stage('Deploy to Production') {
steps {
sh 'echo Run Production deployment here'
}
}
}
}Even though you have explicitly set agent none in the Promote to Production? stage, the agent assigned by agent any at the beginning will be consumed until the user input is given. If your intention is to wait for a longer time period (maybe infinitely or for hours) the above syntax will waste your executors without doing any effective work. In such a situation better to assign agents for individual stages and wait for the user input without consuming an executor.
Using individual agents for each stage
If you require to use agents with different configurations for different stages then it will be good not to assign an agent in the beginning and assign them in stage level.
pipeline {
agent none
stages {
stage('Deploy to QA') {
agent {
label 'NON-PROD'
}
steps {
sh 'echo Run QA deployment here'
}
}
stage('Promote to Production?') {
agent none
input {
message "Do you want to deploy to Production?"
}
steps {
echo "Promoted to Production deployment"
}
}
stage('Deploy to Production') {
agent {
label 'PROD'
}
steps {
sh 'echo Run Production deployment here'
}
}
}
}This method is useful when you have user inputs between the stages also. Here an executor will not be consumed while you are waiting for the user input in thePromote to Production? stage. Hence you can wait infinitely for the user input without any issue.
Using multiple agents for parallel stages
If you have parallel stages and you need to perform some CPU intensive tasks in all the parallel stages then definitely you will need to execute those parallel stages on multiple executors.
pipeline {
agent any
stages {
stage('CI') {
parallel {
stage('Unit Tests') {
steps {
sh 'echo Run your unit tests here'
}
}
stage('Integration Tests') {
steps {
sh 'echo Run your integration tests here'
}
}
}
}
stage('Deploy to QA') {
steps {
sh 'echo Run QA deployment here'
}
}
}
}The above pipeline will execute Unit Tests and Integration Tests stages parallelly. But both the stages will be executed on the same executor. The problem here is if both of these stages require more resources (CPU/memory) then you might face some performance issues here. There is a possibility of agent getting crashed due to high resource utilization also. The solution is you have to run those parallel stages on 2 separate executors. In order to do that, you have to have explicit agent configurations in each parallel stage as follows.
pipeline {
stages {
stage('CI') {
parallel {
stage('Unit Tests') {
agent any
steps {
sh 'echo Run your unit tests here'
}
}
stage('Integration Tests') {
agent any
steps {
sh 'echo Run your integration tests here'
}
}
}
}
stage('Deploy to QA') {
agent any
steps {
sh 'echo Run QA deployment here'
}
}
}
}Scripted environment variables with agent none configuration
If you have scripted environment variables then you need an executor to execute the scripts for environment variables initialization. Therefore you can’t have theagent non configuration in the pipeline root level with pipeline root level scripted environment variables.
The straight forward solution here is moving your scripted environment variable to stage level environment variables block with an agent assigned. But what if you want to access the scripted environment variable with multiple stages? Oops! Then you cannot do this. Here you can solve this problem with a small trick.
Here I have a scripted environment variable called EMAIL_ADDRESS and I want to access it from both stages while having agent none configuration also. So my solution here is we can wrap 2 stages with a parent stage and assign an agent for that parent stage before initializing the scripted environment variable.
Hope I have covered almost all agent configuration needed for most of the common practical use cases.
Concurrency Control
When designing pipeline jobs for various tasks you should have faced many practical scenarios where you need to control the concurrent executions of pipeline instances. In Jenkins declarative pipelines you have the freedom to have a very fined grained concurrency control. If you are coming from a different pipelining tool like GoCD you might be surprised with the abilities provided by Jenkins for concurrency control.
Use cases
- Avoid deployments to the same service from multiple branches at the same time.
- Modifying a shard resource from multiple pipelines.
Disable concurrent builds
pipeline {
agent any
options {
disableConcurrentBuilds()
}
stages {
...
}
...
}This the easiest option to disable the concurrent executions of a Jenkins pipeline. But here you are disabling the concurrent builds for your entire pipeline without much flexibility. And another important thing here is in a multibranch pipeline this disables the concurrent build only for one branch. Let’s say you have two branches as branch_a and branch_b . This will avoid having 2 concurrent builds from branch_a but not one from branch_a and another from branch_b . So this will not help if you want to have only one execution of the pipeline at a moment from all the branches.
Use Jenkins Locks
Jenkins Lockable Resources plugin enables more fine-grained concurrency control across builds. With this plugin, you can acquire a lock for a resource. Here you can use any name (string) as the locking resource. I recommend you to use a unique and easily identifiable name as the locking resource. The real resource (ex: service deployment) you want to avoid concurrent will be a good candidate for the lock resource name. Once the resource is locked by a pipeline run that locked resource cannot be acquired by any other pipeline run. Interesting thing here is this lock is not limited for a single pipeline. That lock is visible for you entire Jenkins server.
Suppose you have 2 separate pipelines which deploy the same service. Then definitely you want to allow only one pipeline execution at a time. With Lockable Resources plugin this is possible. And here you can lock your complete pipeline or a specific stage or a set of stages or only a single step also.
Lock the complete pipeline
pipeline {
agent any
options {
lock(resource: 'backend-service-dev')
}
stages {
...
}
...
}Lock only one stage
pipeline {
agent any
stages {
stage('Deploy') {
options {
lock(resource: 'backend-service-dev-deployment')
}
steps {
sh 'echo Run your deployment here'
}
}
}
}Lock a single step (fine-grained locking)
pipeline {
agent any
stages {
stage('Deploy') {
steps {
sh 'echo Build the code'
lock(resource: 'backend-service-dev-deployment') {
sh 'echo Run your deployment here'
}
sh 'echo Set meta after deployment'
}
}
}
}Cancel the previous build and start the new build
Let’s say you are building a CI pipeline which is triggered for each push user does to the remote branch. If your CI build takes some longer time (let’s say 10 minutes) and if your user also keeps pushing to the branch more often then a lot of builds will be piled up in the Jenkins build queue. The user may also have to wait for a longer time to check the status of the last build as he has to wait until all the builds are complete. In such a scenario you might prefer to cancel all the previous build and start building the latest code immediately when there is a new push to the branch.
Unfortunately, I haven’t found an inbuilt feature from Jenkins for this. The milestone feature is something similar. But it didn’t work for me with multibranch declarative pipelines. Therefore I ended up using a custom script for this purpose and it worked like a charm.
cancelPreviousBuilds function kills all currently running old builds on the same branch (no effect to builds from other branches). Please note that this function cannot be used with disableConcurrentBuilds option because the new pipeline run should be started to execute the above custom script. But disableConcurrentBuilds prevents the new build from starting until the old build is finished.
More tips and tricks
Extract the git revision for pull request merge triggers
If you are using pull request discovery with Github Branch Source Plugin there is an option in Jenkins to use the merged revision instead of branch head revision for the pull request build.
When using this setting, Jenkins’ inbuilt GIT_COMMIT environment variable contains the merge commit performed by Jenkins locally not the real GitHub commit which the pipeline was triggered. But you may want the real commit revision value during the build. For example, if you are pushing to SonarQube you need to set the remote git revision as a property. Setting the merge commit will not work here as that commit is local to Jenkins and not available in your remote GitHub repository. So in such a scenario, you can use a scripted environment variable to extract the remote git commit internally.
pipeline {
agent any
environments {
REVISION = """${
sh(
returnStdout: true,
script: '''
git rev-parse remotes/origin/$BRANCH_NAME
'''
).trim()
}"""
}
srages {
...
}
}The most important thing here is you have to use the remote branch for the command, not the local branch. Because remote/origin/$BRANCH_NAME contains the original commit but $BRANCH_NAME contains the local merge commit performed by Jenkins.
Set custom git commit status checks
When you are using GitHub Branch Source plugin to set commit status checks, you may always get the status check name as continuous-integration/jenkins/*. The last part of the status check varies based on the trigger type you are using as branch, pr-head, pr-merge. But this status check name doesn't make much sense to a user who is not familiar with Jenkins. So we prefer to set custom status check names over this default status check names.
But unfortunately, there is no straightforward way to rename this default status check name. Instead of renaming, we can disable the default status check and set any number of custom status checks as we want. We need to install Disable GitHub Multibranch Status plugin to disabled the default status check. After installing this plugin in the pipeline configurations you will see an option to apply this plugin.
Pipeline -> Config -> Branch Sources -> Behaviours -> Disable GitHub Notifications
After applying this behaviour Jenkins will no longer report the continuous-integration/jenkins/* status check to GitHub. Now you have to set the status checks manually in your Jenkinsfile.
Now if you check the GitHub commit after executing the pipeline you will see that status checks have been set with the custom names (here a status check name is Unit Tests).
Make troubleshooting easy
It’s better to print all the environment variables at the start using a printenv command. Then when something goes wrong you have a clear picture of the environment variables at the time of the issue which will solve your half of the problem.
pipeline {
agent any
stages {
stage('Init') {
steps {
sh 'printenv'
}
}
}
}Above suggestions are from my experience and they may or may not be the best solution. But I have been able to solve most of the problems and make my life easy with them. Suggestions are welcome always for anything better. Happy reading!
