How to implement a retry strategy on your Jenkins pipeline

Implementing a retry strategy can be very easy if you use the Naginator plugin on Jenkins. It’s a bit trickier if you have to deal with a multibranch/multirepo job, in those cases you cannot configure the retry as a post build action on the job configuration. So, how to code a scripted retry strategy? In this article I’ll show a few methods by using the declarative syntax.

1. Just retry()

retry(count: 3) {
  sleep(60)
  doSomething()
}

Pro:

• Easy implementation

Cons:

• Ram eater. Let’s say that we want to retry after 1 minute, because our project might depend on another, whose new implementations haven’t been pushed yet. That means that our job will be active for all that time, even if it’s not doing anything
• If more jobs get stuck waiting for the retry at the same time, Jenkins might exceed the maximum number of concurrent builds. This means that other pipeline that has been triggered after that, and can potentially make the current job succeed (e.g. by building another project on which our current project depends on) will not run, making all this useless
• Can’t keep track graphically of how many times you tried before getting a successful result

2. Call a new job

retry(count: 3) {
  script {
    if (params.retry) {
      def jobBuild = build  job: "$env.JOB_NAME", parameters: [
          booleanParam(name: "retry", value: false)
        ], wait: true, quietPeriod: 60
      params.retry = jobBuild.getResult() != "SUCCESS"
    }
  }
}

Pro:

• Can keep track graphically of how many times you tried before getting a successful result

Cons:

• Same problem with ram
• Same problem with concurrent builds

3. Call recursively a new job

I call again the job with a parameter that count the number of retries, if that catches an error it will call another job recursively. Because of recursion, I don’t need to wait for the downstream job to be completed.

script {
  if ("${params.retriesCounter}".toInteger() < 3) {
    build  job: "$env.JOB_NAME", parameters: [
      string(name: "retriesCounter", value: ("${params.retriesCounter}".toInteger() + 1).toString())
    ], wait: false, quietPeriod: 60
  }
}

Pro:

• Lighter operation possible. The jobs don’t occupy the ram while they’re waiting to retry
• Completely asynchronous, other pipelines can be running during the quietPeriod of a failing job
• Can keep track graphically of how many times you tried before getting a successful result

Cons:

• Every upstream job that triggers this pipeline loses the status information of the downstream job. Example:

  up-job calls test-pipeline(0)
    test-pipeline(0) throws and catches an error
    test-pipeline(0) calls test-pipeline(1)
    test-pipeline(1) completed with success
    test-pipeline(0) completed with failure
  

  Even if my test-pipeline completed with success after one retry, the upstream-job that triggered it failed.

Conclusion

In this article we discussed different strategies to implement a retry on jenkins pipelines, and illustrated pro and cons for each of them