Skip to content

Diagnose a failing or missing job

Find your symptom below. Each entry says what the pipeline looks like when it happens, what causes it, and what to change.

The first question is almost never "why did this job fail". It is "did this job run at all" — a component whose rules: do not match produces no job, no skipped entry and no warning. Check the pipeline graph before reading a log.

My job does not appear in the pipeline at all

Nothing failed. No rule matched, so GitLab created no job. Work through these in order:

  1. Check the trigger matrix. Most components run on merge-request pipelines only. Pushing to a branch with no open merge request runs the Go, Rust and Svelte gates nowhere. See which pipelines each component runs on.
  2. Check the pipeline source. No component runs on a scheduled pipeline except renovate-self, renovate-group, renovate-merge and hugo-pages. The when: never schedule guard is the first rule in every other template and cannot be overridden by any input.
  3. Check your changes filter. If you overrode changes, you replaced the default rather than extending it. An override that omits a path means the gate stops running when that path changes — including .gitlab-ci.yml itself.
  4. Check the stage exists. A stage your stages: list does not declare is a pipeline configuration error, not a silent skip — you would see a red config error rather than a missing job.
  5. Check your workflow: rules. No component rule can run on a pipeline your workflow: declined to create.

To prove a filter is the cause rather than guess, temporarily set changes: ["**/*"] — the universal escape hatch on every component that has the input — and see whether the job appears.

Every job in the pipeline runs twice

Your project is creating both a branch pipeline and a merge-request pipeline for the same push, and the components that run on branch pushes (tofu-lint, tofu-security, tofu-validate, the zensical-pages build) appear in both.

Add the dedup rule to your workflow::

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never
    - if: '$CI_COMMIT_TAG'
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

Keep the $CI_COMMIT_BRANCH guard on the never rule. $CI_COMMIT_BRANCH is empty on a tag pipeline, so the guard is what stops the rule matching there. Writing it as $CI_PIPELINE_SOURCE == "push" instead also matches a tag push and suppresses the entire tag pipeline — your release jobs then never run and nothing goes red.

tofu-plan fails with "no valid credential sources" or a bare AccessDenied

The AWS SDK never reached the web-identity link of its credential chain. Two causes, in order of likelihood:

A hardcoded profile in the provider block. This is the most common OIDC failure by a wide margin. A profile = "..." line sends the SDK down the shared-config link of the chain, which on a runner with no ~/.aws/credentials finds nothing and stops — the web-identity link is never reached. It is the kind of line that arrives from someone's local setup and looks correct in review. Use a variable that is null in CI:

variable "aws_profile" {
  description = "Local AWS CLI profile. Leave null in CI — OIDC is used instead."
  type        = string
  default     = null
}

provider "aws" {
  region  = var.region
  profile = var.aws_profile   # null in CI
}

A trust policy that does not accept this pipeline's subject. The role's trust policy constrains the token's sub claim. If it names refs your pipeline is not running on, AssumeRoleWithWebIdentity returns AccessDenied. Full setup: Set up OIDC for tofu-plan / tofu-apply.

The trust policy looks right and merge-request pipelines still get a 403

Check what the trust policy is matching on. GitLab's sub claim encodes a ref_type, and its only values are branch and tag. There is no mr.

A merge-request pipeline does not run against a merge-request ref — it runs against the source branch, so its token carries ref_type:branch like any other branch pipeline. A condition asking for ref_type:mr matches a value that is never present in any token, so it is never true, and every merge-request pipeline gets a flat AccessDenied no matter how the condition is edited.

Match ref_type:branch and narrow by branch name or project path instead. More generally: when an OIDC trust fails, decode a real token and read the claims that are actually there rather than editing the policy again.

tofu-apply in ref mode fails downloading the plan artifact

The jobs-artifacts API rejected the request. Check which:

  • 401/403 on GitLab Free. CI_JOB_TOKEN cannot authenticate the jobs-artifacts API on Free — that is a Premium/Ultimate capability. Create a personal or group access token with job-artifact read (fine-grained: the Job Artifact resource, Read scope), store it as a Masked CI/CD variable, and pass it as plan_token. job mode never uses plan_token, so this only ever affects ref mode.
  • 404. No artifact exists for plan_job on plan_ref. Either the plan never ran on that ref, or plan_job does not match the plan job's actual name. If you overrode tofu-plan's job_name, the paired tofu-apply must set plan_job to that same value.
  • The artifact expired. tofu-plan sets expire_in: 1 day on tfplan.cache. A tag cut more than a day after the plan was banked has nothing left to download.

tofu apply rejects the plan as stale

Expected, and it is the safety net working. tofu refuses to apply a binary plan when state moved after the plan was produced. Re-run the plan against current state, review it, and apply that.

The Release MR merged, the tag exists, and nothing was published

Check whether a tag pipeline was created at all. Two known causes:

  • A workflow: rule suppressing tag pipelines. A never rule keyed on $CI_PIPELINE_SOURCE == "push" matches a tag push too. Guard it with $CI_COMMIT_BRANCH (see the double-run entry above).
  • The publishing job's stage is .pre or .post. GitLab does not create a pipeline whose jobs are all in a built-in stage. On a release tag every other Tofu job self-excludes, so tofu-module-publish is typically the only job in that pipeline — put it in a built-in stage and the pipeline is never created. The failure is entirely silent: the Release MR merges green, the tag exists, the release page appears, the registry receives nothing.

Fixing the stage does not retroactively publish an existing tag. A tag's .gitlab-ci.yml is frozen at the tagged commit, so a release lost this way needs a fresh one.

osv-scanner exits 127 in go-security with no findings reported

osv_scanner_call_analysis is on. That pass builds your module using the Go bundled in the scanner image under GOTOOLCHAIN=local, so it fails whenever go.mod requires a newer Go than the image ships — and osv-scanner then exits non-zero even with zero remaining findings.

Leave it at its default of false. Reachability analysis is already covered by the dedicated govulncheck job, so turning it off loses no coverage.

I set enable_cross_os: true and no macOS or Windows job appeared

The input is one of two gates. test-macos and test-windows also require the CI variable RUN_CROSS_OS_TESTS to be "true", and they request the saas-macos-medium-m1 / saas-windows-medium-amd64 runner tags. The second gate exists so that a project without eligible runners does not get two permanently-pending jobs blocking its test stage.

Set both, and confirm your project can reach GitLab SaaS macOS/Windows runners. Note that both jobs are allow_failure: true and cannot fail your pipeline by design.

gitleaks reports a secret from a commit that is not mine

Check the scan scope before treating it as a false positive. The security components scope gitleaks to the pipeline's own commit range precisely to avoid cross-branch findings on a shared runner — see Security, always-on. A finding that survives that scoping is in your range.

A component's version jumped several minors and nothing changed

Expected. The components are released as one monorepo under a single tag stream, so every component gets a version on every release whether or not it changed. Read the CHANGELOG entry for the component you actually pin.

Do still read the Renovate MR: pre-1.0, a minor bump may change input shape. See Pin and upgrade a component.

See also