Merge branch 'all-job-templates' into 'main'

Add job template pipeline

Closes #9

See merge request components/opentofu!36

.gitlab/README.md.template

@@ -23,6 +23,10 @@ Read more:
 **Note**: Please make sure to use a released version of this CI/CD component.
 You find all releases on the [Releases Overview Page](https://gitlab.com/components/opentofu/-/releases).
 
+♻️ **Migrating from the Terraform CI/CD templates?** Check **[this](#migrating-from-the-terraform-cicd-templates)** out.
+
+[[_TOC_]]
+
 ## Usage
 
 ```yaml
@@ -69,6 +73,26 @@ include:
 stages: [validate, build, deploy, cleanup]
 ```
 
+Or import all jobs as hidden templates ready to be extended:
+
+```yaml
+include:
+  - component: gitlab.com/components/opentofu/job-templates@<VERSION>
+    inputs:
+      # The version must currently be specified explicitly as an input,
+      # to find the correctly associated images. # This can be removed
+      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
+      version: <VERSION>
+      opentofu_version: <OPENTOFU_VERSION>
+
+stages: [...]
+
+fmt:
+  extends: [.opentofu:fmt]
+
+...
+```
+
 ### Opinionated Templates
 
 This component repository also provides some templates that may often be used, 
@@ -80,7 +104,7 @@ but no destructive actions.
 
 ### Job Templates
 
-Instead of including the `full-pipeline` or another opinionated tempalte, 
+Instead of including the `full-pipeline` or another opinionated template,
 it's also possible to include individual jobs
 and compose your own pipeline, for example, to just run the `fmt` job you can do:
 
@@ -96,6 +120,29 @@ include:
       root_dir: tofu/
 ```
 
+Or you can also include the `job-templates` template, that will include
+all available OpenTofu jobs as hidden job templates prefixed with `.opentofu:`.
+Those are especially useful when you want to minimize your includes and
+you want to extend the jobs:
+
+```yaml
+include:
+  - component: gitlab.com/components/opentofu/job-templates@<VERSON>
+    inputs:
+      # The version must currently be specified explicitly as an input,
+      # to find the correctly associated images. # This can be removed
+      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
+      version: <VERSION>
+      opentofu_version: 1.6.1
+
+plan:
+  extends: [.opentofu:plan]
+  parallel:
+    matrix:
+      - TF_ROOT: test/
+      - TF_ROOT: prod/
+```
+
 Have a look at the [`full-pipeline`](templates/full-pipeline.yml) for how it's constructed.
 
 The following job components exist:
@@ -172,6 +219,26 @@ include:
 This example assumes your GitLab instance is hosted on `gitlab.example.com` and this component
 project is mirrored in the `components/opentofu` project.
 
+## Migrating from the Terraform CI/CD templates
+
+When migrating from the GitLab Terraform CI/CD templates you can use the following migration rules:
+
+- Used `Terraform.gitlab-ci.yml` -> Migrate to `validate-plan-apply`.
+- Used `Terraform/Base.gitlab-ci.yml` -> Migrate to `job-templates`.
+    - Migrate the `.terraform:` job prefix to `.opentofu:`.
+- Used the `kics-iac-sast` job -> Additionally include the `Jobs/SAST-IaC.latest.gitlab-ci.yml` template.
+- Migrate the following job names:
+    - `build` -> `plan`
+    - `deploy` -> `apply`
+- Migrate the `TF_ROOT` variable to the `root_dir` input.
+    - Although the `TF_ROOT` variable is still used and maybe overwritten after the import on individual jobs.
+- Migrate the `TF_STATE_NAME` variable to the `state_name` input.
+    - Although the `TF_STATE_NAME` variable is still used and maybe overwritten after the import on individual jobs.
+- Migrate the `TF_AUTO_DEPLOY` variable to the `auto_apply` input.
+
+The same rules apply for the `latest` templates.
+We also recommend to check out the [Usage](#Usage) section for more details about the available templates and inputs.
+
 ## Contributing
 
 See the [CONTRIBUTING.md](CONTRIBUTING.md) guide.

README.md

@@ -25,6 +25,10 @@ Read more:
 **Note**: Please make sure to use a released version of this CI/CD component.
 You find all releases on the [Releases Overview Page](https://gitlab.com/components/opentofu/-/releases).
 
+♻️ **Migrating from the Terraform CI/CD templates?** Check **[this](#migrating-from-the-terraform-cicd-templates)** out.
+
+[[_TOC_]]
+
 ## Usage
 
 ```yaml
@@ -71,6 +75,26 @@ include:
 stages: [validate, build, deploy, cleanup]
 ```
 
+Or import all jobs as hidden templates ready to be extended:
+
+```yaml
+include:
+  - component: gitlab.com/components/opentofu/job-templates@<VERSION>
+    inputs:
+      # The version must currently be specified explicitly as an input,
+      # to find the correctly associated images. # This can be removed
+      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
+      version: <VERSION>
+      opentofu_version: <OPENTOFU_VERSION>
+
+stages: [...]
+
+fmt:
+  extends: [.opentofu:fmt]
+
+...
+```
+
 ### Opinionated Templates
 
 This component repository also provides some templates that may often be used, 
@@ -82,7 +106,7 @@ but no destructive actions.
 
 ### Job Templates
 
-Instead of including the `full-pipeline` or another opinionated tempalte, 
+Instead of including the `full-pipeline` or another opinionated template,
 it's also possible to include individual jobs
 and compose your own pipeline, for example, to just run the `fmt` job you can do:
 
@@ -98,6 +122,29 @@ include:
       root_dir: tofu/
 ```
 
+Or you can also include the `job-templates` template, that will include
+all available OpenTofu jobs as hidden job templates prefixed with `.opentofu:`.
+Those are especially useful when you want to minimize your includes and
+you want to extend the jobs:
+
+```yaml
+include:
+  - component: gitlab.com/components/opentofu/job-templates@<VERSON>
+    inputs:
+      # The version must currently be specified explicitly as an input,
+      # to find the correctly associated images. # This can be removed
+      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
+      version: <VERSION>
+      opentofu_version: 1.6.1
+
+plan:
+  extends: [.opentofu:plan]
+  parallel:
+    matrix:
+      - TF_ROOT: test/
+      - TF_ROOT: prod/
+```
+
 Have a look at the [`full-pipeline`](templates/full-pipeline.yml) for how it's constructed.
 
 The following job components exist:
@@ -191,6 +238,26 @@ include:
 This example assumes your GitLab instance is hosted on `gitlab.example.com` and this component
 project is mirrored in the `components/opentofu` project.
 
+## Migrating from the Terraform CI/CD templates
+
+When migrating from the GitLab Terraform CI/CD templates you can use the following migration rules:
+
+- Used `Terraform.gitlab-ci.yml` -> Migrate to `validate-plan-apply`.
+- Used `Terraform/Base.gitlab-ci.yml` -> Migrate to `job-templates`.
+    - Migrate the `.terraform:` job prefix to `.opentofu:`.
+- Used the `kics-iac-sast` job -> Additionally include the `Jobs/SAST-IaC.latest.gitlab-ci.yml` template.
+- Migrate the following job names:
+    - `build` -> `plan`
+    - `deploy` -> `apply`
+- Migrate the `TF_ROOT` variable to the `root_dir` input.
+    - Although the `TF_ROOT` variable is still used and maybe overwritten after the import on individual jobs.
+- Migrate the `TF_STATE_NAME` variable to the `state_name` input.
+    - Although the `TF_STATE_NAME` variable is still used and maybe overwritten after the import on individual jobs.
+- Migrate the `TF_AUTO_DEPLOY` variable to the `auto_apply` input.
+
+The same rules apply for the `latest` templates.
+We also recommend to check out the [Usage](#Usage) section for more details about the available templates and inputs.
+
 ## Contributing
 
 See the [CONTRIBUTING.md](CONTRIBUTING.md) guide.

templates/job-templates.yml

+spec:
+  inputs:
+    # Stages
+    stage_validate:
+      default: 'validate'
+      description: 'Defines the validate stage. This stage includes the `fmt` and `validate` jobs.'  
+    stage_build:
+      default: 'build'
+      description: 'Defines the build stage. This stage includes the `plan` job.'
+    stage_deploy:
+      default: 'deploy'
+      description: 'Defines the deploy stage. This stage includes the `apply` job.'
+    stage_cleanup:
+      default: 'cleanup'
+      description: 'Defines the cleanup stage. This stage includes the `destroy` and `delete-state` jobs.'
+
+    # Versions
+    # This version is only required, because we cannot access the context of the component,
+    # see https://gitlab.com/gitlab-org/gitlab/-/issues/438275
+    version:
+      default: 'latest'
+      description: 'Version of this component. Has to be the same as the one in the component include entry.'
+
+    opentofu_version:
+      default: '1.6.1'
+      options:
+        - '$OPENTOFU_VERSION'
+        - '1.6.1'
+        - '1.6.0'
+        - '1.6.0-rc1'
+      description: 'OpenTofu version that should be used.'
+
+    # Images
+    image_registry_base:
+      default: '$CI_REGISTRY/components/opentofu'
+    # FIXME: not yet possible because of https://gitlab.com/gitlab-org/gitlab/-/issues/438722
+    # gitlab_opentofu_image:
+    #   # FIXME: This should reference the component tag that is used.
+    #   #        Currently, blocked by https://gitlab.com/gitlab-org/gitlab/-/issues/438275
+    #   # default: '$CI_REGISTRY/components/opentofu/gitlab-opentofu:$[[ inputs.opentofu_version ]]'
+    #   default: '$CI_REGISTRY/components/opentofu/gitlab-opentofu:$[[ inputs.version ]]-opentofu$[[ inputs.opentofu_version ]]'
+    #   description: 'Tag of the gitlab-opentofu image.'
+
+    # Configuration
+    job_name_prefix:
+      default: '.opentofu:'
+      description: 'Prefix for all job names. Jobs will be hidden if leading with a dot `.`.'
+    root_dir:
+      default: ${CI_PROJECT_DIR}
+      description: 'Root directory for the OpenTofu project.'
+    state_name:
+      default: default
+      description: 'Remote OpenTofu state name.'
+    auto_apply:
+      default: 'false'
+      description: 'Whether the apply job is manual or automatically run.'
+    auto_destroy:
+      default: 'false'
+      description: 'Whether the destroy job is manual or automatically run.'
+
+---
+
+include:
+  - local: '/templates/fmt.yml'
+    inputs:
+      as: '$[[ inputs.job_name_prefix ]]fmt'
+      stage: $[[ inputs.stage_validate ]]
+      version: $[[ inputs.version ]]
+      opentofu_version: $[[ inputs.opentofu_version ]]
+      image_registry_base: $[[ inputs.image_registry_base ]]
+      root_dir: $[[ inputs.root_dir ]]
+  - local: '/templates/validate.yml'
+    inputs:
+      as: '$[[ inputs.job_name_prefix ]]validate'
+      stage: $[[ inputs.stage_validate ]]
+      version: $[[ inputs.version ]]
+      opentofu_version: $[[ inputs.opentofu_version ]]
+      image_registry_base: $[[ inputs.image_registry_base ]]
+      root_dir: $[[ inputs.root_dir ]]
+      state_name: $[[ inputs.state_name ]]
+  - local: '/templates/plan.yml'
+    inputs:
+      as: '$[[ inputs.job_name_prefix ]]plan'
+      stage: $[[ inputs.stage_build ]]
+      version: $[[ inputs.version ]]
+      opentofu_version: $[[ inputs.opentofu_version ]]
+      image_registry_base: $[[ inputs.image_registry_base ]]
+      root_dir: $[[ inputs.root_dir ]]
+      state_name: $[[ inputs.state_name ]]
+  - local: '/templates/apply.yml'
+    inputs:
+      as: '$[[ inputs.job_name_prefix ]]apply'
+      stage: $[[ inputs.stage_deploy ]]
+      version: $[[ inputs.version ]]
+      opentofu_version: $[[ inputs.opentofu_version ]]
+      image_registry_base: $[[ inputs.image_registry_base ]]
+      root_dir: $[[ inputs.root_dir ]]
+      state_name: $[[ inputs.state_name ]]
+      auto_apply: $[[ inputs.auto_apply ]]
+  - local: '/templates/destroy.yml'
+    inputs:
+      as: '$[[ inputs.job_name_prefix ]]destroy'
+      stage: $[[ inputs.stage_cleanup ]]
+      version: $[[ inputs.version ]]
+      opentofu_version: $[[ inputs.opentofu_version ]]
+      image_registry_base: $[[ inputs.image_registry_base ]]
+      root_dir: $[[ inputs.root_dir ]]
+      state_name: $[[ inputs.state_name ]]
+      auto_destroy: $[[ inputs.auto_destroy ]]
+  - local: '/templates/delete-state.yml'
+    inputs:
+      as: '$[[ inputs.job_name_prefix ]]delete-state'
+      stage: $[[ inputs.stage_cleanup ]]
+      state_name: $[[ inputs.state_name ]]

tests/integration-tests/JobTemplates.gitlab-ci.yml

+include:
+  - component: gitlab.com/$CI_PROJECT_PATH/job-templates@$CI_COMMIT_SHA
+    inputs:
+      image_registry_base: $GITLAB_OPENTOFU_IMAGE_BASE
+      version: $CI_COMMIT_SHA
+      opentofu_version: $OPENTOFU_VERSION
+      root_dir: $TEST_TF_ROOT
+      state_name: $TEST_TF_STATE_NAME
+
+stages: [validate, test, build, deploy, cleanup]
+
+# Required to run everything immediately, instead of manually.
+
+fmt:
+  extends: '.opentofu:fmt'
+  rules: [{when: always}]
+
+validate:
+  extends: '.opentofu:validate'
+  rules: [{when: always}]
+
+plan:
+  extends: '.opentofu:plan'
+  rules: [{when: always}]
+
+apply:
+  extends: '.opentofu:apply'
+  rules: [{when: always}]
+
+destroy:
+  extends: '.opentofu:destroy'
+  rules: [{when: always}]
+
+delete-state:
+  extends: '.opentofu:delete-state'
+  needs: [destroy]
+  rules: [{when: always}]
+

tests/integration.gitlab-ci.yml

@@ -10,6 +10,7 @@ component:
   parallel:
     matrix:
       - PIPELINE_NAME: [Defaults]
+      - PIPELINE_NAME: [JobTemplates]
 
 backport-templates:
   stage: test-integration