Deploy new EKS cluster via Janus

This document details the steps to deploy a new EKS cluster via Janus - along with all necessary prerequisites.

Table of contents

Prerequisites

tf-k8s-core (manages core resources for a given environment - ie: kdeploy-dev, kdev, etc).

  • New Environment: If provisioning the EKS cluster in a new environment, tf-k8s-core must be updated to include the environment. Vault secrets must also exist in path secret/keys/<env> . The Jenkins job must then be run post-merge in order to provision the resources via Terraform.
    • examples:
      • PR to add deployer arn configs for kdeploy-dev: https://github.com/mulesoft-ops/tf-k8s-core/pull/289
        • 🚨 important note: A corresponding freeipa group must also exist for your environment. In this case we were renaming our environment from deploy-systems-com-dev to kdeploy-dev. While “deploy-systems-com-dev“ existed as a freeipa group, ”kdeploy-dev“ did not. This required reaching out to members of the mulesoft security team for group creation.
      • PR to add kdeploy-dev environment: https://github.com/mulesoft-ops/tf-k8s-core/pull/288
      • The vault secrets in path secret/keys/kdpeloy-dev were populated by copying over the existing secret/keys/kdev values via the #ms-prodeng-asks thread here: https://salesforce-internal.slack.com/archives/C03UAEK0TNY/p1684435751354739
      • Jenkins job that should be run against the corresponding env/region: https://jenkins.build.msap.io/job/DevOps/job/terraform-new/job/tf-k8s-core/job/master/build?delay=0sec
  • New Region (existing environment): If provisioning the EKS cluster into a new region in an existing environment, update the existing .tfvars file in the profiles directory with any region specific updates if applicable
    • examples:
      • to add us-east-1 subnet data for the kdeploy-dev environment, you would do so here: https://github.com/mulesoft-ops/tf-k8s-core/blob/master/profiles/kdeploy-dev.tfvars#L13). Make sure the “allowedRegions” variable in the Jenkinsfile includes the corresponding region for your environment here: https://github.com/mulesoft-ops/tf-k8s-core/blob/master/Jenkinsfile#L38
      • The Jenkins job will need to be run against the new env/region combo here: https://jenkins.build.msap.io/job/DevOps/job/terraform-new/job/tf-k8s-core/job/master/build?delay=0sec

tf-mulesoft-records (manages top level/root mulesoft DNS records)

  • When adding an entry for a new domain in the tf-k8s-core step above, tf-k8s-core creates the route53 DNS records for your domain in your account, however the root domain (in this case msap.io) will need to be updated to correctly direct queries for your subdomain:
    • example:
      • A new domain "kdeploy-dev.msap.io" was added to tf-k8s-core here: https://github.com/mulesoft-ops/tf-k8s-core/blob/master/profiles/kdeploy-dev.tfvars#L1 this created some route53 records in our aws account, however the upstream SOA/NS records still pointed to the root domain msap.io.
      • To fix this, we added a new entry to tf-mulesoft-records here: https://github.com/mulesoft-ops/tf-mulesoft-records/pull/116 and ran the corresponding Jenkins job here: https://jenkins.build.msap.io/job/DevOps/job/terraform-new/job/tf-mulesoft-records/job/master/build?delay=0sec

core-paas-provisioner (provisioner environment settings - used by subsequent steps)

  • Required configuration: new environments should be added to the terraform/eks config here: https://github.com/mulesoft/core-paas-provisioner/blob/master/terraform/eks.tf
    • example: PR to add kdeploy-dev environment: https://github.com/mulesoft/core-paas-provisioner/pull/686
    • 🚨 important note: once the PR has been merged, find the corresponding master build in Jenkins and note the artifact version tag for the newly built provisioner (example. v5.3.125). This will be used in the https://github.com/mulesoft/core-paas-clusters/blob/master/deployments/<environment>.json file in the following core-paas-clusters config

core-paas-clusters (kubernetes cluster settings)

  • Required configuration: While the core-paas-janus repo/Jenkins job will create PRs against core-paas-clusters, the following settings must be configured for a new environment:
    • Vault secrets in path secret/aws/<env>. The key/value pairs contained in this vault path will need to be an AWS access key and corresponding secret key with admin access
      • example: The vault secrets in secret/aws/kdeploy-dev were populated by having the AWS Access/Secret Key values for our core-paas-clusters IAM user was entered into vault via an #ms-prodeng-asks thread. The following key/value pairs were added:
        • access_key = <AWS access key value>
        • secret_key = <AWS secret key value>
    • Add deployments file for the new environment (
      • example: https://github.com/mulesoft/core-paas-clusters/blob/master/deployments/kdeploy-dev.json
        • 🚨 important note: be sure the provisioner_image_tag value is set to a value equal to (or higher) than the version you created in the previous core-paas-provisioner step (example: https://github.com/mulesoft/core-paas-clusters/blob/master/deployments/kdeploy-dev.json#L6)
    • Add the new environment to the repo’s Jenkinsfile

core-paas-teleport-cli (command line tool used by core-paas-janus to login to the EKS cluster post-creation)

  • Required configuration: Add your new environment to the cmd/accounts.go file
    • example PR adding kdeploy-dev: https://github.com/mulesoft/core-paas-teleport-cli/pull/95

core-paas-base-image-infra-deploy (used by core-paas deploy jobs)

  • Required configuration: Based on the new core-paas-teleport-cli version built in the previous step, update the Dockerfile in this repo to pull your new image/artifact
    • example PR updating the tele cli version: https://github.com/mulesoft/core-paas-base-image-infra-deploy/pull/292/files

kilonova-envs-config (used by core-paas deploy jobs to pick up environment specific settings (secrets, etc) when deploying to your EKS cluster)

Note: this is optional for the EKS cluster creation itself as “deploy” can be unchecked in the core-paas-janus jenkins job which will leave out any deploy steps (more notes on core-paas deploys are including in the core-paas-janus step below)

  • Required configuration for secrets: As noted in the README, each environment needs a .sops.yaml file which contains the ARN for a kms key which will encrypt/decrypt your secret values. The KMS key for your env/region will be named “core-paas” and is created via first step in this guide (tf-k8s-core)
    • example PR adding the required .sops.yaml config along with several other config/secrets files for the kdeploy-dev environment: https://github.com/mulesoft/kilonova-envs-config/pull/24858

core-paas-janus (EKS cluster creation and deployment)

⚠️ 🚧 ⚠️ Note: Known workaround currently required for “deploy” into new environments. Details provided below ⚠️ 🚧 ⚠️

  • New Environment: The Jenkinsfile in the core-paas-janus repo must be updated to include the environment. A Jenkins credential must be created here https://jenkins.build.msap.io/job/core-paas/job/components/job/core-paas-janus/credentials/ named TELE_KEY_JENKINS_ (example: TELE_KEY_JENKINS_KDEPLOY-DEV). The value for this credential can be set to an empty string. The Jenkins job must then be run with deploy unchecked (see note in the examples section below) post-merge in order to provision your EKS cluster.
    • examples:
      • PR to add kdeploy-dev environment: https://github.com/mulesoft/core-paas-janus/pull/475/files
      • The Jenkins credential TELE_KEY_JENKINS_KDEPLOY-DEV (with value set to an empty string) was created via the #ms-prodeng-asks thread here: https://salesforce-internal.slack.com/archives/C03UAEK0TNY/p1684254109468809
      • Jenkins job that should be run against the corresponding env/region: https://jenkins.build.msap.io/job/core-paas/job/components/job/core-paas-janus/job/master/build?delay=0sec
        • parameter settings for the Jenkins job
          • TARGET_ENV: your environment (eg. kdeploy-dev)
          • TARGET_REGION: AWS region (eg. us-west-2)
          • ACTION: (default set to “apply“ - to create the cluster)
          • CLUSTER_NAME: (default set to “eks”, the region will be auto-appended during the Janus job run)
          • DEPLOY: ⚠️ 🚧 ⚠️ uncheck ⚠️ 🚧 ⚠️ (default is checked)
            • note: there is currently a known issue that prevents core-paas deployments to new environments. The EKS cluster will still be created with this unchecked. A core-paas deploy can then be initiated separately using this job: https://jenkins.build.msap.io/job/core-paas/job/deploy/view/change-requests/job/PR-2659/build?delay=0sec
              • example parameters: https://jenkins.build.msap.io/job/core-paas/job/deploy/view/change-requests/job/PR-2659/9/parameters/
            • more info on this deploy workaround:
              • slack thread: https://salesforce-internal.slack.com/archives/C03HW1X2C0G/p1684773337317309?thread_ts=1684511097.251879&cid=C03HW1X2C0G
              • Team dependency: https://gus.lightning.force.com/lightning/r/ADM_Team_Dependency__c/a0nEE000000AR3lYAG/view
              • related GUS Work item: https://gus.lightning.force.com/lightning/r/ADM_Work__c/a07EE00001Qjy2LYAR/view
          • KUBERNETES_DISTRO: (leave default “eks” setting)
          • MULESOFT_CLOUD: commercial or gov
          • MULESOFT_CLUSTER_TYPE: *change to deploy for our teams clusters
          • DRY_RUN: uncheck to create the cluster, leave checked for a dry run
          • remaining parameters can use the default blank values at this time (no change case value is required until we build clusters in higher envs)
  • New Region (existing environment): If provisioning the EKS cluster into a new region in an existing environment, simply make sure your region is already accounted for in the Jenkinsfile and run the Jenkins job with the “TARGET_REGION” parameter set to your region of choice. No new credential or changes from the “New environment” steps are required.

Optional prerequisites (for core-paas app deployments)

  • tf-core-paas-heartbeat (provisions the IAM role required by the core-paas hearbeat app)

    • example
      • PR adding a new environment: https://github.com/mulesoft-ops/tf-core-paas-heartbeat/pull/5
      • Then run the associated Jenkins job against your environment/region to add the role here: https://jenkins.build.msap.io/job/DevOps/job/terraform-new/job/tf-core-paas-heartbeat/job/master/build?delay=0sec

  • Also see Deploying Core Pass Apps for initial core paas app deployment docs

Version: 0.3.110
Last Updated: 2024-07-01T19:32:00+0000