Deploy Systems Documentation

DCP Secrets Management

This guide serves as a resource for secrets management in DCP.

At the time of this writing our secrets are managed in two different ways, Kilonova for bootstrapping and aws-secrets-manager for everything else. This guide will focus on aws-secrets-manager.

Kilonova

Secrets pertaining to bootstrapping the cluster in Janus are managed via kilonova-envs-config.

Documentation for managing SOPS secrets can be found here: getting started with secrets

AWS Secrets Manager

We've choose to use AWS secrets Manager w/ Kubernetes to manage our secrets. This provides us the ability to manage resources declaratively, using IAM Roles and Policies to limit access to specific pods within the EKS cluster.

Using the Kubernetes Secrets Store CSI Driver gives us the ability to mount secrets, keys and certs stored in mounted volumes attached to the containers filesystem. This also gives us the advantage of syncing as Kubernetes secrets from AWS, and natively setting ENV VARs within deployments.

Install the ASCP and Provider

The ASCP driver and Provider are installed as ArgoCD application resources. See deploy-systems-argo-cd as a reference.

Supporting IAM Roles and Policies

In order to give proper access to EKS pods we'll need to define a set of IAM Roles and Policies. The upstream documentation is helpful for different methods of configuration.

The best example of how this configuration works for DCP can be found in this PR: Allow data-access-service to fetch RDS password from secrets manager.

Kubernetes Resources

For EKS pods to be able to authenticate with secrets manager and assume role we need:

1. Kubernetes Service Account.
2. A secret store resource implementing SecretProviderClass.
3. A deployment that mounts the secret store and provides secrets to the POD.

An example PR can be found here: Adding secrets to data-access-service.

Kubernetes Service Account

We need a service account in the same namespace as the pods with a direct annotation to the IAM Role ARN.

There are /three/ key pieces of information here:

1. The namespace must be consistent in the service account with the namespace defined within the IAM aws_iam_policy_document.
2. The ServiceAccount name must also match what is defined within the aws_iam_policy_document.
3. Lastly the Annotation is the key the ties everything together. If the annotation is incorrect the pod will fail to properly mount. The annotation is direct link to the role defined in TF.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: data-access-service-secrets
  namespace: deploy
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::287796791794:role/data-access-service-secrets"

Secret Store Resource

A SecretProviderClass class is required in the same namespace as the EKS pod with a AWS secrets manager object linked directly from the ARN.

Test fetching the mirrored secret with the following:

kubectl get secret dcp-db-password --template='{{.data.rds_password | base64decode}}' 

In this example we're also instructing the driver to mirror the secret locally as a kubernets secret with the secretObjects stanza.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  namespace: deploy
  name: data-access-service-secrets
spec:
  provider: aws
  parameters:
    objects: |
          # This is the ARN to the specific secret you wish to load
        - objectName: "arn:aws:secretsmanager:us-west-2:287796791794:secret:dcp-db-password-QWYC1r"
          objectType: secretsmanager
          objectAlias: dcp-db-password
  # This section provdes support for mirroring aws secrets manager as a local kubernets secret
  # https://secrets-store-csi-driver.sigs.k8s.io/topics/sync-as-kubernetes-secret.html
  secretObjects:
    - secretName:  dcp-db-password
      type: Opaque
      data: 
      - objectName: dcp-db-password
        key: rds_password

Deployment

The container configuration is required to provide a ServiceAccountName, Volume with a map back to the secret store, and a volumeMounts.

spec:
  serviceAccountName: data-access-service-secrets # This must match the kubernetes `ServiceAccount` name.
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "data-access-service-secrets" # This must match the kubernetes `ServiceAccount` name.
...

       containers:
          env:
            - name: "DB_PASSWORD"
              valueFrom:
                secretKeyRef: # Fetch the password from the the local kubernetes secrets store, mirrored from aws-secrets-mangager
                  name: "dcp-db-password" # Uses the objectAlias set in the `SecretProviderClass`
                  key: "rds_password"
          ...

          volumeMounts:
          - name: secrets-store-inline
            mountPath: "/mnt/secrets-store"
            readOnly: true

Validation

IAM Role

We can validate our AWS Roles and policies to ensure we have the proper resources. See below:

$ aws iam get-role --role-name data-access-service-secrets --query Role.AssumeRolePolicyDocument

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::287796791794:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/C0EB4AEEDFD18CBBCBD0686222057E0A"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-2.amazonaws.com/id/C0EB4AEEDFD18CBBCBD0686222057E0A:aud": "sts.amazonaws.com",
                    "oidc.eks.us-west-2.amazonaws.com/id/C0EB4AEEDFD18CBBCBD0686222057E0A:sub": "system:serviceaccount:deploy:data-access-service-secrets"
                }
            }
        }
    ]
}

IAM Policies

$ aws iam list-attached-role-policies --role-name data-access-service-secrets --query 'AttachedPolicies[].PolicyArn' --output text
...
arn:aws:iam::287796791794:policy/data-access-service-secrets-deploy-systems-com-dev

IAM Policies Version

$ aws iam get-policy-version --policy-arn arn:aws:iam::287796791794:policy/data-access-service-secrets-deploy-systems-com-dev --version-id v1
...
{
    "PolicyVersion": {
        "Document": {
            "Statement": [
                {
                    "Action": "secretsmanager:GetSecretValue",
                    "Effect": "Allow",
                    "Resource": "arn:aws:secretsmanager:us-west-2:287796791794:secret:dcp-db-password-QWYC1r",
                    "Sid": "Secrets"
                }
            ],
            "Version": "2012-10-17"
        },
        "VersionId": "v1",
        "IsDefaultVersion": true,
        "CreateDate": "2023-06-06T16:59:36+00:00"
    }
}

Resources:

• Integrating Secrets Manager
• Secrets Provider Class Resource
• Associating IAM Roles with K8s Service Account