How-To configure Crusoe Managed Kubernetes as an OIDC provider so that workloads can access your public cloud resources

Introduction

If your CMK workload pods and resources need to interact with your public cloud account - for example, to store training results in an S3 bucket, or to pull container images - then they will need to authenticate with that cloud provider. In the case of AWS, a convenient method is to supply AWS access keys and secret keys to your workloads by means of Kubernetes secrets or environment variables, but that soon becomes a security and scalability problem, particularly if you need to revoke the credentials for any reason.

It's far better to leverage CMK's ability to act as an OIDC provider, which allows it to create and rotate short-lived JWT tokens for service accounts, which can then be injected into pods. Roles and identities can be easily managed on a per-service-account basis allowing full control and auditing of the actions performed by your workloads. This example focuses on AWS, where the method is called IRSA (IAM Roles for Service Accounts). The same principles apply to other cloud providers such as GCP.

Note: This use case is not the same as that described in other Crusoe documentation related to OIDC. The CMK OIDC feature exposed in the Crusoe console is for managing end-user access to your cluster using identities from a third-party provider, namely OKTA. In this article, the CMK cluster itself acts as the OIDC provider, validating the identities of the workloads running on it to external services.

Prerequisites

• Admin-level CLI access to a Crusoe Cloud account with sufficient resources to create the CMK cluster that meets your requirements
• Console and CLI access to your AWS account with sufficient authority to create IAM providers, polices and roles, and S3 buckets and objects.
• The means to host OIDC Document Discovery data (JWKS keys generated by your CMK cluster) on a publicly-accessible web server with a valid TLS certificate (here we use a public AWS S3 bucket, but alternatively you can host them in the CMK cluster itself using a reverse proxy solution like this)

Step-by-Step Instructions

1. Create a public S3 bucket (for hosting the OIDC Discovery Document and JWKS)

• In your AWS S3 console, create a new S3 bucket and un-check the 'Block all public access' option
• Determine the FQDN of your new bucket. For example, if the bucket is called my-cmk-oidc-discovery-123456789100-us-east-1-an and is in the us-east-1 region, its FQDN will be my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com and the corresponding service-account-issuer URL will be https://my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com. In the following instructions, replace that example FQDN with your own bucket FQDN.
• Edit the bucket policy to allow all objects to be read:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-cmk-oidc-discovery-123456789100-us-east-1-an/*"
        }
    ]
}

    2. Create a new CMK cluster using the Crusoe CLI

Specify the cluster name, region, VPC subnet ID, the add-ons that you want and the extra args as shown below (extra args cannot be supplied when creating a CMK cluster in the Crusoe web console)

crusoe kubernetes clusters create --name my-cluster --cluster-version "1.34.5-cmk.11" \
    --subnet-id 727c6a4d-14cb-4026-8bb8-000000000000 \
    --location us-east1-a \
    --add-ons nvidia_gpu_operator,nvidia_network_operator,crusoe_csi,active_health_checks,autoclusters \
    --apiserver-extra-args "api-audiences=sts.amazonaws.com,service-account-issuer=https://my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com,service-account-jwks-uri=https://my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com/openid/v1/jwks"

• Allow 15 minutes or so for cluster creation to complete
• Download the kubeconfig of the new cluster
• Create nodepools as needed (CPU-only c1a nodes are sufficient for testing this solution)

     3. Obtain the OIDC discovery document and JWKS keys

Copy the OIDC discovery document and JWKS data to the public bucket that you created in step 1 (using the same folder paths as shown in the examples below, but substituting your own bucket name):

kubectl get --raw /.well-known/openid-configuration > openid-configuration.json
kubectl get --raw /openid/v1/jwks > keys.json

aws s3 cp keys.json s3://my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com/openid/v1/jwks
aws s3 cp openid-configuration.json s3://my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com/.well-known/openid-configuration

In your browser, access the public URL corresponding to the keys.json object, which should exactly match the service-account-jwks-uri argument that you supplied when creating the CMK cluster, and verify that the JWKS contents are visible:

4. Add the CMK cluster as an OpenID Connect provider in AWS

• In your AWS IAM console, select Access Management -> Identity providers and add a new Identity provider of type 'OpenID Connect'. Set its URL to match the value of the service-account-issuer arg provided when creating the cluster (i.e https://my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com). Set the audience to be sts.amazonaws.com
• Under IAM Roles, create a new role for the provider, with a Trust Policy like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456789100:oidc-provider/my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com:sub": "system:serviceaccount:default:amazon-access",
                    "my-cmk-oidc-discovery-123456789100-us-east-1-an.s3.us-east-1.amazonaws.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

• Attach relevant permissions to the role. The right choice of permissions will be determined by your use case, but to use the example below, attach permissions that allow S3 buckets to be created and written to.
• For granular access control, create several roles, each with their own set of permissions, and federate them all to the CMK OIDC provider that you just added to AWS. Later on, you can map a different service account to each one and then assign a service account to each workload depending on the access it should have.
• Configuration is now complete. You can now create service accounts that map onto the newly-created roles, and inject them into pods that will then be able to use the permissions that you granted to the role. This is shown in the following example.

Workload Example

Copy the example at https://github.com/crusoecloud/solutions-library/blob/main/cmk-as-oidc-provider/pod-using-s3-with-oidc-sa.yaml and edit the role-arn annotation in two places (the Service Account annotation and the AWS_ROLE_ARN environment variable in the pod spec). Save your edited file and apply it:

kubectl apply -f pod-using-s3-with-oidc-sa.yaml

#after running, view the pod logs
kubectl logs amazon-s3-test
==> Creating bucket: s3://cmk-oidc-test-1778781475
make_bucket: cmk-oidc-test-1778781475
==> Writing test data
==> Listing bucket contents
2026-05-14 17:57:58         58 hello.txt
==> Done

 View the logs of the pod and the contents of your AWS S3 storage to see that the pod created a bucket and uploaded a small file to it. This proves that IRSA is working - and you didn't need to add a single AWS access key or secret to your CMK cluster! If you were to exec into the still-running pod and cat the contents of /var/run/secrets/sts/token, you would see the very long JWT that represents the pod's short-lived credentials.