Ratify on Azure: Allow only signed images to be deployed on AKS with Ratify

Signing container images ensure their authenticity and integrity. By deploying only signed images on Azure Kubernetes Service (AKS), you can ensure that the images come from a trusted origin and have not been altered since they were created.

With Azure Container Registry (ACR), you can store and distribute images with signatures together. You can use Azure Key Vault (AKV) to keep your signing keys and certificates safe, and then use tools like Notation or Cosign to sign your container images with them.

This article walks you through an end-to-end workflow of deploying only signed images on AKS with Ratify.

In this article:

• Prerequisites
• Prepare container images in ACR
• Sign container images in ACR
  • Use Notation with certificates stored in AKV
  • Use Cosign with keys stored in AKV
• Set up an Azure workload identity
• Set up your AKS cluster
• Install OPA Gatekeeper and Ratify
• Configure Ratify
• Deploy container images in AKS

Please note that the examples and commands provided in this document are specifically designed for the Linux operating system.

Prerequisites

• Create or use an ACR for storing container images and signatures
• Create or use an AKV for storing keys and certificates
• Create or use an AKS for deploying container images
• Install Cosign for signing container images with Cosign signatures
• Install Notation and Notation AKV plugin for signing container images with Notary Project signatures
• Install and configure the latest Azure CLI, or run commands in the Azure Cloud Shell.

Configure environment variables

# Azure related variables
export TENANT_ID=<your Tenant ID>
export SUB_ID=<your subscription id>
# AKV related variables
export AKV_RG=<your AKV resource group>
export AKV_NAME=<your AKV name>
# ACR related variables
export ACR_RG=<your ACR resource group>
export ACR_NAME=<your ACR name>
export IMAGE_SIGNED="$ACR_NAME.azurecr.io/ratify-demo/net-monitor:v1"
export IMAGE_SIGNED_SOURCE=https://github.com/wabbit-networks/net-monitor.git#main
export IMAGE_UNSIGNED="$ACR_NAME.azurecr.io/ratify-demo/net-watcher:v1"
export IMAGE_UNSIGNED_SOURCE=https://github.com/wabbit-networks/net-watcher.git#main
# Workload identity used by Ratify to access ACR and AKV
export IDENTITY_NAME=<name of the identity to be created>
export IDENTITY_RG=<your identity resource group name>
# AKS related variables
export AKS_RG=<your AKS resource group>
export AKS_NAME=<your AKS name>
export RATIFY_NAMESPACE="gatekeeper-system"

Prepare container images in ACR

You can skip this section if you already built and pushed images to ACR.

1. Sign in with Azure CLI

   az login

   To learn more about Azure CLI and how to sign in with it, see Sign in with Azure CLI.

2. Ensure the logged-in identity has both acrpush and acrpull roles assigned.

   export USER_OBJECT_ID="$(az ad signed-in-user show --query id -o tsv)"
   az role assignment create \
       --assignee-object-id ${USER_OBJECT_ID} \
       --role acrpull --role acrpush \
       --scope subscriptions/${SUB_ID}/resourceGroups/${ACR_RG}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}

3. Log into ACR

   az acr login --name ${ACR_NAME}

4. Build and push an image that will be signed in later steps.

   az acr build -r ${ACR_NAME} -t ${IMAGE_SIGNED} ${IMAGE_SIGNED_SOURCE} --no-logs

5. Build and push an image that will not be signed

   az acr build -r ${ACR_NAME} -t ${IMAGE_UNSIGNED} ${IMAGE_UNSIGNED_SOURCE} --no-logs

Sign container images in ACR

Notation and Cosign are two options for signing container images. They produce different kinds of signatures and store them in the ACR. Depending on the tool you use, you need to configure Ratify accordingly to verify the signatures from each tool.

You can skip this section if you have already used Notation or Cosign to sign your container images in ACR.

Use Notation with certificates stored in AKV

Depending on the type of certificates you use, you can refer to different documents to sign container images with Notation and AKV.

For self-signed certificates, see Sign container images with Notation and Azure Key Vault using a self-signed certificate.

For CA issued certificates, see Sign container images with Notation and Azure Key Vault using a CA issued certificate.

As a result, the image named ${IMAGE_SIGNED} should be signed successfully with Notation and AKV. Configure the following environment variables for later usage.

export CERT_NAME=<name of signing/leaf certificate>
export SUBJECT_DN=<subject DN of the signing/leaf certificate>
export CERT_KEY_ID=<key identity for the signing/leaf certificate>

Use Cosign with keys stored in AKV

1. Configure environment variables

# The key used for Cosign signing
export KEY_NAME=<name of the key to be created for Cosign signing>

1. Log into Azure

   az login

2. Assign role Key Vault Crypto Officer to logged in identity for creating a key

   export USER_OBJECT_ID="$(az ad signed-in-user show --query id -o tsv)"
   az role assignment create --role "Key Vault Crypto Officer" --assignee ${USER_OBJECT_ID} \
   --scope "/subscriptions/${SUB_ID}/resourceGroups/${AKV_RG}/providers/Microsoft.KeyVault/vaults/${AKV_NAME}"

3. Create a key in your AKV

   az keyvault key create --vault-name ${AKV_NAME} -n ${KEY_NAME} --protection software

   Get the key id and retrieve the key version:

   az keyvault key show --name ${KEY_NAME} --vault-name ${AKV_NAME} --query "key.kid"

   An example output:

   https://<your akv name>.vault.azure.net/keys/<your key name>/<your key version>

   Configure an environment variable for the version for later usage.

   export KEY_VER=<your key version>

4. Sign an image stored in your ACR

   Confirm no signatures before signing

   cosign tree ${IMAGE_SIGNED}

   Sign the image

   cosign sign --key azurekms://$AKV_NAME.vault.azure.net/${KEY_NAME}/${KEY_VER} --tlog-upload=false ${IMAGE_SIGNED}

   In this article, use flag --tlog-upload=false to skip upload the signature to the transparent log (Rekor by default).

   Sign using a key in AKV does not necessarily require the role Key Vault Crypto Officer, you can use another identity and assign the role Key Vault Crypto User for signing action only.

   Confirm the signature is pushed and associated with the image in ACR

   cosign tree ${IMAGE_SIGNED}

Set up an Azure workload identity

Ratify pulls artifacts from an ACR using Workload Federated Identity in an AKS cluster. For an overview on how workload identity operates in Azure, refer to the documentation. You can use workload identity federation to configure an Azure AD app registration or user-assigned managed identity. The following workflow includes the Workload Identity configuration.

Create a Workload Federated Identity.

az identity create --name "${IDENTITY_NAME}" --resource-group "${IDENTITY_RG}" --location "${LOCATION}" --subscription "${SUB_ID}"

export IDENTITY_OBJECT_ID="$(az identity show --name "${IDENTITY_NAME}" --resource-group "${IDENTITY_RG}" --query 'principalId' -otsv)"
export IDENTITY_CLIENT_ID=$(az identity show --name ${IDENTITY_NAME} --resource-group ${IDENTITY_RG} --query 'clientId' -o tsv)

Authoring access to ACR

Configure the user-assigned managed identity and assign AcrPull role to the workload identity.

az role assignment create \
    --assignee-object-id ${IDENTITY_OBJECT_ID} \
    --role acrpull \
    --scope subscriptions/${SUB_ID}/resourceGroups/${ACR_RG}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}

Authoring access to AKV

Image signed with Notation and certificates in AKV

Ratify requires secret permissions to retrieve the root CA certificate from the entire certificate chain, please set private keys to Non-exportable at certificate creation time to avoid security risk. Learn more about non-exportable keys here

For security or other reasons (such as you are from a different organization), you may not be able to access AKV and get the root CA certificates. In that case, you can use the inline certificate provider to specify the root CA certificate value directly, without needing AKV.

Assign Key Vault Secrets User role to this identity for accessing AKV

az role assignment create --role "Key Vault Secrets User" --assignee ${IDENTITY_OBJECT_ID} \
--scope "/subscriptions/${SUB_ID}/resourceGroups/${AKV_RG}/providers/Microsoft.KeyVault/vaults/${AKV_NAME}"

Images signed with Cosign and keys in AKV

Assign Key Vault Crypto User role to this identity for accessing AKV

az role assignment create --role "Key Vault Crypto User" --assignee $IDENTITY_OBJECT_ID \
--scope "/subscriptions/${SUB_ID}/resourceGroups/${AKV_RG}/providers/Microsoft.KeyVault/vaults/${AKV_NAME}"

Set up your AKS cluster

1. Create an OIDC enabled AKS cluster. You can skip this step if you have an AKS cluster with both OIDC and workload identity enabled.

   
   az aks create \
       --resource-group "${AKS_RG}" \
       --name "${AKS_NAME}" \
       --node-vm-size Standard_DS3_v2 \
       --node-count 1 \
       --generate-ssh-keys \
       --enable-workload-identity \
       --attach-acr ${ACR_NAME} \
       --enable-oidc-issuer
   
   # Connect to the AKS cluster:
   az aks get-credentials --resource-group ${AKS_RG} --name ${AKS_NAME}
   
   export AKS_OIDC_ISSUER="$(az aks show -n ${AKS_NAME} -g ${AKS_RG} --query "oidcIssuerProfile.issuerUrl" -otsv)"

   The official steps for setting up Workload Identity on AKS can be found here.

   This step above may take around 10 minutes to complete.

2. Update an existing AKS cluster with OIDC and workload identity enabled. You can skip this step if you have an AKS cluster with both OIDC and workload identity enabled.

   az aks update -g ${AKS_RG} -n ${AKS_NAME} --enable-oidc-issuer --enable-workload-identity
   export AKS_OIDC_ISSUER="$(az aks show -n ${AKS_NAME} -g ${AKS_RG} --query "oidcIssuerProfile.issuerUrl" -otsv)"

3. Establish federated identity credential for namespace ${RATIFY_NAMESPACE} where you deploy Ratify:

   az identity federated-credential create \
   --name ratify-federated-credential \
   --identity-name "${IDENTITY_NAME}" \
   --resource-group "${IDENTITY_RG}" \
   --issuer "${AKS_OIDC_ISSUER}" \
   --subject system:serviceaccount:"${RATIFY_NAMESPACE}":"ratify-admin"

Install Gatekeeper and Ratify in AKS

Run az aks show -g "${AKS_RG}" -n "${AKS_NAME}" --query addonProfiles.azurepolicy to verify if the AKS cluster has azure policy addon enabled, learn more at use azure policy

When Azure Policy Addon is not enabled

1. Install Gatekeeper from helm chart:

   helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
   
   helm install gatekeeper/gatekeeper  \
   --name-template=gatekeeper \
   --namespace gatekeeper-system --create-namespace \
   --set enableExternalData=true \
   --set validatingWebhookTimeoutSeconds=5 \
   --set mutatingWebhookTimeoutSeconds=2 \
   --set externaldataProviderResponseCacheTTL=10s

2. Install Ratify on AKS from helm chart:

   # Add a Helm repo
   helm repo add ratify https://ratify-project.github.io/ratify
   
   # Install Ratify
   helm install ratify \
       ratify/ratify --atomic \
       --namespace ${RATIFY_NAMESPACE} --create-namespace \
       --set featureFlags.RATIFY_CERT_ROTATION=true \
       --set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID}

3. Enforce Gatekeeper policy to allow only signed images can be deployed on AKS:

   kubectl apply -f https://ratify-project.github.io/ratify/library/default/template.yaml
   kubectl apply -f https://ratify-project.github.io/ratify/library/default/samples/constraint.yaml

When Azure Policy Addon is enabled on AKS

1. Ensure your AKS cluster is 1.26+

2. Run az feature register -n AKS-AzurePolicyExternalData --namespace Microsoft.ContainerService

3. Install Ratify on AKS from helm chart:

   # Add a Helm repo
   helm repo add ratify https://ratify-project.github.io/ratify
   helm repo update
   
   # Install Ratify
   helm install ratify \
       ratify/ratify --atomic \
       --namespace gatekeeper-system --create-namespace \
       --set provider.enableMutation=false \
       --set featureFlags.RATIFY_CERT_ROTATION=true \
       --set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID}

4. Create and assign azure policy on your cluster:

   export CUSTOM_POLICY=$(curl -L https://raw.githubusercontent.com/deislabs/ratify/main/library/default/customazurepolicy.json)
   export DEFINITION_NAME="ratify-default-custom-policy"
   export POLICY_SCOPE=$(az aks show -g "${AKS_RG}" -n "${AKS_NAME}" --query id -o tsv)
   
   export DEFINITION_ID=$(az policy definition create --name "${DEFINITION_NAME}" --rules "$(echo "${CUSTOM_POLICY}" | jq .policyRule)" --params "$(echo "${CUSTOM_POLICY}" | jq .parameters)" --mode "Microsoft.Kubernetes.Data" --query id -o tsv)
   
   export ASSIGNMENT_ID=$(az policy assignment create --policy "${DEFINITION_ID}" --name "${DEFINITION_NAME}" --scope "${POLICY_SCOPE}" --query id -o tsv)
   
   echo "Please wait policy assignment with id ${ASSIGNMENT_ID} taking effect"
   echo "It often requires 15 min"
   echo "You can run 'kubectl get constraintTemplate ratifyverification' to verify the policy takes effect"

Configure Ratify

Create a custom resource for accessing ACR

1. Create a configuration file for a Store custom resource named store-oras:

   cat <<EOF > store_config.yaml
   apiVersion: config.ratify.deislabs.io/v1beta1
   kind: Store
   metadata:
     name: store-oras
   spec:
     name: oras
     parameters:
       authProvider:
         name: azureWorkloadIdentity
         clientID: $IDENTITY_CLIENT_ID
       cosignEnabled: true
   EOF

2. Apply the configuration

   kubectl apply -f store_config.yaml

3. Confirm the configuration is applied successful.

   kubectl get Store store-oras

   Make sure the ISSUCCESS value is true in the results of above three commands. If it is not, you need to check the detailed error logs by using kubectl describe commands. For example,

   kubectl describe Store store-oras

Create a custom resource for accessing AKV

1. Create a configuration file for a keymanagementprovider custom resource named keymanagementprovider-akv:

   For verifying images signed with Notation using certificates in AKV, create the following configuration:

   cat <<EOF > kmp_config.yaml
   apiVersion: config.ratify.deislabs.io/v1beta1
   kind: KeyManagementProvider
   metadata:
     name: keymanagementprovider-akv
   spec:
     type: azurekeyvault
     parameters:
       vaultURI: https://${AKV_NAME}.vault.azure.net/
       certificates:
         - name: ${CERT_NAME}
           version: ${CERT_KEY_ID}
       tenantID: ${TENANT_ID}
       clientID: ${IDENTITY_CLIENT_ID}
   EOF

   For verifying images signed with Cosign using keys in AKV, create the following configuration:

   cat <<EOF > kmp_config.yaml
   apiVersion: config.ratify.deislabs.io/v1beta1
   kind: KeyManagementProvider
   metadata:
     name: keymanagementprovider-akv
   spec:
     type: azurekeyvault
     parameters:
       vaultURI: https://${AKV_NAME}.vault.azure.net/
       keys:
         - name: ${KEY_NAME}
           version: ${KEY_VER}
       tenantID: ${TENANT_ID}
       clientID: ${IDENTITY_CLIENT_ID}
   EOF

You may combine the configuration into one KeyManagementProvider resource for both keys and certificates if they are stored in the same AKV.

1. Apply the configuration

   kubectl apply -f kmp_config.yaml

2. Confirm the configuration is applied successful.

   kubectl get KeyManagementProvider keymanagementprovider-akv

   Make sure the ISSUCCESS value is true in the results of above three commands. If it is not, you need to check the detailed error logs by using kubectl describe commands. For example,

   kubectl describe KeyManagementProvider keymanagementprovider-akv

Configure the Notation verifier resource for verifying images signed with Notation

1. Create a configuration file for a Verifier custom resource named verifier-notation:

   cat <<EOF > notation_config.yaml
   apiVersion: config.ratify.deislabs.io/v1beta1
   kind: Verifier
   metadata:
     name: verifier-notation
   spec:
     name: notation
     artifactTypes: application/vnd.cncf.notary.signature
     parameters:
       verificationCertStores:
         ca:
           ca-certs:
             - keymanagementprovider-akv
       trustPolicyDoc:
         version: "1.0"
         trustPolicies:
           - name: default
             registryScopes: 
               - "*"
             signatureVerification:
               level: strict
             trustStores:
               - ca:ca-certs
             trustedIdentities:
               - "x509.subject: ${SUBJECT_DN}"
   EOF

2. Apply the configuration

   kubectl apply -f notation_config.yaml

3. Confirm the configuration is applied successful.

   kubectl get Verifier verifier-notation

   Make sure the ISSUCCESS value is true in the results of above three commands. If it is not, you need to check the detailed error logs by using kubectl describe commands. For example,

   kubectl describe Verifier verifier-notation

Configuration for images signed with Cosign using keys in AKV

1. Create a configuration file for a Verifier custom resource named verifier-cosign:

   cat <<EOF > cosign_config.yaml
   apiVersion: config.ratify.deislabs.io/v1beta1
   kind: Verifier
   metadata:
     name: verifier-cosign
   spec:
     name: cosign
     artifactTypes: application/vnd.dev.cosign.artifact.sig.v1+json
     parameters:
       trustPolicies:
         - name: default
           scopes:
             - "*"
           keys:
             - provider: keymanagementprovider-akv
   EOF

2. Apply the verification configuration

   kubectl apply -f cosign_config.yaml

3. Confirm the configuration is applied successful.

   kubectl get Verifier verifier-cosign

   Make sure the ISSUCCESS value is true in the results of above three commands. If it is not, you need to check the detailed error logs by using kubectl describe commands. For example,

   kubectl describe Verifier verifier-cosign

Deploy container images in AKS

Run the following command, since $IMAGE_SIGNED is signed with the key configured in Ratify, so this image was allowed for deployment after signature verification succeeded.

kubectl run demo-signed --image=$IMAGE_SIGNED

Run the following command, since $IMAGE_UNSIGNED is not signed, so this image was NOT allowed for deployment.

kubectl run demo-unsigned --image=$IMAGE_UNSIGNED

Other scenarios

Fine-tuned trust policy for Cosign verifier

You can configure different trust policies for images from various registry scope for the Cosign verifier. For example, you have two ACR: $ACR_NAME1 and $ACR_NAME2. For $ACR_NAME1, you want to use keymanagementprovider-akv1 resource, For $ACR_NAME2, you want to use keymanagementprovider-akv2 resource. You can update Cosign Verifier resource as the following:

apiVersion: config.ratify.deislabs.io/v1beta1
kind: Verifier
metadata:
  name: verifier-cosign
spec:
  name: cosign
  artifactTypes: application/vnd.dev.cosign.artifact.sig.v1+json
  parameters:
    trustPolicies:
      - name: $ACR_NAME1
        scopes:
          - "$ACR_NAME1.azurecr.io/*"
        keys:
          - provider: keymanagementprovider-akv1
      - name: $ACR_NAME2
        scopes:
          - "$ACR_NAME2.azurecr.io/*"
        keys:
          - provider: keymanagementprovider-akv2

For more information, please refer to the scopes of Cosign verifier .

Rotate the key used by Cosign

Keys in AKV may be rotated regularly as security best practice. If the key is rotated with a new version, you can update the KeyManagementProvider resource by adding the new version of the key, as Ratify (v1.2.0 or before) does not support reconciling key resources regularly. For example, if the new version of key is set to environment variable $KEY_VER_NEW, you can do the following:

1. Add the new key $KEY_VER_NEW for KeyManagementProvider resource

   apiVersion: config.ratify.deislabs.io/v1beta1
   kind: KeyManagementProvider
   metadata:
     name: keymanagementprovider-akv
   spec:
     type: azurekeyvault
   parameters:
     vaultURI: https://$AKV_NAME.vault.azure.net/
     keys:
       - name: $KEY_NAME
         version: $KEY_VER 
         version: $KEY_VER_NEW
     tenantID: $TENANT_ID
     clientID: $CLIENT_ID

2. Apply the new configuration

   kubectl apply -f verification_config.yaml

3. Confirm the new configuration is applied successfully

   kubectl get KeyManagementProvider keymanagementprovider-akv

Disable the specific version of key used by Cosign

In some cases, you may need to disable a specific version of key. For example, the specific version of key is leaked. So, images signed using the specific version of key should not be trusted and the deployment of those images should be denied. As Ratify (v1.2.0 or before) does not support reconciling key resources regularly, so you need to manually remove the version of key from KeyManagementProvider resource. For example, if version $KEY_VER is leaked, what you need to do is:

1. Disable the specific version from AKV, in this case, version $KEY_VER is disabled.

2. Rotate the key to a new version $KEY_VER_NEW, so that your images will be signed with new version.

3. Update KeyManagementProvider resource to add the new version of key and remove the disabled version

   The KeyManagementProvider resource will look like the following as disabled version $KEY_VER was removed.

   apiVersion: config.ratify.deislabs.io/v1beta1
   kind: KeyManagementProvider
   metadata:
     name: keymanagementprovider-akv
   spec:
     type: azurekeyvault
   parameters:
     vaultURI: https://$AKV_NAME.vault.azure.net/
     keys:
       - name: $KEY_NAME
         version: $KEY_VER_NEW
     tenantID: $TENANT_ID
     clientID: $CLIENT_ID

   Apply the new configuration

   kubectl apply -f verification_config.yaml

   Confirm the new configuration is applied successfully

   kubectl get KeyManagementProvider keymanagementprovider-akv