Fix a Helm "Secret Too Long" Bug

When try to upgrade a Helm chart, it fails with an error like below.

Error: UPGRADE FAILED: create: failed to create: 
Secret "sh.helm.release..." is invalid: 
data: Too long: must have at most 1048576 bytes

Helm by default uses Kubernetes Secrets to store release information. For what is “release information”, basically it includes the source files of the chart.

The release information includes the contents of charts and values files

See Storage backends for more details.

In my case, certificates (after encryption) are stored in the chart files, each of which is like hundreds KB big. And what’s more, each certificates has several variants for each deployment environments, like alpha, beta and production. All these files are in one helm chart. When to bring in a new certificate into the chart, the chart becomes too big, so the above error happens.

To solve this error for my chart, a simple solution to create a script to dynamically “helm ignore” files which are not needed for a Helm release for a particular deployment environment. For example, before do Helm release for the alpha environment, (using CI scripts to) add files which are for production and beta into the .helmignore file to exclude these files from the Helm chart.

HELM_ENVS=(alpha beta prod)
for HELM_ENV in ${HELM_ENVS[@]}; do
  if [ "$HELM_ENV" != "$DEPLOY_ENV" ]; then
      echo "helm ignore this folder in chart: secrets/$HELM_ENV/"
      echo "secrets/$APL_ENV/" >> .helmignore
  fi
done

Helm also has a beta feature called “SQL storage backend” to store release information in a database, for a big chart.

Using such a storage backend is particularly useful if your release information weighs more than 1MB