In this post, I’d like to describe the issue I’ve recently encountered when using Helm umbrella charts. Long story short, it turned out that subcharts are not completely isolated from each other, contrary to what we probably would expect. This feature has also some important consequences for building umbrella charts and combining different dependencies into one parent chart, which I want to outline as well.

Umbrella charts basics

Helm umbrella charts are an easy and powerful way of installing multiple components as a single one. They allow us to set up pretty complex configurations like Kafka cluster or Elastic stack with minimal effort - by installing just a single chart (one helm install call).

All components (dependencies) an umbrella chart contain, are also Helm charts - this makes it in fact a chart of charts. Those dependencies could be defined within the Chart.yaml file inside a dedicated section called dependencies (in Helm 2 compatible syntax, this has to be placed in the requirements.yaml file) like this:

# parent/Chart.yaml

dependencies:
  - name: sub1
    repository: https://charts.thisisjustanexample.com/examplerepo
    version: 1.1.0
  - name: sub2
    repository: https://charts.thisisjustanexample.com/examplerepo
    version: 1.3.2
  - name: sub3
    repository: https://charts.thisisjustanexample.com/examplerepo
    version: 0.1.0

This approach should feel familiar to all the developers using dependency management tools like Maven or npm, where one dependency can introduce additional transitive packages too. However, one could say that it’s not the most accurate analogy in this case since things being deployed in Helm & Kubernetes are rather isolated from each other, while packages may run into conflicts in some cases (for example when two different versions of the same library are in use). If you also share this feeling, then the upcoming example may change your mind.

Reproducing the issue

Let’s assume we want to build an umbrella chart containing two services. Each of the services will have a dependency on the MongoDB chart. However, in both cases, the MongoDB chart will come from a bit different distribution. As a result, we would like to create a structure like in the diagram below:

As you can see, there would be no direct dependency between svc1 and svc2. We can say, that those services should know nothing about each other.

Side note: don’t get me wrong - I’m far from strongly recommending keeping the databases within k8s in general. MongoDB appeared here only to better illustrate the issue. If you’re interested in that topic, there is a nice Google Cloud Blog post covering that.

Starting from service definitions:

• svc1 would have a single dependency on the latest Bitnami’s MongoDB from Helm Hub

  # svc1/Chart.yaml
  dependencies:
      - name: mongodb
        repository: "https://charts.bitnami.com/bitnami"
        version: 8.1.1
  

• svc2 would have a single dependency on the latest MongoDB from the deprecated stable repo

  # svc2/Chart.yaml
  dependencies:
      - name: mongodb
        repository: "https://charts.helm.sh/stable"
        version: 7.4.6
  

Our parent chart called umbrella-chart needs dependencies on svc1 and svc2:

# umbrella-chart/Chart.yaml
dependencies:
     - name: svc1
       repository: "https://charts.thisisjustanexample.com/examplerepo"
       version: 1.2.3
     - name: svc2
       repository: "https://charts.thisisjustanexample.com/examplerepo"
       version: 2.3.4

Additionally, to avoid name conflicts between secrets, etc., we should provide different naming for both MongoDB installations. The easiest way to achieve that is to override dependant charts’ properties from within the umbrella chart’s values.yaml file:

# umbrella-chart/values.yaml
svc1:
  mongodb:
    nameOverride: "svc1-mongodb"
    fullnameOverride: "svc1-mongodb"

svc2:
  mongodb:
    nameOverride: "svc2-mongodb"
    fullnameOverride: "svc2-mongodb"

The complete configuration for this example could be found on GitHub.

Since everything is already set up, we can simply install our umbrella chart with helm install umbrella-chart ./umbrella-chart. Unfortunately, we will run into the following error:

Error: template: umbrella-chart/charts/svc2/charts/mongodb/templates/NOTES.txt:68:4: executing "umbrella-chart/charts/svc2/charts/mongodb/templates/NOTES.txt" at <include "mongodb.validateValues" .>: error calling include: template: umbrella-chart/charts/svc1/charts/mongodb/templates/_helpers.tpl:174:35: executing "mongodb.validateValues" at <include "mongodb.validateValues.architecture" .>: error calling include: template: umbrella-chart/charts/svc1/charts/mongodb/templates/_helpers.tpl:190:12: executing "mongodb.validateValues.architecture" at <ne .Values.architecture "standalone">: error calling ne: incompatible types for comparison

The error message suggest a problem related to the templates defined within _helpers.tpl files, but it does not provide a clear reason for the failure. What happened?

Lack of templates isolation

As stated few times in Helm’s docs:

An important detail to keep in mind when naming templates: template names are global. If you declare two templates with the same name, whichever one is loaded last will be the one used. Because templates in subcharts are compiled together with top-level templates, you should be careful to name your templates with chart-specific names.

Due to this behavior, in order to avoid conflicts and ensure at least partial isolation, the docs advise:

One popular naming convention is to prefix each defined template with the name of the chart: {{ define "mychart.labels" }}. By using the specific chart name as a prefix we can avoid any conflicts that may arise due to two different charts that implement templates of the same name.

However, both MongoDB charts from our example already prefix their templates defined within _helpers.tpl. The problem is, they both use the same mongodb prefix.

It should not be surprising at this point, that in our setup there may exist templates with the same name but different definitions. As an example, let’s compare the definitions of the mongodb.fullname templates. In stable/mongodb we have:

{{- define "mongodb.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}

while in bitnami/mongodb:

{{- define "mongodb.fullname" -}}
{{- include "common.names.fullname" . -}}
{{- end -}}

We can just hope, that in the end, both definitions lead to the same fullname value, but it’s unclear after taking just a quick look. A further comparison would reveal, that are some templates being defined in one chart distribution, but not in the other - just like with the architecture in our example.

The exact reason for the error message we’ve seen when installing the example umbrella chart is the lack of isolation combined with the order in which Helm processes the subchart. Lucky for us, installation failed before actually doing anything on the cluster. Unfortunately, this is not always to be the case.

Inverting the order

Helm docs state clearly, that for template resolution the last loaded wins strategy is used. Unfortunately, it seems to be undocumented, in which order the subcharts are being processed when installing the parent umbrella chart. This encouraged me to experiment even further.

My first attempt was to just change the order of services within the dependencies section of the Chart.yaml file (to make svc2 appear before svc1). However, it made no difference and the same error message appeared.

It seems to be that the subcharts’ templates are being loaded alphabetically (at least in our example setup). If we invert the dependencies of svc1 and svc2, so svc1 would declare the dependency from the stable repo and svc2 would use the Bitnami’s MongoDB flavor, we would observe, that helm install command execution will succeed. In this case, no error message does not have to mean that everything is alright. The templates defined in one service have overridden the templates of the other one and the final result may be very hard to predict.

At the time of creating this article, Helm (checked with v3.4.1 and v2.17) do not provide any hints or warnings for any of the described conflict scenarios. However, some discussions within the project’s GitHub issues gives hope, that Helm v4 may actually improve the situation in the future.

Avoiding conflicts

The best what we have for now is to avoid template conflicts when designing our umbrella charts. To do so, the following rules should be followed within the scope of a single umbrella chart:

• If possible, do not declare a dependency on the same chart more than once.
• If the same dependent chart has to be declared multiple times, make sure that exactly the same version is being used consistently.
• If the different versions of the same chart have to be used, make sure that they have the same template definitions (identical _helpers.tpl) or just “rewire” one of them.

The “rewiring” of the subchart would mean to edit it in a way, that a different prefix would be used for all its templates. This may not be a big deal when the dependency is our own chart (we are the maintainers), but in the case of some publically available charts, it may be quite challenging.

Summary

It should be kept in mind, that the parent chart’s dependencies are not fully isolated from each other. Because all templates defined in subcharts are considered global, name conflicts may occur. Since the tooling does not provide any warnings currently, umbrella chart maintainers have to be careful when including the dependencies - otherwise, the results may be very hard to predict.