The power of ArgoCD can be enhanced by the use of so called plugins. A plugin will operate during a refresh operation. It has access to the checked out repository and may manipulate data before syncing it to your Kubernetes clusters. I use it for encrypt my git-crypt encrypted repositories, before manifests will get applied to my Kubernetes cluster.
There are two possible options to implement ArgoCD, one is the option to implement it via a Configmap, the second is, to run the plugin within a sidecar container of the ArgoCD repo server. Lately I stumbled over the information, that the Argo project decided to deprecate the option of using plugins via Configmap with the upcoming version 2.6. This is how I have implemented my plugins at the moment. So, to be prepared, I decided to have a look into running plugins within a sidecar container. I want to share this learning process with you here, cause the official documentation lacks on many information in my opinion.
Let’s take one step back first. If you wonder what plugins might be useful for, here are two examples what I use them for:
- Decrypting my git-crypt encrypted repositories. Else Argo can not process encrypted manifests obviously.
- Templating of my Git repos (I’m using gomplate for that) before the manifests will get applied
We will use the example plugin from the official Argo website (with some changes though, cause this will not work as described). Also, there is the possibility, to “bake in” the actual plugin manifest to the sidecar container image. I decided against that cause my plugins are not that complex, and I don’t want to build a new image every time I change something in my script. So we will still configure the plugin manifest within a Configmap.
Let’s start with that. Let’s create a Configmap that holds the plugin manifest. Although it is of kind: ConfigManagementPlugin, it is not a CRD. It’s only something that Argo can parse later and therefore has a fixed structure:
apiVersion: v1
kind: ConfigMap
metadata:
name: cmp-plugin
namespace: argocd
data:
plugin.yaml: |
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: cmp-plugin
spec:
version: v1.0
init:
command: [sh, -c, 'echo "Initializing..."']
generate:
command: [sh, -c, 'echo "{\"kind\": \"ConfigMap\", \"apiVersion\": \"v1\", \"metadata\": { \"name\": \"$ARGOCD_APP_NAME\", \"namespace\": \"$ARGOCD_APP_NAMESPACE\", \"annotations\": {\"Foo\": \"$ARGOCD_ENV_FOO\", \"KubeVersion\": \"$KUBE_VERSION\", \"KubeApiVersion\": \"$KUBE_API_VERSIONS\",\"Bar\": \"baz\"}}}"']
discover:
fileName: "./plugin-run"I want to share some more words on the ConfigManagementPlugin. As you might have recognized, it’s divided in three sections: discover, init and generate. Having a discover section is mandatory when running a plugin in a sidecar. The reason for that is, that you no longer are allowed to specify which plugin to run within your Application or ApplicationSet. All plugins will run in random order, if it will actually be used is defined by the result of the discover script. You really should avoid that two or more plugins can be triggered for one Application. In the above example, the discover part will check for a file plugin-run. If it exists (in the application repository path), the plugin will get triggered to run and the next step will be defined in the init section. Here you can do more or less what you want, having an init section is optional in my experience. The third and last step (generate) is again mandatory. It has one requirement, the result of this section needs to generate something that is able to be processed by Kubernetes, so it needs to be valid YAML or JSON. Last but not least, you may wonder about the environment variables I’m using in the example (they differ from the official documentation). Every user defined environment variable in an Application needs to be prefixed with ARGOCD_ENV_ within the plugin.
The above example plugin should create a JSON formated manifest of a ConfigMap.
Next, we need to update the ArgoCD deployment and add something like the following to it. It describes the sidecar that should run in the ArgoCD Repo Server pod. In my Github repository, you can find a Kustomize overlay, that will do the patching of the original deployment for you.
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
containers:
- name: cmp
securityContext:
runAsNonRoot: true
runAsUser: 999
image: ghcr.io/thedatabaseme/kube-tools:0.1.1
imagePullPolicy: IfNotPresent
command: [/var/run/argocd/argocd-cmp-server]
volumeMounts:
- mountPath: /var/run/argocd
name: var-files
- mountPath: /home/argocd/cmp-server/plugins
name: plugins
- mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: plugin.yaml
name: cmp-plugin
- mountPath: /tmp
name: cmp-tmp
volumes:
- name: cmp-plugin
configMap:
name: cmp-plugin
- emptyDir: {}
name: cmp-tmpI use my own built kube-tools image for running the sidecar container. It has some additional tooling installed which you might find handy. Because this demo is not using any tools that are not present on most container images, you may want to change it to busybox or alpine.
Having the sidecar running under the user ID 999 is a must again. Else you will have issues using the repository data that is getting checked out and processed by the plugin.
The next and final step is to create an application for ArgoCD. An example app is part of the kustomization overlay in my Github repo. If you want to use it, you need to add my cookbook git repo to your ArgoCD repositories.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: plugin-demo
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
namespace: default
project: default
source:
path: kubernetes/argocd/manifests/plugin-example
plugin:
env:
- name: FOO
value: bar
repoURL: https://github.com/thedatabaseme/cookbooks.git
targetRevision: HEAD
syncPolicy:
automated:
prune: true
selfHeal: trueNow apply all the manifests (best using the kustomization.yaml from my repo first). If all goes well, you should see something like this in your Argo UI.
You might recognize something interesting. The path within the application points to another kustomization.yaml which would normally deploy a simple busybox pod to your cluster. With the running plugin, there is only the configmap created which we specified within the plugin. This is, because as I wrote before, the plugin output needs to be YAML or JSON. But we leave out the original application YAML and just deliver the configmap as output. If you want to also add the rest of your application, you need to append something like && kustomize build . -o kustomize.out && cat kustomize.out to your plugin.
If you check the logs of the sidecar container, you can see log output like the following:
time="2022-12-02T15:01:19Z" level=info msg="ArgoCD ConfigManagementPlugin Server is starting" built="2022-11-28T16:51:33Z" commit=0c7de210ae66bf631cc4f27ee1b5cdc0d04c1c96 version=v2.5.3+0c7de21
time="2022-12-02T15:01:19Z" level=info msg="argocd-cmp-server v2.5.3+0c7de21 serving on /home/argocd/cmp-server/plugins/cmp-plugin-v1.0.sock"
time="2022-12-02T15:02:13Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=MatchRepository grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-12-02T15:02:13Z" grpc.time_ms=2.851 span.kind=server system=grpc
time="2022-12-02T15:02:13Z" level=info msg="Generating manifests with no request-level timeout"
time="2022-12-02T15:02:13Z" level=info msg="sh -c echo \"Initializing...\"" dir=/tmp/_cmp_server/7b5268f4-24c7-483b-af1b-a7af12323e09/kubernetes/argocd/manifests/plugin-example execID=7688b
time="2022-12-02T15:02:13Z" level=info msg="sh -c echo \"{\\\"kind\\\": \\\"ConfigMap\\\", \\\"apiVersion\\\": \\\"v1\\\", \\\"metadata\\\": { \\\"name\\\": \\\"$ARGOCD_APP_NAME\\\", \\\"namespace\\\": \\\"$ARGOCD_APP_NAMESPACE\\\", \\\"annotations\\\": {\\\"Foo\\\": \\\"$ARGOCD_ENV_FOO\\\", \\\"KubeVersion\\\": \\\"$KUBE_VERSION\\\", \\\"KubeApiVersion\\\": \\\"$KUBE_API_VERSIONS\\\",\\\"Bar\\\": \\\"baz\\\"}}}\"" dir=/tmp/_cmp_server/7b5268f4-24c7-483b-af1b-a7af12323e09/kubernetes/argocd/manifests/plugin-example execID=ff873
time="2022-12-02T15:02:13Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=plugin.ConfigManagementPluginService grpc.start_time="2022-12-02T15:02:13Z" grpc.time_ms=50.778 span.kind=server system=grpc
The final configmap looks like this, as you can see, our environment variable was injected correctly:
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
Bar: baz
Foo: bar
KubeVersion: '1.24'
labels:
app.kubernetes.io/instance: plugin-demo
name: plugin-demo
namespace: defaultUpdate 08.02.2023: The deprecation of CMPs configured within a configmap has been delayed until Argo 2.7. Also there was the decision, to reimplement the possibility to specify which plugin should be used within the Application / Applicationset manifest as spec.plugin.name. This comes with a price though.
- Different from the description above, your
ConfigManagementPluginmust not have adiscoverysection when you want to specify it viaplugin.namein the Application. - You cannot specify which plugin to be used by
plugin.name, as long as you have a version specified in yourConfigManagementPlugin. If you want to have different versions for your CMPs, you need to specify the version within theplugin.nameas<metadata.name>-<spec.version>. I for myself removed the version from the CMP specification.
Philip
