Automating Munki with munki-promoter

Munki is an amazing software management solution for macOS that, paired with AutoPkg, can be fed with patch definitions for software automatically. But, once those patch definitions are in your Munki repo, how do you get them rolled out to your endpoints? Lots of implementations of Munki rely on promoting patch definitions through catalogs manually (perhaps every week, two weeks, or monthly), but this can lead to delays between patches being released, and patches reaching production. Depending on how vendor release days line up with your own promotion days, some software will make it through this pipeline faster than others.

The problem⌗

Consider a Munki implementation with 3 catalogs: autopkg, staging, and production. Software is fed into the autopkg catalog by nightly AutoPkg runs, and every week, an administrator promotes software from staging to production, and from autopkg to staging.

In the best case scenario, a patch definition is released and imported into the autopkg catalog on the morning that this task is carried out. It will then be promoted to staging, and a week later to production, having spent less than 24 hours in the autopkg catalog. It will have spent two weeks in the pipeline before it made it to endpoints.

In the worst-case scenario, a patch definition gets released and imported into the autopkg catalog on the morning after this task is carried out. It then spends 6 days in the autopkg catalog on top of the two weeks it spends in the rest of the release pipeline, for a total of almost 3 weeks.

Because not all vendors will release their patches once a week the night before your AutoPkg runs, you will end up with certain applications spending longer in the release pipeline than other applications, creating inconsistency in how quickly some apps are patched compared to others.

Basic auto-promotion⌗

It’s not super difficult to imagine a script that automates the above flow, where every two weeks the contents of the staging catalog gets moved to production, and a new staging catalog is created from the contents of the autopkg catalog.

However, this script only solves half the problem - the toil of promotion can be reduced, but the inconsistency between patch application durations still remains. Perhaps if there was a way of ensuring that items only spend a specific amount of time in a given catalog, we could also correct for this inconsistency.

The Munki _metadata dictionary⌗

Fortunately, Munki provides a way of at least determining the age of an item. Each patch definition in Munki has its metadata (such as post-install scripts) stored in a pkginfo file, stored in the repo. This XML file also contains a special _metadata key, which contains some information about the environment the patch definition was imported in.

<key>_metadata</key>
	<dict>
		<key>created_by</key>
		<string>autopkg_runner</string>
		<key>creation_date</key>
		<date>2022-12-13T15:52:20Z</date>
		<key>munki_version</key>
		<string>5.7.3.4443</string>
		<key>os_version</key>
		<string>12.5.1</string>
	</dict>

Most interestingly for us, this information also provides a timestamp for the moment that an item was added to the Munki repository.

A promotion workflow⌗

Given that we can ascertain when an item was added to the repo (which then basically serves as an approximation of the software release date if you’re running AutoPkg daily), we can build out a workflow that determines when an item was imported, and promote it to the right catalog based on its age.

Using our 3-catalog implementation from earlier, we can think about the following workflow to promote items out of the autopkg catalog:

With this in mind, I wrote munki-promoter (based on an initial Python2 script by Arjen van Bochoven). The script implements the workflow described in this article, and can be found here.

Note that you’ll probably need to fork and modify this script to fit an existing Munki environment.

The basic premise is the following:

• You run munki-promoter for the “staging to production” promotion run
  • munki-promoter looks at your staging catalog for items older than 14 days
  • munki-promoter then promotes these items to the production catalog
• You run munki-promoter for the “autopkg to staging” promotion run
  • munki-promoter looks at your autopkg catalog for items older than 7 days
  • munki-promoter then promotes these items to the staging catalog

Now, with the above in place, your imported items will uniformly each be promoted a set time after they are imported. You can also then run this workflow as frequently as you’d like (we run ours daily), as it will only take action on items that meet the requirements for promotion.

Implementing this workflow⌗

In our Munki implementation, all of the Munki pkgsinfo and manifests are kept in version control¹. We can then run munki-promoter as an additional CI job², where we clone the Munki repo, make our changes, and then commit and push these changes back into the Munki repo.

An important consideration with any kind automation is that, ideally, you should have some kind of visibility into what’s going on. This gives you the information you need to correlate tickets with events that have taken place in your environment. To achieve this with our AutoPkg pipeline, we set up a Slack channel that gets fed with information about which software is being imported, using a fork of Ben Reilly’s Slacker AutoPkg processor.

With munki-promoter, we want to know when items are being promoted, and when the promotion pipeline fails. Like with AutoPkg, we use Slack for this, sending a nice summary of each promotion run. If the script finds a set SLACK_WEBHOOK environment variable, then it will attempt to send a webhook to the URL specified in that variable.

You run munki-promoter by running it once for each migration, like below:

./munki-promoter.py --name=stagingtoproduction --auto
./munki-promoter.py --name=autopkgtostaging --auto

Note that we’re working on the staging catalog first - starting with the autopkg item promotions would mean we added items to staging, and then immediately promoted these items to production when we ran the second run.

If you want to, you can omit the --auto flag (which triggers automatic runs without confirmations). Doing so drops you into an interactive prompt, like this:

10/04/2023 11:05:09 AM - WARNING: No configuration.yml file was found. Proceeding with defaults...
10/04/2023 11:05:09 AM - WARNING: The 'SLACK_WEBHOOK' environment variable is undefined. Webhooks will not be sent.
***
* Promoting the catalogs of the following pkgsinfo files to ['staging']
***
LogiOptionsPlus - 1.48.0.437015
Firefox - 116.0.2
Firefox - 116.0.1
Firefox - 116.0
Docker-arm64 - 4.22.0
GAMOAuth - 1.0
Cyberduck - 8.6.2
TablePlus - 5.3.9
GoogleChrome - 115.0.5790.170
Microsoft Advertising Editor - 11.30.15573.644
Microsoft Advertising Editor - 11.30.15559.48627
Microsoft Advertising Editor - 11.30.15547.51970
Microsoft Advertising Editor - 11.30.15565.36126
Microsoft Advertising Editor - 11.30.15533.34380
Spotify - 1.2.17.834.g26ee1129
Spotify - 1.2.17.834.g26ee1129
TeamViewer Quick Support - 15.44.5
Rectangle - 0.70
munkitools - 6.3.3.4593
munkitools_admin - 6.3.3.4593
munkitools_python - 3.10.11.4589
munkitools_core - 6.3.3.4593
munkitools_app_usage - 6.3.3.4593
Zoom - 5.15.6.21146
Dropbox - 179.4.4985
Do you want to promote these? [y/n]

Here, you can review the changes that will be made, and confirm that these are the ones you want to make before you write the files out.

Configuration⌗

There might be some cases where you don’t want to promote apps as slowly or as quickly as after 7 days. Perhaps you have a different release schedule, or want a particular app to spend longer in staging before release.

For this use-case, munki-promoter can read configuration from a configuration.yml file in the repository with the script, which stores information about how long a specific Munki item should be in each catalog.

For each promotion run, you can define when that run will happen to a given item. That file looks like this:

# For each promotion you want to configure, 
# add the item name and the number of 
# days the item should live in each catalog
autopkgtostaging:
  GoogleChrome: 3 # Chrome spends 3 days in the autopkg catalog
stagingtoproduction:
  GoogleChrome: 4 # Chrome then spends 4 days in staging

munki-promoter uses the name attribute from an items pkginfo to match these configurations to specific items.

This approach means we don’t have to add any information to pkginfo files or AutoPkg recipes, and munki-promoter worked with our existing Munki repo. If there isn’t any configuration for an app, munki-promoter will just use the defaults of 7 days per catalog.

An argument against auto-promotion⌗

In some environments, automatic promotion might not be suitable. If you’re deploying specific user-requested software that has designated functional requirements (e.g. something like SPSS), there may be a designated software owner who has to carry out testing before greenlighting the release of that update to the general user population. You may also be required to carry out updates within a certain change window, and/or be able to state which versions of apps are being rolled out on certain dates, which is something that is not addressed in munki-promoter.

That doesn’t mean you can’t get something out of a process like this though - based on the import date of Munki items, you can (outside of emergency changes for CVEs) still effectively forecast when software will manually be released according to your release schedule. You may even be able to tie this into your ticketing system, to create change tickets automatically. These are all useful automations in and of themselves that you can use by leveraging the data you already have in your Munki repository.

Future work⌗

This script works well for a pretty simple setup, and is admittedly tailored to how we use Munki. But, it has room for improvement.

• Not everyone uses Munki catalogs in the same way that we do. Some use a lot more catalogs than we do, which would require more modifications to this script than just defining some extra promotions.
• This script is unable to determine (by itself) the order of promotions to run. I’m working on figuring out some logic for this, but at the moment this means we also have hard-coded promotion names in the source for determining when to promote items.
• This script doesn’t “do it all” in one go. Maybe (with the points above) this would be a better approach, where munki-promoter can do the whole promotion step atomically.

Footnotes⌗