Munki is an amazing software management solution for macOS that, paired with AutoPkg, can be fed with patch definitions for software automatically. But, once those patch definitions are in your Munki repo, how do you get them rolled out to your endpoints? Lots of implementations of Munki rely on promoting patch definitions through catalogs manually (perhaps every week, two weeks, or monthly), but this can lead to delays between patches being released, and patches reaching production. Depending on how vendor release days line up with your own promotion days, some software will make it through this pipeline faster than others.
Consider a Munki implementation with 3 catalogs:
Software is fed into the
autopkg catalog by nightly AutoPkg runs, and every week, an administrator promotes software from
production, and from
In the best case scenario, a patch definition is released and imported into the
autopkg catalog on the morning that this task is carried out.
It will then be promoted to
staging, and a week later to
production, having spent less than 24 hours in the
It will have spent two weeks in the pipeline before it made it to endpoints.
In the worst-case scenario, a patch definition gets released and imported into the
autopkg catalog on the morning after this task is carried out.
It then spends 6 days in the
autopkg catalog on top of the two weeks it spends in the rest of the release pipeline, for a total of almost 3 weeks.
Because not all vendors will release their patches once a week the night before your AutoPkg runs, you will end up with certain applications spending longer in the release pipeline than other applications, creating inconsistency in how quickly some apps are patched compared to others.
It’s not super difficult to imagine a script that automates the above flow, where every two weeks the contents of the
staging catalog gets moved to
production, and a new
staging catalog is created from the contents of the
However, this script only solves half the problem - the toil of promotion can be reduced, but the inconsistency between patch application durations still remains. Perhaps if there was a way of ensuring that items only spend a specific amount of time in a given catalog, we could also correct for this inconsistency.
Fortunately, Munki provides a way of at least determining the age of an item.
Each patch definition in Munki has its metadata (such as post-install scripts) stored in a
pkginfo file, stored in the repo. This XML file also contains a special
_metadata key, which contains some information about the environment the patch definition was imported in.
<key>_metadata</key> <dict> <key>created_by</key> <string>autopkg_runner</string> <key>creation_date</key> <date>2022-12-13T15:52:20Z</date> <key>munki_version</key> <string>184.108.40.20643</string> <key>os_version</key> <string>12.5.1</string> </dict>
Most interestingly for us, this information also provides a timestamp for the moment that an item was added to the Munki repository.
A promotion workflow⌗
Given that we can ascertain when an item was added to the repo (which then basically serves as an approximation of the software release date if you’re running AutoPkg daily), we can build out a workflow that determines when an item was imported, and promote it to the right catalog based on its age.
Using our 3-catalog implementation from earlier, we can think about the following workflow to promote items out of the
Note that you’ll probably need to fork and modify this script to fit an existing Munki environment.
The basic premise is the following:
- You run
munki-promoterfor the “staging to production” promotion run
munki-promoterlooks at your
stagingcatalog for items older than 14 days
munki-promoterthen promotes these items to the
- You run
munki-promoterfor the “autopkg to staging” promotion run
munki-promoterlooks at your
autopkgcatalog for items older than 7 days
munki-promoterthen promotes these items to the
Now, with the above in place, your imported items will uniformly each be promoted a set time after they are imported. You can also then run this workflow as frequently as you’d like (we run ours daily), as it will only take action on items that meet the requirements for promotion.
Implementing this workflow⌗
In our Munki implementation, all of the Munki pkgsinfo and manifests are kept in version control1. We can then run
munki-promoter as an additional CI job2, where we clone the Munki repo, make our changes, and then commit and push these changes back into the Munki repo.
An important consideration with any kind automation is that, ideally, you should have some kind of visibility into what’s going on.
This gives you the information you need to correlate tickets with events that have taken place in your environment.
To achieve this with our AutoPkg pipeline, we set up a Slack channel that gets fed with information about which software is being imported, using a fork of Ben Reilly’s
Slacker AutoPkg processor.
munki-promoter, we want to know when items are being promoted, and when the promotion pipeline fails. Like with AutoPkg, we use Slack for this, sending a nice summary of each promotion run. If the script finds a set
SLACK_WEBHOOK environment variable, then it will attempt to send a webhook to the URL specified in that variable.
munki-promoter by running it once for each migration, like below:
./munki-promoter.py --name=stagingtoproduction --auto ./munki-promoter.py --name=autopkgtostaging --auto
Note that we’re working on the
stagingcatalog first - starting with the
autopkgitem promotions would mean we added items to
staging, and then immediately promoted these items to
productionwhen we ran the second run.
If you want to, you can omit the
--auto flag (which triggers automatic runs without confirmations). Doing so drops you into an interactive prompt, like this:
10/04/2023 11:05:09 AM - WARNING: No configuration.yml file was found. Proceeding with defaults... 10/04/2023 11:05:09 AM - WARNING: The 'SLACK_WEBHOOK' environment variable is undefined. Webhooks will not be sent. *** * Promoting the catalogs of the following pkgsinfo files to ['staging'] *** LogiOptionsPlus - 220.127.116.117015 Firefox - 116.0.2 Firefox - 116.0.1 Firefox - 116.0 Docker-arm64 - 4.22.0 GAMOAuth - 1.0 Cyberduck - 8.6.2 TablePlus - 5.3.9 GoogleChrome - 115.0.5790.170 Microsoft Advertising Editor - 11.30.15573.644 Microsoft Advertising Editor - 11.30.15559.48627 Microsoft Advertising Editor - 11.30.15547.51970 Microsoft Advertising Editor - 11.30.15565.36126 Microsoft Advertising Editor - 11.30.15533.34380 Spotify - 18.104.22.1684.g26ee1129 Spotify - 22.214.171.1244.g26ee1129 TeamViewer Quick Support - 15.44.5 Rectangle - 0.70 munkitools - 126.96.36.19993 munkitools_admin - 188.8.131.5293 munkitools_python - 184.108.40.20689 munkitools_core - 220.127.116.1193 munkitools_app_usage - 18.104.22.16893 Zoom - 22.214.171.12446 Dropbox - 179.4.4985 Do you want to promote these? [y/n]
Here, you can review the changes that will be made, and confirm that these are the ones you want to make before you write the files out.
There might be some cases where you don’t want to promote apps as slowly or as quickly as after 7 days. Perhaps you have a different release schedule, or want a particular app to spend longer in
staging before release.
For this use-case,
munki-promoter can read configuration from a
configuration.yml file in the repository with the script, which stores information about how long a specific Munki item should be in each catalog.
For each promotion run, you can define when that run will happen to a given item. That file looks like this:
# For each promotion you want to configure, # add the item name and the number of # days the item should live in each catalog autopkgtostaging: GoogleChrome: 3 # Chrome spends 3 days in the autopkg catalog stagingtoproduction: GoogleChrome: 4 # Chrome then spends 4 days in staging
munki-promoter uses the
name attribute from an items pkginfo to match these configurations to specific items.
This approach means we don’t have to add any information to pkginfo files or AutoPkg recipes, and
munki-promoter worked with our existing Munki repo. If there isn’t any configuration for an app,
munki-promoter will just use the defaults of 7 days per catalog.
An argument against auto-promotion⌗
In some environments, automatic promotion might not be suitable.
If you’re deploying specific user-requested software that has designated functional requirements (e.g. something like SPSS), there may be a designated software owner who has to carry out testing before greenlighting the release of that update to the general user population.
You may also be required to carry out updates within a certain change window, and/or be able to state which versions of apps are being rolled out on certain dates, which is something that is not addressed in
That doesn’t mean you can’t get something out of a process like this though - based on the import date of Munki items, you can (outside of emergency changes for CVEs) still effectively forecast when software will manually be released according to your release schedule. You may even be able to tie this into your ticketing system, to create change tickets automatically. These are all useful automations in and of themselves that you can use by leveraging the data you already have in your Munki repository.
This script works well for a pretty simple setup, and is admittedly tailored to how we use Munki. But, it has room for improvement.
- Not everyone uses Munki catalogs in the same way that we do. Some use a lot more catalogs than we do, which would require more modifications to this script than just defining some extra promotions.
- This script is unable to determine (by itself) the order of promotions to run. I’m working on figuring out some logic for this, but at the moment this means we also have hard-coded promotion names in the source for determining when to promote items.
- This script doesn’t “do it all” in one go. Maybe (with the points above) this would be a better approach, where
munki-promotercan do the whole promotion step atomically.
In practice, you don’t need to have Munki in version control to do this, nor do you need to have any CI/CD pipelines. You could run this task as a cron job or LaunchDaemon on your Munki server itself. ↩︎
You can just as well include this task in your daily AutoPkg run, or even a scheduled job on the Munki repo itself. We don’t, in order to have a clear seperation of duty between all of our Munki-related repositories. ↩︎