Moving Munki Middleware to the Cloud

Middleware for Munki is pretty great. You can simplify your Munki server deployment using cloud-native object storage from your cloud provider of choice, instead of hosting a web server. The middleware out there¹²³⁴⁵ will help authenticate Munki against those providers, so that you also don’t have to publicly serve your Munki repo on the internet.

Middleware, however, is not always so fun to deploy. Some middleware requires you to install extra Python modules for Munki’s built-in Python, which (if you use signed and notarized Munki app bundles) may not work. This usually means compiling Munki (or at least its framework) from source (which can also be challenging).

About a month ago, while I was struggling dealing with Munki middleware, Nate Walck posted something interesting in the #munki channel on the Mac Admins Slack.

Nate had had the amazing idea of responding to authenticated HTTP GET requests with a redirect to a signed GCS URL, meaning you could leverage the native authentication methods for Munki, instead of using middleware. He wasn’t yet able to share his code, but given the perfect timing of seeing this, I decided I’d try and see whether I could recreate it.

A primer on Google Cloud Storage and Signed URLs⌗

Google Cloud Storage (GCS) is Google Cloud Platforms answer to Amazon’s S3 object storage. It is (relatively) cheap, fast, and can be scaled to serve clients across multiple geographical regions.

If you don’t have your GCS bucket open to the world, you’ll need to authenticate requests against it. With Google’s command-line tools, this is simple:

gsutil signurl gs://datboi-munki-prod/catalogs/production

This returns a “signed URL”, which is essentially just a URL with a temporary, time-limited authentication built-in. For a short (configurable) time after this link is generated, anyone can use it to download the linked file.

However, the CLI tools weigh in at about half a gigabyte, are not trivial to install, and you run the risk of messing up the $PATH of your developers. Also, Munki can’t really leverage this, which is where Wade Robson’s gcs-auth middleware comes in. Munki basically passes outgoing web requests to the middleware first, which returns a request that Munki then executes. This means the middleware generates the signed URL for Munki, and Munki sends a GET request while the URL is still valid.

Middleware for GCS requires a Google Service Account set up to handle signing URLs, and the credentials for this service account need to be deployed to Macs using Munki. While this isn’t strictly a security risk (you can limit your service account to have the minimum set of privileges required to sign URLs), you run the risk that accidentally adding roles or permissions to that service account could cause a headache.

Writing a web service to authenticate against⌗

Munki natively supports authentication methods such as Basic Auth⁶, and SSL Client Certificates⁷, with no middleware required. So, how do you write a service that accepts authenticated requests, and returns a signed URL in response?

The above is a simplified explanation, but it’s also not much simpler than the code itself⁸.

1. An incoming request is received by the server
2. The authentication is validated by the server
3. A response code is returned by the server, along with the signed URL if the authentication succeeded.

The function that does the heavy lifting is shown below, along with some comments to explain what’s going on a little bit more:

func (app *application) create_signed_url(w http.ResponseWriter, r *http.Request) {
	log.Printf("- %s - %s - %s", r.RemoteAddr, r.RequestURI, r.Header.Get("User-Agent"))
	ctx := context.Background()

	// Creates a GCS client.
	client, err := storage.NewClient(ctx)
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}
	// Close it later
	defer client.Close()

	// Escape any urlencoding from the URI (otherwise you get invalid signed URLs)
	decoded_uri, err := url.QueryUnescape(r.RequestURI[1:])

	if err != nil {
		log.Fatalf("Failed to decode request: %v", err)
	}

	// Set URL expiry to 15 minutes
	expires := time.Now().Add(time.Minute * 15)
	// Create the signed GET URL
	url, err := client.Bucket(app.gcp_conf.gcs_bucket_name).SignedURL(decoded_uri, &storage.SignedURLOptions{
		Method:  "GET",
		Expires: expires,
		Scheme:  storage.SigningSchemeV4,
	})
	// If there is an error, throw a 500 Internal Server Error
	if err != nil {
		http.Error(w, "Internal Server Error", http.StatusInternalServerError)
		fmt.Println("Error " + err.Error())
	}

	// we use StatusFound (302) here because StatusMovedPermanently (301) can mean the redirect gets cached
	// Chrome does this, we don't want expired tokens to get cached client-side
	// Return the redirect
	http.Redirect(w, r, url, http.StatusFound)

}

It’s worth noting here that this works because Munki respects HTTP status codes - returning a redirect to a Munki query means that Munki simply follows the provided redirect. We return a 302 (Found) instead of a 301 (Moved Permanently) because this is The Proper Way - signed URLs are not permanent. Some clients (Google Chrome being an example) will cache 301 responses, as they assume they will not change. Returning a 301 runs the risk that a client caches an expired URL, which is less than ideal, whereas 302 responses should not be cached.

It’s also worth mentioning that if you request a non-existent object from the server, GCS will still sign a URL for it. This actually saves us some work, as instead of validating whether the object being requested exists, we can return this URL - it will 404 anyways.

A final advantage is that, because we are limiting what the container does to this one function, inadvertent expansion of service account privileges doesn’t allow for easy and immediate exploitation by a malicious actor.

Running the service⌗

The service presented here has been written to work with Google Cloud Run⁹. You’ll notice, looking through the source code, that GCP credentials are never required. Google Cloud Run (like a lot of other Google Services) lets you attach a service account to a deployment¹⁰, which essentially injects credentials into it. If you assign the service account you were previously using with middleware to the Cloud Run deployment, it will work “out of the box”.

You will need to provide the credentials you want to use for basic authentication, as well as the name of the bucket containing your Munki repo, as environment variables in Cloud Run.

If you decide you want to use this with Munki, please let me know! I’d be curious to see whether people prefer this over middleware. As always, you can send me an email if you have questions or ideas, or find me over on the Mac Admins Slack as @jc0b. munki-gcs-redirector⁸ is my first foray into Golang, so feel free to contribute any suggestions or improvements!

Lastly, a massive thanks again to Nate for giving me the idea to write this myself, it was a lot of fun!

References⌗