admin: Identity management, remote admin, config loaders (#3994)

This commits dds 3 separate, but very related features:

1. Automated server identity management

How do you know you're connecting to the server you think you are? How do you know the server connecting to you is the server instance you think it is? Mutually-authenticated TLS (mTLS) answers both of these questions. Using TLS to authenticate requires a public/private key pair (and the peer must trust the certificate you present to it).

Fortunately, Caddy is really good at managing certificates by now. We tap into that power to make it possible for Caddy to obtain and renew its own identity credentials, or in other words, a certificate that can be used for both server verification when clients connect to it, and client verification when it connects to other servers. Its associated private key is essentially its identity, and TLS takes care of possession proofs.

This configuration is simply a list of identifiers and an optional list of custom certificate issuers. Identifiers are things like IP addresses or DNS names that can be used to access the Caddy instance. The default issuers are ZeroSSL and Let's Encrypt, but these are public CAs, so they won't issue certs for private identifiers. Caddy will simply manage credentials for these, which other parts of Caddy can use, for example: remote administration or dynamic config loading (described below).

2. Remote administration over secure connection

This feature adds generic remote admin functionality that is safe to expose on a public interface.

- The "remote" (or "secure") endpoint is optional. It does not affect the standard/local/plaintext endpoint.
- It's the same as the [API endpoint on localhost:2019](https://caddyserver.com/docs/api), but over TLS.
- TLS cannot be disabled on this endpoint.
- TLS mutual auth is required, and cannot be disabled.
- The server's certificate _must_ be obtained and renewed via automated means, such as ACME. It cannot be manually loaded.
- The TLS server takes care of verifying the client.
- The admin handler takes care of application-layer permissions (methods and paths that each client is allowed to use).\
- Sensible defaults are still WIP.
- Config fields subject to change/renaming.

3. Dyanmic config loading at startup

Since this feature was planned in tandem with remote admin, and depends on its changes, I am combining them into one PR.

Dynamic config loading is where you tell Caddy how to load its config, and then it loads and runs that. First, it will load the config you give it (and persist that so it can be optionally resumed later). Then, it will try pulling its _actual_ config using the module you've specified (dynamically loaded configs are _not_ persisted to storage, since resuming them doesn't make sense).

This PR comes with a standard config loader module called `caddy.config_loaders.http`.

Caddyfile config for all of this can probably be added later.

COMMITS:

* admin: Secure socket for remote management

Functional, but still WIP.

Optional secure socket for the admin endpoint is designed
for remote management, i.e. to be exposed on a public
port. It enforces TLS mutual authentication which cannot
be disabled. The default port for this is :2021. The server
certificate cannot be specified manually, it MUST be
obtained from a certificate issuer (i.e. ACME).

More polish and sensible defaults are still in development.

Also cleaned up and consolidated the code related to
quitting the process.

* Happy lint

* Implement dynamic config loading; HTTP config loader module

This allows Caddy to load a dynamic config when it starts.

Dynamically-loaded configs are intentionally not persisted to storage.

Includes an implementation of the standard config loader, HTTPLoader.
Can be used to download configs over HTTP(S).

* Refactor and cleanup; prevent recursive config pulls

Identity management is now separated from remote administration.

There is no need to enable remote administration if all you want is identity
management, but you will need to configure identity management
if you want remote administration.

* Fix lint warnings

* Rename identities->identifiers for consistency

11 files changed, 862 insertions, 214 deletions

diff --git a/admin.go b/admin.go
index f539b44..f333657 100644
--- a/admin.go
+++ b/admin.go
@@ -17,6 +17,10 @@ package caddy
 import (
 	"bytes"
 	"context"
+	"crypto"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/base64"
 	"encoding/json"
 	"errors"
 	"expvar"
@@ -35,12 +39,11 @@ import (
 	"sync"
 	"time"
 
+	"github.com/caddyserver/certmagic"
 	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/zap"
 )
 
-// TODO: is there a way to make the admin endpoint so that it can be plugged into the HTTP app? see issue #2833
-
 // AdminConfig configures Caddy's API endpoint, which is used
 // to manage Caddy while it is running.
 type AdminConfig struct {
@@ -58,54 +61,131 @@ type AdminConfig struct {
 	// If true, CORS headers will be emitted, and requests to the
 	// API will be rejected if their `Host` and `Origin` headers
 	// do not match the expected value(s). Use `origins` to
-	// customize which origins/hosts are allowed.If `origins` is
+	// customize which origins/hosts are allowed. If `origins` is
 	// not set, the listen address is the only value allowed by
-	// default.
+	// default. Enforced only on local (plaintext) endpoint.
 	EnforceOrigin bool `json:"enforce_origin,omitempty"`
 
 	// The list of allowed origins/hosts for API requests. Only needed
 	// if accessing the admin endpoint from a host different from the
 	// socket's network interface or if `enforce_origin` is true. If not
 	// set, the listener address will be the default value. If set but
-	// empty, no origins will be allowed.
+	// empty, no origins will be allowed. Enforced only on local
+	// (plaintext) endpoint.
 	Origins []string `json:"origins,omitempty"`
 
-	// Options related to configuration management.
+	// Options pertaining to configuration management.
 	Config *ConfigSettings `json:"config,omitempty"`
+
+	// Options that establish this server's identity. Identity refers to
+	// credentials which can be used to uniquely identify and authenticate
+	// this server instance. This is required if remote administration is
+	// enabled (but does not require remote administration to be enabled).
+	// Default: no identity management.
+	Identity *IdentityConfig `json:"identity,omitempty"`
+
+	// Options pertaining to remote administration. By default, remote
+	// administration is disabled. If enabled, identity management must
+	// also be configured, as that is how the endpoint is secured.
+	// See the neighboring "identity" object.
+	//
+	// EXPERIMENTAL: This feature is subject to change.
+	Remote *RemoteAdmin `json:"remote,omitempty"`
 }
 
-// ConfigSettings configures the, uh, configuration... and
-// management thereof.
+// ConfigSettings configures the management of configuration.
 type ConfigSettings struct {
 	// Whether to keep a copy of the active config on disk. Default is true.
+	// Note that "pulled" dynamic configs (using the neighboring "load" module)
+	// are not persisted; only configs that are pushed to Caddy get persisted.
 	Persist *bool `json:"persist,omitempty"`
+
+	// Loads a configuration to use. This is helpful if your configs are
+	// managed elsewhere, and you want Caddy to pull its config dynamically
+	// when it starts. The pulled config completely replaces the current
+	// one, just like any other config load. It is an error if a pulled
+	// config is configured to pull another config.
+	//
+	// EXPERIMENTAL: Subject to change.
+	LoadRaw json.RawMessage `json:"load,omitempty" caddy:"namespace=caddy.config_loaders inline_key=module"`
 }
 
-// listenAddr extracts a singular listen address from ac.Listen,
-// returning the network and the address of the listener.
-func (admin AdminConfig) listenAddr() (NetworkAddress, error) {
-	input := admin.Listen
-	if input == "" {
-		input = DefaultAdminListen
-	}
-	listenAddr, err := ParseNetworkAddress(input)
-	if err != nil {
-		return NetworkAddress{}, fmt.Errorf("parsing admin listener address: %v", err)
-	}
-	if listenAddr.PortRangeSize() != 1 {
-		return NetworkAddress{}, fmt.Errorf("admin endpoint must have exactly one address; cannot listen on %v", listenAddr)
-	}
-	return listenAddr, nil
+// IdentityConfig configures management of this server's identity. An identity
+// consists of credentials that uniquely verify this instance; for example,
+// TLS certificates (public + private key pairs).
+type IdentityConfig struct {
+	// List of names or IP addresses which refer to this server.
+	// Certificates will be obtained for these identifiers so
+	// secure TLS connections can be made using them.
+	Identifiers []string `json:"identifiers,omitempty"`
+
+	// Issuers that can provide this admin endpoint its identity
+	// certificate(s). Default: ACME issuers configured for
+	// ZeroSSL and Let's Encrypt. Be sure to change this if you
+	// require credentials for private identifiers.
+	IssuersRaw []json.RawMessage `json:"issuers,omitempty" caddy:"namespace=tls.issuance inline_key=module"`
+
+	issuers []certmagic.Issuer
+}
+
+// RemoteAdmin enables and configures remote administration. If enabled,
+// a secure listener enforcing mutual TLS authentication will be started
+// on a different port from the standard plaintext admin server.
+//
+// This endpoint is secured using identity management, which must be
+// configured separately (because identity management does not depend
+// on remote administration). See the admin/identity config struct.
+//
+// EXPERIMENTAL: Subject to change.
+type RemoteAdmin struct {
+	// The address on which to start the secure listener.
+	// Default: :2021
+	Listen string `json:"listen,omitempty"`
+
+	// List of access controls for this secure admin endpoint.
+	// This configures TLS mutual authentication (i.e. authorized
+	// client certificates), but also application-layer permissions
+	// like which paths and methods each identity is authorized for.
+	AccessControl []*AdminAccess `json:"access_control,omitempty"`
+}
+
+// AdminAccess specifies what permissions an identity or group
+// of identities are granted.
+type AdminAccess struct {
+	// Base64-encoded DER certificates containing public keys to accept.
+	// (The contents of PEM certificate blocks are base64-encoded DER.)
+	// Any of these public keys can appear in any part of a verified chain.
+	PublicKeys []string `json:"public_keys,omitempty"`
+
+	// Limits what the associated identities are allowed to do.
+	// If unspecified, all permissions are granted.
+	Permissions []AdminPermissions `json:"permissions,omitempty"`
+
+	publicKeys []crypto.PublicKey
+}
+
+// AdminPermissions specifies what kinds of requests are allowed
+// to be made to the admin endpoint.
+type AdminPermissions struct {
+	// The API paths allowed. Paths are simple prefix matches.
+	// Any subpath of the specified paths will be allowed.
+	Paths []string `json:"paths,omitempty"`
+
+	// The HTTP methods allowed for the given paths.
+	Methods []string `json:"methods,omitempty"`
 }
 
 // newAdminHandler reads admin's config and returns an http.Handler suitable
 // for use in an admin endpoint server, which will be listening on listenAddr.
-func (admin AdminConfig) newAdminHandler(addr NetworkAddress) adminHandler {
-	muxWrap := adminHandler{
-		enforceOrigin:  admin.EnforceOrigin,
-		enforceHost:    !addr.isWildcardInterface(),
-		allowedOrigins: admin.allowedOrigins(addr),
-		mux:            http.NewServeMux(),
+func (admin AdminConfig) newAdminHandler(addr NetworkAddress, remote bool) adminHandler {
+	muxWrap := adminHandler{mux: http.NewServeMux()}
+
+	// secure the local or remote endpoint respectively
+	if remote {
+		muxWrap.remoteControl = admin.Remote
+	} else {
+		muxWrap.enforceHost = !addr.isWildcardInterface()
+		muxWrap.allowedOrigins = admin.allowedOrigins(addr)
 	}
 
 	addRouteWithMetrics := func(pattern string, handlerLabel string, h http.Handler) {
@@ -197,18 +277,18 @@ func (admin AdminConfig) allowedOrigins(addr NetworkAddress) []string {
 	return allowed
 }
 
-// replaceAdmin replaces the running admin server according
-// to the relevant configuration in cfg. If no configuration
-// for the admin endpoint exists in cfg, a default one is
-// used, so that there is always an admin server (unless it
-// is explicitly configured to be disabled).
-func replaceAdmin(cfg *Config) error {
+// replaceLocalAdminServer replaces the running local admin server
+// according to the relevant configuration in cfg. If no configuration
+// for the admin endpoint exists in cfg, a default one is used, so
+// that there is always an admin server (unless it is explicitly
+// configured to be disabled).
+func replaceLocalAdminServer(cfg *Config) error {
 	// always be sure to close down the old admin endpoint
 	// as gracefully as possible, even if the new one is
 	// disabled -- careful to use reference to the current
 	// (old) admin endpoint since it will be different
 	// when the function returns
-	oldAdminServer := adminServer
+	oldAdminServer := localAdminServer
 	defer func() {
 		// do the shutdown asynchronously so that any
 		// current API request gets a response; this
@@ -236,19 +316,20 @@ func replaceAdmin(cfg *Config) error {
 	}
 
 	// extract a singular listener address
-	addr, err := adminConfig.listenAddr()
+	addr, err := parseAdminListenAddr(adminConfig.Listen, DefaultAdminListen)
 	if err != nil {
 		return err
 	}
 
-	handler := adminConfig.newAdminHandler(addr)
+	handler := adminConfig.newAdminHandler(addr, false)
 
 	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
 	if err != nil {
 		return err
 	}
 
-	adminServer = &http.Server{
+	localAdminServer = &http.Server{
+		Addr:              addr.String(), // for logging purposes only
 		Handler:           handler,
 		ReadTimeout:       10 * time.Second,
 		ReadHeaderTimeout: 5 * time.Second,
@@ -258,7 +339,7 @@ func replaceAdmin(cfg *Config) error {
 
 	adminLogger := Log().Named("admin")
 	go func() {
-		if err := adminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+		if err := localAdminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
 			adminLogger.Error("admin server shutdown for unknown reason", zap.Error(err))
 		}
 	}()
@@ -276,6 +357,252 @@ func replaceAdmin(cfg *Config) error {
 	return nil
 }
 
+// manageIdentity sets up automated identity management for this server.
+func manageIdentity(ctx Context, cfg *Config) error {
+	if cfg == nil || cfg.Admin == nil || cfg.Admin.Identity == nil {
+		return nil
+	}
+
+	oldIdentityCertCache := identityCertCache
+	if oldIdentityCertCache != nil {
+		defer oldIdentityCertCache.Stop()
+	}
+
+	// set default issuers; this is pretty hacky because we can't
+	// import the caddytls package -- but it works
+	if cfg.Admin.Identity.IssuersRaw == nil {
+		cfg.Admin.Identity.IssuersRaw = []json.RawMessage{
+			json.RawMessage(`{"module": "zerossl"}`),
+			json.RawMessage(`{"module": "acme"}`),
+		}
+	}
+
+	// load and provision issuer modules
+	if cfg.Admin.Identity.IssuersRaw != nil {
+		val, err := ctx.LoadModule(cfg.Admin.Identity, "IssuersRaw")
+		if err != nil {
+			return fmt.Errorf("loading identity issuer modules: %s", err)
+		}
+		for _, issVal := range val.([]interface{}) {
+			cfg.Admin.Identity.issuers = append(cfg.Admin.Identity.issuers, issVal.(certmagic.Issuer))
+		}
+	}
+
+	logger := Log().Named("admin.identity")
+	cmCfg := cfg.Admin.Identity.certmagicConfig(logger)
+
+	// issuers have circular dependencies with the configs because,
+	// as explained in the caddytls package, they need access to the
+	// correct storage and cache to solve ACME challenges
+	for _, issuer := range cfg.Admin.Identity.issuers {
+		// avoid import cycle with caddytls package, so manually duplicate the interface here, yuck
+		if annoying, ok := issuer.(interface{ SetConfig(cfg *certmagic.Config) }); ok {
+			annoying.SetConfig(cmCfg)
+		}
+	}
+
+	// obtain and renew server identity certificate(s)
+	return cmCfg.ManageAsync(ctx, cfg.Admin.Identity.Identifiers)
+}
+
+// replaceRemoteAdminServer replaces the running remote admin server
+// according to the relevant configuration in cfg. It stops any previous
+// remote admin server and only starts a new one if configured.
+func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
+	if cfg == nil {
+		return nil
+	}
+
+	remoteLogger := Log().Named("admin.remote")
+
+	oldAdminServer := remoteAdminServer
+	defer func() {
+		if oldAdminServer != nil {
+			go func(oldAdminServer *http.Server) {
+				err := stopAdminServer(oldAdminServer)
+				if err != nil {
+					Log().Named("admin").Error("stopping current secure admin endpoint", zap.Error(err))
+				}
+			}(oldAdminServer)
+		}
+	}()
+
+	if cfg.Admin == nil || cfg.Admin.Remote == nil {
+		return nil
+	}
+
+	addr, err := parseAdminListenAddr(cfg.Admin.Remote.Listen, DefaultRemoteAdminListen)
+	if err != nil {
+		return err
+	}
+
+	// make the HTTP handler but disable Host/Origin enforcement
+	// because we are using TLS authentication instead
+	handler := cfg.Admin.newAdminHandler(addr, true)
+
+	// create client certificate pool for TLS mutual auth, and extract public keys
+	// so that we can enforce access controls at the application layer
+	clientCertPool := x509.NewCertPool()
+	for i, accessControl := range cfg.Admin.Remote.AccessControl {
+		for j, certBase64 := range accessControl.PublicKeys {
+			cert, err := decodeBase64DERCert(certBase64)
+			if err != nil {
+				return fmt.Errorf("access control %d public key %d: parsing base64 certificate DER: %v", i, j, err)
+			}
+			accessControl.publicKeys = append(accessControl.publicKeys, cert.PublicKey)
+			clientCertPool.AddCert(cert)
+		}
+	}
+
+	// create TLS config that will enforce mutual authentication
+	cmCfg := cfg.Admin.Identity.certmagicConfig(remoteLogger)
+	tlsConfig := cmCfg.TLSConfig()
+	tlsConfig.NextProtos = nil // this server does not solve ACME challenges
+	tlsConfig.ClientAuth = tls.RequireAndVerifyClientCert
+	tlsConfig.ClientCAs = clientCertPool
+
+	// convert logger to stdlib so it can be used by HTTP server
+	serverLogger, err := zap.NewStdLogAt(remoteLogger, zap.DebugLevel)
+	if err != nil {
+		return err
+	}
+
+	// create secure HTTP server
+	remoteAdminServer = &http.Server{
+		Addr:              addr.String(), // for logging purposes only
+		Handler:           handler,
+		TLSConfig:         tlsConfig,
+		ReadTimeout:       10 * time.Second,
+		ReadHeaderTimeout: 5 * time.Second,
+		IdleTimeout:       60 * time.Second,
+		MaxHeaderBytes:    1024 * 64,
+		ErrorLog:          serverLogger,
+	}
+
+	// start listener
+	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
+	if err != nil {
+		return err
+	}
+	ln = tls.NewListener(ln, tlsConfig)
+
+	go func() {
+		if err := remoteAdminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+			remoteLogger.Error("admin remote server shutdown for unknown reason", zap.Error(err))
+		}
+	}()
+
+	remoteLogger.Info("secure admin remote control endpoint started",
+		zap.String("address", addr.String()))
+
+	return nil
+}
+
+func (ident *IdentityConfig) certmagicConfig(logger *zap.Logger) *certmagic.Config {
+	if ident == nil {
+		// user might not have configured identity; that's OK, we can still make a
+		// certmagic config, although it'll be mostly useless for remote management
+		ident = new(IdentityConfig)
+	}
+	cmCfg := &certmagic.Config{
+		Storage: DefaultStorage, // do not act as part of a cluster (this is for the server's local identity)
+		Logger:  logger,
+		Issuers: ident.issuers,
+	}
+	if identityCertCache == nil {
+		identityCertCache = certmagic.NewCache(certmagic.CacheOptions{
+			GetConfigForCert: func(certmagic.Certificate) (*certmagic.Config, error) {
+				return cmCfg, nil
+			},
+		})
+	}
+	return certmagic.New(identityCertCache, *cmCfg)
+}
+
+// IdentityCredentials returns this instance's configured, managed identity credentials
+// that can be used in TLS client authentication.
+func (ctx Context) IdentityCredentials(logger *zap.Logger) ([]tls.Certificate, error) {
+	if ctx.cfg == nil || ctx.cfg.Admin == nil || ctx.cfg.Admin.Identity == nil {
+		return nil, fmt.Errorf("no server identity configured")
+	}
+	ident := ctx.cfg.Admin.Identity
+	if len(ident.Identifiers) == 0 {
+		return nil, fmt.Errorf("no identifiers configured")
+	}
+	if logger == nil {
+		logger = Log()
+	}
+	magic := ident.certmagicConfig(logger)
+	return magic.ClientCredentials(ctx, ident.Identifiers)
+}
+
+// enforceAccessControls enforces application-layer access controls for r based on remote.
+// It expects that the TLS server has already established at least one verified chain of
+// trust, and then looks for a matching, authorized public key that is allowed to access
+// the defined path(s) using the defined method(s).
+func (remote RemoteAdmin) enforceAccessControls(r *http.Request) error {
+	for _, chain := range r.TLS.VerifiedChains {
+		for _, peerCert := range chain {
+			for _, adminAccess := range remote.AccessControl {
+				for _, allowedKey := range adminAccess.publicKeys {
+					// see if we found a matching public key; the TLS server already verified the chain
+					// so we know the client possesses the associated private key; this handy interface
+					// doesn't appear to be defined anywhere in the std lib, but was implemented here:
+					// https://github.com/golang/go/commit/b5f2c0f50297fa5cd14af668ddd7fd923626cf8c
+					comparer, ok := peerCert.PublicKey.(interface{ Equal(crypto.PublicKey) bool })
+					if !ok || !comparer.Equal(allowedKey) {
+						continue
+					}
+
+					// key recognized; make sure its HTTP request is permitted
+					for _, accessPerm := range adminAccess.Permissions {
+						// verify method
+						methodFound := accessPerm.Methods == nil
+						for _, method := range accessPerm.Methods {
+							if method == r.Method {
+								methodFound = true
+								break
+							}
+						}
+						if !methodFound {
+							return APIError{
+								HTTPStatus: http.StatusForbidden,
+								Message:    "not authorized to use this method",
+							}
+						}
+
+						// verify path
+						pathFound := accessPerm.Paths == nil
+						for _, allowedPath := range accessPerm.Paths {
+							if strings.HasPrefix(r.URL.Path, allowedPath) {
+								pathFound = true
+								break
+							}
+						}
+						if !pathFound {
+							return APIError{
+								HTTPStatus: http.StatusForbidden,
+								Message:    "not authorized to access this path",
+							}
+						}
+					}
+
+					// public key authorized, method and path allowed
+					return nil
+				}
+			}
+		}
+	}
+
+	// in theory, this should never happen; with an unverified chain, the TLS server
+	// should not accept the connection in the first place, and the acceptable cert
+	// pool is configured using the same list of public keys we verify against
+	return APIError{
+		HTTPStatus: http.StatusUnauthorized,
+		Message:    "client identity not authorized",
+	}
+}
+
 func stopAdminServer(srv *http.Server) error {
 	if srv == nil {
 		return fmt.Errorf("no admin server")
@@ -286,7 +613,7 @@ func stopAdminServer(srv *http.Server) error {
 	if err != nil {
 		return fmt.Errorf("shutting down admin server: %v", err)
 	}
-	Log().Named("admin").Info("stopped previous server")
+	Log().Named("admin").Info("stopped previous server", zap.String("address", srv.Addr))
 	return nil
 }
 
@@ -302,10 +629,15 @@ type AdminRoute struct {
 }
 
 type adminHandler struct {
+	mux *http.ServeMux
+
+	// security for local/plaintext) endpoint, on by default
 	enforceOrigin  bool
 	enforceHost    bool
 	allowedOrigins []string
-	mux            *http.ServeMux
+
+	// security for remote/encrypted endpoint
+	remoteControl *RemoteAdmin
 }
 
 // ServeHTTP is the external entry point for API requests.
@@ -318,6 +650,12 @@ func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		zap.String("remote_addr", r.RemoteAddr),
 		zap.Reflect("headers", r.Header),
 	)
+	if r.TLS != nil {
+		log = log.With(
+			zap.Bool("secure", true),
+			zap.Int("verified_chains", len(r.TLS.VerifiedChains)),
+		)
+	}
 	if r.RequestURI == "/metrics" {
 		log.Debug("received request")
 	} else {
@@ -330,6 +668,14 @@ func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 // be called more than once per request, for example if a request
 // is rewritten (i.e. internal redirect).
 func (h adminHandler) serveHTTP(w http.ResponseWriter, r *http.Request) {
+	if h.remoteControl != nil {
+		// enforce access controls on secure endpoint
+		if err := h.remoteControl.enforceAccessControls(r); err != nil {
+			h.handleError(w, r, err)
+			return
+		}
+	}
+
 	if strings.Contains(r.Header.Get("Upgrade"), "websocket") {
 		// I've never been able demonstrate a vulnerability myself, but apparently
 		// WebSocket connections originating from browsers aren't subject to CORS
@@ -363,8 +709,6 @@ func (h adminHandler) serveHTTP(w http.ResponseWriter, r *http.Request) {
 		w.Header().Set("Access-Control-Allow-Origin", origin)
 	}
 
-	// TODO: authentication & authorization, if configured
-
 	h.mux.ServeHTTP(w, r)
 }
 
@@ -372,20 +716,16 @@ func (h adminHandler) handleError(w http.ResponseWriter, r *http.Request, err er
 	if err == nil {
 		return
 	}
-	if err == ErrInternalRedir {
-		h.serveHTTP(w, r)
-		return
-	}
 
 	apiErr, ok := err.(APIError)
 	if !ok {
 		apiErr = APIError{
-			Code: http.StatusInternalServerError,
-			Err:  err,
+			HTTPStatus: http.StatusInternalServerError,
+			Err:        err,
 		}
 	}
-	if apiErr.Code == 0 {
-		apiErr.Code = http.StatusInternalServerError
+	if apiErr.HTTPStatus == 0 {
+		apiErr.HTTPStatus = http.StatusInternalServerError
 	}
 	if apiErr.Message == "" && apiErr.Err != nil {
 		apiErr.Message = apiErr.Err.Error()
@@ -393,11 +733,11 @@ func (h adminHandler) handleError(w http.ResponseWriter, r *http.Request, err er
 
 	Log().Named("admin.api").Error("request error",
 		zap.Error(err),
-		zap.Int("status_code", apiErr.Code),
+		zap.Int("status_code", apiErr.HTTPStatus),
 	)
 
 	w.Header().Set("Content-Type", "application/json")
-	w.WriteHeader(apiErr.Code)
+	w.WriteHeader(apiErr.HTTPStatus)
 	encErr := json.NewEncoder(w).Encode(apiErr)
 	if encErr != nil {
 		Log().Named("admin.api").Error("failed to encode error response", zap.Error(encErr))
@@ -418,8 +758,8 @@ func (h adminHandler) checkHost(r *http.Request) error {
 	}
 	if !allowed {
 		return APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("host not allowed: %s", r.Host),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("host not allowed: %s", r.Host),
 		}
 	}
 	return nil
@@ -433,14 +773,14 @@ func (h adminHandler) checkOrigin(r *http.Request) (string, error) {
 	origin := h.getOriginHost(r)
 	if origin == "" {
 		return origin, APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("missing required Origin header"),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("missing required Origin header"),
 		}
 	}
 	if !h.originAllowed(origin) {
 		return origin, APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("client is not allowed to access from origin %s", origin),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("client is not allowed to access from origin %s", origin),
 		}
 	}
 	return origin, nil
@@ -480,7 +820,7 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 
 		err := readConfig(r.URL.Path, w)
 		if err != nil {
-			return APIError{Code: http.StatusBadRequest, Err: err}
+			return APIError{HTTPStatus: http.StatusBadRequest, Err: err}
 		}
 
 		return nil
@@ -495,8 +835,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 		if r.Method != http.MethodDelete {
 			if ct := r.Header.Get("Content-Type"); !strings.Contains(ct, "/json") {
 				return APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("unacceptable content-type: %v; 'application/json' required", ct),
+					HTTPStatus: http.StatusBadRequest,
+					Err:        fmt.Errorf("unacceptable content-type: %v; 'application/json' required", ct),
 				}
 			}
 
@@ -507,8 +847,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 			_, err := io.Copy(buf, r.Body)
 			if err != nil {
 				return APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("reading request body: %v", err),
+					HTTPStatus: http.StatusBadRequest,
+					Err:        fmt.Errorf("reading request body: %v", err),
 				}
 			}
 			body = buf.Bytes()
@@ -523,8 +863,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 
 	default:
 		return APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method %s not allowed", r.Method),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method %s not allowed", r.Method),
 		}
 	}
 
@@ -555,46 +895,17 @@ func handleConfigID(w http.ResponseWriter, r *http.Request) error {
 	parts = append([]string{expanded}, parts[3:]...)
 	r.URL.Path = path.Join(parts...)
 
-	return ErrInternalRedir
-}
-
-func handleStop(w http.ResponseWriter, r *http.Request) error {
-	err := handleUnload(w, r)
-	if err != nil {
-		Log().Named("admin.api").Error("unload error", zap.Error(err))
-	}
-	if adminServer != nil {
-		// use goroutine so that we can finish responding to API request
-		go func() {
-			err := stopAdminServer(adminServer)
-			var exitCode int
-			if err != nil {
-				exitCode = ExitCodeFailedQuit
-				Log().Named("admin.api").Error("failed to stop admin server gracefully", zap.Error(err))
-			}
-			Log().Named("admin.api").Info("stopping now, bye!! 👋")
-			os.Exit(exitCode)
-		}()
-	}
 	return nil
 }
 
-// handleUnload stops the current configuration that is running.
-// Note that doing this can also be accomplished with DELETE /config/
-// but we leave this function because handleStop uses it.
-func handleUnload(w http.ResponseWriter, r *http.Request) error {
+func handleStop(w http.ResponseWriter, r *http.Request) error {
 	if r.Method != http.MethodPost {
 		return APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method not allowed"),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
 		}
 	}
-	Log().Named("admin.api").Info("unloading")
-	if err := stopAndCleanup(); err != nil {
-		Log().Named("admin.api").Error("error unloading", zap.Error(err))
-	} else {
-		Log().Named("admin.api").Info("unloading completed")
-	}
+	exitProcess(Log().Named("admin.api"))
 	return nil
 }
 
@@ -806,9 +1117,9 @@ func (f AdminHandlerFunc) ServeHTTP(w http.ResponseWriter, r *http.Request) erro
 // and client responses. If Message is unset, then
 // Err.Error() will be serialized in its place.
 type APIError struct {
-	Code    int    `json:"-"`
-	Err     error  `json:"-"`
-	Message string `json:"error"`
+	HTTPStatus int    `json:"-"`
+	Err        error  `json:"-"`
+	Message    string `json:"error"`
 }
 
 func (e APIError) Error() string {
@@ -818,20 +1129,44 @@ func (e APIError) Error() string {
 	return e.Message
 }
 
+// parseAdminListenAddr extracts a singular listen address from either addr
+// or defaultAddr, returning the network and the address of the listener.
+func parseAdminListenAddr(addr string, defaultAddr string) (NetworkAddress, error) {
+	input := addr
+	if input == "" {
+		input = defaultAddr
+	}
+	listenAddr, err := ParseNetworkAddress(input)
+	if err != nil {
+		return NetworkAddress{}, fmt.Errorf("parsing listener address: %v", err)
+	}
+	if listenAddr.PortRangeSize() != 1 {
+		return NetworkAddress{}, fmt.Errorf("must be exactly one listener address; cannot listen on: %s", listenAddr)
+	}
+	return listenAddr, nil
+}
+
+// decodeBase64DERCert base64-decodes, then DER-decodes, certStr.
+func decodeBase64DERCert(certStr string) (*x509.Certificate, error) {
+	derBytes, err := base64.StdEncoding.DecodeString(certStr)
+	if err != nil {
+		return nil, err
+	}
+	return x509.ParseCertificate(derBytes)
+}
+
 var (
-	// DefaultAdminListen is the address for the admin
+	// DefaultAdminListen is the address for the local admin
 	// listener, if none is specified at startup.
 	DefaultAdminListen = "localhost:2019"
 
-	// ErrInternalRedir indicates an internal redirect
-	// and is useful when admin API handlers rewrite
-	// the request; in that case, authentication and
-	// authorization needs to happen again for the
-	// rewritten request.
-	ErrInternalRedir = fmt.Errorf("internal redirect; re-authorization required")
+	// DefaultRemoteAdminListen is the address for the remote
+	// (TLS-authenticated) admin listener, if enabled and not
+	// specified otherwise.
+	DefaultRemoteAdminListen = ":2021"
 
 	// DefaultAdminConfig is the default configuration
-	// for the administration endpoint.
+	// for the local administration endpoint.
 	DefaultAdminConfig = &AdminConfig{
 		Listen: DefaultAdminListen,
 	}
@@ -869,4 +1204,8 @@ var bufPool = sync.Pool{
 	},
 }
 
-var adminServer *http.Server
+// keep a reference to admin endpoint singletons while they're active
+var (
+	localAdminServer, remoteAdminServer *http.Server
+	identityCertCache                   *certmagic.Cache
+)
diff --git a/caddy.go b/caddy.go
index 000cd6f..70135ff 100644
--- a/caddy.go
+++ b/caddy.go
@@ -130,8 +130,8 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	newCfg, err := json.Marshal(rawCfg[rawConfigKey])
 	if err != nil {
 		return APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("encoding new config: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("encoding new config: %v", err),
 		}
 	}
 
@@ -146,14 +146,14 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	err = indexConfigObjects(rawCfg[rawConfigKey], "/"+rawConfigKey, idx)
 	if err != nil {
 		return APIError{
-			Code: http.StatusInternalServerError,
-			Err:  fmt.Errorf("indexing config: %v", err),
+			HTTPStatus: http.StatusInternalServerError,
+			Err:        fmt.Errorf("indexing config: %v", err),
 		}
 	}
 
 	// load this new config; if it fails, we need to revert to
 	// our old representation of caddy's actual config
-	err = unsyncedDecodeAndRun(newCfg)
+	err = unsyncedDecodeAndRun(newCfg, true)
 	if err != nil {
 		if len(rawCfgJSON) > 0 {
 			// restore old config state to keep it consistent
@@ -233,8 +233,10 @@ func indexConfigObjects(ptr interface{}, configPath string, index map[string]str
 // it as the new config, replacing any other current config.
 // It does NOT update the raw config state, as this is a
 // lower-level function; most callers will want to use Load
-// instead. A write lock on currentCfgMu is required!
-func unsyncedDecodeAndRun(cfgJSON []byte) error {
+// instead. A write lock on currentCfgMu is required! If
+// allowPersist is false, it will not be persisted to disk,
+// even if it is configured to.
+func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 	// remove any @id fields from the JSON, which would cause
 	// loading to break since the field wouldn't be recognized
 	strippedCfgJSON := RemoveMetaFields(cfgJSON)
@@ -245,6 +247,19 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 		return err
 	}
 
+	// prevent recursive config loads; that is a user error, and
+	// although frequent config loads should be safe, we cannot
+	// guarantee that in the presence of third party plugins, nor
+	// do we want this error to go unnoticed (we assume it was a
+	// pulled config if we're not allowed to persist it)
+	if !allowPersist &&
+		newCfg != nil &&
+		newCfg.Admin != nil &&
+		newCfg.Admin.Config != nil &&
+		newCfg.Admin.Config.LoadRaw != nil {
+		return fmt.Errorf("recursive config loading detected: pulled configs cannot pull other configs")
+	}
+
 	// run the new config and start all its apps
 	err = run(newCfg, true)
 	if err != nil {
@@ -259,7 +274,8 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 	unsyncedStop(oldCfg)
 
 	// autosave a non-nil config, if not disabled
-	if newCfg != nil &&
+	if allowPersist &&
+		newCfg != nil &&
 		(newCfg.Admin == nil ||
 			newCfg.Admin.Config == nil ||
 			newCfg.Admin.Config.Persist == nil ||
@@ -311,14 +327,14 @@ func run(newCfg *Config, start bool) error {
 
 	// start the admin endpoint (and stop any prior one)
 	if start {
-		err = replaceAdmin(newCfg)
+		err = replaceLocalAdminServer(newCfg)
 		if err != nil {
 			return fmt.Errorf("starting caddy administration endpoint: %v", err)
 		}
 	}
 
 	if newCfg == nil {
-		return nil
+		newCfg = new(Config)
 	}
 
 	// prepare the new config for use
@@ -400,7 +416,7 @@ func run(newCfg *Config, start bool) error {
 	}
 
 	// Start
-	return func() error {
+	err = func() error {
 		var started []string
 		for name, a := range newCfg.apps {
 			err := a.Start()
@@ -420,6 +436,64 @@ func run(newCfg *Config, start bool) error {
 		}
 		return nil
 	}()
+	if err != nil {
+		return err
+	}
+
+	// now that the user's config is running, finish setting up anything else,
+	// such as remote admin endpoint, config loader, etc.
+	return finishSettingUp(ctx, newCfg)
+}
+
+// finishSettingUp should be run after all apps have successfully started.
+func finishSettingUp(ctx Context, cfg *Config) error {
+	// establish this server's identity (only after apps are loaded
+	// so that cert management of this endpoint doesn't prevent user's
+	// servers from starting which likely also use HTTP/HTTPS ports;
+	// but before remote management which may depend on these creds)
+	err := manageIdentity(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// replace any remote admin endpoint
+	err = replaceRemoteAdminServer(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// if dynamic config is requested, set that up and run it
+	if cfg != nil && cfg.Admin != nil && cfg.Admin.Config != nil && cfg.Admin.Config.LoadRaw != nil {
+		val, err := ctx.LoadModule(cfg.Admin.Config, "LoadRaw")
+		if err != nil {
+			return fmt.Errorf("loading config loader module: %s", err)
+		}
+		loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+		if err != nil {
+			return fmt.Errorf("loading dynamic config from %T: %v", val, err)
+		}
+
+		// do this in a goroutine so current config can finish being loaded; otherwise deadlock
+		go func() {
+			Log().Info("applying dynamically-loaded config", zap.String("loader_module", val.(Module).CaddyModule().ID.Name()))
+			currentCfgMu.Lock()
+			err := unsyncedDecodeAndRun(loadedConfig, false)
+			currentCfgMu.Unlock()
+			if err == nil {
+				Log().Info("dynamically-loaded config applied successfully")
+			} else {
+				Log().Error("running dynamically-loaded config failed", zap.Error(err))
+			}
+		}()
+	}
+
+	return nil
+}
+
+// ConfigLoader is a type that can load a Caddy config. The
+// returned config must be valid Caddy JSON.
+type ConfigLoader interface {
+	LoadConfig(Context) ([]byte, error)
 }
 
 // Stop stops running the current configuration.
@@ -462,20 +536,6 @@ func unsyncedStop(cfg *Config) {
 	cfg.cancelFunc()
 }
 
-// stopAndCleanup calls stop and cleans up anything
-// else that is expedient. This should only be used
-// when stopping and not replacing with a new config.
-func stopAndCleanup() error {
-	if err := Stop(); err != nil {
-		return err
-	}
-	certmagic.CleanUpOwnLocks()
-	if pidfile != "" {
-		return os.Remove(pidfile)
-	}
-	return nil
-}
-
 // Validate loads, provisions, and validates
 // cfg, but does not start running it.
 func Validate(cfg *Config) error {
@@ -486,6 +546,72 @@ func Validate(cfg *Config) error {
 	return err
 }
 
+// exitProcess exits the process as gracefully as possible,
+// but it always exits, even if there are errors doing so.
+// It stops all apps, cleans up external locks, removes any
+// PID file, and shuts down admin endpoint(s) in a goroutine.
+// Errors are logged along the way, and an appropriate exit
+// code is emitted.
+func exitProcess(logger *zap.Logger) {
+	if logger == nil {
+		logger = Log()
+	}
+	logger.Warn("exiting; byeee!! 👋")
+
+	exitCode := ExitCodeSuccess
+
+	// stop all apps
+	if err := Stop(); err != nil {
+		logger.Error("failed to stop apps", zap.Error(err))
+		exitCode = ExitCodeFailedQuit
+	}
+
+	// clean up certmagic locks
+	certmagic.CleanUpOwnLocks(logger)
+
+	// remove pidfile
+	if pidfile != "" {
+		err := os.Remove(pidfile)
+		if err != nil {
+			logger.Error("cleaning up PID file:",
+				zap.String("pidfile", pidfile),
+				zap.Error(err))
+			exitCode = ExitCodeFailedQuit
+		}
+	}
+
+	// shut down admin endpoint(s) in goroutines so that
+	// if this function was called from an admin handler,
+	// it has a chance to return gracefully
+	// use goroutine so that we can finish responding to API request
+	go func() {
+		defer func() {
+			logger = logger.With(zap.Int("exit_code", exitCode))
+			if exitCode == ExitCodeSuccess {
+				logger.Info("shutdown complete")
+			} else {
+				logger.Error("unclean shutdown")
+			}
+			os.Exit(exitCode)
+		}()
+
+		if remoteAdminServer != nil {
+			err := stopAdminServer(remoteAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop remote admin server gracefully", zap.Error(err))
+			}
+		}
+		if localAdminServer != nil {
+			err := stopAdminServer(localAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop local admin server gracefully", zap.Error(err))
+			}
+		}
+	}()
+}
+
 // Duration can be an integer or a string. An integer is
 // interpreted as nanoseconds. If a string, it is a Go
 // time.Duration value such as `300ms`, `1.5h`, or `2h45m`;
diff --git a/caddyconfig/configadapters.go b/caddyconfig/configadapters.go
index 1665fa0..ccac5f8 100644
--- a/caddyconfig/configadapters.go
+++ b/caddyconfig/configadapters.go
@@ -35,6 +35,14 @@ type Warning struct {
 	Message   string `json:"message,omitempty"`
 }
 
+func (w Warning) String() string {
+	var directive string
+	if w.Directive != "" {
+		directive = fmt.Sprintf(" (%s)", w.Directive)
+	}
+	return fmt.Sprintf("%s:%d%s: %s", w.File, w.Line, directive, w.Message)
+}
+
 // JSON encodes val as JSON, returning it as a json.RawMessage. Any
 // marshaling errors (which are highly unlikely with correct code)
 // are converted to warnings. This is convenient when filling config
diff --git a/caddyconfig/httploader.go b/caddyconfig/httploader.go
new file mode 100644
index 0000000..aabd103
--- /dev/null
+++ b/caddyconfig/httploader.go
@@ -0,0 +1,151 @@
+package caddyconfig
+
+import (
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"time"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(HTTPLoader{})
+}
+
+// HTTPLoader can load Caddy configs over HTTP(S). It can adapt the config
+// based on the Content-Type header of the HTTP response.
+type HTTPLoader struct {
+	// The method for the request. Default: GET
+	Method string `json:"method,omitempty"`
+
+	// The URL of the request.
+	URL string `json:"url,omitempty"`
+
+	// HTTP headers to add to the request.
+	Headers http.Header `json:"header,omitempty"`
+
+	// Maximum time allowed for a complete connection and request.
+	Timeout caddy.Duration `json:"timeout,omitempty"`
+
+	TLS *struct {
+		// Present this instance's managed remote identity credentials to the server.
+		UseServerIdentity bool `json:"use_server_identity,omitempty"`
+
+		// PEM-encoded client certificate filename to present to the server.
+		ClientCertificateFile string `json:"client_certificate_file,omitempty"`
+
+		// PEM-encoded key to use with the client certificate.
+		ClientCertificateKeyFile string `json:"client_certificate_key_file,omitempty"`
+
+		// List of PEM-encoded CA certificate files to add to the same trust
+		// store as RootCAPool (or root_ca_pool in the JSON).
+		RootCAPEMFiles []string `json:"root_ca_pem_files,omitempty"`
+	} `json:"tls,omitempty"`
+}
+
+// CaddyModule returns the Caddy module information.
+func (HTTPLoader) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "caddy.config_loaders.http",
+		New: func() caddy.Module { return new(HTTPLoader) },
+	}
+}
+
+// LoadConfig loads a Caddy config.
+func (hl HTTPLoader) LoadConfig(ctx caddy.Context) ([]byte, error) {
+	client, err := hl.makeClient(ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	method := hl.Method
+	if method == "" {
+		method = http.MethodGet
+	}
+
+	req, err := http.NewRequest(method, hl.URL, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header = hl.Headers
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode >= 400 {
+		return nil, fmt.Errorf("server responded with HTTP %d", resp.StatusCode)
+	}
+
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	result, warnings, err := adaptByContentType(resp.Header.Get("Content-Type"), body)
+	if err != nil {
+		return nil, err
+	}
+	for _, warn := range warnings {
+		ctx.Logger(hl).Warn(warn.String())
+	}
+
+	return result, nil
+}
+
+func (hl HTTPLoader) makeClient(ctx caddy.Context) (*http.Client, error) {
+	client := &http.Client{
+		Timeout: time.Duration(hl.Timeout),
+	}
+
+	if hl.TLS != nil {
+		var tlsConfig *tls.Config
+
+		// client authentication
+		if hl.TLS.UseServerIdentity {
+			certs, err := ctx.IdentityCredentials(ctx.Logger(hl))
+			if err != nil {
+				return nil, fmt.Errorf("getting server identity credentials: %v", err)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = certs
+		} else if hl.TLS.ClientCertificateFile != "" && hl.TLS.ClientCertificateKeyFile != "" {
+			cert, err := tls.LoadX509KeyPair(hl.TLS.ClientCertificateFile, hl.TLS.ClientCertificateKeyFile)
+			if err != nil {
+				return nil, err
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = []tls.Certificate{cert}
+		}
+
+		// trusted server certs
+		if len(hl.TLS.RootCAPEMFiles) > 0 {
+			rootPool := x509.NewCertPool()
+			for _, pemFile := range hl.TLS.RootCAPEMFiles {
+				pemData, err := ioutil.ReadFile(pemFile)
+				if err != nil {
+					return nil, fmt.Errorf("failed reading ca cert: %v", err)
+				}
+				rootPool.AppendCertsFromPEM(pemData)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.RootCAs = rootPool
+		}
+
+		client.Transport = &http.Transport{TLSClientConfig: tlsConfig}
+	}
+
+	return client, nil
+}
+
+var _ caddy.ConfigLoader = (*HTTPLoader)(nil)
diff --git a/caddyconfig/load.go b/caddyconfig/load.go
index 4855b46..7a390d0 100644
--- a/caddyconfig/load.go
+++ b/caddyconfig/load.go
@@ -69,8 +69,8 @@ func (al adminLoad) Routes() []caddy.AdminRoute {
 func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	if r.Method != http.MethodPost {
 		return caddy.APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method not allowed"),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
 		}
 	}
 
@@ -81,8 +81,8 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	_, err := io.Copy(buf, r.Body)
 	if err != nil {
 		return caddy.APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("reading request body: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("reading request body: %v", err),
 		}
 	}
 	body := buf.Bytes()
@@ -90,45 +90,21 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	// if the config is formatted other than Caddy's native
 	// JSON, we need to adapt it before loading it
 	if ctHeader := r.Header.Get("Content-Type"); ctHeader != "" {
-		ct, _, err := mime.ParseMediaType(ctHeader)
+		result, warnings, err := adaptByContentType(ctHeader, body)
 		if err != nil {
 			return caddy.APIError{
-				Code: http.StatusBadRequest,
-				Err:  fmt.Errorf("invalid Content-Type: %v", err),
+				HTTPStatus: http.StatusBadRequest,
+				Err:        err,
 			}
 		}
-		if !strings.HasSuffix(ct, "/json") {
-			slashIdx := strings.Index(ct, "/")
-			if slashIdx < 0 {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("malformed Content-Type"),
-				}
-			}
-			adapterName := ct[slashIdx+1:]
-			cfgAdapter := GetAdapter(adapterName)
-			if cfgAdapter == nil {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("unrecognized config adapter '%s'", adapterName),
-				}
-			}
-			result, warnings, err := cfgAdapter.Adapt(body, nil)
+		if len(warnings) > 0 {
+			respBody, err := json.Marshal(warnings)
 			if err != nil {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("adapting config using %s adapter: %v", adapterName, err),
-				}
-			}
-			if len(warnings) > 0 {
-				respBody, err := json.Marshal(warnings)
-				if err != nil {
-					caddy.Log().Named("admin.api.load").Error(err.Error())
-				}
-				_, _ = w.Write(respBody)
+				caddy.Log().Named("admin.api.load").Error(err.Error())
 			}
-			body = result
+			_, _ = w.Write(respBody)
 		}
+		body = result
 	}
 
 	forceReload := r.Header.Get("Cache-Control") == "must-revalidate"
@@ -136,8 +112,8 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	err = caddy.Load(body, forceReload)
 	if err != nil {
 		return caddy.APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("loading config: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("loading config: %v", err),
 		}
 	}
 
@@ -146,6 +122,47 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	return nil
 }
 
+// adaptByContentType adapts body to Caddy JSON using the adapter specified by contenType.
+// If contentType is empty or ends with "/json", the input will be returned, as a no-op.
+func adaptByContentType(contentType string, body []byte) ([]byte, []Warning, error) {
+	// assume JSON as the default
+	if contentType == "" {
+		return body, nil, nil
+	}
+
+	ct, _, err := mime.ParseMediaType(contentType)
+	if err != nil {
+		return nil, nil, caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("invalid Content-Type: %v", err),
+		}
+	}
+
+	// if already JSON, no need to adapt
+	if strings.HasSuffix(ct, "/json") {
+		return body, nil, nil
+	}
+
+	// adapter name should be suffix of MIME type
+	slashIdx := strings.Index(ct, "/")
+	if slashIdx < 0 {
+		return nil, nil, fmt.Errorf("malformed Content-Type")
+	}
+
+	adapterName := ct[slashIdx+1:]
+	cfgAdapter := GetAdapter(adapterName)
+	if cfgAdapter == nil {
+		return nil, nil, fmt.Errorf("unrecognized config adapter '%s'", adapterName)
+	}
+
+	result, warnings, err := cfgAdapter.Adapt(body, nil)
+	if err != nil {
+		return nil, nil, fmt.Errorf("adapting config using %s adapter: %v", adapterName, err)
+	}
+
+	return result, warnings, nil
+}
+
 var bufPool = sync.Pool{
 	New: func() interface{} {
 		return new(bytes.Buffer)
diff --git a/go.mod b/go.mod
index 825f874..a4335d9 100644
--- a/go.mod
+++ b/go.mod
@@ -6,14 +6,14 @@ require (
 	github.com/Masterminds/sprig/v3 v3.1.0
 	github.com/alecthomas/chroma v0.8.2
 	github.com/aryann/difflib v0.0.0-20170710044230-e206f873d14a
-	github.com/caddyserver/certmagic v0.12.1-0.20210107224522-725b69d53d57
+	github.com/caddyserver/certmagic v0.12.1-0.20210126230115-267fdad76a0f
 	github.com/dustin/go-humanize v1.0.1-0.20200219035652-afde56e7acac
 	github.com/go-chi/chi v4.1.2+incompatible
 	github.com/google/cel-go v0.6.0
 	github.com/klauspost/compress v1.11.3
 	github.com/klauspost/cpuid/v2 v2.0.1
 	github.com/lucas-clemente/quic-go v0.19.3
-	github.com/mholt/acmez v0.1.2
+	github.com/mholt/acmez v0.1.3
 	github.com/naoina/go-stringutil v0.1.0 // indirect
 	github.com/naoina/toml v0.1.1
 	github.com/prometheus/client_golang v1.9.0
diff --git a/go.sum b/go.sum
index cc11013..7349790 100644
--- a/go.sum
+++ b/go.sum
@@ -99,8 +99,8 @@ github.com/bombsimon/wsl/v2 v2.0.0/go.mod h1:mf25kr/SqFEPhhcxW1+7pxzGlW+hIl/hYTK
 github.com/boombuler/barcode v1.0.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
 github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
 github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
-github.com/caddyserver/certmagic v0.12.1-0.20210107224522-725b69d53d57 h1:eslWGgoQlVAzOGMUfK3ncoHnONjCUVOPTGRD9JG3gAY=
-github.com/caddyserver/certmagic v0.12.1-0.20210107224522-725b69d53d57/go.mod h1:yHMCSjG2eOFdI/Jx0+CCzr2DLw+UQu42KbaOVBx7LwA=
+github.com/caddyserver/certmagic v0.12.1-0.20210126230115-267fdad76a0f h1:uJoft/gLxPvKq+ojfq3k7w8deji/xt/1RSWN7OAk6Ng=
+github.com/caddyserver/certmagic v0.12.1-0.20210126230115-267fdad76a0f/go.mod h1:CUPfwomVXGCyV77EQbR3v7H4tGJ4pX16HATeR55rqws=
 github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ=
 github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
 github.com/census-instrumentation/opencensus-proto v0.2.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
@@ -175,6 +175,7 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7
 github.com/fatih/color v1.6.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fatih/color v1.8.0/go.mod h1:3l45GVGkyrnYNl9HoIjnp2NnNWvh6hLAqD8yTfGjnw8=
+github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 h1:BHsljHzVlRcyQhjrss6TZTdY2VfCqZPbv5k3iBFa2ZQ=
 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
 github.com/francoispqt/gojay v1.2.13/go.mod h1:ehT5mTG4ua4581f1++1WLG0vPdaA9HaiDsoyrBGkyDY=
 github.com/franela/goblin v0.0.0-20200105215937-c9ffbefa60db/go.mod h1:7dvUGVsVBjqR7JHJk0brhHOZYGmfBYOrK0ZhYMEtBr4=
@@ -379,8 +380,6 @@ github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/u
 github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
-github.com/jsternberg/zap-logfmt v1.2.0 h1:1v+PK4/B48cy8cfQbxL4FmmNZrjnIMr2BsnyEmXqv2o=
-github.com/jsternberg/zap-logfmt v1.2.0/go.mod h1:kz+1CUmCutPWABnNkOu9hOHKdT2q3TDYCcsFy9hpqb0=
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/juju/ansiterm v0.0.0-20180109212912-720a0952cc2a h1:FaWFmfWdAUKbSCtOU2QjDaorUexogfaMgbipgYATUMU=
 github.com/juju/ansiterm v0.0.0-20180109212912-720a0952cc2a/go.mod h1:UJSiEoRfvx3hP73CvoARgeLjaIOjybY9vj8PUPPFGeU=
@@ -459,8 +458,8 @@ github.com/mattn/go-sqlite3 v1.10.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsO
 github.com/mattn/goveralls v0.0.2/go.mod h1:8d1ZMHsd7fW6IRPKQh46F2WRpyib5/X4FOpevwGNQEw=
 github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
-github.com/mholt/acmez v0.1.2 h1:26ncYNBt59D+59cMUHuGa/Fzjmu6FFrBm6kk/8hdXt0=
-github.com/mholt/acmez v0.1.2/go.mod h1:8qnn8QA/Ewx8E3ZSsmscqsIjhhpxuy9vqdgbX2ceceM=
+github.com/mholt/acmez v0.1.3 h1:J7MmNIk4Qf9b8mAGqAh4XkNeowv3f1zW816yf4zt7Qk=
+github.com/mholt/acmez v0.1.3/go.mod h1:8qnn8QA/Ewx8E3ZSsmscqsIjhhpxuy9vqdgbX2ceceM=
 github.com/microcosm-cc/bluemonday v1.0.1/go.mod h1:hsXNsILzKxV+sX77C5b8FSuKF00vh2OMYv+xgHpAMF4=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
 github.com/miekg/dns v1.1.30 h1:Qww6FseFn8PRfw07jueqIXqodm0JKiiKuK0DeXSqfyo=
@@ -785,7 +784,6 @@ go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A=
 go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
 go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4=
 go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
-go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
 go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
diff --git a/modules/caddytls/acmeissuer.go b/modules/caddytls/acmeissuer.go
index 64725c9..bbcd5d7 100644
--- a/modules/caddytls/acmeissuer.go
+++ b/modules/caddytls/acmeissuer.go
@@ -59,6 +59,13 @@ type ACMEIssuer struct {
 	// other than ACME transactions.
 	Email string `json:"email,omitempty"`
 
+	// If you have an existing account with the ACME server, put
+	// the private key here in PEM format. The ACME client will
+	// look up your account information with this key first before
+	// trying to create a new one. You can use placeholders here,
+	// for example if you have it in an environment variable.
+	AccountKey string `json:"account_key,omitempty"`
+
 	// If using an ACME CA that requires an external account
 	// binding, specify the CA-provided credentials here.
 	ExternalAccount *acme.EAB `json:"external_account,omitempty"`
@@ -98,15 +105,26 @@ func (ACMEIssuer) CaddyModule() caddy.ModuleInfo {
 func (iss *ACMEIssuer) Provision(ctx caddy.Context) error {
 	iss.logger = ctx.Logger(iss)
 
+	repl := caddy.NewReplacer()
+
 	// expand email address, if non-empty
 	if iss.Email != "" {
-		email, err := caddy.NewReplacer().ReplaceOrErr(iss.Email, true, true)
+		email, err := repl.ReplaceOrErr(iss.Email, true, true)
 		if err != nil {
 			return fmt.Errorf("expanding email address '%s': %v", iss.Email, err)
 		}
 		iss.Email = email
 	}
 
+	// expand account key, if non-empty
+	if iss.AccountKey != "" {
+		accountKey, err := repl.ReplaceOrErr(iss.AccountKey, true, true)
+		if err != nil {
+			return fmt.Errorf("expanding account key PEM '%s': %v", iss.AccountKey, err)
+		}
+		iss.AccountKey = accountKey
+	}
+
 	// DNS providers
 	if iss.Challenges != nil && iss.Challenges.DNS != nil && iss.Challenges.DNS.ProviderRaw != nil {
 		val, err := ctx.LoadModule(iss.Challenges.DNS, "ProviderRaw")
@@ -161,6 +179,7 @@ func (iss *ACMEIssuer) makeIssuerTemplate() (certmagic.ACMEManager, error) {
 		CA:                iss.CA,
 		TestCA:            iss.TestCA,
 		Email:             iss.Email,
+		AccountKeyPEM:     iss.AccountKey,
 		CertObtainTimeout: time.Duration(iss.ACMETimeout),
 		TrustedRoots:      iss.rootPool,
 		ExternalAccount:   iss.ExternalAccount,
diff --git a/modules/caddytls/tls.go b/modules/caddytls/tls.go
index fd3473e..489d87f 100644
--- a/modules/caddytls/tls.go
+++ b/modules/caddytls/tls.go
@@ -306,9 +306,11 @@ func (t *TLS) Manage(names []string) error {
 // requires that the automation policy for r.Host has an issuer of type
 // *certmagic.ACMEManager, or one that is ACME-enabled (GetACMEIssuer()).
 func (t *TLS) HandleHTTPChallenge(w http.ResponseWriter, r *http.Request) bool {
+	// no-op if it's not an ACME challenge request
 	if !certmagic.LooksLikeHTTPChallenge(r) {
 		return false
 	}
+
 	// try all the issuers until we find the one that initiated the challenge
 	ap := t.getAutomationPolicyForName(r.Host)
 	type acmeCapable interface{ GetACMEIssuer() *ACMEIssuer }
@@ -320,6 +322,16 @@ func (t *TLS) HandleHTTPChallenge(w http.ResponseWriter, r *http.Request) bool {
 			}
 		}
 	}
+
+	// it's possible another server in this process initiated the challenge;
+	// users have requested that Caddy only handle HTTP challenges it initiated,
+	// so that users can proxy the others through to their backends; but we
+	// might not have an automation policy for all identifiers that are trying
+	// to get certificates (e.g. the admin endpoint), so we do this manual check
+	if challenge, ok := certmagic.GetACMEChallenge(r.Host); ok {
+		return certmagic.SolveHTTPChallenge(t.logger, w, r, challenge.Challenge)
+	}
+
 	return false
 }
 
diff --git a/sigtrap.go b/sigtrap.go
index 4ad94c1..0fce6d0 100644
--- a/sigtrap.go
+++ b/sigtrap.go
@@ -47,37 +47,15 @@ func trapSignalsCrossPlatform() {
 			}
 
 			Log().Info("shutting down", zap.String("signal", "SIGINT"))
-			go gracefulStop("SIGINT")
+			go exitProcessFromSignal("SIGINT")
 		}
 	}()
 }
 
-// gracefulStop exits the process as gracefully as possible.
-// It always exits, even if there are errors shutting down.
-func gracefulStop(sigName string) {
-	exitCode := ExitCodeSuccess
-	defer func() {
-		Log().Info("shutdown done", zap.String("signal", sigName))
-		os.Exit(exitCode)
-	}()
-
-	err := stopAndCleanup()
-	if err != nil {
-		Log().Error("stopping config",
-			zap.String("signal", sigName),
-			zap.Error(err))
-		exitCode = ExitCodeFailedQuit
-	}
-
-	if adminServer != nil {
-		err = stopAdminServer(adminServer)
-		if err != nil {
-			Log().Error("stopping admin endpoint",
-				zap.String("signal", sigName),
-				zap.Error(err))
-			exitCode = ExitCodeFailedQuit
-		}
-	}
+// exitProcessFromSignal exits the process from a system signal.
+func exitProcessFromSignal(sigName string) {
+	logger := Log().With(zap.String("signal", sigName))
+	exitProcess(logger)
 }
 
 // Exit codes. Generally, you should NOT
diff --git a/sigtrap_posix.go b/sigtrap_posix.go
index a8e4cec..0e4dda3 100644
--- a/sigtrap_posix.go
+++ b/sigtrap_posix.go
@@ -35,12 +35,12 @@ func trapSignalsPosix() {
 			switch sig {
 			case syscall.SIGQUIT:
 				Log().Info("quitting process immediately", zap.String("signal", "SIGQUIT"))
-				certmagic.CleanUpOwnLocks() // try to clean up locks anyway, it's important
+				certmagic.CleanUpOwnLocks(Log()) // try to clean up locks anyway, it's important
 				os.Exit(ExitCodeForceQuit)
 
 			case syscall.SIGTERM:
 				Log().Info("shutting down apps then terminating", zap.String("signal", "SIGTERM"))
-				gracefulStop("SIGTERM")
+				exitProcessFromSignal("SIGTERM")
 
 			case syscall.SIGUSR1:
 				Log().Info("not implemented", zap.String("signal", "SIGUSR1"))