At the previous company I contracted to, we stored all of our secrets in Bitwarden (not in source control as some naughty teams might do!). We also used AWX to perform deployment and management of our large infrastructure. While AWX is a handy tool for centralised Ansible operations, it is just a Django application and we decided that it was not a suitable place to store secrets. We were also trying to avoid having secrets stored in multiple places, so it became apparent that we needed a way for AWX to retrieve secrets from Bitwarden.

It is possible to use a Bitwarden CLI client to pull secrets from Bitwarden at runtime, but that still requires that AWX store Bitwarden credentials. Since Bitwarden doesn’t support short-lived tokens, those credentials would be some user’s username and password - not ideal. Hashicorp Vault, on the other hand, has excellent support for short-lived tokens, and (like Bitwarden) is purpose-built for secret storage and management. One option that we considered at this point was migrating from Bitwarden to Vault; however, the secrets were mostly used by humans rather than direct application access, and for simple human access Bitwarden is simpler than Vault (not to mention already installed, reviewed, and understood).

So I came up with an alternative solution. I made use of the extensibility of Vault and wrote a backend that pulls secrets from Bitwarden on demand. This allowed us to use short-lived tokens with strict access limits for deployment activities while leaving all the secrets as they were in Bitwarden.

Because Bitwarden uses full client-side encryption, there is no simple and expressive API for pulling secrets from a server - the server doesn’t even know the names of the secrets that it is storing. Instead, for Vault to obtain secrets it must first download the entire collection of secrets that the user has access to and decrypt them to find the one it needs. Bitwarden uses several different types of encryption for various aspects of the process; there is hashing on the password (PBKDF2) to generate the AES256 master encryption key for the secret collection, then further hashing to verify authorisation to access the stored secrets. There is RSA2048-OAEP asymmetric encryption to enable sharing of secrets in groups within an organisation. And the encryption all uses HMAC for data integrity. All of this needed to be implemented in the Vault backend to gain access to the secrets held in Bitwarden.

This project also gave me an opportunity to try out some Go. The Mock Secrets Plugin provided an excellent foundation to build from, showing what boilerplate is required so that I could focus on the functionality of the plugin.

In building this plugin, I made extensive use of the Bitwarden source code and the excellent breakdown of the Bitwarden API written by Joshua Stein.

Paths

The backend has three paths: config/, auth/, and secrets/.

config/

The config/ path stores the simple configuration of the backend - just the URL for the Bitwarden instance. This value can be both written and read, and just for good measure it can be deleted too.

func (b *backend) operationConfigUpdate(
        ctx context.Context, 
        req *logical.Request, 
        data *framework.FieldData
) (*logical.Response, error) {
	var url string
	if urlIfc, ok := data.GetOk("url"); ok {
		url = urlIfc.(string)
	} else {
		return nil, logical.CodedError(http.StatusBadRequest, "url is required")
	}
	logger.Info(fmt.Sprintf("Updating base URL to '%s'", url))

	entry, err := logical.StorageEntryJSON("config", bwConfig{
		URL: url,
	})
	if err != nil {
		return nil, fmt.Errorf("Failed to create StorageEntryJSON: %w", err)
	}
	if err := req.Storage.Put(ctx, entry); err != nil {
		return nil, fmt.Errorf("Failed to Put entry into storage: %w", err)
	}
	return nil, nil
}

auth/

The auth/ path is asymmetrical. The write-side takes credentials for each user (email, password, and 2FA token) and authenticates against the Bitwarden instance, then securely stores the resulting tokens and encryption key.

func (b *backend) operationAuthUpdate(
        ctx context.Context, 
        req *logical.Request, 
        data *framework.FieldData
) (*logical.Response, error) {
	// Extract request details
	email := data.Get("email").(string)
	password := data.Get("password").(string)
	code := data.Get("2fa").(string)
	// Drop trailing newlines so that this works properly in the case
	// where the user passed their password via stdin
	password = strings.TrimRight(password, "\n\r")
	if email == "" || password == "" || code == "" {
		return nil, logical.CodedError(http.StatusBadRequest, "'email', 'password', and '2fa' must be supplied")
	}

	// Get the bitwarden client
	client, err := b.getClient(ctx, req)
	if err != nil {
		return nil, err
	}

	// Send login request to bitwarden
	err = client.login(email, password, code, b.uuid)
	if err != nil {
		return nil, err
	}

	// Store the retrieved auth data
	if err = storeClient(ctx, req, client); err != nil {
		return nil, err
	}

	return nil, nil
}

Reading from the auth/ path only tells the user whether they have already authenticated against Bitwarden and if so, how long until the token expires. As is standard security practice, there is no mechanism to read out any sensitive details (the tokens or encryption key).

func (b *backend) operationAuthRead(
        ctx context.Context, 
        req *logical.Request, 
        _ *framework.FieldData
) (*logical.Response, error) {
	client, err := b.getClient(ctx, req)
	if err != nil {
		return nil, err
	}
	authenticated := client.isAuthenticated()
	responseData := map[string]interface{}{
		"authenticated": authenticated,
	}
	if authenticated {
		expiry := client.getRefreshExpiry()
		responseData["expiry"] = fmt.Sprint(expiry)
		remaining := expiry.Sub(time.Now())
		responseData["remaining time"] = formatRemaining(remaining)
		responseData["note"] = "The expiry will be extended by reading a secret"
	}

	return &logical.Response{
		Data: responseData,
	}, nil
}

The auth/ path also supports the DELETE operation, to drop the tokens and key so that Vault can no longer access Bitwarden as the current user.

secrets/

Finally, there is the secrets/ path, which does all the heavy lifting and is the purpose of this backend. Reading from a path that starts with secrets/ causes Vault to pull a secret from Bitwarden. For example, if I were to read from secrets/openshift_prod, the backend would find the secret in Bitwarden called openshift_prod and return it.

func (b *backend) operationSecretRead(
        ctx context.Context, 
        req *logical.Request, 
        data *framework.FieldData
) (*logical.Response, error) {
	// Extract the name of the secret
	nameIfc, ok := data.GetOk("name")
	if !ok {
		return nil, logical.CodedError(http.StatusBadRequest, "No secret name given")
	}
	name := nameIfc.(string)

	client, err := b.getClient(ctx, req)
	if err != nil {
		return nil, err
	}

	// Make a call out to BW
	secret, err := client.getSecret(name)
	if err != nil {
		if _, ok := err.(*NotFoundError); ok {
			// Not an error to not find anything, there's just
			// nothing there, so return nothing
			return nil, nil
		}
		return nil, err
	}

	// Convert between "types" (come on, Go...)
	responseData := make(map[string]interface{})
	for key, val := range secret {
		responseData[key] = val
	}

	// Format the response
	return &logical.Response{
		Data: responseData,
	}, nil
}

This enjoys all the regular features of Vault - I can generate a short-lived token that has permission to access only certain paths and supply that to AWX for its deployment operations. If AWX were to be compromised, the only parameters that would be found on previous jobs would be expired (i.e., useless) tokens and the exposure caused by any live tokens would be limited to only certain secrets (a.k.a. limited blast radius).

Reading secrets from the secrets/ path is straightforward, but it does a lot of work behind the scenes. The basic breakdown of operations is:

  1. Check token validity
  2. Download the whole Bitwarden vault blob. This blob includes
    • The user’s private key, encrypted with the user’s encryption key
    • Organisation encryption keys for any shared secrets, encrypted with the user’s public key
    • All secrets the user has access to, encrypted with either the user’s encryption key or the relevant organisation encryption key
  3. Use the user’s encryption key to decrypt the user’s private key
  4. Use the user’s private key to decrypt all the organisation encryption keys
  5. Iterate through the secrets, decrypting each one’s name with the appropriate key
  6. Once the requested secret is identified, decrypt the secret details and return them to the caller. I only read the username, password, and notes fields since these are the only ones we needed; it wouldn’t be difficult to extend the logic to handle all possible fields.

The backend doesn’t store any of this information between calls to avoid returning stale data after a secret is updated, so all of these steps need to happen for every secret request.

There is no support for writing to (or deleting from) the secrets/ path as we didn’t need that functionality.

Encryption

Since I am not a trained cryptographer, I did the sensible thing and used libraries for the actual cryptographic operations; the Go standard library includes almost all the cryptographic functionality that I needed, the only extra package that is required is for PBKDF2 (which is also part of the Go library, just not the core):

import (
	"crypto/aes"
	"crypto/cipher"
	"crypto/hmac"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha1"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"errors"
	"fmt"
	"regexp"
	"strconv"
	"strings"

	"golang.org/x/crypto/pbkdf2"
)

That left me the job of plumbing it all together. Bitwarden stores all its encrypted content in “cipher-strings” that include the encryption type, the initialisation vectors, the actual encrypted payload, and the HMAC signature. So the decryption process starts with extracting and verifying all these details:

func decryptCipherstring(cipherstring string, key []byte) ([]byte, error) {
	m := re.FindStringSubmatch(cipherstring)
	if m == nil {
		// Don't include the cipherstring in the error message, as it is
		// a secret, even if it is encrypted.
		return nil, fmt.Errorf("Bad format for cipherstring")
	}
	encType, errType := strconv.Atoi(m[1])
	iv, errIV := base64.StdEncoding.DecodeString(m[2])
	payload, errData := base64.StdEncoding.DecodeString(m[3])
	receivedMac, errMac := base64.StdEncoding.DecodeString(m[4])
	if errType != nil || errIV != nil || errData != nil || errMac != nil {
		return nil, fmt.Errorf("Failed to decode cipherstring: %w", first(errType, errIV, errData, errMac))
	}
	if encType != 2 {
		// 2 = AesCbc256_HmacSha256_B64
		return nil, fmt.Errorf("Unsupported cipherstring type %d", encType)
	}

	// Split the encryption key into its two parts
	encKey := key[:32]
	macKey := key[32:]

	// Verify MAC
	macData := append(iv, payload...)
	mac := hmac.New(sha256.New, macKey)
	mac.Write(macData)
	computedMac := mac.Sum(nil)
	if !hmac.Equal(receivedMac, computedMac) {
		return nil, errors.New("MAC verification failed")
	}

	// Decrypt the payload
	return decryptAES(payload, iv, encKey)
}

With these details identified and verified (importantly, this includes verifying the HMAC signature), the actual decryption is fairly straightforward - the aes and cipher libraries do all the work:

func decryptAES(payload []byte, iv []byte, key []byte) ([]byte, error) {
	block, err := aes.NewCipher(key)
	if err != nil {
		return nil, fmt.Errorf("Failed to create AES cipher: %w", err)
	}
	cbc := cipher.NewCBCDecrypter(block, iv)
	plaintext := make([]byte, len(payload))
	cbc.CryptBlocks(plaintext, payload)
	// Remove padding
	padding := plaintext[len(plaintext)-1]
	plaintext = plaintext[:len(plaintext)-int(padding)]
	return plaintext, nil
}

If you’re reading these code snippets, you will have noticed that the decryptCipherstring() function takes a key as a parameter. This key will either be the user’s main encryption key (for secrets that belong to that user) or an organisation encryption key (for secrets that are shared via an organisation). Organisation encryption keys are themselves encrypted by the user’s public key, thus they are decrypted by the user’s private key with this series of error-checked operations:

func decryptOrgKey(cipherString string, privateKeyDER []byte) ([]byte, error) {
	parts := strings.Split(cipherString, ".")
	if len(parts) != 2 {
		return nil, errors.New("Bad cipherstring format")
	}
	if parts[0] != "4" {
		// 4 = Rsa2048_OaepSha1_B64
		return nil, fmt.Errorf("Unsupported orgkey cipherstring type: %s", parts[0])
	}
	rawPayload, err := base64.StdEncoding.DecodeString(parts[1])
	if err != nil {
		return nil, fmt.Errorf("Failed to decode orgkey payload: %w", err)
	}
	privateKeyIfc, err := x509.ParsePKCS8PrivateKey(privateKeyDER)
	if err != nil {
		return nil, fmt.Errorf("Failed to parse private key: %w", err)
	}
	privateKey, ok := privateKeyIfc.(*rsa.PrivateKey)
	if !ok {
		return nil, fmt.Errorf("Private key is not an RSA private key apparently")
	}
	orgKey, err := rsa.DecryptOAEP(sha1.New(), rand.Reader, privateKey, rawPayload, []byte{})
	if err != nil {
		return nil, fmt.Errorf("Failed to decrypt org key: %w", err)
	}
	return orgKey, nil
}

This private key is delivered in the data blob from the Bitwarden server, encrypted with the main encryption key. That main encryption key is obtained at log-in time, decrypted using the master key:

func (c *BwClient) login(email string, password string, code string, deviceUUID string) error {
	iterations, err := prelogin(c.baseURL, email)
	if err != nil {
		return err
	}
	masterKey := makeMasterKey(password, email, iterations)
	hashedPassword := hashPassword(masterKey, password)

	reqBody := url.Values{
		// ... snip ...
	}

	resp, err := http.PostForm(c.baseURL+"/identity/connect/token", reqBody)
	if err != nil {
		return fmt.Errorf("Error posting to login endpoint: %w", err)
	}
	
	defer resp.Body.Close()
	if resp.StatusCode != 200 {
		return handleErrorResponse(resp)
	}

	decodedResponse := responseType{}
	err = json.NewDecoder(resp.Body).Decode(&decodedResponse)
	if err != nil {
		// I'm not including the underlying error in this message
		// just in case one of the tokens ends up in the message.
		return fmt.Errorf("Failed to decode login response")
	}

	stretchedMasterKey := stretchMasterKey(masterKey)
	encryptionKey, err := decryptCipherstring(decodedResponse.Key, stretchedMasterKey)

	// ... snip ...
}

The master key in turn is derived from the user’s password (using PBKDF2):

func makeMasterKey(password string, email string, iterations int) []byte {
	return pbkdf2.Key([]byte(password), []byte(strings.ToLower(email)), iterations, 256/8, sha256.New)
}

func stretchMasterKey(masterKey []byte) []byte {
	return append(hkdfExpand(masterKey, "enc"), hkdfExpand(masterKey, "mac")...)
}

func hkdfExpand(prk []byte, info string) []byte {
	msg := append([]byte(info), 1)
	mac := hmac.New(sha256.New, prk)
	mac.Write(msg)
	return mac.Sum(nil)
}

That covers most of the interesting bits of logic, I think.

It was certainly interesting to explore how the different cryptographic parts fit together. I don’t claim to be a cryptography expert: if you can see I’ve done anything insecure please do let me know!

Extending the plugin

This approach worked well for our use case; if you want to see more implementation details or make use of this backend yourself, the complete source can be found in the Gitlab repository. If you find it useful I’d love to hear about it!

There are a few limitations that I have alluded to above as I only created it for our use-case. The largest is probably that it only supports read access to Bitwarden as that was all that we required. If you need more functionality, please feel free to extend it; I have no plans to do so myself at this stage, but I am happy to consider PRs.