Why do we hash user passwords in our databases? So that if bad actors somehow gain access to our database, they will have a hard time stealing our secrets. But suppose we need to securely store other sensitive data in our database? Encrypt it. But how do we go about that? Assymetric, public/private key encryption with PGP would undoubtedly be the best way to go, but in the case at hand, it’s too impractical. Faced with this situation, I reasoned that symmetrical encryption was the next best option: use one key for both encryption and decryption. But how to store the encryption key securely? As a plain text file sitting on the same server as the database username and password? Which, by the way, are also sitting there in plain text, if not in plain sight?
Disclosures, Disclaimers, Excuses
Now that I have your attention, a quick digression. I am not a security expert — I just try to pay attention to those who are. So please, use your own judgment at your own risk yadda yadda yadda. Moreover, I’m not even officially a web application developer. My job title has nothing to do with that. Long story short: I stumbled into coding some 20 years ago because in my workplace we could find no software, commerical or otherwise, that met our needs in managing a busy federal court interpreters office. So we (OK, I) rolled our own, and have been rolling ever since. Now I’m working on the next iteration of our great project, using Zend Framework 3 and Doctrine, and that’s the context in which this security problem arises.
Wandering back to our main topic: we need to store some sensitive data of contract court interpreters in a database, we want to encrypt it symmetrically, and we need to keep the secret encryption key secret.
Hashicorp Vault to the rescue
The very clever people at Hashicorp generously provide, among other things, a secret-management tool aptly named Vault. As their website puts it,
Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.
The use of Vault is a complex subject, and the following discussion is no substitute for the documentation. We should just point out that secrets (yes, that’s the technical term) can be stored in several different types of storage backends, and authentication can happen by way of a number of different authentication backends. These things are identified and accessed by way of paths similar to a filesystem. And that unified interface mentioned above is an HTTP API. You talk to Vault by sending HTTP requests with an authentication token in your request headers, and get JSON responses. Of course, this makes it a snap for applications to communicate with Vault with the language of your choosing.
The challenging part is devising a good security model, and there is a vast — if not to say baffling — array of choices and decisions to be made. I’m still not convinced my current solution is the best, so it’s subject to change. But let’s step through it as it is at the moment. Of course, you’ll first need to install and configure and run Vault, and they make that pretty damn easy. (Though you do have to decide what machine(s) to put it on. There’s replication and high availability and all that sexy stuff for large-scale deployments. Ours happens to be a modest configuration, involving a small set of users behind our organization’s firewall.)
Creating Policies
In Vault, you create authentication tokens that have policies attached to them, determining what the bearer of the token is allowed to do. I want our authenticated application not to be authorized to read our ultimate secret, but to grant an auth token bound to a policy that does allow it. So we need a policy for reading the secret. We also need a policy for creating the token that’s attached to the policy for reading the secret. Sound complicated? Don’t worry: it is! But the underlying principle is a basic one: grant access that is as permissive as necessary, and no more. So I took out my quill and wrote policies, thus:
read-secret.hcl
path "secret/data" {
policy = "read"
}
create-token.hcl
path "auth/token/create/read-secret" {
policy = "write"
}
Incidentally, that .hcl extension you see refers to Hashicorp Configuration Language. It’s a lot like JSON and you’ll pick it up immediately. Then, after logging into Vault with sufficient privileges, we wrote our policies to Vault via its CLI:
vault policy-write read-secret read-secret.hcl
and
vault policy-write create-token create-token.hcl
and finally, the rather more esoteric one:
vault write auth/token/roles/read-secret allowed_policies=read-secret
which magically allows the “role” to bestow access to something that the role itself cannot access.
Authenticating the application
As implied above, I decided Step One should be that not the user but the application authenticates itself against Vault using the TLS backend. To that end, I created my certificates (thank you, https://jamielinux.com/docs/openssl-certificate-authority/ for help with that), stored them in Vault,
vault write auth/cert/certs/web display_name=web policies=create-token \
certificate=@/path/to/your/cert.pem
and tested it out with the Vault cli,
vault auth -method=cert -client-cert=/path/to/your/cert.pem \
-client-key=/path/to/your/key.pem
then with the HTTP API using our good friend curl. By the way, before thinking too much about integration with ZF, I followed the same procedure with all this token stuff until I was able to go from zero to reading the secret (encryption key) from the command line. I predicted (correctly!) that the heavy lifing was setting up the Vault stuff, and getting it to play well with ZF would be more like fun than work.
Getting a token for getting the secret
After the previous command, now that we’re authenticated via TLS, we can go
vault token-create -role=read-secret
and get a response something like
Key Value
--- -----
token ef2fb0d1-1644-937f-5326-3c6270abc3ba
token_accessor 522c0a9d-7897-a670-e511-650d37ea6d20
token_duration 768h0m0s
token_renewable true
token_policies [default read-cipher]Getting the secret
And finally, authenticate with the token we just got:
vault auth ef2fb0d1-1644-937f-5326-3c6270abc3ba
and go for the gold
vault read secret/data
resulting in
Key Value
--- -----
refresh_interval 768h0m0s
cipher b862600aaeba7c4ccd74006d2e616083ffb7031a3b088e743080bcf32e90f3b4
and that “cipher” you see is the encryption key for encrypting/decrypting our sensitive database fields, a step we will perform in our PHP application.
In all honesty, it’s a bit more complicated than this because we are going to be using response wrapping. When a client sends the header so indicating, the response is not the thing requested, but another auth token with which you can get the thing requested. And that token is a single-use token: use it once and then it’s revoked. You can also limit the time-to-live on this token, so that whether it gets used or expires, it is not going to be hanging around for long as a valid token. This exemplifies the Vault (and general security) principle of limiting the exposure of a secret.
Plugging it into ZF
The front end
At last, the fun part. (Not to say that Vault isn’t loads of fun — it absolutely is! But after dealing and struggling with something unfamiliar, it’s comforting like a warm bath to return to the familiar, isn’t it?) I decided that this application would have its Vault capabilities in a separate module to make it easy to enable or disable. If it’s disabled, they simply don’t get to work with sensitive data. Let’s work from the front end to the back. The form has a good number of fields, but the ones we’re concerned with look like this:
because there’s no reason to expose this data if they don’t ask. The encrypted values are tucked away as hidden fields. When they click those lock thingies, we display a modal dialog inviting the already-authenticated user to re-authenticate (in case they load the form, wander off to the bathroom, and just then an identity thief comes up to their workstation). A couple of Javascript xhr calls later, if all goes well, the dialog goes away and these fields are re-populated with the decrypted values.
The Controller
And the above-mentioned xhr sends a POST request containing the encrypted values to /vault/decrypt, a route mapped to this action method in our module/Vault/src/Controller/VaultController.php:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | public function decryptAction() { $params = $this->params()->fromPost(); try { $key = $this->vaultService->getEncryptionKey(); $cipher = new BlockCipher(new Openssl()); $cipher->setKey($key); $decrypted = [ 'ssn' => $cipher->decrypt($params['ssn']), 'dob' => $cipher->decrypt($params['dob']) ]; return new JsonModel($decrypted) ; } catch (VaultException $e) { return new JsonModel(['error'=>$e->getMessage()]); } } |
The part involving Vault gets our encryption key, and it’s a one-liner. The next bit simply makes use of zend-crypt to take care of decryption.
You notice $this->vaultService. That’s a dependency injected via the constructor, so there’s a factory which pulls it from the container. Boring, so we’ll skip that part. Let’s have a look at the module’s module.config.php (ooh, exciting!)
Configuration
<?php namespace SDNY\Vault; use Zend\Router\Http\Literal; return [ 'vault' => [ // override these with a local configuration 'vault_address' => 'https://vault.example.org:8200', 'sslcafile' => '/usr/share/ca-certificates/ca-chain.cert.pem', // these settings must match the configuration set in Vault 'ssl_key' => '/path/to/your-private-key.key.pem', 'ssl_cert' => '/path/to/your-cert.pem', 'path_to_secret' => '/path/to/your/secret', // including leading slash // but do not change this adapter 'adapter' => 'Zend\Http\Client\Adapter\Curl', ], 'service_manager' => [ 'factories' => [ Service\Vault::class => Service\Factory\VaultServiceFactory::class, ] ], 'controllers' => [ 'factories' => [ Controller\VaultController::class => Controller\Factory\VaultControllerFactory::class, ] ], ]; |
So the module’s config sets some default/dummy values, with the expectation that they will be overridden by a local configuration file called something like config/autoload/vault.local.php, consistent with the ZF convention. No reason to put this in a public repository.
The Vault service
We have a VaultServiceFactory that injects all that config for us when it instantiates our VaultService, which is what does the talking to Vault.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 | <?php /** * module/Vault/src/Service/Vault.php */ namespace SDNY\Vault\Service; use Zend\Http\Client; /** * Extension of Zend\Http\Client for communciating with Hashicorp Vault * * The purpose is enable us to store sensitive data in MySQL using symmetrical * encryption while avoiding having to store the encryption key in plain text * anywhere at any time. All the configuration has to be correctly set before * instantiation. Error-checking is left up to the consumer. * */ use Zend\EventManager\EventManagerAwareInterface; use Zend\EventManager\EventManagerAwareTrait; class Vault extends Client implements EventManagerAwareInterface { use EventManagerAwareTrait; protected $events; /** * mapping of string keys to CURL integer constants * * we need this because if a config array key is an integer * unfortunate things happen when the framework merges the configs * * @var array */ private static $curlopt_keys = [ 'ssl_key' => \CURLOPT_SSLKEY, 'ssl_cert'=> \CURLOPT_SSLCERT, ]; /** * vault authentication token * * @var string */ private $token; // some more instance variables omitted for brevity /** * constructor * * @param array $config */ public function __construct(Array $config) { $this->vault_address = $config['vault_address'] . $this->prefix; $this->path_to_secret = isset($config['path_to_secret']) ? $config['path_to_secret'] : null; $curloptions = []; foreach ($config as $key => $value) { if (key_exists($key, self::$curlopt_keys)) { $curloptions[self::$curlopt_keys[$key]] = $value; } } $config['curloptions'] = $curloptions; parent::__construct(null, $config); $this->getRequest() ->getHeaders() ->addHeaderLine('Accept: application/json'); } // some setters/getters omitted for brevity /** * checks response for errors * @param Array $response * @return boolean true if error */ public function isError(Array $response) { return key_exists('errors',$response); } /** * resets request, response, etc, and restores * request header for JSON responses * * @return \SDNY\Vault\Service\Vault */ public function reset() { parent::reset(); $this->getRequest() ->getHeaders() ->addHeaderLine('Accept: application/json'); return $this; } /** * attempts Vault TLS authentication * * this will attempt to authenticate using TLS certificates, which have to * have been installed and set in our configuration up front. * * @link https://www.vaultproject.io/docs/auth/cert.html * * @return Vault * @throws VaultException */ public function authenticateTLSCert($options = []) { $this->setMethod('POST') ->setUri($this->vault_address .'/auth/cert/login') ->send(); $response = $this->responseToArray($this->getResponse()->getBody()); if ($this->isError($response)) { $this->getEventManager()->trigger(__FUNCTION__, $this, []); throw new VaultException($response['errors'][0]); } $this->token = $response['auth']['client_token']; return $this; } /** * Attempts to acquire access token that is authorized to read the cipher * we use for symmetrical encryption/decryption of sensitive Interpreter * data. * * @return Vault * @throws VaultException */ public function requestCipherAccessToken() { $this->getRequest()->getHeaders() ->addHeaderLine("X-Vault-Token:$this->token") ->addHeaderLine("X-Vault-Wrap-TTL: 10s"); $endpoint = $this->vault_address . '/auth/token/create/read-cipher'; $this->getRequest()->setContent(json_encode( [ // maybe reconsider these settings 'ttl' => '5m', 'num_uses' => 3, ] )); $this->setMethod('POST')->setUri($endpoint)->send(); $response = $this->responseToArray($this->getResponse()->getBody()); if ($this->isError($response)) { $this->getEventManager()->trigger(__FUNCTION__, $this, [ 'message' => 'failed to get token for cipher access' ]); throw new VaultException($response['errors'][0]); } $this->token = $response['wrap_info']['token']; return $this; } /** * unwraps a wrapped response and returns it as an array * * @param string $token * @return array */ public function unwrap() { $this->reset(); $endpoint = $this->vault_address . '/sys/wrapping/unwrap'; $this->setAuthToken($this->token); $this->setMethod('POST')->setUri($endpoint)->send(); $response = $this->responseToArray($this->getResponse()->getBody()); if ($this->isError($response)) { $this->getEventManager()->trigger(__FUNCTION__, $this, [ 'message' => 'failed to unwrap response' ]); throw new VaultException($response['errors'][0]); } if (isset($response['auth'])) { $this->setAuthToken($response['auth']['client_token']); } return $response; } /** * requests response-wrapped encryption key * * @param string $token authentication token * @return Vault * @throws VaultException */ public function requestWrappedEncryptionKey() { if (! $this->path_to_secret) { throw new VaultException('path to secret has to be set before calling '.__FUNCTION__); } $endpoint = $this->vault_address . $this->path_to_secret; $this->getRequest()->getHeaders()->addHeaderLine("X-Vault-Wrap-TTL: 10s"); $this->setMethod('GET')->setUri($endpoint)->send(); $response = $this->responseToArray($this->getResponse()->getBody()); if ($this->isError($response)) { $this->getEventManager()->trigger(__FUNCTION__, $this, [ 'message' => 'failed to get wrapped encryption-key response' ]); throw new VaultException($response['errors'][0]); } $this->setAuthToken($response['wrap_info']['token']); return $this; } /** * gets encryption key. * * convenience method that wraps the several * steps into one. * * @param string $token authentication token * @return string * @throws VaultException */ public function getEncryptionKey() { $this->authenticateTLSCert() ->requestCipherAccessToken() ->unwrap(); $this->requestWrappedEncryptionKey(); $key = $this->unwrap()['data']['cipher']; return $key; } /** * sets Vault authentication token header * and instance variable * * @param string $token * @return \SDNY\Service\Vault */ public function setAuthToken($token) { $this->getRequest() ->getHeaders() ->addHeaderLine("X-Vault-Token:$token"); $this->token = $token; return $this; } /** * converts json to array * * @param string $json * @return Array */ public function responseToArray($json) { return json_decode($json,true); } /** * attempts user/password authentication * * this will attempt to authenticate user against Vault's * userpass auth backend. NOTE: looks like we won't be using this auth * method after all, so this method is not currently used. * @link https://www.vaultproject.io/docs/auth/userpass.html * * @param string $user * @param string $password * @return array Vault response as array */ public function authenticateUser($user,$password) { $uri = $this->vault_address . "/auth/userpass/login/$user"; $this->getRequest()->setContent(json_encode(['password'=>$password])); $this->setUri($uri)->setMethod('POST')->send(); return $this->responseToArray($this->getResponse()->getBody()); } } |
This Vault service extends Zend\Http\Client because it basically is a specialized http client. The general pattern is that most of these methods attempt to get the token we need, store it in $this->token, and return $this so the lazy coder using the class can save some keystrokes.
We make use of the handy Zend\EventManager\EventManagerAwareInterface and Zend\EventManager\EventManagerAwareTrait and do a fair amount of $this->getEventManager()->trigger(). It’s most definitely a work in progress, but by the time this project is done (or pretty close) there are going to be event listeners responsible for logging things and generally paying attention. On the Vault side, I plan to (figure out a way to) monitor the audit log with the ultimate goal of trying to detect a possible breach and hit the panic button.
So we’ve demonstrated decryption, how about encryption? Coming soon! But as you can see, it will be straightforward.
Further thoughts on Vault and PHP Applications
Remember we started out with a reference to the practice of storing database credentials in plain text? Vault provides a database secret backend so you can avoid doing that. Vault also has a very straightforward user/password authentication backend that makes it tempting to consider implementing a Zend\Authentication\Adapter\AdapterInterface that uses it.
All of this is to say that in terms of security, I think Vault offers us ways to take Zend Framework applications, and PHP applications generally, to the next level.
Conclusion
As noted at the beginning, this solution to my challenge of keeping our encryption key secure may not be the best possible. But it’s a start. The first objective was to avoid leaving our secret lying around in plain text, and this accomplishes that. My holy grail is to make it so that even in a worst case scenario, where an attacker gains root access to the machine where this application is installed, my precious social security numbers and dates of birth would still not be compromised. Like anything, this model is not guaranteed bullet-proof against a breach that severe, but again, it’s a start.
Hopefully this demonstrates that once you have Vault going, it’s quite easy to integrate with your ZF application. As a personal note, I would add that this is actually my debut blogging about Zend Framework, and I do feel a little self-conscious about presuming to address an audience with so many badasses in it. Stage fright notwithstanding, I am truly interested in hearing your comments, so please feel free.
References