Painless Password Rotation with HashiCorp Vault (2023)

For more in-depth tutorials and documentation on using HashiCorp Vault for password rotation, seevault trackon HashiCorp Learn.

introduction: Nobody likes passwords. They're hard to remember, and complex rules and rotation requirements can make security worse.

Bad practices abound because password management is so complicated:

• Users copy annoying, hard-to-remember passwords onto sticky notes.
• Sometimes system passwords are stored in clear text on a wiki page or shared document.
• Credentials are often shared between multiple users, or the same username and password can be used to access multiple systems.

Password management is a thorn in the side of many system administrators. Fortunately, Vault is a system that automates most of the headaches associated with key and password rotation. With built-in tools you already have installed on your servers (Bash or Powershell), you can automatically generate strong passwords for Linux or Windows servers and store them securely in the vault.

HashiCorp Solutions Engineer Sean Carolan demonstrates some ways you can clean up and automate password management with Vault.

speaker

• 

  Alte KarolaSolutions Engineer, HashiCorp

  • 
  • 

transcription

Good morning My name is Sean Carolan and I am a former or former system administrator. Today I am a Solutions Engineer at HashiCorp. I love building things so now I travel and help users get started with HashiCorp tools.

My first computer was a Commodore 64. Old or legacy systems, right? This is a fairly recent photo but there is a Commodore 64 in the photo. This was captured at the Living Computer Museum in Seattle. I recommend if you are in Seattle visit this place. Make a pilgrimage to the Living Computer Museum. There's nothing that makes you feel older than seeing your childhood toys in a museum.

So password rotation. Court! Not everyone looks so excited. Let's be honest, okay? Nobody came to this talk because you like changing passwords, right? We all woke up this morning and said, "I'm going to change my password today." NO. Let's look at an example.

The pain of changing passwords

Anyone else do this? I'll do that. “I've got two full weeks. I don't want to change my password. I've still got 2 weeks to go and it took me at least 3 weeks to memorize the damn thing. You know that? I'm busy right now. Come back later. There may be a day when I change this password, but today is not that day. Why can't I keep my password? I really like my password. ACCORDINGLY. He's fine. We are until the last moment. I'll do that this afternoon, I promise. Nobody likes that, do they? And that's just 1 password on 1 machine.

So why do we engage in this crazy ritual every few months and how did we get here? More importantly, who can we blame? I did some research and found a modern computer password finder. So here it is. This handsome gentleman is Fernando Corbató. He is one of the pioneers of timesharing computer systems and led the team that developed the first compatible mainframe timesharing system. And he is also credited with inventing the modern computer password.

You see, back then, computers were these huge, heavy machines that filled entire rooms. These powerful machines were large and expensive to run. Users wrote their programs on punch cards and then sent the punch cards to operators for execution. So remember, the next time you implement your code, bring a giant box of punch cards and keep them on the sysadmin's desk.

So these programs were queued and run serially at first, which meant you had to wait days or hours for your program to run and come back later to get the results. It wasn't practical to have developers constantly connected to the mainframe because what are we doing most of the time? We write, think things through, refactor. We're not actually using the machine's CPU.

So Fernando and his team were tasked with solving this problem and created their timeshare concept. Multiple users can now connect to the same system and one user's idle time can be used as another user's active time. Finally, multiple users can take full advantage of the mainframe.

Passwords: Trouble from the start

Therefore, the password was not originally invented for security reasons. In 1961, its main purpose was to keep the archives of the different researchers separate. So, to Dr. Corbató: "The main problem was that we set up multiple terminals that were used by multiple people, but each person had their own private files and assigning a password as a lock to each individual user seemed like a very simple problem."

And if Hollywood has taught us anything, it's that when the scientist's beautiful experiment begins, you know something is going to go horribly wrong, right? And of course yes. Luckily Fernando is healthy, he's still with us and he didn't turn into a fly so it didn't go that bad.

Less than a year after the invention of the password, however, the first problems have already arisen. So we had the first case of password theft in 1962, less than a year after it was invented. You see, these machines were very expensive to run, so each user had 4 hours of computing time. And one of the other researchers on the project, Dr. Allan Scherr was frustrated with this 4 hour limit. However, I did know that there is an offline printing feature where you can mail in a punch card and print files overnight and pick them up in the morning.

So, on Friday night, Dr. Scherr submitted a print job, showed up bright and early Saturday morning, and collected the entire list of usernames and passwords from the system. Immediately, the list of passwords was shared with the other researchers on the team, and one of the other users immediately began deceiving the administrator by sending messages to his system account. Then in 1962 we had the first internet troll.

And much like the sci-fi movie plot, the brilliant scientist's creation eventually came back to haunt him. Doctor Corbató said: "It became something of a nightmare. I don't think anyone can remember all passwords. So if the MIT graduate student in computer science thinks we're hopeless, what hope is there for the rest of us, right?

Fast forward the clock to the 1970s: while researching for this talk, I wanted to find the oldest machine I could log into. And this is Miss Piggy. Miss Piggy is a PDP-11 running in the same museum, the Living Computer Museum. Old school right?

So this system and other systems in the 1970s used the same password file, but there were some minor improvements. So we started encrypting the passwords, which means we encrypt them, make it a little harder for someone to get away with that list, and start using other people's passwords.

This is an example of a hashed password file. A little harder to get to, but the same basic concept is there. Every user has some kind of password on the system and they are stored in this file here. Later we improved this so-called salted hash by adding some random characters and encrypting the password. But over time, computers have become more and more powerful.

Does anyone remember when a 6-digit password was considered secure? Or 8? Good? Eight digits are no longer safe. A modern computer can figure this out quickly. Computers are becoming more and more powerful and better at guessing passwords. And that has made it harder and harder for users because now we have to remember longer and more complex passwords.

How do we set passwords now

So let's choose a password. Let's go through a day in the life of a user. That user is me and I'm choosing a new password. This is my demon cat. I heard that you can earn internet points by including a cat in your presentation and she didn't want any. But this is Mochiko. So Mochiko is very cute and fluffy. Mochiko means rice flour. And I'll use my pet's name as the password because it's easy to remember.

We'll start with your short name. Easy to remember, but unfortunately very easy to decipher. I ran a password cracking tool against this password. Can you guess how long it took? Yes, less than 2 minutes, very short, very easy to guess. And that in a very small instance that costs 7 cents an hour to run.

let's make it longer Mochiko's full name is a little better, but it's still only 7 characters long, and the security team say they need some numbers. Okay, let's add a number at the end. Are we really kidding someone here? You also need some symbols; this is part of our password requirements. Okay, let's change a zero to an O and put an exclamation point. Now we have this odd hodgepodge of characters that we hope to remember.

Let's make sure we never forget this password. You've never seen that at your workplace, have you? Why is that happend? We like to torment our users with these really weird password complexity requirements. It turns out that on average people can reliably remember a group of 7 digits, about 6 letters or 5 words, plus or take 2. They researched this. On average, people can remember about 7 digits, which happens to be the length of a phone number if you don't include the area code.

Can someone remember 5 phone numbers here? I see a pair of hands. Believe it or not, there was a time when many of us could recite a dozen or more phone numbers from memory. I can barely remember 3 phone numbers, and some of my friends don't even remember their own phone number. However, we told our poor users, "You need to remember this 8-digit string of random characters, letters, and symbols." How about we give them a set of 4 or 5 words to learn instead?

So this is the xkcd comic that illustrates this. Above we have your classic number/letter/symbol password, which unfortunately turns out to be pretty easy for a computer to guess. And in the comics he does the math and estimates that if you keep cracking this it could take about 3 days to crack that password. Below we have "correct horse battery clamp". "Correct Horse Battery Clamp". You will never forget it. And it's very difficult for computers to guess that because of the higher amount of entropy.

In other words, a really long password made up of words is easier for humans to remember but harder for computers to guess. Unfortunately, these super-long passwords are not supported by many systems because they do not meet the complexity requirements.

Then we turn to the password vault. Anyone use password security? Most of us, right? It's a necessity in the modern world. So it's an application or program or maybe a user's web browser that allows you to securely store passwords. And your passwords end up on your laptop, smartphone or in the cloud. Usually you use a master password to unlock it and crack all other passwords.

Now this is fine for personal use. But what if you need to manage hundreds of systems in the cloud or data center? This usually doesn't scale that well.

So we looked at the history of password security. Let's take a quick look at the present and the future.

Automation is the key to password nirvana

Here is a possible solution to the password management problem. What if you could provide usernames and passwords that expire automatically? Like an old spy movie or TV show where the tape self-destructed and exploded in smoke and flames. Users and applications can simply ask for new credentials when they need them, use them for a specified period of time, and then the credentials automatically disappear when they expire. Sounds great right?

I like the example of a ski resort. If you want to ski you can buy a pass and this pass has an expiration date and access levels. So maybe get a beginner ski pass that gives you access to the Berghasen run, or you can get a year-round pass that's valid for a few months and lets you ski to the black diamonds.

Thendynamic credentialsare very similar, and this is something that HashiCorpbovedacan do for you, either with your databases or in the cloud. It works the same way. You order these credentials, get them, use them for a short time, and they expire.

Sounds great right? But I can hear you thinking, "But Sean, I'm stuck with all these outdated machines that I have to manage and I can't do all this fancy dynamic stuff."

Dealing with older systems

Let's talk about legacy systems. Does anyone have a machine like Miss Piggy on their network? Yes? wow right So who manages your company's legacy systems? please get up Do not be shy. get up get up Let's look at this legacy on any machine older than 5 years, for example. OK, we see some. Let's help these people. Thanks. And that could be all of the awards they've received throughout the year. So be kind to your system administrators.

The reason these legacy systems still exist is because they still work. They were very well built and still serve a certain function. So what do we do with these systems? Some of my clients have started giving them new names.

A few weeks ago, an IT manager introduced me to his "classic" environment. So they are no longer legacy machines; They are classic machines. But I think my favorite is the Heritage collection. Perhaps unsurprisingly, it's a bank. So this is what you can put on your resume: "Heritage Systems Curator".

If there's one thing we know about these legacy systems, it's that they're not going away anytime soon. These dinosaurs have survived for so long and someone needs to take care of them. But the security team says, "You must rotate root and admin passwords on these boxes every 90 days." How will we meet this challenge? So I set out to design a system that would securely automate the rotation and storage of your system's passwords. These are the design requirements.

First, no external software. Nobody running that old Linux or Windows server wants to know anything about Docker. I am right? Why not put it in a container, spray some DevOps on it, and move it to the cloud? We want to keep it as simple as possible. bash or powershell.

"We need a reliable source of entropy for password generation." Obviously, letting the human choose the password, as you saw on the slide with my cat, is not a good strategy. We ended up with a weird password that could probably be cracked fairly quickly.

The first version of this tool used OpenSSL or PowerShell to generate passwords, but there are some problems with that. We weren't sure we could trust that each machine had the necessary tools. Again, this is beyond our control. So we will use Vault to generate new passwords. We want each computer to be responsible for its own password rotation. You are busy. You've come to a talk called Painless Password Rotation, so we don't want any manual steps. This should be something you can set and forget. We do this so that each machine takes care of itself.

Flexible Passwortunterstützung

We also want to support different types of passwords. Therefore, the system must support long and complex strings of numbers and letters, as well as the password style of the password. All passwords must be encrypted both in transit and at rest, and our servers must have write access only. In other words, we form a one-way door. When a new password is generated and stored in the vault, we don't want the same computer to come back and read the password again. It is not necessary.

we want to saveNorteNumber of previous password versions. What if I need to go back and retrieve the old password that was set a week or a few days ago? People should have read-only access to these passwords by policy, and we want our passwords to automatically rotate all of themXdays, which is the main point of this talk.

These are our design requirements. Let's discuss what you can do to configure this in your own environment.

Step 1, or Step 0 if you prefer, is to get a vault. It's easier than ever to set up the HashiCorp Vault. You can download for free. You can even unlock it with a cloud provider so you can set up your HA-enabled Vault cluster. The second step is the installationthis password generator plugin. This was created by Seth Vargo at Google, a very smart guy, and he created this password generator plugin that you can use with Vault and make Vault your password generator.

The next step is to enable a key-value store or secret mechanism. We go with version 2 of the KV store because it supports versioning of our passwords.

And we finally need somethingguidelinesto determine how our machines can access the vault and how our humans can access the vault. So at the top we have the Rotate Linux policy. We grant access to write new passwords in Vault and we also grant access to API endpoints that generate new passwords. Here it is the same for Windows. And all this source code is online. I leave the link at the end of the presentation.

Now we need a token. Here we will generate a Vault Token that will allow our computer to run our password rotation script. We use a periodic token that officially has no expiration date but requires regular registration by the system. So this token has a period of 24 hours, which means that if a computer doesn't log in within 24 hours, this token will expire and you will have to create a new one. That helps with cleaning. So if you ever take that legacy computer offline, the token will expire after a short time.

And then the next step is to configure the scripts. So we have something for Linux and Windows. Linux is a bash script, Windows is a PowerShell script. And there are 2 environment variables that you set to make this work.

Then it sets the vault address and vault token, and then runs the scripts. How do these scripts work? I designed them to be all-in-one with no external dependencies, so the first thing we do in our script is update our token. We run a cURL command or call with Bash or call RestMethod with PowerShell. And we're basically saying to Vault, "Hey, I'd like to keep using this token. Please renew it for me.”

After the token renewal, we proceed and generate our new password. So let's make an API call to Vault again and access the passphrase or password terminal. Vault will respond with a new password. Finally, we store this new password on the local system, but we need to do it securely. We don't want to cut the branch we're sitting on and lock ourselves out of the system by changing the password and not sending it to the vault.

So let's do it backwards. First we generate a secure password, verify that it is verified in the vault, and then write it to the local system. That way, if you accidentally entered the password into the vault but it didn't make it to your local system, you can still revert to the previous version without it getting locked on your system.

The final step is to schedule the script, put it in a cron job or windows scheduled task, and then forget about it.

A live demo

Let's do a live demo. Here is my legacy box. This password rotates every minute. You can see that we are using the long password style here. And if I need to generate new passwords, I now have these API endpoints at my disposal. Then I can do a Vault recording.

And we can do the same here with passwords. So if you need a long and complex password, you can also do it by choosing the length. If you are not a fan of symbols, you can also limit the number of symbols.

This is our password source. And then on the client side we have a Windows PowerShell script. We just changed my personal password on this machine. So it works with any username on this system. And if you try to rotate a password for a non-existent user, the script will fail.

That was the demo. So it was very short, and you can see why I had to include the computer's password history in this presentation, otherwise we would have been done in 2 minutes. However, I still have a few slides. There are always security guards in the room. "And yes?" And we can't be mad at them because that's their job to say, "What if?" So I'll go through some of the dos and ifs.

What happens if an attacker gets their hands on a system token? what could they do Well, you can write any data to your Vault instance, either Linux or Windows path. You cannot replace the existing password, only fill your password storage and delete old versions.

Another scenario, a little less likely, but they could write so much data that it would completely fill the KV memory. As an alternative scenario, a DDoS or failure of your Vault cluster could theoretically lock you out of all your systems. So there are some risks and also some steps you can take to reduce them.

One would be to configure the vault token to only be usable for a very short period of time. It might take 2-3 minutes when you run your scheduled task, that's the only small window during the day for these scripts to run.

Another method you can use is to limit the size of the data that can be written to the key-value store. These are the features of Vault Enterprise.

A third would be using Vault in HA mode and using the DR feature, Disaster Recovery, for emergencies. And that way, if the primary vault cluster fails, you can always switch to disaster recovery or the replicated vault cluster, and that way you're not locked out of your systems. Finally, you can monitor your Vault audit logs for unusual activity on systemcreds Linux or Windows endpoints.

Now one last note: we do a different kind of password rotation. If this is a challenge or an issue in your world, starting with Vault 0.11 it will also be integrated into Vault. We can also rotate Active Directory passwords. This uses lazy rotation. Passwords have a TTL but are not rotated until the TTL expires and then the password is requested. Therefore, it is designed for high-load environments where you have multiple instances or services using the same system password.

You can download the codeHere. This will also be put online. Both the presentation and the scripts are available atthis depot.

Thank you for coming. That was Painless Password Rotation with HashiCorp Vault.