- kube-scheduler: The Kubernetes scheduler - Fri, Sep 29 2023
- Kubernetes logs: Pod logs, container logs, Docker logs, kubelet logs, and master node logs - Mon, Sep 25 2023
- Kubernetes DaemonSets - Wed, Sep 6 2023
Two-factor authentication (or 2FA in short) provides an extra layer of security because, in addition to a username and password, it requires you to enter a temporary one-time code, which is usually generated in an authenticator app installed on a mobile device.
I will use Ubuntu Linux and the Google Authenticator app on a smartphone to generate a time-based one-time password (TOTP). You can use the same idea to enable 2FA on any other Linux distro and use any other authenticator app instead of Google.
Create a temporary admin user [optional]
Note that this step is optional. We are creating a temporary admin user on the Linux system to avoid getting locked out permanently if SSH access breaks due to an incorrect 2FA setup. If you can access your Linux system without SSH (KVM, for instance, then you can skip this step, but if your Linux system is in the cloud and there is no alternative to SSH, it is highly recommended to create a temporary admin user.
To create a temporary admin user, run the following commands:
sudo adduser tempadmin sudo usermod -aG sudo tempadmin
Once you successfully confirm that the 2FA setup is working as expected, delete this user.
Time synchronization
Before enabling 2FA, it is really important that the clocks on your Linux system and mobile device are synchronized, which means that both should have the same time in their clocks. However, a nominal time difference of a few seconds (e.g., 30 seconds or less) is usually acceptable.
The best way to keep the time synchronized is using the network time protocol (NTP). By default, Ubuntu uses timedatectl/timesyncd for time synchronization, and we will stick with the default. To verify whether the NTP service is enabled and the system's clock is properly synchronized, use the timedatectl command.
If you see System clock synchronized: no and NTP service: inactive, as shown in the screenshot, run the following command:
sudo timedatectl set-ntp true
This command enables the NTP service and synchronizes the system's clock. Now repeat the timedatectl command once again and confirm that the output is as shown in the screenshot.
You will notice that the system clock is now properly synchronized. Similarly, you can also set your mobile device to use a network-provided time instead of manually setting it.
Install Google Authenticator
After synchronizing the clocks on your Linux system and smartphone, head over to the Linux console and run the following command to install the Google Authenticator PAM packages:
sudo apt install libpam-google-authenticator -y
Install the Authenticator app on a mobile device
The next step is to install the Google Authenticator app on your mobile phone or tablet. By the way, you can also use any other authenticator app, such as Microsoft Authenticator or Authy. I will install the Google Authenticator on my Android phone.
Add the Linux system to the Authenticator app
Now run the Google Authenticator app on the mobile device. Click the Get started button, and you will see two ways of adding a device:
- Scan a QR code
- Enter a setup key
In the Linux console, run the Google Authenticator package you installed earlier on your Linux server. By default, it stores the secret key in the user's home directory (in ~/.google_authenticator file), but we will save it in ~/.ssh directory. If ~/.ssh directory doesn't already exist on your Linux system, simply use the ssh-keygen command to create it before actually running the next command. Now run the following command in the Linux console:
google-authenticator -s ~/.ssh/google_authenticator
This command will run the Google Authenticator PAM module. The -s option allowed us to save the secret key in a nonstandard location, and we specified the ~/.ssh directory where SSH keys are kept. When you run this command, you will see a QR code on screen if QR code libraries are supported. Now open the Google Authenticator app in your mobile, tap the Scan a QR code option, and scan the QR code displayed on the Linux server console.
If your Linux server doesn't support QR codes or your mobile device can't scan the QR code, you could use the secret key option. To do so, tap the Enter a setup key option in the Google Authenticator app, and use the secret key displayed on the Linux console to add the account in the Authenticator app.
Once your account is added to the Authenticator app, type the TOTP from the Authenticator app in the Linux console and press Enter. When the code is confirmed, you will see the emergency scratch codes (or backup codes) displayed on the Linux console. Store these backup codes in a safe place, as you might need them to get your SSH access back if something goes wrong with your phone in the future. Furthermore, you will be prompted by a series of questions during the setup. Make sure you answer these questions, as shown in the screenshot:
Configure 2FA for SSH
The next step is to modify the PAM configuration file for the SSH daemon (/etc/pam.d/sshd). Run the following command to open the config file:
sudo nano /etc/pam.d/sshd Now add the following line at the end: auth required pam_google_authenticator.so secret=/home/${USER}/.ssh/google_authenticator nullok
Notice how the path of the google_authenticator secret key is supplied using secret=/home/${USER}/.ssh/google_authenticator. If you store your secret key file in a different location, make sure you supply the correct path here. Also note that the nullok option will allow users to log in with just the username and password until they complete the 2FA setup. Once the 2FA configuration is successfully completed for all users, it is a good idea to remove the nullok option to enforce the use of 2FA for everyone. If you are the only user on this system, you can skip adding the nullok option.
Now, open the SSH daemon configuration file (/etc/ssh/sshd_config) using the following command:
sudo nano /etc/ssh/sshd_config
- If you're running Ubuntu 22.04 or later, add the following line in the sshdconfig file:
KbdInteractiveAuthentication yes
- If you're running Ubuntu 21.10 or earlier, add the following line instead:
ChallengeResponseAuthentication yes
In either case, if the suggested line already exists in the sshdconfig file, make sure you uncomment it and change it to "yes". By doing so, you're essentially enabling keyboard-interactive (or challenge response) authentication for the SSH daemon. See the following screenshots for reference:
Finally, restart the SSH daemon using the following command:
sudo systemctl restart sshd
Make sure you don't close the current Linux console session until you confirm the 2FA setup is properly working. If you close this session and 2FA doesn't work, you might get yourself locked out. If this happens, you need to log in with the temporary admin user we created earlier.
Verify the 2FA setup
At this point, your Ubuntu Linux system is ready with two-factor authentication. To verify if everything is working, launch a new terminal, and try to access the Ubuntu system using SSH. Assuming you already have an SSH or console session open to your Ubuntu system, you can run the sudo tail -f /var/log/auth.log command to view the SSH access logs at the same time.
Alongside, open a new SSH session to the Ubuntu system, and you will notice that after entering your username and password, you will now be prompted for a Verification Code (see the screenshot below).
You need to use the verification code generated in the Google Authenticator app on your mobile device for successful authentication. If you enter the wrong verification code, it will prompt you for a password again. Now, if you take a look at the SSH access logs (shown in the screenshot below), you will see a log indicating invalid verification code. This is because I intentionally typed the wrong verification code the first time. When the correct verification code is entered, authentication is successful, and the SSH session is opened.
This is it. The SSH access on your Ubuntu Linux system is now protected with 2FA.
Enable 2FA for public key authentication [optional]
The above steps enabled two-factor authentication for SSH access using password authentication. If you are using public key authentication and want to protect it with 2FA, open the /etc/ssh/sshd_config file again and add these lines at the end:
PasswordAuthentication no PubkeyAuthentication yes AuthenticationMethods publickey,keyboard-interactive
Troubleshooting
If you can't access the Linux system using SSH, make sure you followed all the steps correctly, and the clocks on both the Linux server and the mobile device are properly synced. If the clock on your Linux system is properly synced using NTP, but you still see an invalid verification code in the SSH logs, the problem is most likely with the time synchronization on your mobile device. Follow these steps to fix it:
Subscribe to 4sysops newsletter!
- On your mobile device, open the Settings of the Google Authenticator app.
- Tap the Time correction for codes option.
- Tap the Sync now option. This will sync the app's time with the Google servers, of course, without affecting your phone's time.
Conclusion
That was it for this guide. You just learned how to enable two-factor authentication for SSH login in Linux. When 2FA starts working properly, don't forget to delete the temporary admin user that we created at the beginning, and make sure you disable root login via SSH to further improve the security.
Very nice article.
– I have been doing this on my linux servers. Just to add, if you use a server admin tool such as Webmin in the Debian distros it has a module for MFA.
– The RHL derivatives use a similar tool such as cockpit. I have not been able to set up MFA on that.So I use SSH and MFA.
This is great!
In the article you point out the following…
“Once your account is added to the Authenticator app, type the TOTP from the Authenticator app in the Linux console and press Enter. When the code is confirmed, you will see the emergency scratch codes (or backup codes) displayed on the Linux console. Store these backup codes in a safe place, as you might need them to get your SSH access back if something goes wrong with your phone in the future. Furthermore, you will be prompted by a series of questions during the setup. Make sure you answer these questions, as shown in the screenshot:”
Do you have recommendations where to save these? I can see doing this for a cloud based install where you might want consultants or MSP to have access to the system. It seems like a vault of some type in a business owned on-prem location would be best but I am wondering if there is a standard.
Also, what happens if the google service is down or the app doesn’t have proper access to the Internet? (i.e. phone is dead/broken) What is plan B?
As I understand it, the app does not need internet connection afterwards. The authenticator on Linux and the Google authenticator both generate random 6 digit code based on algorithm such as sha1 or sha256 they use a common salt and the time variable (that is why it is important to sync as Surender has explained) to generate the codes. At any instant these independently generated codes must match.
– Storing the emergency codes is a good practice. May be relevant to the discussion, you can also use any TOTP generator – e.g., I use Microsoft authenticator, KeepPassXC or even an online generator such as https://totp.danhersam.com/. Make sure you store the secret during the set up safe.
– I have an article on managing the MFA code/secret: https://www.linkedin.com/pulse/how-manage-mfa-accounts-note-myself-ratan-mohapatra/
Hi @jobc,
The emergency scratch codes are the plan B here. Each user that you enable 2FA for, gets a ~/.ssh/google_authenticator file containing backup codes that you could use to login in case of emergency when you can’t use your phone.
Storing backup codes in the cloud could be debatable. I prefer storing mine in VeraCrypt encrypted vaults that are completely isolated from rest of network/internet for a complete peace of mind.
Let me share a different approach:
You can use a putty variant called pageant (“putty ageant 0.77”) to use SmartCards that hold the keys for domain accounts. That is less complicated to setup, if your goal is to connect windows->ssh->linux
https://github.com/NoMoreFood/putty-cac/releases/tag/0.77u1
How does this accommodate MFA? Using a smartcard (something you have) instead of the smart phone.
In looking at the documentation for pagent.exe wouldn’t this be just storing a key on a system?
https://tartarus.org/~simon/putty-snapshots/htmldoc/Chapter9.html#pageant
Yeah, MFA is the best security I can think of at present- may be in future the will use a biometric option (useless on VMs anyway). Storing the passwords on a computer or a software agent also restricts your access from that computer only. Options such as public key (stored on Linux) and private-key (stored on the computer from where you access) is also used popularly. Again there is no substitute for the freedom of access with MFA. I need not bother about super-complex passwords or change them often them as long as I handle my MFA code safely.
How? The Smartcard works with a PIN.