Renewing a reverse proxy cert with certbot

Renewing a cert on a loadbalancer with certbot is fairly easy, but there is a minor trick to it.  Since your reverse proxy is not hosting any website, and instead proxying all requests, certbot is unable to post the verification key file on the server.

To get around this, shutdown the reverse proxy server temporarily and have certbot run as a standalone server in order to facilitate the authentication:

# certbot renew --standalone

This allows certbot to run its own server on port 80,443 while it authenticates and shuts down immediately once its done.  You can then restart your own reverse proxy server.

Running virt-manager on wayland

Fedora 26 introduces Wayland as the new display server protocol, and this introduces new challenges.

The wayland developers have chosen not to support running a GUI application as root: which breaks virt-manager and virt-viewer.  Both applications need to run as root, in order for them to execute commands such as mount and etc...

There seems to be very little way around this, except to temporarily allow root execution using:

$ xhost +si:localuser:root

Permission can later be revoked using:

$ xhost -si:localuser:root

I hope that a more elegant solution will be forthcoming from either the wayland or libvirt developers.

Recording Audio in CentOS

Today, I needed a tool to quickly and effectively record audio from a microphone connected to my computer, in order to test it.  I didn't have time to start playing around with Audacity settings to figure out how to get it to work properly, so I decided to look around for something else.  As it turns out, there is a nice little command line tool which allows for quick and dirty recording: sox.

All I had to do was:

$ sox -t alsa default out.wav

Bingo, instant recording for very basic purposes.  I'm sure the manual page will yield much more interesting features.

Kerberos User Principal not found. Do you have a valid Credential Cache?

NOTE "Kerberos User Principal not found. Do you have a valid Credential Cache?"

Reading through Sander Van Vugt's book (RHCSA/RHCE 7), I came across an issue while setting up Kerberos for NFS.  It is detailed in Appendix D on the CD that came with the book: Appendix D, "Preparing Your IPA Server for Kerberized NFS".

Step 8 of the book states:

"On server1, type ipa-getkeytab -s ipa.example.com -p nfs/server1.
example.com -k /etc/krb5.keytab."

Following these instructions only yielded the error below:

Kerberos User Principal not found. Do you have a valid Credential Cache?

According to the Kerberos documentation it is necessary to request a ticket before proceeding, therefore running the following command prior to Step 8 should resolve the issue:

[root@server1 ~]# kinit admin
Password for admin@EXAMPLE.COM: (enter the password)

You can then retry the command from Step 8.

[root@server1 ~]# ipa-getkeytab -s ipa.example.com -p nfs/server1.example.com -k /etc/krb5.keytab

Failed to parse result: Failed to decode GetKeytab Control.
Retrying with pre-4.0 keytab retrieval method...
Keytab successfully retrieved and stored in: /etc/krb5.keytab

The command was successful... this is another reminder for me that books can and will contain mistakes - and that the online documentation (manual pages) are an excellent resource.

Updating system time using chrony on CentOS 7

HOW TO: If the current system time is completely off by several months or years, using chrony to update the time can be a bit tricky.  The online documentation does not clearly explain how to handle this type of scenario, and probably with good reason.  Why would you use an NTP system to update a device that has gone off the correct time by more than several minutes to hours?  Any further wider time drift means that something could be seriously wrong hardware-wise, and whats worse data and security could have been affected.  That's not what ntpd or chronyd are designed for; rather they are designed to prevent time-drift by seconds, perhaps minutes.

The correct way to update the time on a RHEL/CentOS 7 that is completely off, is to set it manually and then use an NTP daemon to maintain it.

Setting the time and date manually to the current time:

# timedatectl set-time "2017-09-16 11:51:58"

Next, simply make sure chronyd or ntpd is running - not both:

# systemctl status chronyd

---

Ok, suppose we really want to update a system clock that has gone off by a ridiculous amount of time, lets say a year or more... there is a way:

Note: It is tricky to correct such a gap using chrony because it works by increasing the clock speed or reducing the clock speed to catch up.  Obviously this is not efficient, hence why you should set it manually as demonstrated above.

However, ignoring all of the above, the first step is to restart the chronyd service:

# systemctl restart chronyd

In most cases this will not immediately update the system time, the service will be aware of the correct time internally.  Using timedatectl will show that the system time has not been changed yet.

Next you will need to force update the time by "making the step":

# chronyc -a makestep

At this point the system time should be fully updated and you can verify this with:

# timedatectl

Finally you can update the hardware clock with:

# hwclock --systohc

I've asked myself more than once while writing this post, why would this even matter... Realistically, one can find herself/himself in a position where this type of knowledge can sometimes be useful and its nearly impossible to know when.

Rescue a CentOS 7 system with a deleted /boot directory

HOW TO: Your /boot directory is missing or deleted on CentOS 7, you can't boot!  Imagine this type of situation happening on a real production system.  It is unlikely to happen, but it is always good to know how to recover from such a disastrous failure.  Even the most resilient systems can have storage failures.  Here's an article on silent corruption: http://perspectives.mvdirona.com/2012/02/observations-on-errors-corrections-trust-of-dependent-systems/

This scenario is based on several tests I've performed on KVM based virtual machines.

Boot the system with a rescue DVD (or ISO for a VM).

At the CentOS 7 boot CD prompt, choose "Troubleshooting" and "Rescue a CentOS system".  Next choose "Continue" to allow the rescue environment to mount the machine's file systems under "/mnt/sysimage". 

At the prompt you will be in a shell loaded by the boot CD.

sh-4.2# ...

Since we want to work directly with the broken system we will chroot to the mounted FS.

# chroot /mnt/sysimage

Check the state of the boot directory:

# ls -la /boot

At this point if the boot partition was corrupted you could run either parted, gdisk or fdisk to recreate the partition.  You could also run fsck to run a filesystem check.

In my case /boot was fine, but empty.

HOW TO FIX A MISSING KERNEL:

Now we need to re-install the kernel... However, the kernel version installed is later than the one on the installation CD.  There are several things we can do at this point, but I will outline two:

1. Install the old kernel from the CD or,
2. Start the network and install the latest kernel from yum.

NOTE: Once the kernel is reinstalled through RPM or YUM, the installation triggers dracut which re-generates the necessary initramfs files.

RE-INSTALL THE KERNEL FROM THE BOOT CD: (skip if you want to use yum and the network)

Mount the boot CD to the /run/install/repo directory:

# mount /dev/sr0 /run/install/repo

# rpm -ivh --force /run/install/repo/Packages/kernel-<...version and arch...>

RE-INSTALL THE KERNEL FROM THE NETWORK: (skip if you re-installed the kernel from the CD already)


Luckily the network configuration is sound so we can simply start the network device and use yum to reinstall the kernel:

# service network start

Run a yum clean all just in case.

# yum clean all

Reinstall the kernel.

# yum reinstall kernel

RE-INSTALL GRUB:

Run ls -la /boot to verify the /boot directory and you should see the new kernel and associated files listed.  Most of the /boot directory's missing files and directories will be created.  One key portion that will be missing is Grub2.

# ls -la /boot

So we now need to reinstall grub2 and to recreate the configuration.  This process is fairly simple.

Install grub2 on /dev/  -- in my case on a KVM it's /dev/vda

# grub2-install /dev/vda

If no errors were reported, you are ready to reconfigure grub (otherwise you'll need to troubleshoot why you can't write to your device.):

# grub2-mkconfig -o /etc/grub2.cfg

(While the real grub2.cfg file is actually in the /boot/ partition, /etc/grub2.cfg is a symlink and easier to reference - especially if you are using UEFI.  If you are using UEFI the grub2.cfg filename is actually /etc/grub2-efi.cfg -> ../boot/efi/EFI/centos/grub.cfg)

 Next since you are in a chroot shell you need to exit before you can reboot:

# exit

# reboot

In theory your system should now be able to boot just fine, but the SeLinux relabeling will have been triggered and may take some time to complete.  Once done your system will reboot automatically one more time.

If you had multiple kernels installed, but chose to fix this system by installing the base one from the CD, you can install your version again by running:

# yum reinstall kernel-

If you don't know which kernels you had previously installed, you can get the version from the rpm query command:

# rpm -q kernel

There you are...

-----

There are other steps we could have taken to restore / install a kernel, however in general they are quite similar.  Mainly, the differences would be where to get the Kernel RPM from.  Since the version originally installed on the system can be different from the ones available on media or through  yum, it may sometimes be necessary to download a specific kernel and install it manually.

One could even re-compile the kernel but its probably not such a great idea if we are working on a production server.  The main problem is that it would require downloading all the sources and headers required, as well as compilation tools.  Due to security concerns, it would be best not to install compilation tools on a production server as they could be used to gain elevated privileges in the event of a limited intrusion.

CVE-2017-1000251 - bluetooth vulnerability

Given the scope of this vulnerability, its probably a good idea to disable bluetooth until all devices are patched.

https://access.redhat.com/security/vulnerabilities/blueborne

On RH based systems:

Mask the service, just in case, this will prevent another systemd unit from attempting to load it.

# systemctl mask bluetooth.service

Stop the service if it is running:

# systemctl stop bluetooth.service

ssh-chat - irc-like chat client over SSH

How to securely chat over SSH?  In this post I discuss one of the latest solutions I've discovered - A very nice piece of software: ssh-chat.

For a long time I've been using and maintaining an active "talk" client/server on one of my systems in order to be able to communicate and collaborate securely over SSH with whoever I needed to.

Unfortunately this is not a perfect solution for many reasons.  I've been thinking of setting up a local IRC server but there are weaknesses with that as well.

Recently I found an interesting project on github, created by a very ingenious programmer who goes by the alias of shazow.  His project, written in Go: ssh-chat.

https://github.com/shazow/ssh-chat

It uses the go libraries for most of the SSH client/server code, but it adds a custom terminal which has a similar look and feel as IRC.

It's very well written and requires very little work to get it up and running.  The only thing I did was install it on a small KVM server and got it to start up automatically.

Here is the systemd file I created for it, located at:

/etc/systemd/system/ssh-chat.service

Content:

[Unit]
Description=SSH-CHAT service

[Service]
Type=simple
ExecStart=//ssh-chat/ssh-chat -i //.ssh/id_rsa

[Install]
WantedBy=multi-user.target

A couple of issues with this solution:

1) While it is an interesting idea, I need to keep an eye out on the golang SSH client/server libraries to make sure security vulnerabilities are kept at bay.

2) Keep in mind that like 'talk,' the conversations on the local server are not necessarily encrypted and could potentially be captured if the server is compromised.

Apart from these reservations, I really like this client and will look into it further as a potential solution.