Kerberos User Principal not found. Do you have a valid Credential Cache?

NOTE "Kerberos User Principal not found. Do you have a valid Credential Cache?"

Reading through Sander Van Vugt's book (RHCSA/RHCE 7), I came across an issue while setting up Kerberos for NFS.  It is detailed in Appendix D on the CD that came with the book: Appendix D, "Preparing Your IPA Server for Kerberized NFS".

Step 8 of the book states:

"On server1, type ipa-getkeytab -s ipa.example.com -p nfs/server1.
example.com -k /etc/krb5.keytab."

Following these instructions only yielded the error below:

Kerberos User Principal not found. Do you have a valid Credential Cache?

According to the Kerberos documentation it is necessary to request a ticket before proceeding, therefore running the following command prior to Step 8 should resolve the issue:

[root@server1 ~]# kinit admin
Password for admin@EXAMPLE.COM: (enter the password)

You can then retry the command from Step 8.

[root@server1 ~]# ipa-getkeytab -s ipa.example.com -p nfs/server1.example.com -k /etc/krb5.keytab

Failed to parse result: Failed to decode GetKeytab Control.
Retrying with pre-4.0 keytab retrieval method...
Keytab successfully retrieved and stored in: /etc/krb5.keytab

The command was successful... this is another reminder for me that books can and will contain mistakes - and that the online documentation (manual pages) are an excellent resource.

Updating system time using chrony on CentOS 7

HOW TO: If the current system time is completely off by several months or years, using chrony to update the time can be a bit tricky.  The online documentation does not clearly explain how to handle this type of scenario, and probably with good reason.  Why would you use an NTP system to update a device that has gone off the correct time by more than several minutes to hours?  Any further wider time drift means that something could be seriously wrong hardware-wise, and whats worse data and security could have been affected.  That's not what ntpd or chronyd are designed for; rather they are designed to prevent time-drift by seconds, perhaps minutes.

The correct way to update the time on a RHEL/CentOS 7 that is completely off, is to set it manually and then use an NTP daemon to maintain it.

Setting the time and date manually to the current time:

# timedatectl set-time "2017-09-16 11:51:58"

Next, simply make sure chronyd or ntpd is running - not both:

# systemctl status chronyd

---

Ok, suppose we really want to update a system clock that has gone off by a ridiculous amount of time, lets say a year or more... there is a way:

Note: It is tricky to correct such a gap using chrony because it works by increasing the clock speed or reducing the clock speed to catch up.  Obviously this is not efficient, hence why you should set it manually as demonstrated above.

However, ignoring all of the above, the first step is to restart the chronyd service:

# systemctl restart chronyd

In most cases this will not immediately update the system time, the service will be aware of the correct time internally.  Using timedatectl will show that the system time has not been changed yet.

Next you will need to force update the time by "making the step":

# chronyc -a makestep

At this point the system time should be fully updated and you can verify this with:

# timedatectl

Finally you can update the hardware clock with:

# hwclock --systohc

I've asked myself more than once while writing this post, why would this even matter... Realistically, one can find herself/himself in a position where this type of knowledge can sometimes be useful and its nearly impossible to know when.

Rescue a CentOS 7 system with a deleted /boot directory

HOW TO: Your /boot directory is missing or deleted on CentOS 7, you can't boot!  Imagine this type of situation happening on a real production system.  It is unlikely to happen, but it is always good to know how to recover from such a disastrous failure.  Even the most resilient systems can have storage failures.  Here's an article on silent corruption: http://perspectives.mvdirona.com/2012/02/observations-on-errors-corrections-trust-of-dependent-systems/

This scenario is based on several tests I've performed on KVM based virtual machines.

Boot the system with a rescue DVD (or ISO for a VM).

At the CentOS 7 boot CD prompt, choose "Troubleshooting" and "Rescue a CentOS system".  Next choose "Continue" to allow the rescue environment to mount the machine's file systems under "/mnt/sysimage". 

At the prompt you will be in a shell loaded by the boot CD.

sh-4.2# ...

Since we want to work directly with the broken system we will chroot to the mounted FS.

# chroot /mnt/sysimage

Check the state of the boot directory:

# ls -la /boot

At this point if the boot partition was corrupted you could run either parted, gdisk or fdisk to recreate the partition.  You could also run fsck to run a filesystem check.

In my case /boot was fine, but empty.

HOW TO FIX A MISSING KERNEL:

Now we need to re-install the kernel... However, the kernel version installed is later than the one on the installation CD.  There are several things we can do at this point, but I will outline two:

1. Install the old kernel from the CD or,
2. Start the network and install the latest kernel from yum.

NOTE: Once the kernel is reinstalled through RPM or YUM, the installation triggers dracut which re-generates the necessary initramfs files.

RE-INSTALL THE KERNEL FROM THE BOOT CD: (skip if you want to use yum and the network)

Mount the boot CD to the /run/install/repo directory:

# mount /dev/sr0 /run/install/repo

# rpm -ivh --force /run/install/repo/Packages/kernel-<...version and arch...>

RE-INSTALL THE KERNEL FROM THE NETWORK: (skip if you re-installed the kernel from the CD already)


Luckily the network configuration is sound so we can simply start the network device and use yum to reinstall the kernel:

# service network start

Run a yum clean all just in case.

# yum clean all

Reinstall the kernel.

# yum reinstall kernel

RE-INSTALL GRUB:

Run ls -la /boot to verify the /boot directory and you should see the new kernel and associated files listed.  Most of the /boot directory's missing files and directories will be created.  One key portion that will be missing is Grub2.

# ls -la /boot

So we now need to reinstall grub2 and to recreate the configuration.  This process is fairly simple.

Install grub2 on /dev/  -- in my case on a KVM it's /dev/vda

# grub2-install /dev/vda

If no errors were reported, you are ready to reconfigure grub (otherwise you'll need to troubleshoot why you can't write to your device.):

# grub2-mkconfig -o /etc/grub2.cfg

(While the real grub2.cfg file is actually in the /boot/ partition, /etc/grub2.cfg is a symlink and easier to reference - especially if you are using UEFI.  If you are using UEFI the grub2.cfg filename is actually /etc/grub2-efi.cfg -> ../boot/efi/EFI/centos/grub.cfg)

 Next since you are in a chroot shell you need to exit before you can reboot:

# exit

# reboot

In theory your system should now be able to boot just fine, but the SeLinux relabeling will have been triggered and may take some time to complete.  Once done your system will reboot automatically one more time.

If you had multiple kernels installed, but chose to fix this system by installing the base one from the CD, you can install your version again by running:

# yum reinstall kernel-

If you don't know which kernels you had previously installed, you can get the version from the rpm query command:

# rpm -q kernel

There you are...

-----

There are other steps we could have taken to restore / install a kernel, however in general they are quite similar.  Mainly, the differences would be where to get the Kernel RPM from.  Since the version originally installed on the system can be different from the ones available on media or through  yum, it may sometimes be necessary to download a specific kernel and install it manually.

One could even re-compile the kernel but its probably not such a great idea if we are working on a production server.  The main problem is that it would require downloading all the sources and headers required, as well as compilation tools.  Due to security concerns, it would be best not to install compilation tools on a production server as they could be used to gain elevated privileges in the event of a limited intrusion.

CVE-2017-1000251 - bluetooth vulnerability

Given the scope of this vulnerability, its probably a good idea to disable bluetooth until all devices are patched.

https://access.redhat.com/security/vulnerabilities/blueborne

On RH based systems:

Mask the service, just in case, this will prevent another systemd unit from attempting to load it.

# systemctl mask bluetooth.service

Stop the service if it is running:

# systemctl stop bluetooth.service

ssh-chat - irc-like chat client over SSH

How to securely chat over SSH?  In this post I discuss one of the latest solutions I've discovered - A very nice piece of software: ssh-chat.

For a long time I've been using and maintaining an active "talk" client/server on one of my systems in order to be able to communicate and collaborate securely over SSH with whoever I needed to.

Unfortunately this is not a perfect solution for many reasons.  I've been thinking of setting up a local IRC server but there are weaknesses with that as well.

Recently I found an interesting project on github, created by a very ingenious programmer who goes by the alias of shazow.  His project, written in Go: ssh-chat.

https://github.com/shazow/ssh-chat

It uses the go libraries for most of the SSH client/server code, but it adds a custom terminal which has a similar look and feel as IRC.

It's very well written and requires very little work to get it up and running.  The only thing I did was install it on a small KVM server and got it to start up automatically.

Here is the systemd file I created for it, located at:

/etc/systemd/system/ssh-chat.service

Content:

[Unit]
Description=SSH-CHAT service

[Service]
Type=simple
ExecStart=//ssh-chat/ssh-chat -i //.ssh/id_rsa

[Install]
WantedBy=multi-user.target

A couple of issues with this solution:

1) While it is an interesting idea, I need to keep an eye out on the golang SSH client/server libraries to make sure security vulnerabilities are kept at bay.

2) Keep in mind that like 'talk,' the conversations on the local server are not necessarily encrypted and could potentially be captured if the server is compromised.

Apart from these reservations, I really like this client and will look into it further as a potential solution.  

Compiling minetest on CentOS 7

HOW TO: The default gcc version of CentOS 7.3.1611 is gcc 4.8.5.  Minetest 0.4.16 requires a minimum of gcc 4.9 to compile.

--

Minetest is probably one of the best Open Source games of all times.  The project website is https://www.minetest.net/

While minetest can be compiled on many different OS'es, its easier on some than on others...  I found it easiest to compile on Fedora - all the libraries are readily available and the compilers are cutting edge.

This post doesn't go into resolving dependencies, rather it is a step-by-step document describing compiling minetest with an appropriate compiler.  The default gcc version of CentOS 7.3.1611 is gcc 4.8.5.  Minetest 0.4.16 requires a minimum of gcc 4.9 to compile:

Regarding dependencies, I think there are enough articles on the net that describe how to get what you need.  I will say this however: The third party repos that I normally use on CentOS are: Epel and RPMFusion.  Once these two are installed, getting the required dependencies is as easy as running either: yum search or yum provides */

COMPILING MINETEST:

Attempting to run cmake on the minetest project, renders:

"Insufficient gcc version, found 4.8.5.  Version 4.9 or higher is required. "

So, here is what we do in CentOS 7:

- Install the Software Collections "devtoolset-6"
- Provide CMAKE with the devtoolset compiler locations

Installing the "software collections" repositories,

# yum install centos-release-scl*

Enable the collection repository that we need:

# yum-config-manager --enable rhel-server-rhscl-7-rpms

Install SCL devtoolset-6 to get a newer version of GCC and G++.  

# yum install devtoolset-6

Go into your minetest build folder and enable the toolset's version of bash:

# scl enable devtoolset-6 bash

Unfortunately, since cmake is not included in the toolsets, we need to tell the base one where to find the right compilers, otherwise it tries to use the system's default.

The only method that I found which worked, was:

$ CXX=/opt/rh/devtoolset-6/root/usr/bin/g++ CC=/opt/rh/devtoolset-6/root/usr/bin/gcc cmake . -DRUN_IN_PLACE=TRUE ...

Now, you can exit the devtoolset-6 bash shell and compile normally.  Cmake generated all the necessary information to use the correct compilers regardless of your environment:

$ exit

$ make -j <# cpus>

Happy compiling!

P.S. Remember that when you run scl enable devtoolset-6 bash, you are in a new bash session with