Enabling the talk daemon on Fedora 20

well, it's been a few years and as technology changes, so does the methods used to configure a system.

I still use the talk program on a regular basis.  Here are the instructions for enabling it:

# yum install xinetd talk-server talk

# systemctl enable xinetd.service
# systemctl enable ntalk.service

At this poing, simply starting the xinetd and ntalk services does not seem to allow the talk program to function.  At the moment, the only solution I had was to reboot the system.  If someone has a better way, I would very much like to know.

# reboot

Talk should now work.  However, there is a chance that SELinux will deny it.  Check your logs:

# grep -i denied /var/log/audit/audit.log

If you do get a denial you will need to build a new policy.  Make sure you have the following utility installed: checkpolicy

# yum install checkpolicy

# grep in.ntalkd /var/log/audit/audit.log | audit2allow -M mypol

# semodule -i mypol.pp

That's it.

Block access to files by IP using X-Forwarded-For

What is the purpose of blocking by X-Forwarded-For IP, instead of the REMOTE_ADDR?

Sometimes a site may be behind a reverse proxy and it may not be possible to add a rule to block a file by IP at the reverse proxy level.  If the reverse proxy is passing the remote client IP in a header like X-Forwarded-For, you can still block by client IP.

Match the header to an IP address and assign it to an environment variable in Apache.  Here is an example of a complete configuration to block remote access to a wordpress login page, except for a certain range of IPs:

<files wp-login.php>
order deny,allow
deny from all
SetEnvIf X-Forwarded-For "192\.168\..*" LocalAccess
SetEnvIf X-Forwarded-For "10\..*" LocalAccess
Allow from env=LocalAccess
</files>

If the IP contained in the X-Forwarded-For header matches one of the regular expressions, it will populate the "LocalAccess" environment variable.

Misc Notes

Securely delete files in linux:

# srm --help

# shred --help

---------

Bash history tweaks:

1) Increase history size in Redhat / Fedora / CentOS:

# vim /etc/profile
...
HISTSIZE=100000
...

2) Append history from multiple terminals:

# vim /etc/bashrc
    ...
    fi
    # Turn on append history
    shopt -s histappend
    history -a
    # Turn on checkwinsize
    ...

---------

Copy your public key to a remote host:

$ ssh-copy-id -i ./.ssh/id_rsa.pub remote.domain.com

Modify MySQL variables without restarting the server

Reminder note on changing MySQL variables live and without restarting the service.

mysql> show global variables where variable_name like '%engine%';
+---------------------------+--------+
| Variable_name             | Value  |
+---------------------------+--------+
| engine_condition_pushdown | ON     |
| storage_engine            | MyISAM |
+---------------------------+--------+
2 rows in set (0.00 sec)

mysql> SET GLOBAL storage_engine=InnoDB;
Query OK, 0 rows affected (0.00 sec)

mysql> show global variables where variable_name like '%engine%';
+---------------------------+--------+
| Variable_name             | Value  |
+---------------------------+--------+
| engine_condition_pushdown | ON     |
| storage_engine            | InnoDB |
+---------------------------+--------+
2 rows in set (0.00 sec)

Very simple, but I can never remember this and it tends to be buried somewhere in MySQL's massive documentation.

Granular permissions through sudoers

A quick example on how to provide root permissions on specific commands to a specific group of users.

You can create command aliases, which can be very useful when formatting and controlling access to these.

For example:

Cmnd_Alias vi    = /usr/bin/vim

This will match both /usr/bin/vim or just plain vim.

Assigning ROOT permissions to run this command alias to a specific user:

username ALL=(root) vi

And the same for a group:

%groupname ALL=(root) vi

In my example below, I provide access to use all the NGINX service commands on a redhat 6 system, to a new group called nginxadm.

Open up the sudoers file using visudo.

## NGINX USERS - should be part of nginxadm group
# Usage: nginx {start|stop|restart|condrestart|try-restart|force-reload|upgrade|reload|status|help|configtest}
Cmnd_Alias NG           = /sbin/service nginx
Cmnd_Alias NGRES        = /sbin/service nginx restart
Cmnd_Alias NGSTA        = /sbin/service nginx start
Cmnd_Alias NGSTO        = /sbin/service nginx stop
Cmnd_Alias NGSTS        = /sbin/service nginx status
Cmnd_Alias NGCDR        = /sbin/service nginx condrestart
Cmnd_Alias NGTRS        = /sbin/service nginx try-restart
Cmnd_Alias NGFRL        = /sbin/service nginx force-reload
Cmnd_Alias NGUPG        = /sbin/service nginx upgrade
Cmnd_Alias NGRLD        = /sbin/service nginx reload
Cmnd_Alias NGHLP        = /sbin/service nginx help
Cmnd_Alias NGCFG        = /sbin/service nginx configtest
%nginxadm ALL=(root)    NG,NGRES,NGSTA,NGSTO,NGSTS,NGCDR,NGTRS,NGFRL,NGUPG,NGRLD,NGHLP,NGCFG

Thanks to FACLs in Linux, we can also give granular permissions to the NGINX configuration files.

IBM DB2 connector - executable stack error

I ran into this issue following the installation of a new IBM DB2 connector library for use with PHP on an Apache server.

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/ibm_db2.so' - libdb2.so.1: cannot enable executable stack as shared object requires: Permission denied in Unknown on line 0

As it turns out, Dan Walsh already wrote up something about this on his blog in 2011:

http://danwalsh.livejournal.com/38736.html

Following his instructions, I've opted to clear the executable stack flag.  I've modified his one liner command to execute the 'clear' command on all the files found.  It would be tedious to change them one by one.

find /opt/ibm/db2/V9.7/lib64 -exec execstack -q {} \; -print 2> /dev/null | grep ^X | cut --delimiter=' ' -f2 | xargs -n 1 execstack -c

This fixed the problem.

List users in linux

Here is a quick tip on how to more or less reliably list all non-system users and print out status information about their accounts, in linux.

grep -i ':/home/' /etc/passwd | cut -d: -f1 | xargs -n 1 passwd -S

Step 1:
We use grep to look for entries in the /etc/passwd file where the /home/ directory is specified.  System users don't normally have such a directory.

Step 2:
We then use cut to retain only the username, and we specify : (colon) as a delimiter.

Step 3:
We run the usernames through xargs to pass each username as a single argument to the passwd -S command.

passwd -S prints out short status of the username.  Whether it's locked, expired, etc...

Using GDB to explore a core file

Before we go any further, please note that these are instructions for a Centos 6.3 OS.  They should work in a similar way on Redhat and Fedora.

Recently, I've been experiencing issues with pacemaker immediately after ugprading from one version to another.  To make a long story short, crmd, which is part of the pacemaker package, was crashing every 15 minutes and dumping a core file.  It became necessary to read the details of the core dump using gdb.  To be fair though, the service itself stayed up fine but kept respawning child processes.  So, even though it complains and dumps a core file, it is still robust enough to continue operating.

Now the issue becomes, how to get the error content out of the core file?  We first need to make sure the right repositories are available for our OS:

# DEBUGINFO
[debuginfo]
name=debuginfo
baseurl=http://debuginfo.centos.org/$releasever/$basearch
enabled=0
gpgcheck=0

You will need a tool called 'debuginfo-install' which is part of a yum utility package.

# yum install yum-utils

Install gdb

# yum install gdb

Run gdb against your executable and it's core file.  For example:

# gdb /usr/libexec/pacemaker/crmd ./core.26688

If you are missing any debuginfo file, gdb will let you know which ones.  You can then install them as required.  For example:

Missing separate debuginfos, use: debuginfo-install audit-libs-2.2-2.el6.x86_64

Simply run the suggested command to get the right debuginfo files.

# debuginfo-install audit-libs-2.2-2.el6.x86_64 --enablerepo=debuginfo

Once you've got all your debug info files, run the gdb debugger once again.

# gdb /usr/libexec/pacemaker/crmd ./core.26688
...
Core was generated by `/usr/libexec/pacemaker/crmd'.
Program terminated with signal 6, Aborted.
#0  0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install libtool-ltdl-2.2.6-15.5.el6.x86_64
(gdb)

At this point you are in the gdb prompt.  Use "bt" to output the backtrace of the error.

(gdb) bt
#0  0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f81896ae085 in abort () at abort.c:92
#2  0x00007f818bb8a56b in crm_abort (file=0x7f818bba9d58 "xml.c", function=0x7f818bbab6b4 "string2xml", line=650,
    assert_condition=0x7f818bbaa01a "String parsing error", do_core=, do_fork=) at utils.c:1073
#3  0x00007f818bb933af in string2xml (
    input=0x1e745f8 "#4  0x00007f818b76a2fc in lrmd_ipc_dispatch (buffer=, length=, userdata=0x1e72910) at lrmd_client.c:310
#5  0x00007f818bba2e90 in mainloop_gio_callback (gio=, condition=G_IO_IN, data=0x1e73be0) at mainloop.c:585
#6  0x00007f8188fbbf0e in g_main_dispatch (context=0x1d4f120) at gmain.c:1960
#7  IA__g_main_context_dispatch (context=0x1d4f120) at gmain.c:2513
#8  0x00007f8188fbf938 in g_main_context_iterate (context=0x1d4f120, block=1, dispatch=1, self=) at gmain.c:2591
#9  0x00007f8188fbfd55 in IA__g_main_loop_run (loop=0x1e734a0) at gmain.c:2799
#10 0x00000000004052ce in crmd_init () at main.c:154
#11 0x00000000004055cc in main (argc=1, argv=0x7fffe77a4f88) at main.c:120
(gdb)

This is the information we were looking for.

Thanks to Andrew Beekhof of  http://clusterlabs.org/ for pointing me in the right direction with GDB.

Log file system changes using Audit

Adding an audit rule to log file-system activity:

# auditctl -w /home/something/ -p rwa

Flags:

-w Insert watch
-p Set the permission filter

If you search the audit log, you will only get results if there has been activity:

# ausearch -f /home/something/
<no matches>

Now, if we create a new file and search again:

# touch /home/something/testing1

# ausearch -f /home/something/
----
time->Tue Apr  9 08:53:08 2013
type=PATH msg=audit(1365511988.313:969510): item=1 name="/home/something/testing1" inode=54411 dev=08:15 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=user_u:object_r:user_home_t:s0
type=PATH msg=audit(1365511988.313:969510): item=0 name="/home/something/" inode=54374 dev=08:15 mode=040700 ouid=1041 ogid=1041 rdev=00:00 obj=user_u:object_r:user_home_dir_t:s0
type=CWD msg=audit(1365511988.313:969510):  cwd="/root"
type=SYSCALL msg=audit(1365511988.313:969510): arch=c000003e syscall=2 success=yes exit=0 a0=7fff217f4cb6 a1=941 a2=1b6 a3=32cc35410c items=2 ppid=10799 pid=26986 auid=1041 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=130958 comm="touch" exe="/bin/touch" subj=user_u:system_r:unconfined_t:s0 key=(null)

Let's delete the file and see what happens to the logs.

# rm /home/something/testing1

Run the search again:

# ausearch -f /home/something/

Notice the syscall key which is displayed in each entry.  What does the code mean?  Let's make things more 'readable' by setting the -i flag.

# ausearch -f /home/something/ -i
----
type=PATH msg=audit(04/09/2013 09:00:52.093:969611) : item=1 name=/home/something/testing1 inode=54411 dev=08:15 mode=file,644 ouid=root ogid=root rdev=00:00 obj=user_u:object_r:user_home_t:s0
type=PATH msg=audit(04/09/2013 09:00:52.093:969611) : item=0 name=/home/something/ inode=54374 dev=08:15 mode=dir,700 ouid=something ogid=something rdev=00:00 obj=user_u:object_r:user_home_dir_t:s0
type=CWD msg=audit(04/09/2013 09:00:52.093:969611) :  cwd=/root
type=SYSCALL msg=audit(04/09/2013 09:00:52.093:969611) : arch=x86_64 syscall=unlink success=yes exit=0 a0=7fff2c3b2cbc a1=1 a2=2 a3=168f7610 items=2 ppid=10799 pid=27302 auid=something uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=130958 comm=rm exe=/bin/rm subj=user_u:system_r:unconfined_t:s0 key=(null) 

The timestamps are now all converted to human-readable formats.  The UIDs and GIDs are converted to the name of the user or group and finally, the system call codes show something intelligible.  For example, syscall=87 now reads syscall=unlink which which we can interpret as 'delete'.

You can search by system call codes as well.  Instead of displaying all activity on all files and reading through each entry one by one, you can search for 'unlink' system calls.

The flag is -sc <syscall>

For example, the following command will return the log(s) entry(ies) showing an 'unlink' call.

# ausearch -f /home/something/testing1 -i -sc unlink

To remove a watch, use the -W flag.  Note, when using this flag, the remove (-W) command must match the rule.  If you don't know the exact rule, you can list them:

# auditctl -l

LIST_RULES: exit,always dir=/home/something (0xe) perm=rwa

Now we can delete it using:

# auditctl -W /home/something -p rwa