Activate memory in Linux at run time using bash - vmware

In following with my previous post about "Adding scsi device at runtime on linux guest VM," I am adding some information here on how to use a bash "for-loop" to activate memory; it was added at runtime on a VMWare guest.

I've tested this on a CentOS 5.5 system.

First, add the new memory using VMWare VSphere.

Second, find the new memory that is currently listed as "offline".

[root@... ~]# grep offline /sys/devices/system/memory/*/state
/sys/devices/system/memory/memory40/state:offline
/sys/devices/system/memory/memory41/state:offline
/sys/devices/system/memory/memory42/state:offline
/sys/devices/system/memory/memory43/state:offline
/sys/devices/system/memory/memory44/state:offline
/sys/devices/system/memory/memory45/state:offline
/sys/devices/system/memory/memory46/state:offline
/sys/devices/system/memory/memory47/state:offline
/sys/devices/system/memory/memory48/state:offline
/sys/devices/system/memory/memory49/state:offline
/sys/devices/system/memory/memory50/state:offline
/sys/devices/system/memory/memory51/state:offline
/sys/devices/system/memory/memory52/state:offline
/sys/devices/system/memory/memory53/state:offline
/sys/devices/system/memory/memory54/state:offline
/sys/devices/system/memory/memory55/state:offline
/sys/devices/system/memory/memory56/state:offline
/sys/devices/system/memory/memory57/state:offline
/sys/devices/system/memory/memory58/state:offline
/sys/devices/system/memory/memory59/state:offline
/sys/devices/system/memory/memory60/state:offline
/sys/devices/system/memory/memory61/state:offline
/sys/devices/system/memory/memory62/state:offline
/sys/devices/system/memory/memory63/state:offline
/sys/devices/system/memory/memory64/state:offline
/sys/devices/system/memory/memory65/state:offline
/sys/devices/system/memory/memory66/state:offline
/sys/devices/system/memory/memory67/state:offline
/sys/devices/system/memory/memory68/state:offline
/sys/devices/system/memory/memory69/state:offline
/sys/devices/system/memory/memory70/state:offline
/sys/devices/system/memory/memory71/state:offline


Then, use a for loop to activate that memory:

[root@... ~]# for memcount in {40..71}; do echo online > /sys/devices/system/memory/memory$memcount/state; done


Check the new memory is active:

[root@... ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          8045        854       7190          0          8         87

Pre-allocating RAM on a Virtualbox guest

One of the problems with guest VMs in Virtualbox is that RAM is dynamically allocated by the host as the guest uses an increasing quantity of memory.  This is fine if you run many VMs which do not always need all the memory allocated to them at once; but you will find it inconvenient if you really need to allocate specific (read: large) amounts of memory to a VM.  This is especially true when running a guest on a host like windows 7, where superfetch has already allocated chunks of memory to different applications.  When the guest requests more memory, the OS does not give it, because it is only available to other programs which may not necessarily need it.

-- The solution then...

Forcing VirtualBox to "grab" all of the guest's memory at startup is possible.  This will attempt to allocate the entire guest memory from the host.  If that memory isn't really free, then the guest will not start at all.

There is a boolean flag which can be set as follows:

VBoxManage setextradata <VM NAME> VBoxInternal/RamPreAlloc 1

That's all.

Documentation: Unfortunately there is very poor documentation on the "VBoxInternal" keyset; probably because it is mainly used for development purposes and not necessarily "real" day-to-day use.  Consider this then, a hack.

The possible variables that can be set using VBoxInternal seem to be defined in the following C++ header file:

http://www.virtualbox.org/svn/vbox/trunk/src/VBox/VMM/include/PGMInternal.h

Here is another interesting file worth reading:

http://www.virtualbox.org/svn/vbox/trunk/include/VBox/err.h

I would be very careful with attempting to set any of these without a good understanding of the VirtualBox codebase.

Adding or resizing scsi device at runtime on linux guest VM

It's always a tricky thing to add more disk space to a VM when you don't want to take down it's services to reboot the box.  Linux doesn't need to be rebooted just to know that a new device has been attached, but how do you get it to recognize it?

As per a great blog post by Vivek Gite from NixCraft on: http://www.cyberciti.biz/tips/vmware-add-a-new-hard-disk-without-rebooting-guest.html

The basic command to re-scan scsi devices is:

echo "- - -" > /sys/class/scsi_host/<host#>/scan

fdisk -l

tail -f /var/log/message

If you do not have a new disk, but instead have increased the size of an existing disk, then you must rescan the device.  Note that this may not be appropriate if the device is used for the /boot partition.

# echo "1" > /sys/class/scsi_device/<device>/device/rescan

Parse Apache Logs by Date Range

Parsing apache logs by date and by date ranges can be fairly simple with a bit of awk scripting.

We use AWK to compare date fields in order to retrieve specific rows.

The date fields between access logs and error logs can vary, so some adjustments are needed:

Note that the date field is contained within a single column in the access_log file, therefore we can do a comparison against a single column.  Typically column #4.

AWK Date Range for access logs:

$ awk '$4>"[09/Aug/2012:15:00:" && $4<"[09/Aug/2012:15:59:"' ./access_log | less

The date field in the error log is in separate columns.  Example: [Thu Aug 09 15:30:...  That in itself is four columns.  They must be combined in order to be compared effectively.  To do this, we assign a combination of those four columns to two variables: $from and $two.  We then use these two variables for the comparison.  See below:

AWK Date Range for error logs:

$ awk '$from>"[Thu Aug 09 15:30:00" && $to<"[Thu Aug 09 15:59:00"' from='$1 " " $2 " " $3 " " $4' to='$1 " " $2 " " $3 " " $4' ./error_log | less

yum - Error: database disk image is malformed

If you've ever gotten this cryptic error using yum, you'll find that it's very difficult to pinpoint the cause. The message itself, "database disk image is malformed," refers to a corrupted sqlite file. However, the RPM and YUM systems use a variety of different such files; therefore finding the right one can be difficult.

The best thing to do, is to start by attempting to fix this using the available command line tool:

# yum clean all

This should solve the problem in most cases.  If the problem continues, perhaps the RPM database files are corrupted. One of my previous articles talks about rebuilding these, but I will go over it again here:

The database files are located in "/var/lib/rpm" and are named __db.001 __db.002 etc... etc...

Delete those files:

# rm -f /var/lib/rpm/__db*

Rebuild the database:

# rpm --rebuilddb

Then try to clean the yum cache as per the above command and try your yum command again.  If this continues to fail, try deleting your yum cache manually:

# rm -Rf /var/cache/yum

Now try the command again.  This should have gotten rid of the last sqlite files yum could possibly use.  The command should be able to rebuild all the databases correctly at this point.

Force Virtualbox Display in Fullscreen

I have a dual display setup with virtualbox.  In fullscreen I found that virtualbox would switched onto my smaller display.

This can be adjusted.

1) Switch to fullscreen with Virtualbox

2) Use the combination of your "host" key + "home" key. (right-ctrl + home) in my case.

3) Go to the "view" menu item -> "virtual screen"

4) Select the appropriate monitor.

5) If the size of your display varies,  you will have to go out of fullscreen and back into fullscreen to re-adjust the workspace to the correct size.

yum crashed with python import error - fixed corrupted rpm database

I ran into an interesting error while trying to find out, which repository one of my installed packages came from.  Before we proceed, let me explain that I had need to use the "repoquery" utility which is part of the "yum-utils" package.  I proceeded to install this one as I did not yet have it.  The installation worked perfectly well and did not install any dependencies.

# yum install yum-utils -y

Using the repoquery command, I attempted to query which repo my php53 package came from:

$ repoquery -i php53

Instead of getting the information I wanted, the script crashed with the following error:

File "/usr/bin/repoquery", line 38, in

from yum.i18n import to_unicode

cannot import name to_unicode

Googling around, many blog posts and site talked about the yum installation being broken.  That may well be, but I decided to see if maybe there was something a bit simpler at play here.  First I needed to find out which yum packages are already installed on this system:

# rpm -qa | grep -i yum

yum-fastestmirror-1.1.16-16.el5.centos
yum-3.2.22-37.el5.centos
yum-updatesd-0.9-2.el5
yum-utils-1.1.16-16.el5.centos
yum-metadata-parser-1.1.2-2.el5

NOTE: The listed version numbers do not reflect the original numbers I had on my system.  I went ahead and attempted to update all of the above listed packages:

# yum update yum yum-fastestmirror yum-updatesd yum-metadata-parser

Yum only found that yum and yum-fastestmirror needed to be updated.  I proceeded with the update.

After the update, the repoquery command started working perfectly well.  However, a completely unrelated problem occurred which I will discuss very briefly.

# repoquery -i php53

Instead of getting a nice listing of information from the RPM database, I received an error message saying the database was corrupted.  The next step then was to rebuild the dabase.  

The database files are located in "/var/lib/rpm" and are named __db.001 __db.002 etc... etc...

Delete those files:

# rm -f /var/lib/rpm/__db*

Rebuild the database:

# rpm -vv --rebuilddb

Once completed, I tried the repoquery command once again:

# repoquery -i php53

Name        : php53
Version     : 5.3.3
Release     : 1.el5_7.6
Architecture: x86_64
Size        : 3591477
Packager    : None
Group       : Development/Languages
URL         : http://www.php.net/
Repository  : updates
Summary     : PHP scripting language for creating dynamic web sites
Description :
PHP is an HTML-embedded scripting language. PHP attempts to make it
easy for developers to write dynamically generated webpages. PHP also
offers built-in database integration for several commercial and
non-commercial database management systems, so writing a
database-enabled webpage with PHP is fairly simple. The most common
use of PHP coding is probably as a replacement for CGI scripts.

The php package contains the module which adds support for the PHP
language to Apache HTTP Server.

Convert Blocks to Bytes

Very easily 

=========== CONVERT FROM BYTES TO BLOCKS =========== 

For example:

100 Megabytes = 1024 x 1024 x 100

---------------------------------

1 Megabyte = 1024 x 1024 x 1

1 Kilobyte = 1024 x 1

1 Byte = 1

A block is a set quantity of bytes. For example, a mounted partition could have blocks of 4096 bytes or 4k.

Typically however, quotas will have a block size of 1024 bytes ( 1k ).

To calculate the quantity of blocks for quotas you should allocate blocks using the following formula:

Allocate 100Mb of space to a user:

1024 x 1024 x 100 = 104,857,600 (in bytes) = 100Mb

Divide the number of bytes by the block size:

Block size of 4096: 104,857,600 / 4096 = 25,600 blocks

Block size of 1024: 104,857,600 / 1024 = 102,400 blocks

25,600  blocks = 100Mb if your block size is 4096 (unlikely for quotas)

102,400 blocks = 100mb if your block size is 1024

=========== CONVERT FROM BLOCKS TO BYTES ===========

Now determine how many Megabytes a quota of 262144 blocks equals:

1) 262144 multiplied by the size of the blocks:

262144 x 1024 = 268,435,456 (number of total bytes)

2) Since we are dealing with megabytes, divide by 1024 x 1024

( 268,435,456 / 1048576 ) = 256