How to debug Linux Init Scripts (in Ubuntu) which fail at bootime

How to debug Linux Init Scripts (Ubuntu).

Often you have the problem that you hacked an init script or have to modify an existing one which will work whenever you manually start it but the process seems not be there after rebooting.

The problem is that your system is not in normal "user mode" during bootime but uses a slim version of your system. For example in most cases for me it is always that my script relies on environment variables such those I usally set in /etc/environment which are loaded after executing the init scripts.

But here is my actual troubleshooting for init scripts which dont work at bootime (manually executing often works):

1) you havent set up any runlevels for your script (solution: update-rc.d)
2) you havent made them executable (solution: chmod +x)
3) your script relies on environment variables which are not set at this stage of bootime
4) your script also does not work at manually running - this meens you have general problems with it - script ERRORS -use bashdb tool

So what to do if it does not run at bootime but runs manually?

Answer: simulate a bootime environment. You can do this by using the env command [1]

cd /
env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" /etc/init.d/daemon start

this simulates the absolut exact situation which are at boottime!!!!

But often if you run modern scripts it does not output anything also you run it in a bootime-like environment.
So what I usally do I use the bashdebugger bashdb tool [2]. In Ubuntu it is easy to install because it is in the repository under "bashdb".

So now I run the full thing like


cd /
env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" bashdb /etc/init.d/daemon start

and it works like a charme to see whats going on.

Often now I can trace down errors but in some cases it was not possible to get the errornous output. Often this is because the error lies in lines which contain the "start-stop-daemon" program which often modern init scripts rely on. But a solution is near [3]. Just single step to the line in bashdb which contains the start-stop-daemon execution and print it out in the shell e.g. the line is called:


"start-stop-daemon -S -p/var/run/jetty.pid -cjetty -d/path/to/solr -b -m -a /usr/bin/java -- -Dsolr.solr.home=/path/to/solr/solr -Djetty.logs=/path/to/solr/logs -Djetty.home=/path/to/solr -Djava.io.tmpdir=/tmp -jar /path/to/solr/start.jar --daemon"

so in bashdb i hop to the line, hold on at there and use the command "x" :

x start-stop-daemon -S -p"$JETTY_PID" $CH_USER -d"$JETTY_HOME" -b -m -a "$JAVA" -- "${RUN_ARGS[@]}" --daemon

and it will print me out the translated string code:

sudo start-stop-daemon -S -p/var/run/jetty.pid -cjetty -d/path/to/solr -v -b -m -a /usr/bin/java -- -Dsolr.solr.home=/path/to/solr/solr -Djetty.logs=/path/to/solr/logs -Djetty.home=/path/to/solr -Djava.io.tmpdir=/tmp -jar /path/to/solr/start.jar --daemon

so I usally just execut this line then in the shell but often I cannot see any output again:-(. But using this blogpost here [3] I see that I have to get rid of the -b parameter in order to see the output.

sudo start-stop-daemon -S -p/var/run/jetty.pid -cjetty -d/path/to/solr -v -m -a /usr/bin/java -- -Dsolr.solr.home=/path/to/solr/solr -Djetty.logs=/path/to/solr/logs -Djetty.home=/path/to/solr -Djava.io.tmpdir=/tmp -jar /path/to/solr/start.jar --daemon

So in my case the JAVA_HOME was set in /etc/environment but could not be seen by this script so it did not know where JAVA_HOME was. So I putted the JAVA_HOME path in the top of the init script and everything worked fine then.


[1] http://stackoverflow.com/questions/1421607/erlang-daemon-init-d-script-fails-to-start
[2] http://bashdb.sourceforge.net/
[3] http://greenash.net.au/thoughts/2011/02/solr-jetty-and-daemons-debugging-jettysh/

Installing jetty hightide 7.4.5 under Ubuntu 10.04.3 LTS (Lucid Lynx) with port 80 support (setuid)

Installing jetty hightide 7.4.5 under Ubuntu 10.04.3 LTS (Lucid Lynx) with port 80 support (setuid)

Since I needed an javax.servlet container on our brand new vServer setup and tomcat needs too much resources in my opinion
I wanted to install jetty on this clean ubuntu LTS system.
So I installed jetty6 from package with "apt-get install jetty". After a bit of configuring the /etc/default/jetty it was
working like a charme but I couldn't get it run on port 80 because my vServer kernel (openvz) did not support NAT out of the box
for port forwarding and also I did not want to install apache2 for port forwarding because apache2 would also be needing some amount
of memory and since our resources are limited and I need the full power/memory for all the other server applications (mysql etc.) I searched
for a alternative. So jetty hightide to the rescue!
Jetty 7 hightide has a new feature called setuid so one can run the webserver with normal user under port 80!!!
Since I did not find any useful FULL description on how to get it to run I decided to write my own. My basic idea comes from website [1] and
since I needed to find out why things were not working I found this beautiful blog entry [2]
So here are my details to install the latest jetty 7 under Lucid Lynx server:


#login with your root account on server
#first lets make an update
apt-get update
#ok my openvz system did not even had these "standard" programs :-D...
#install java
# open /etc/apt/sources.list and put in:
deb http://archive.canonical.com/ubuntu lucid partner
deb-src http://archive.canonical.com/ubuntu lucid partner
#so we can install java
apt-get install less wget unzip sun-java6-jdk
#make your shell aware of java: open /etc/environment and alter line:PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
#put in:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/lib/jvm/java-6-sun"
#add the line:
JAVA_HOME=/usr/lib/jvm/java-6-sun

#download latest jetty hightide from codehaus

wget http://dist.codehaus.org/jetty/jetty-hightide-7.4.5/jetty-hightide-7.4.5.v20110725.zip
unzip jetty-hightide-7.4.5.v20110725.zip
#copy the start script to the init scripts so it will later run on server startup
cp jetty-hightide-7.4.5.v20110725/bin/jetty.sh /etc/init.d/jetty
#move the jetty to system dir
mkdir /opt/jetty
mv jetty-hightide-7.4.5.v20110725/* /opt/jetty
#add a user for security reasons one should not run as root
useradd jetty
#change permissions
chown -R jetty /opt/jetty
chmod -R ugo+rw /opt/jetty
mkdir -p /var/log/jetty
chown jetty /var/log/jetty -R
#create an jetty config file
touch /etc/default/jetty
vi /etc/default/jetty

#set in this file
DEBUG=1
JAVA=$JAVA_HOME/bin/java
JETTY_HOME=/opt/jetty
JETTY_LOGS=/var/log/jetty
JETTY_PORT=8080 #we will change this later
JETTY_USER=jetty #we will change this later

#add t server startup init levels
update-rc.d jetty defaults

#ok after this jetty is ready to start "/etc/init.d/jetty start" so test it under port 8080 before proceeding

Now to the second part: enable port 80 setuid feature
#move the seuid jar to a different location since it is at the wrong one in the hightide package (see [5] for a why)
cp /opt/jetty/lib/setuid/jetty-setuid-java-7.4.5.v20110725.jar /opt/jetty/lib/ext/


#open /etc/default/jetty again and add the following line:
JETTY_SETUID=/opt/jetty/etc/jetty-setuid.xml
#change the following line:
JETTY_PORT=8080 => JETTY_PORT=80
#this is f**** crucial and took me a long time: you have to start jetty now as "root" user but it will actually set
#the jetty user as owner of the jetty server...without setting to root the setuid feature will not work
JETTY_USER=root

#now open /etc/init.d/jetty
#after those lines:

#####################################################
# Add jetty properties to Java VM options.
#####################################################
JAVA_OPTIONS+=("-Djetty.home=$JETTY_HOME" "-Djava.io.tmpdir=$TMPDIR")

[ -f "$JETTY_HOME/etc/start.config" ] && JAVA_OPTIONS=("-DSTART=$JETTY_HOME/etc/start.config" "${JAVA_OPTIONS[@]}")


#insert the following lines:

##################################################
# enable setuid
##################################################
if [ -z "$JETTY_SETUID" ]
then
JETTY_SETUID=
fi


#change the line: RUN_ARGS=(${JAVA_OPTIONS[@]} -jar "$JETTY_START" $JETTY_ARGS $"${CONFIGS[@]}") to

RUN_ARGS=(${JAVA_OPTIONS[@]} -jar "$JETTY_START" $JETTY_SETUID $JETTY_ARGS $"${CONFIGS[@]}")



#now go back to the shell
id jetty
#write out for uid for example mine was: uid=1000(jetty) gid=1000(jetty) groups=1000(jetty)
vi /opt/jetty/etc/jetty-setuid.xml
#insert following code (replace USERID with uid from the command above)
#actually the original description on jetty homepage [3]does no longer work for hightide (was written for jetty6)
#one has to change the XML attribute id from tag Configure from "Server" to "org.eclipse.jetty.server.Server"
#otherwise there will be errors (thanks the post on [4])


<Configure id="org.eclipse.jetty.server.Server" class="org.mortbay.setuid.SetUIDServer">
<Set name="uid">UID</Set>
</Configure>

#now change the following lines using your actual uid of user jetty with the lines above:

<Configure id="org.eclipse.jetty.server.Server" class="org.mortbay.setuid.SetUIDServer">
<Set name="uid">1000</Set>
</Configure>


chown jetty /opt/jetty/etc/jetty-setuid.xml

restart jetty and then port80 will work (/etc/init.d/jetty restart)
You can make proof of this by going to the stderr logfile less /var/log/jetty/20xx_xx_xx.stderrout.log after you restarted the server and there you will see:

2011-08-03 10:26:18.478:INFO::Started SelectChannelConnector@0.0.0.0:80 STARTING
2011-08-03 10:26:18.479:INFO::Setting UID=1000
2011-08-03 10:26:18.487:INFO::jetty-7.4.5.v20110725


Note:
In any case there are errors always "tail" the "/var/log/jetty/" start.out and error log files. Also for some errors produced by startup script
its best to debug the jetty init script, see [2] for a very good guideline to do so!


Sources:
[1]http://webcache.googleusercontent.com/search?q=cache:-YW2xPMZYk0J:ubuntuforums.org/showthread.php%3Ft%3D466143+jetty+init+script&cd=1&hl=de&ct=clnk&gl=de&client=firefox-a&source=www.google.de
[2]http://greenash.net.au/thoughts/2011/02/solr-jetty-and-daemons-debugging-jettysh/
[3]http://docs.codehaus.org/display/JETTY/port80
[4]http://dev.eclipse.org/mhonarc/lists/jetty-users/msg00787.html
[5]http://groups.google.com/group/cometd-users/browse_thread/thread/8ae7677f60c5100b

How to install Centos 5.5 on any GPT partition table harddrive by patching the CentOS 5 anaconda installer

Centos is a free derivate of the most used server linux red hat (commercial).
Therefore I do not understand why the Centos people have disabled GPT
support for their latest Centos 5.5 release because almost every modern
server have big raid disks which often are bigger than 2.2 TB so no msdos
partition table can be used here....
They have disabled it because Centos uses an old Grub version which cannot handle
GPT tables thats all....in general the CentOS 5.5 installer called anaconda can install CentOS on GPT tables (as you will see).
But there are ways to fix this limitations on your own.
I first heard about it from Richards blog where he showed a nice way on how to install Centos 5.0 on large GPT partitions because the CentOS 5.0 version did not check for GPT tables
and he uses a patched Grub from the LinuxRescueCD Project to fix the grub (which does not support GPT) installed by centos.
So this just works like a charme in most cases.

But unfortunately Richard's approach didn"t work on my HP proliant DL 380G6 Server. As I need to install Centos 5 on our servers and I could not install Centos5.0 because the smart array raid system build into the server hardware is not supported from the centos5.0 kernel I
just found out another approach on my own on how to install centos5.5 on every machine.
As I am mainly a programmer and not a system administrator it was very easy for me to tweak the anaconda installer routine (installer scripts are written in python) a bit to let me and possible you reading
this install centos5.5 on any gpt tables disk. So tweaking anaconda was the easy part but understanding the anaconda installation cd build process in the first plance, understanding anaconda concepts such as buildstamps, rpm packaging, re-generating new installer and generating a new cd was the harder part for me :-) (I am not a linux guru....yet)

Here is a screenshot of what happens if you try to install Centos > 5.0 on a machine with a GPT partitioning table.


Everything works fine at the begin of the installation but when it comes to partitionating your harddrive the following error occurs:
"The following critical errors exist with your requested partitioning scheme. These errors must be corrected prior to continuing with your install of CentOS. Your boot partition is on a disk using the GPT partitioning scheme but this machine cannot boot using GPT."
and you cannot go on!

So the solution is to get rid of this error message in the anaconda installer by modifing the anaconda installation code.

Here are an overview of the concept and steps I used.
1. First partitionate your GPT harddisk using a linux rescue disk ...(mklabel gpt etc....)
you can read richards blog on how to do this in general.
2. I use a virtualbox environment to hack the centos5.5 installer (anaconda),
you can also use a normal centos5.5 install but for tweaking and hacking
I find a virtual environment more safe as to worki on a real install
as you can mess around without any side-effects.
2a) Install Centos5.5 in a virtual environment or boot up a normal installation.
if you install it new for this purpose, deselect all packages, choose custom
and select:
Development libs, Development tools, Editors...
you only need to install anaconda, a text editor and the squash fs system support..... you do not need more
3) mount the centos5.5 installation cd and modify the anaconda pyhton code
4) rebuild the installation cd...we will create a bootable iso
5) install centos5.5 with the new modified installer...no gpt errors will occur and installation can be made
6) after installation of centos5.5 you will have to patch the grub loader
7) reboot into your new centos5.5 working on a gpt table and enjoy

Here are all the EXACT steps I did.

after booting up a centos5.5 installationand login as root use the following commands:

#some prequisites
#change to homedir first
cd
yum install anaconda anaconda-runtime anaconda-helper
PYTHONPATH=/usr/lib/anaconda
PATH="$PATH:/usr/lib/anaconda-runtime"
#choose your arch: i386 or x86_64
ARCH=x86_64
BUILDBASE=`pwd`
mkdir -p $BUILDBASE/linux/$ARCH/SRPMS

#insert and mount the first centos5.5 install cd in your centos5.5 installation
mkdir /media/cdrom
mount /dev/cdrom /media/cdrom
#make a copy we can edit
rsync -avr /media/cdrom/ $BUILDBASE/linux/$ARCH
#now mount the full anaconda suite from the installation cd into a new directory
mkdir /media/anaconda

mount -o loop -t squashfs $BUILDBASE/linux/$ARCH/images/stage2.img /media/anaconda
#copy the anaconda mount to a place we can edit
mkdir $BUILDBASE/anaconda
rsync -avr /media/anaconda/ $BUILDBASE/anaconda to get rid of the GPT check
#now edit the following anaconda python script file:
emacs $BUILDBASE/anaconda/usr/lib/anaconda/partitions.py

#In the editor go to line:1082 and change:

errors.append(
_("Your boot partition is on a disk "


change to=>
warnings.append(
_("Your boot partition is on a disk "

#After that save and close the editor.
#now back at the shell do:

yum install squashfs-tools
cd $BUILDBASE/anaconda
mksquashfs . ../stage2.img

#backup original stage2
mkdir $BUILDBASE/backup
cp $BUILDBASE/linux/$ARCH/images/stage2.img $BUILDBASE/backup

#now rebuild the cd
DISCINFO=`head -1 $BUILDBASE/linux/$ARCH/.discinfo`
createrepo -u “media://$DISCINFO” -g $BUILDBASE/linux/$ARCH/repodata/comps.xml $BUILDBASE/linux/$ARCH/

#copy the new stage2 we generated to the cd installation
cp $BUILDBASE/anaconda/../stage2.img $BUILDBASE/linux/$ARCH/images/stage2.img

#now make a new iso
ISOFILENAME = $BUILDBASE/c5.5-custom.iso
cd $BUILDBASE/linux/$ARCH
mkisofs -r -R -J -T -v \
-no-emul-boot -boot-load-size 4 -boot-info-table \
-V “Centos 5.5 GPT fix″ -p “Oli P” \
-A “Centos 5.5 GPT fix - 2010/21/09″ \
-b isolinux/isolinux.bin -c isolinux/boot.cat \
-x “lost+found” -o $ISOFILENAME $BUILDBASE/linux/$ARCH

implantisomd5 $ISOFILENAME

#transfer the iso to a place where you can use it in virtual box or burn the cd of it

Now partitionate your GPT harddrive with a Linux resue CD or pmagic linux cd see Richards Blog for an overview
(Install CentOS onto large partitions using gpt disk layout).
#boot the rescue cd
#select your harddrive e.g. /dev/sda or when using pmagic cd: /cciss/c0d0 etc.
#here is my configuration of parted

#after setting up partitons quit parted "type q" and in the shell format your new partitions as follow:
mkfs.ext3 /dev/sda1
mkfs.ext3 /dev/sd3
mkswap /dev/sda2
#the last partition "data" will take a long time so pick up a coffee.....
mkfs.ext3 /dev/sda4

reboot and insert the modified Centos 5.5 installer cd.
At the Welcome to CentOS 5.5 screen press OK :-D
Choose Language and keyboard layout.

Now at the next screen "Partitioning type" choose "Create custom layout".

Select the partitons and create mount points (/boot,/,/data etc.) but do not chose format, leave this because you have already formatted everything.




And now the next screen:

oh no this thing didnt work....did it?
If you watch closely and compare to the original message at the beginning of this document you will see a difference in the heading: it was "Error" no it is "Warning" and we have a "YES" button to continue!!!
Error will stop the installer going any further, warning message can be ignored and you can go on in the installer!!! So click OK NOW!

So you will no longer get the Error that you cannot do this on a GPT table....but instead get a nag screen with a warning only :-) ...successfully hacked...because its a warning only the system let you continue
installing !!!!!
so next steps are to Install Centos:
Next important steps are "Use Grub Boot loader" and after some other steps the next important step is "Boot Loader Configuration".

Question is "Where do you want to install the boot loader?" HEre you have to chose "First sector of boot partition" THIS IS IMPORTANT
DO NOT INSTALL IT IN THE MBR!! WE USE BOOT SO WE CAN FIX IT LATER!!!

After Installation has finally finished remove Centos5.5 cd and insert RescueCD and reboot to fix the GRUB.
If you boot without fixing grub the boot partition and grub will just not be recognised so you have to do the following steps....

I fixed it using the steps at http://www.rodsbooks.com/gdisk/booting.html (section Linux, GRUB, and LILO) because the instructions at
http://www.sysresccd.org/Sysresccd-Partitioning-EN-The-new-GPT-disk-layout (section Installing a patched Grub-0.97) did not produce fast results.
Here are my commands:
at sysrescue bootup chose your architecture type (I use 64-bit kernel with standards).
Then when you got a shell type:

#instructions from site:http://www.rodsbooks.com/gdisk/booting.html (section Linux, GRUB, and LILO)
#copy the boot partition from the live cd to a safe place

mv /boot /boot_livecd
mkdir /boot
#my boot partition is on sda1 to find out yours: parted with print command
mount /dev/sda1 /boot
#make a backup of grub...in a worst case scenario...always make a copy of this... never just move to grub!
cp -r /boot/grub /boot/grub_original
#edit /boot/grub/devices.map and replace entry "cc0011/c0d0" with "sda"...in general replace your harddisk device name with sda
nano /boot/grub/devices.map
#now install the fixed grub from the rescue cd on your boot partition
grub-install /dev/sda

this was the file /boot/grub/devices.map showing before editing
you have to replace your actual device name with sda so in my case the complete new
line will be (without the quotes):
"(hd0) /dev/sda"

after running grub-install you should get the following success status:

reboot....your gpt table will load and finally centos5.5 is working!!!


Note: I can share the modified installer CD on a one-click-hoster if there is high demand for it.

References:
[1] Distribution Hacking 101: disecting anaconda, http://k12linux.mesd.k12.or.us/at/roswell-ltsp.html
[1] Build a custom CentOS 5 install CD, http://www.sirius.pe.kr/v5/tt
[3] Install CentOS onto large partitions using gpt disk layout, http://richardjhblog.wordpress.com/2010/08/25/install-centos-onto-large-partitions-using-gpt-disk-layout/
[4] Booting from GPT (Linux, Grub and Lilo), http://www.rodsbooks.com/gdisk/booting.html

How to install Centos 5 onto large partitions using GPT layout (HP proliant DL 3xx G6 server system)

UPDATE: this approach is deprecated and does not work on SINGLE GPT partitions bigger
than 2.2 TB.
Use my updated guide to install Centos5.5 instead:
http://oliverpelz.blogspot.com/2010/09/how-to-install-centos-55-on-any-gpt.html

I completely wasted a lot of time trying to get Centos5 running on some HP proliant server system.
After reading endless tutorials and tips I gave up...none of them worked...its impossible (at the moment) to install Centos5.x on a GPT system like mine (as others also stated).
In the end after partitoning or doing installer tricks such as erasing MBR with dd to pass the GPT check etc. I always stuck at the point that the installer does not recognize my smart array HP raid5.
So after all this approaches do not work I found the following solution:

-Install centos 4.8 which works perfectly well with GPT layout.
But before you can do this you have to upgrade your firmware to the latest edition because otherwise centos anaconda installer will fail with some GPT/MSDOS recognition problems.
-So I updated my clusters which come with a firmware from march2010 to the latest 9.03.

-Use a boot partition and install grub in this partition.
-After installation of some base packages immediately put in the latest linux rescue cd (http://www.sysresccd.org) and install new grub with GPT support because the one shipped with 4.8 does not support GPT partitions and grub will not load (red screen of death) the system.

Immediately after installation has finished installing insert linux rescue cd, reboot, I have 64 so chose 64bit version from the rescue cd menue and install a patched grub with gpt support:

mv /boot /boot_livecd
mkdir /boot
mount /dev/sda1 /boot (my boot partition is on sda1 to find out yours: parted /dev/sda print)
#make a backup of grub...for worst case scenario..
cp -r /boot/grub /boot/grub_original
edit /boot/grub/devices.map and replace entry "cc0011/c0d0" with "sda1"
grub-install /dev/sda
reboot

grub will work and load the OS

Reboot and update your centos installation from 4 to 5 using a good other tutorial which can be found plenty of on the net.

Caused by: org.hibernate.MappingException: Could not determine type for: XXX, at table: XXXX, for columns: YYYY

This is a common hibernate mapping exception and in most cases you simply forgot to implement

toString()
hashCode()
equals()

in your class you are using a relation to another class (only when using annotations)

Another thing what could be the cause of this could be to mix the annotations between above the fields and above the getter MEthods.
Try to put all the annotations consistent above the getter methods AND not above the data fields.

So:
instead of :
@Id
@GeneratedValue(strategy = GenerationType.AUTO)
@Column(name = "count")
public long id;

public long getId() {
return id;
}

alter all of them to:


public long id;

@Id
@GeneratedValue(strategy = GenerationType.AUTO)
@Column(name = "count")
public long getId() {
return id;
}

Running Jetty 5 under Ubuntu 9.04 Jaunty

Ok another weird thing just happened.
I have installed jetty5 and some war applications via jetty webapps folder as also via maven2 jetty plugin but just cant connect through any webbrowser via localhost:8080 (mvn2) or localhost:8280 (jetty). So i tried and tried but just cant connect to it. Here is what happened: Though jetty describes itself as a complete full integrated http webserver I think it is not configured right on ubuntu because i just found out that you need to install apache2 package in order to run localhost jetty applications!

Using Joomla and the Strato website provider

Today I had the problem that I had to alter my php.ini because the joomla pony gallery plugin needed extra memory when generating thumbnails and the strato memory size was too low for processing.

So I did some research and found out that Strato allows own php.ini files which can overwrite their values. Problem was only that I could not get my own php.ini to work with the website or backend, I got a lot of internal server failures. So I did some research again and here are the correct settings to use own customized .htaccess and php.ini:

create a .htaccess in your root dir :

Directoryindex index.php , index.htm, index.html
AddType application/x-httpd-php5 .php .php4 .php3 .php5

next create a php.ini in your root dir and in your administrator dir where you now do what you like and it actually will work:

register_globals = Off
safe_mode = Off
memory_limit = 16M
upload_max_filesize = 10M
max_execution_time = 30   
max_input_time = 60

so have phun with this!!