Saving the RAID May 22, 2008
Posted by idimmu in linux.I managed to screw up my home machine when upgrading from Dapper to Hardy somehow, so decided to flatten and reinstall Hardy from scratch.
This is all well and good but I have a RAID 5 array on my machine which stores all my important documents and stuff (I really hate drive failure). Ubuntu didn't suddenly detect my RAID array and figure everything out, which was not entirely unsurprising but was a little scary.
After some light reading, and a few sighs and heart flutters I worked out the magic runes to tell the system I have a RAID array, where it was and how to use it!
root@server:/root# mdadm -A /dev/md0 -v /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 4.
mdadm: added /dev/sdb to /dev/md0 as 1
mdadm: added /dev/sdc to /dev/md0 as 2
mdadm: added /dev/sdd to /dev/md0 as 3
mdadm: added /dev/sde to /dev/md0 as 4
mdadm: added /dev/sda to /dev/md0 as 0
mdadm: /dev/md0 has been started with 5 drives.
root@server:/root# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda[0] sde[4] sdd[3] sdc[2] sdb[1]
976793600 blocks level 5, 128k chunk, algorithm 2 [5/5] [UUUUU]
unused devices: <none>
root@server:/root# mdadm -Es
ARRAY /dev/md0 level=raid5 num-devices=5 UUID=19c77b42:c5e86159:c23e7b08:7f8ceabd
xend refusing to start March 10, 2008
Posted by idimmu in linux.We recently had a few power outages at work, some scheduled, some not, and this played havoc with our xen servers.
One of the problems we had was that xend would not start (and thus xendomains would also not start).
Checking /var/log/xen/xend.log gave us the following snippet:
inst = XendNode()
File "/usr/lib/python2.5/site-packages/xen/xend/XendNode.py", line 164, in __init__
saved_pifs = self.state_store.load_state('pif')
File "/usr/lib/python2.5/site-packages/xen/xend/XendStateStore.py", line 104, in
load_state
dom = minidom.parse(xml_path)
File "xml/dom/minidom.py", line 1913, in parse
File "xml/dom/expatbuilder.py", line 924, in parse
File "xml/dom/expatbuilder.py", line 211, in parseFile
ExpatError: no element found: line 1, column 0
[2008-03-10 21:37:40 18122] INFO (__init__:1094) Xend exited with status 1.
A quick google of that error revealed several people that had come across the same problem, but no actual answer!
It looks like xen is having problems parsing an xml file, so some quick mental inspiration, and the find command, yielded /var/lib/xend/state/pif.xml which was a 0 byte file! A comparison to a working server showed that it should (or atleast could) contain this:
<?xml version="1.0" ?>
<pifs/>
A copy and paste later and we had a working xend! However it refused to create any of the xenlets:
root@xen0:/etc/xen# xm create server0.cfg
Using config file "./server0.cfg".
Error: The privileged domain did not balloon!
Despite their being plenty of RAM!
root@xen0:/var/log/xen# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 7928 8 r----- 832.8
root@xen0:/var/log/xen# free
total used free shared buffers cached
Mem: 8119416 393028 7726388 0 11344 58832
-/+ buffers/cache: 322852 7796564
Swap: 15631224 0 15631224
An strace of the process revealed xen did think it had less memory available than it actually had ..
[2008-03-10 21:47:48 18620] DEBUG (__init__:1094) Balloon: 131064 KiB free; 0 to scrub;
need 524288; retries: 20.
As we had a working xend finally we decided to implement a technique we'd learned from working with Windows machines and rebooted the server. This magically fixed the memory issue, it would have been nice to know what actually caused it and if there was a proper fix though.
Changing deb package architecture February 5, 2008
Posted by idimmu in linux.We have several PHP applications and libraries incorrectly packaged for i386 rather than all, which means they wont install to our nice new shiny AMD64 servers. Unfortunately we don't have (or at least we can't find) the original package sources, so we need to 'frobble' the packages and change the arch by hand!
Deb files are just an ar package so we must first extract it! (FYI ar is like tar, but for pirates .. :( )
rus@absinthe:~/$ mkdir phplibs
rus@absinthe:~/$ mv phplibs_1.0-1_i386.deb phplibs
rus@absinthe:~/$ cd phplibs
rus@absinthe:~/phplibs$ ar xv phplibs_1.0-1_i386.deb
x - debian-binary
x - control.tar.gz
x - data.tar.gz
rus@absinthe:~/phplibs$ ls
control.tar.gz data.tar.gz phplibs_1.0-1_i386.deb debian-binary
The control.tar.gz is the one that contains all our precious arch information, so we must extract it!
rus@absinthe:~/phplibs$ tar -xzf control.tar.gz
rus@absinthe:~/phplibs$ ls
control control.tar.gz data.tar.gz phplibs_1.0-1_i386.deb debian-binary md5sums
open up the control file and edit in the arch you want (in our case, all)
Package: phplibs
Version: 1.0-2
Section: unknown
Priority: optional
Architecture: all
Installed-Size: 12
Maintainer: rus <rus@internet.com>
Description: shared PHP libraries
Required system php libs
I'm going to increment the package version to -2 as well to signify that it's just the packaging that has changed.
Now we delete the original control.tar.gz file and rebuild it.
rus@absinthe:~/phplibs$ rm control.tar.gz
rus@absinthe:~/phplibs$ tar -czf control.tar.gz control md5sums
We now just need to use ar again to create our debian package!
rus@absinthe:~/phplibs$ ar -r phplibs_1.0-2_all.deb debian-binary control.tar.gz data.tar.gz
ar: creating phplibs_1.0-2_all.deb
And there we have it, a shiny new package of arch all that can be installed anywhere!
Turning CPAN modules in to deb with dh perl make February 1, 2008
Posted by idimmu in linux.Some of our PERL tools require some CPAN modules that are not part of the standard Ubuntu distribution. It's obviously possible to install the module using CPAN but I like using deb packages where possible as then you only have one repository to manage. Fortunately with dh-make-perl it is possible to quickly turn any CPAN module in to a debian package!
First make sure dh-make-perl is installed
apt-get install dh-make-perl
Then download the PERL module you wish to package and extract it
rus@absinthe:~$ wget http://www.cpan.org/modules/by-module/Crypt/Crypt-RC5-2.00.tar.gz
rus@absinthe:~$ tar -pzxvf Crypt-RC5-2.00.tar.gz
Crypt-RC5-2.00/
Crypt-RC5-2.00/Changes
Crypt-RC5-2.00/Makefile.PL
Crypt-RC5-2.00/MANIFEST
Crypt-RC5-2.00/RC5.pm
Crypt-RC5-2.00/README
Crypt-RC5-2.00/test.pl
Now run dh-make-perl on the directory to create the files needed for the package
rus@absinthe:~$ dh-make-perl Crypt-RC5-2.00/
cat: /etc/mailname: No such file or directory
Use of uninitialized value in concatenation (.) or string at /usr/bin/dh-make-perl line 527.
Found: Crypt::RC5 2.00 (libcrypt-rc5-perl arch=all)
Package does not provide a long description - Please fill it in manually.
Using maintainer: rus <rus@>
Found changelog: Changes
Found docs: README
Using rules: /usr/share/dh-make-perl/rules.MakeMaker.noxs
Done
This will have created a debian directory
rus@absinthe:~/Crypt-RC5-2.00$ ls
Changes debian Makefile.PL MANIFEST RC5.pm README test.pl
rus@absinthe:~/Crypt-RC5-2.00$ ls debian/
changelog compat control copyright rules
Then cd in to the directory and run debuild to actually build the package
rus@absinthe:~$ cd Crypt-RC5-2.00/
rus@absinthe:~/Crypt-RC5-2.00$ debuild
This package has a Debian revision number but there does not seem to be
an appropriate original tar file or .orig directory in the parent directory;
(expected libcrypt-rc5-perl_2.00.orig.tar.gz or Crypt-RC5-2.00.orig)
continue anyway? (y/n) y
fakeroot debian/rules clean
dh_testdir
dh_testroot
# Add commands to clean up after the build process here
[ ! -f Makefile ] || /usr/bin/make realclean
dh_clean build-stamp install-stamp
dpkg-source -b Crypt-RC5-2.00
dpkg-source: warning: source directory './Crypt-RC5-2.00' is not <sourcepackage>-<upstreamversion> 'libcrypt-rc5-perl-2.00'
dpkg-source: building libcrypt-rc5-perl in libcrypt-rc5-perl_2.00-1.tar.gz
dpkg-source: building libcrypt-rc5-perl in libcrypt-rc5-perl_2.00-1.dsc
debian/rules build
dh_testdir
# Add commands to compile the package here
/usr/bin/perl Makefile.PL INSTALLDIRS=vendor
Checking if your kit is complete...
Looks good
Writing Makefile for Crypt::RC5
/usr/bin/make OPTIMIZE="-Wall -O2 -g"
make[1]: Entering directory `/home/rus/Crypt-RC5-2.00'
cp RC5.pm blib/lib/Crypt/RC5.pm
Manifying blib/man3/Crypt::RC5.3pm
make[1]: Leaving directory `/home/rus/Crypt-RC5-2.00'
touch build-stamp
fakeroot debian/rules binary
dh_testdir
dh_testroot
dh_clean -k
# Add commands to install the package into debian/ACKAGE_NAME here
/usr/bin/make test
make[1]: Entering directory `/home/rus/Crypt-RC5-2.00'
PERL_DL_NONLAZY=1 /usr/bin/perl "-Iblib/lib" "-Iblib/arch" test.pl
1..1
# Running under perl version 5.008008 for linux
# Current time local: Fri Feb 1 16:23:07 2008
# Current time GMT: Fri Feb 1 16:23:07 2008
# Using Test.pm version 1.25
ok 1
make[1]: Leaving directory `/home/rus/Crypt-RC5-2.00'
/usr/bin/make install DESTDIR=/home/rus/Crypt-RC5-2.00/debian/libcrypt-rc5-perl PREFIX=/usr
make[1]: Entering directory `/home/rus/Crypt-RC5-2.00'
Manifying blib/man3/Crypt::RC5.3pm
Installing /home/rus/Crypt-RC5-2.00/debian/libcrypt-rc5-perl/usr/share/perl5/Crypt/RC5.pm
Installing /home/rus/Crypt-RC5-2.00/debian/libcrypt-rc5-perl/usr/share/man/man3/Crypt::RC5.3pm
make[1]: Leaving directory `/home/rus/Crypt-RC5-2.00'
# As this is a architecture independent package, we are not
# supposed to install stuff to /usr/lib. MakeMaker creates
# the dirs, we delete them from the deb:
rmdir --ignore-fail-on-non-empty --parents /home/rus/Crypt-RC5-2.00/debian/libcrypt-rc5-perl/usr/lib/perl5
touch install-stamp
dh_testdir
dh_testroot
dh_installdocs README
dh_installchangelogs Changes
dh_perl
dh_link
dh_strip
dh_compress
dh_fixperms
dh_installdeb
dh_gencontrol
dpkg-gencontrol: warning: unknown substitution variable ${misc:Depends}
dh_md5sums
dh_builddeb
dpkg-deb: building package `libcrypt-rc5-perl' in `../libcrypt-rc5-perl_2.00-1_all.deb'.
dpkg-genchanges
dpkg-genchanges: including full source code in upload
dpkg-buildpackage (debuild emulation): full upload; Debian-native package (full source is included)
Now signing changes and any dsc files...
Could not find a signing program (pgp or gpg)!
debuild: fatal error at line 1174:
running debsign failed
rus@absinthe:~/Crypt-RC5-2.00$
You should then find a nice debian package a directory above!
rus@absinthe:~/Crypt-RC5-2.00$ ls .. | grep libcrypt | grep deb
libcrypt-rc5-perl_2.00-1_all.deb
Apache2 with SSL and Tomcat5.5 on Ubuntu January 31, 2008
Posted by idimmu in linux.One of the newer features to our site is an access control mechanism to force specific paths to only be delivered over SSL when our customers have particularly sensitive data. We already use Apache2 with mod_jk to talk to the Tomcat5.5 instance running our app so the only part left is to enable SSL!
First make sure mod_ssl is enabled:
root@reltest-tcj0:/var/log/apache2# a2enmod
Which module would you like to enable?
Your choices are: actions asis auth_anon auth_dbm auth_digest auth_ldap cache cern_meta cgid cgi dav_fs dav deflate disk_cache expires ext_filter file_cache headers imap include info jk ldap mem_cache mime_magic proxy_connect proxy_ftp proxy_http proxy rewrite speling ssl suexec unique_id userdir usertrack vhost_alias
Module name? ssl
This module is already enabled!
Then we configure mod_jk to pass it's SSL environment variables to Tomcat by adding the following to apache2.conf
JkExtractSSL On
JkHTTPSIndicator HTTPS
JkSESSIONIndicator SSL_SESSION_ID
JkCIPHERIndicator SSL_CIPHER
JkCERTSIndicator SSL_CLIENT_CERT
Tell Apache2 to listen on the SSL port by editing ports.conf
Listen 443
We want to make sure we have the latest common CA certificates in order to establish a trusted root for our new shiny signed certificate!
apt-get install ca-certificates
If you have a lovely genuinely signed certificate like we do you might need to then add it's intermediate certificate to the ca-certificates system. Move the certificate to /usr/share/ca-certificates then add it's location to /etc/ca-certificates.conf
Now run update-ca-certificates to update the system's certificate store (located in /etc/ssl/certs/ca-certificates.crt).
root@reltest-tcj0:/etc/apache2/sites-enabled# update-ca-certificates
Updating certificates in /etc/ssl/certs....done.
We want the same site to simply be available over SSL I'm going to duplicate the existing VirtualHost for that site specifying the use of port 80 for the original vhost and port 443 for the new one that uses SSL. The only change that needs to be made to the new vhost are the following SSL directives:
SSLEngine On
SSLCertificateFile /etc/apache2/ssl/domain.com.crt
SSLCertificateKeyFile /etc/apache2/ssl/domain.com.key
SSLCACertificateFile /etc/ssl/certs/ca-certificates.crt
Obviously making sure the keys are in the right place!
And lastly make sure that NameVirtualHost settings exist for both port 80 and port 443!
NameVirtualHost *:80
NameVirtualHost *:443
et voila.
