yum error: Couldn’t fork Cannot allocate memory

Yum Couldn't Fork Cannot Allocate Memory I’ve been doing some awesome things to a new VM for work, namely installing CouchDB, Apache and running Node.JS apps along side a WordPress plugin using Angular.JS. It’s pretty cool. But computer’s are dicks so when it came down to installing Monit to ensure everything was lovely I got the following error: Couldn’t fork %pre(monit-5.5-1.el6.rf.x86_64): Cannot allocate memory. Bum.

error: Couldn’t fork %pre(monit-5.5-1.el6.rf.x86_64): Cannot allocate memory

Seem’s simple enough, for whatever reason Yum cannot allocate memory, so lets take a peak


root@bridge opt]# free
total used free shared buffers cached
Mem: 1020376 832736 187640 0 3988 81256
-/+ buffers/cache: 747492 272884
Swap: 0 0 0

Man there’s totally enough memory there, 187MB of RAM is free, Quake took less than that and is way more complicated than some stupid RPMs.. maybe it’s something else!

Quite often this error is caused because the RPM database has duplicates or got corrupted in some way, so lets try and clean that up.


[root@bridge ~]# package-cleanup --cleandupes
Loaded plugins: fastestmirror, protectbase
Loading mirror speeds from cached hostfile
* base: mirror.checkdomain.de
* epel: mirrors.n-ix.net
* extras: mirror.checkdomain.de
* rpmforge: mirror1.hs-esslingen.de
* updates: mirror.checkdomain.de
1490 packages excluded due to repository protections
No duplicates to remove
[root@bridge ~]# rpm --rebuilddb

Well no duplicates and the RPM database is all cool, so lets try again ..


[root@bridge ~]# yum install monit
Loaded plugins: fastestmirror, protectbase

Running Transaction
Error in PREIN scriptlet in rpm package monit-5.5-1.el6.rf.x86_64
error: Couldn't fork %pre(monit-5.5-1.el6.rf.x86_64): Cannot allocate memory
error: install: %pre scriptlet failed (2), skipping monit-5.5-1.el6.rf
Verifying : monit-5.5-1.el6.rf.x86_64 1/1

Failed:
monit.x86_64 0:5.5-1.el6.rf

Complete!

Man, haters gonna hate!

Solving error: Couldn’t fork %pre(monit-5.5-1.el6.rf.x86_64): Cannot allocate memory

Ok, lets step back a minute and assume the error is legit, lets turn some stuff off ..


[root@bridge ~]# /etc/init.d/couchdb stop
Stopping database server couchdb
[root@bridge ~]# /etc/init.d/httpd stop
Stopping httpd: [ OK ]

And try again!


[root@bridge ~]# yum install monit
Loaded plugins: fastestmirror, protectbase

Downloading Packages:
monit-5.5-1.el6.rf.x86_64.rpm | 267 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : monit-5.5-1.el6.rf.x86_64 1/1
Verifying : monit-5.5-1.el6.rf.x86_64 1/1

Installed:
monit.x86_64 0:5.5-1.el6.rf

Complete!

Sweet that did it. So it was a bonafide legit error and shutting some services down freed up enough memory to allow us to install RPMs again.


root@bridge ~]# free
total used free shared buffers cached
Mem: 1020376 510972 509404 0 11632 146780
-/+ buffers/cache: 352560 667816
Swap: 0 0 0

mmm 509MB free, thats a lot more.. I guess Yum actually needs a ton of RAM to actually do anything. Weird. If you guys get this problem, try turning some services off and on again 😉

sudo: sorry, you must have a tty to run sudo

sudo: sorry, you must have a tty to run sudo We’re using an old version of Upstart, on Centos, to manage stopping and starting our Node.js daemons, and one of the things the script does, like any good deamon, is change the user of the deamon process from root to something more applicable, security and all that 😉

The scripts look a little like this


!upstart
description "Amazing Node.js Daemon"
author "idimmu"

start on runlevel [2345]
stop on shutdown

env PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
env NAME=”amazing-daemon”

script
export HOME=”/root”
cd /opt/idimmu/$NAME
echo $$ > /var/run/$NAME.pid
exec sudo -u idimmu /usr/bin/node /opt/idimmu/$NAME/server.js >> /var/log/$NAME/stdout.log 2>&1
end script

pre-start script
echo “[`date -u +%Y-%m-%dT%T.%3NZ`] (upstart) Starting $NAME” >> /var/log/$NAME/stdout.log
end script

pre-stop script
rm /var/run/$NAME.pid
echo “[`date -u +%Y-%m-%dT%T.%3NZ`] (upstart) Stopping $NAME” >> /var/log/$NAME/stdout.log
end script

Which is nice, as it means we can use Upstart to stop/start/status deamons really nicely. The equivalent init.d script looked really horrible.

But there’s one massive caveat, which we always encounter when building a brand new box, from scratch.


2013-09-27T10:50:10.174Z] (upstart) Starting amazing-daemon
sudo: sorry, you must have a tty to run sudo

sudo: sorry, you must have a tty to run sudo

So it all falls apart due to the following error:

sudo: sorry, you must have a tty to run sudo

Basically sudo is stopping the process from running because Upstart doesn’t have a TTY. This is easily fixable. Just edit /etc/sudoers using visudo and comment out


Defaults requiretty

i.e.


#Defaults requiretty

Now we can use Upstart to start the daemon and check it’s status to confirm it’s running! More recent versions of Upstart don’t need this hack. One day I’ll live in the future, but not today.


deploy:amazing root$ start amazing
amazing start/running, process 3965
deploy:amazing root$ status amazing
amazing start/running, process 3965

Bamo, problem solved!

Installing OpenSIPS on Redhat Enterprise Linux 5

We’ve got to set up a SIP Registrar for a fantastic new project that one of our clients recently announced 😉 on RHEL5 so we’re going to trial OpenSIPS, backed with MySQL to see how it performs.

There’s a RHEL5 repository mirror here http://centos.leurent.eu/ that I’ve mirrored locally. If you don’t know how to create your own repositories you can read about that here.

First off I installed the opensips-mysql.x86_64 package to guarantee we pull in OpenSIPS as well as it’s MySQL dependencies.


[root@dollrose repos]# yum install opensips-mysql.x86_64
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package opensips-mysql.x86_64 0:1.7.1-0 set to be updated
--> Processing Dependency: opensips = 1.7.1 for package: opensips-mysql
--> Running transaction check
---> Package opensips.x86_64 0:1.7.1-0 set to be updated
--> Processing Dependency: perl(Frontier::RPC2) for package: opensips
--> Running transaction check
---> Package perl-Frontier-RPC.noarch 0:0.07b4-5.el5 set to be updated
--> Processing Dependency: perl(HTTP::Status) for package: perl-Frontier-RPC
--> Processing Dependency: perl(HTTP::Daemon) for package: perl-Frontier-RPC
--> Processing Dependency: perl(Apache::Constants) for package: perl-Frontier-RPC
--> Processing Dependency: perl(HTTP::Request) for package: perl-Frontier-RPC
--> Processing Dependency: perl(XML::Parser) for package: perl-Frontier-RPC
--> Processing Dependency: perl(LWP::UserAgent) for package: perl-Frontier-RPC
--> Running transaction check
---> Package mod_perl.x86_64 0:2.0.4-6.el5 set to be updated
--> Processing Dependency: perl(BSD::Resource) for package: mod_perl
---> Package perl-XML-Parser.x86_64 0:2.34-6.1.2.2.1 set to be updated
---> Package perl-libwww-perl.noarch 0:5.805-1.1.1 set to be updated
--> Processing Dependency: perl-HTML-Parser >= 3.33 for package: perl-libwww-perl
--> Processing Dependency: perl(HTML::Entities) for package: perl-libwww-perl
--> Processing Dependency: perl(Compress::Zlib) for package: perl-libwww-perl
--> Running transaction check
---> Package perl-BSD-Resource.x86_64 0:1.28-1.fc6.1 set to be updated
---> Package perl-Compress-Zlib.x86_64 0:1.42-1.fc6 set to be updated
---> Package perl-HTML-Parser.x86_64 0:3.55-1.fc6 set to be updated
--> Processing Dependency: perl-HTML-Tagset >= 3.03 for package: perl-HTML-Parser
--> Processing Dependency: perl(HTML::Tagset) for package: perl-HTML-Parser
--> Running transaction check
---> Package perl-HTML-Tagset.noarch 0:3.10-2.1.1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================================================================================
Package Arch Version Repository Size
=================================================================================================================================================================================================================
Installing:
opensips-mysql x86_64 1.7.1-0 opensips 96 k
Installing for dependencies:
mod_perl x86_64 2.0.4-6.el5 rhel 4.0 M
opensips x86_64 1.7.1-0 opensips 5.7 M
perl-BSD-Resource x86_64 1.28-1.fc6.1 rhel 29 k
perl-Compress-Zlib x86_64 1.42-1.fc6 rhel 52 k
perl-Frontier-RPC noarch 0.07b4-5.el5 epel 40 k
perl-HTML-Parser x86_64 3.55-1.fc6 rhel 92 k
perl-HTML-Tagset noarch 3.10-2.1.1 rhel 14 k
perl-XML-Parser x86_64 2.34-6.1.2.2.1 rhel 210 k
perl-libwww-perl noarch 5.805-1.1.1 rhel 376 k

Transaction Summary
=================================================================================================================================================================================================================
Install 10 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 11 M
Is this ok [y/N]: y
Downloading Packages:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 131 MB/s | 11 MB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : perl-Compress-Zlib 1/10
Installing : perl-BSD-Resource 2/10
Installing : mod_perl 3/10
Installing : perl-HTML-Tagset 4/10
Installing : perl-HTML-Parser 5/10
Installing : perl-libwww-perl 6/10
Installing : perl-XML-Parser 7/10
Installing : perl-Frontier-RPC 8/10
Installing : opensips 9/10
Installing : opensips-mysql 10/10

Installed:
opensips-mysql.x86_64 0:1.7.1-0

Dependency Installed:
mod_perl.x86_64 0:2.0.4-6.el5 opensips.x86_64 0:1.7.1-0 perl-BSD-Resource.x86_64 0:1.28-1.fc6.1 perl-Compress-Zlib.x86_64 0:1.42-1.fc6 perl-Frontier-RPC.noarch 0:0.07b4-5.el5
perl-HTML-Parser.x86_64 0:3.55-1.fc6 perl-HTML-Tagset.noarch 0:3.10-2.1.1 perl-XML-Parser.x86_64 0:2.34-6.1.2.2.1 perl-libwww-perl.noarch 0:5.805-1.1.1

Complete!

Create the opensips user


adduser opensips

Then edit /etc/opensips/opensipsctlrc and make the following changes


SIP_DOMAIN=localhost
DBENGINE=MYSQL
DBHOST=localhost
DBNAME=opensips
DBRWUSER="opensipsrw"
DBRWPW="opensips"
DBROOTUSER="root"
USERCOL="username"
INSTALL_EXTRA_TABLES=ask
INSTALL_PRESENCE_TABLES=ask

We can now run opensipsdbctl create and create the database and tables


[root@dollrose opensips]# opensipsdbctl create
MySQL password for root:
INFO: test server charset
INFO: creating database opensips ...
INFO: Core OpenSIPS tables succesfully created.
Install presence related tables? (y/n): y
INFO: creating presence tables into opensips ...
INFO: Presence tables succesfully created.
Install tables for imc cpl siptrace domainpolicy carrierroute userblacklist? (y/n): y
INFO: creating extra tables into opensips ...
INFO: Extra tables succesfully created.

Edit /etc/opensips/opensips.cfg and enable the following options


loadmodule "db_mysql.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
modparam("usrloc", "db_mode", 2)
modparam("usrloc", "db_url",
"mysql://opensips:opensipsrw@localhost/opensips")
modparam("auth_db", "calculate_ha1", yes)
modparam("auth_db", "password_column", "password")
modparam("auth_db", "db_url",
"mysql://opensips:opensipsrw@localhost/opensips")
modparam("auth_db", "load_credentials", "")

Now we can create a couple of new soft phone accounts using the opensipctl add command


[root@dollrose opensips]# opensipsctl add 1234 1234
new user '1234' added
[root@dollrose opensips]# opensipsctl add 5678 5678
new user '5678' added

We can then create the MySQL user and grant him credentials on the database.


[root@dollrose opensips]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 417
Server version: 5.0.77 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> GRANT ALL PRIVILEGES ON opensips.* TO opensips@localhost IDENTIFIED BY 'opensipsrw';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON opensips.* TO opensips@127.0.0.1 IDENTIFIED BY 'opensipsrw';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

We can now start OpenSIPS!


[root@dollrose opensips]# /etc/init.d/opensips start
Starting opensips:


root@dollrose opensips]# netstat -lpn | grep 5060
tcp 0 0 10.20.0.11:5060 0.0.0.0:* LISTEN 4253/opensips
tcp 0 0 87.252.58.21:5060 0.0.0.0:* LISTEN 4253/opensips
tcp 0 0 127.0.0.1:5060 0.0.0.0:* LISTEN 4253/opensips
udp 0 0 10.20.0.11:5060 0.0.0.0:* 4253/opensips
udp 0 0 87.252.58.21:5060 0.0.0.0:* 4253/opensips
udp 0 0 127.0.0.1:5060 0.0.0.0:* 4253/opensips

I win! :D

List Contents Of An RPM Package

Often when building RedHat RPM packages I want to make sure that the package built correctly and all the files are present before I actually install the RPM, but I do keep forgetting the command to do this, so here it is for prosperity!


rpm -qlp

e.g.


root@build:/usr/src/redhat/SPECS# rpm -qlp ../RPMS/noarch/servicemap-0.16-1.noarch.rpm
/var/www/html/servicemap
/var/www/html/servicemap/amq.png
/var/www/html/servicemap/amqfail.png
/var/www/html/servicemap/amqreportingprov1prov1.png
/var/www/html/servicemap/amqreportingprov1prov2.png
/var/www/html/servicemap/amqreportingprov2prov1.png
/var/www/html/servicemap/amqreportingprov2prov2.png
/var/www/html/servicemap/amqsdrprov1prov1.png
/var/www/html/servicemap/amqsdrprov1prov2.png
/var/www/html/servicemap/amqsdrprov2prov1.png
/var/www/html/servicemap/amqsdrprov2prov2.png
/var/www/html/servicemap/amqslee1fail.png
/var/www/html/servicemap/amqslee1prov1.png
/var/www/html/servicemap/amqslee1prov2.png
/var/www/html/servicemap/amqslee2fail.png
/var/www/html/servicemap/amqslee2prov1.png
/var/www/html/servicemap/amqslee2prov2.png
/var/www/html/servicemap/amqvqeprov1prov1.png
/var/www/html/servicemap/amqvqeprov1prov2.png
/var/www/html/servicemap/amqvqeprov2prov1.png
/var/www/html/servicemap/amqvqeprov2prov2.png
/var/www/html/servicemap/amqwebserviceprov1prov1.png
/var/www/html/servicemap/amqwebserviceprov1prov2.png
/var/www/html/servicemap/amqwebserviceprov2prov1.png
/var/www/html/servicemap/amqwebserviceprov2prov2.png
/var/www/html/servicemap/checks.php
/var/www/html/servicemap/config.php
/var/www/html/servicemap/hash.png
/var/www/html/servicemap/index.php
/var/www/html/servicemap/nagios.php
/var/www/html/servicemap/script.php
/var/www/html/servicemap/sleevipactivenotok.png
/var/www/html/servicemap/style.php
/var/www/html/servicemap/ttprov1.png
/var/www/html/servicemap/ttprov1prov2.png
/var/www/html/servicemap/ttprov2.png
/var/www/html/servicemap/ttprov2prov1.png
/var/www/html/servicemap/ttprovfail.png
/var/www/html/servicemap/ttslee1.png
/var/www/html/servicemap/ttslee1slee2.png
/var/www/html/servicemap/ttslee2.png
/var/www/html/servicemap/ttsleefail.png
/var/www/html/servicemap/ttvipactivenotok.png
/var/www/html/servicemap/ttvipactiveok.png

Fedora Linux O’Reilly’s Fedora Linux covers everything you need to know about dealing with Redhat systems and is a recommended read for anyone who wants to seriously get in to Redhat administration.

Error: Could not open command file ‘/var/log/nagios/rw/nagios.cmd’ for update! on RedHat

I’m currently setting up some new clusters under RedHat and each cluster is getting it’s own Nagios instance, trying to use the web based management interface however threw and error.

I’ve seen this error before on Ubuntu and was getting it again under RedHat. Of course I revisited my Ubuntu solution and realised that it didn’t help at all, due to using dpkg overrides, also the situation was very different!


root@nagios:/var/log/nagios/rw# ls -al
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:37 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:40 ..

The file didn’t exist 🙁

A quick scan of a working system showed :


prw-rw---- 1 nagios nagcmd 0 2009-10-15 13:32 nagios.cmd

It’s a pipe! Woohoo.

So we need to create it


root@nagios:/var/log/nagios/rw# mknod nagios.cmd p
root@nagios:/var/log/nagios/rw# chown nagios:apache nagios.cmd
root@nagios:/var/log/nagios/rw# chmod 660 nagios.cmd
root@nagios:/var/log/nagios/rw# ls -la
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:37 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:43 ..
prw-rw---- 1 nagios apache 0 Oct 30 13:37 nagios.cmd
root@nagios:/var/log/nagios/rw# /etc/init.d/nagios restart

A quick check of the site and .. same error? Still broken? Noooooooooooo

Lets have another look ..


root@mm2su0:/var/log/nagios/rw# ls -al
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:45 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:45 ..
prw-rw---- 1 nagios nagios 0 Oct 30 13:45 nagios.cmd

Nice, nagios changed the permissions for us so apache can’t write to it. I’m not setting apache to run as the nagios user 🙁

A look in the init file for nagios shows that it actually manages the file itself, so we didn’t need to actually make it (strange that it wasn’t there when I looked then ..) The init file actually handles creation and removal of the file :


root@nagios:/var/log/nagios/rw# /etc/init.d/nagios stop
Stopping nagios: done.
root@nagios:/var/log/nagios/rw# ls
root@nagios:/var/log/nagios/rw# /etc/init.d/nagios start
Starting nagios: done.
root@nagios:/var/log/nagios/rw# ls
nagios.cmd
root@nagios:/var/log/nagios/rw# ls
nagios.cmd
root@nagios:/var/log/nagios/rw# ls -al
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:52 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:52 ..
prw-rw---- 1 nagios nagios 0 Oct 30 13:52 nagios.cmd

To fix this I’m going to stick the apache user in the nagios group.

[/code]
root@nagios:/var/log/nagios/rw# usermod -G nagios apache
root@nagios:/var/log/nagios/rw# /etc/init.d/httpd restart
[/code]

No more error, problem solved!

Nagios

If you want to become a more advanced Nagios administrator, I recommend Nagios by O’Reilly. It’s full of best practice implementation advice.

Creating A Local and HTTP RedHat Yum Repository

We’re rolling out a RedHat platform for a major product delivery to a client in the next couple of weeks, which is a great chance for me to build a fresh platform using all the skills I’ve acquired and actually do things properly (TM).

We’re using RedHat Enterprise Linux 5 because this is a very critical deployment and we want an Operating System that is certified to both our hardware and software applications, something that unfortunately Ubuntu does not deliver.

The first stage of the deployment is to create our own RedHat RPM repository on the local network, so we can easily create servers.

I’m going to cover setting up 2 types of repository, initially a file system based repo and then an http based repo so that other servers can talk to the repo and snag RPMs from it.

To start with a repository is just a path with RPMs in it, regardless of file hierarchy, so we need to create this. I’m working with the RHEL 5 DVD so my first task is to simply copy the contents of the DVD on to the server’s hard drive.

First we mount the DVD


[rus@redhat ~]$ sudo mkdir /media/cdrom
[rus@redhat ~]$ sudo mount /dev/cdrom /media/cdrom/

And then we copy across the entire of the DVD to where we want the file system based repo to be!


[rus@redhat ~]$ sudo mkdir -p /var/repo/rhel5
[rus@redhat ~]$ sudo cp -r /media/cdrom/* /var/repo/rhel5
[rus@redhat ~]$ find /var/repo/rhel5 -name *.rpm | wc -l
3180

Ok, now we have a lovely directory structure with over 3000 RPMs just gagging to be installed in it 😉

Next up we need to create the Yum repository metadata, to do this we first need to install a tool called createrepo. Fortunately this is on the DVD image we just copied to our hard drive so we can sneakily install it with rpm -i 😉


[rus@redhat Server]$ cd /var/repo/rhel5/Server/
[rus@redhat Server]$ sudo rpm -i createrepo-0.4.11-3.el5.noarch.rpm
warning: createrepo-0.4.11-3.el5.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 37017186

Tada, createrepo is now installed! Do not worry about the NOKEY error, that will be explained and fixed shortly!

To create the metadata we simply run createrepo . in the repo directory.


[rus@redhat Server]$ cd /var/repo/
[rus@redhat repo]$ sudo createrepo .
3180/3180 - rhel5/VT/Virtualization-it-IT-5.2-11.noarch.rpm .rpm3.x86_64.rpm
Saving Primary metadata
Saving file lists metadata
Saving other metadata
[rus@redhat repo]$ ls
repodata rhel5
[rus@redhat repo]$ ls -al repodata/
total 13460
drwxr-xr-x 2 root root 4096 Oct 20 23:00 .
drwxr-xr-x 4 root root 4096 Oct 20 23:00 ..
-rw-r--r-- 1 root root 3178251 Oct 20 23:00 filelists.xml.gz
-rw-r--r-- 1 root root 9431119 Oct 20 23:00 other.xml.gz
-rw-r--r-- 1 root root 1108342 Oct 20 23:00 primary.xml.gz
-rw-r--r-- 1 root root 951 Oct 20 23:00 repomd.xml

As you can see we have lovely metadata now for our 3180 RPMs!

repomd.xml this is the file that describes the other metadata files. It is like an index file to point to the other files. It contains timestamps and checksums for the other files. This lets a client download this one, small file and know if anything else has changed. This also means that cryptographically (ex: gpg) signing this one file can ensure repository integrity.

primary.xml.gz this file stores the primary metadata information. This includes information such as name, epoch, version, release, architecture, file size, file location, description, summary, format, checksums header byte-ranges, dependencies, provides, conflicts, obsoletes, suggests, recommends,file lists for the package for CERTAIN files – specifically files matching: /etc*, *bin/*, /usr/lib/sendmail

filelists.xml.gz this file stores the complete file and directory listings for the packages. The package is identified by: name, epoch, version, release, architecture and package checksum id.

other.xml.gz this file currently only stores the changelog data from packages. However, this file could be used for any other additional metadata that could be useful for clients.

Ok, we now have our file system based repo, so we now need to tell Yum all about it!

Yum repository information is located in /etc/yum.repos.d as a series of files, it is possible to define multiple repositories in a single file, but I prefer to limit each file to one repository as in my opinion this makes it easier to manage, especially when implementing tools like Puppet.

Our repository is a local rhel repo, so create it vi /etc/yum.repo.d/rhel-local.repo


[rhel-local]
gpgcheck=1
name=Red Hat Linux $releasever - $basearch - DVD
baseurl=file:///var/repo

What’s going on here is quite simple:

We’ve created a repo labeled rhel-local. We’ve told Yum that the RPMs are signed using GPG, we’ve given it a name, and we’ve defined the baseurl of the repo to be a local file path!

To test we simply run yum list and you’ll see that packages are now available in the rhel-local repo!

That’s great, but we have to get on to the real task of creating a HTTP based Yum repository so we can use it to start building servers!! We’re going to serve the repo via the Apache HTTPD server so the first logical thing to do is to install it, using our new fancy pants local repo 😉


rus@redhat ~]$ sudo yum install httpd
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.3-31.el5 set to be updated
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd
--> Running transaction check
---> Package apr.x86_64 0:1.2.7-11.el5_3.1 set to be updated
---> Package apr-util.x86_64 0:1.2.7-7.el5_3.2 set to be updated
--> Processing Dependency: libpq.so.4()(64bit) for package: apr-util
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.1.11-1.el5_1.1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved
=========================================================================
Package Arch Version Repository Size
=========================================================================
Installing:
httpd x86_64 2.2.3-31.el5 rhel-local 1.2 M
Installing for dependencies:
apr x86_64 1.2.7-11.el5_3.1 rhel-local 119 k
apr-util x86_64 1.2.7-7.el5_3.2 rhel-local 75 k
postgresql-libs x86_64 8.1.11-1.el5_1.1 rhel-local 195 k
Transaction Summary
=========================================================================
Install 4 Package(s)
Update 0 Package(s)
Remove 0 Package(s)
Total download size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
————————————————————————-
Total 324 MB/s | 1.6 MB 00:00
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 37017186
Public key for apr-util-1.2.7-7.el5_3.2.x86_64.rpm is not installed

Oh god no, what is this error? Wait, that looks similar to the rpm NOKEY error we saw earlier! Relax, it’s simple!

Remember we set the repository to be GPG signed? Well, it is, but we don’t have the GPG key installed that we need to use to compare to the signings! This is easily fixable as the GPG keys came on the DVD, which we copied to the hard drive 😉


[rus@redhat ~]$ sudo rpm --import /var/repo/rhel5/RPM-GPG-KEY-redhat-beta /var/repo/rhel5/RPM-GPG-KEY-redhat-release

Now we can retry!


[rus@redhat ~]$ sudo yum install httpd
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.3-31.el5 set to be updated
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd
--> Running transaction check
---> Package apr.x86_64 0:1.2.7-11.el5_3.1 set to be updated
---> Package apr-util.x86_64 0:1.2.7-7.el5_3.2 set to be updated
--> Processing Dependency: libpq.so.4()(64bit) for package: apr-util
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.1.11-1.el5_1.1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved
==============================================================================
Package Arch Version Repository Size
==============================================================================
Installing:
httpd x86_64 2.2.3-31.el5 rhel-local 1.2 M
Installing for dependencies:
apr x86_64 1.2.7-11.el5_3.1 rhel-local 119 k
apr-util x86_64 1.2.7-7.el5_3.2 rhel-local 75 k
postgresql-libs x86_64 8.1.11-1.el5_1.1 rhel-local 195 k
Transaction Summary
==============================================================================
Install 4 Package(s)
Update 0 Package(s)
Remove 0 Package(s)
Total download size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
——————————————————————————
Total 559 MB/s | 1.6 MB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : apr 1/4
Installing : postgresql-libs 2/4
Installing : apr-util 3/4
Installing : httpd 4/4
Installed:
httpd.x86_64 0:2.2.3-31.el5
Dependency Installed:
apr.x86_64 0:1.2.7-11.el5_3.1 apr-util.x86_64 0:1.2.7-7.el5_3.2
postgresql-libs.x86_64 0:8.1.11-1.el5_1.1
Complete!

Disco, all done.

RedHat has a quirk though, a default install of most services means that the service will not start if the box is rebooted, in fact after installing an RPM of a service, like Apache it doesn’t even start the daemon! So we must do both, register the service to be started each boot and then start the server up!


[rus@redhat ~]$ sudo /sbin/chkconfig httpd on
[rus@redhat ~]$ sudo /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[ OK ]

Using chkconfig we can tell the system to start the service on boot!

Now we just need to configure Apache to use the repo we’ve previously set up!

There are an infinite number of ways to configure Apache to serve data, but I’m going to do it the simple and quick way and just use an Alias to do it. Create /etc/httpd/conf.d/repo.conf


Alias /repo/ "/var/repo/"

Now we just need to reload Apache


[rus@redhat ~]$ sudo /etc/init.d/httpd reload
Reloading httpd: [ OK ]

And we should be able to access the all important repomd.xml file with lynx :


[rus@redhat ~]$ lynx http://localhost/repo/repodata/repomd.xml

If Apache throws a 403 forbidden error error this could very well be SELinux getting in the way. If you are using SELinux you will need to change the Security Context of all the repo files so that Apache can read them!


[rus@redhat ~]$ sudo chcon -R -t httpd_sys_content_t /var/repo/

Lynx should now be able to download that file successfully.

The last thing to do now that we can retrieve files is to configure Yum to use the web based repo!

First we’re going to disable the existing local file server repo /etc/yum.repos.d/rhel-local.repo with the enabled=0 flag:


[rhel-local]
gpgcheck=1
enabled=0
name=Red Hat Linux $releasever - $basearch - DVD
baseurl=file:///var/repo

And create our new HTTP repo /etc/yum.repos.d/rhel-www.repo


[rhel-www]
gpgcheck=1
name=Red Hat Linux $releasever - $basearch - DVD
baseurl=http://localhost/repo/

To check it’s working we then just issue the following Yum commands to first clean up the repo and to then display their status!


[rus@redhat yum.repos.d]$ sudo yum clean all
Loaded plugins: rhnplugin, security
Cleaning up Everything
[rus@redhat yum.repos.d]$ sudo yum repolist all
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
rhel-www | 951 B 00:00
rhel-www/primary | 1.1 MB 00:00
rhel-www 3180/3180
repo id repo name status
rhel-debuginfo Red Hat Enterprise Linux 5Server - x86_64 - Debu disabled
rhel-local Red Hat Linux 5Server - x86_64 - DVD disabled
rhel-www Red Hat Linux 5Server - x86_64 - DVD enabled: 3,180
repolist: 3,180

Voila, one HTTP based RedHat Enterprise Linux RPM repository!

Fedora Linux O’Reilly’s Fedora Linux covers everything you need to know about dealing with Redhat systems and is a recommended read for anyone who wants to seriously get in to Redhat administration.

rpmdb: Lock table is out of available locker

I had a crazy weird bug today whilst running Puppet on one of our CentOS boxes where no packages were being installed. A quick investigation with yum yielded the following scenario!

Id I tried to use yum as root or with sudo it gave the following error ..


[idimmu@server ~]$ sudo yum check-update
rpmdb: Lock table is out of available locker entries
error: db4 error(22) from db->close: Invalid argument
error: cannot open Providename index using db3 - Cannot allocate memory (12)
Repository update is listed more than once in the configuration
Repository base is listed more than once in the configuration
Setting up repositories
http://www.mirrorservice.org/sites/mirror.centos.org/Null/updates/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Cannot open/read repomd.xml file for repository: update
failure: repodata/repomd.xml from update: [Errno 256] No more mirrors to try.
Error: failure: repodata/repomd.xml from update: [Errno 256] No more mirrors to try.

But worked fine if I did it as a non root user ..


[idimmu@server ~]$ yum check-update
Repository update is listed more than once in the configuration
Repository base is listed more than once in the configuration
Setting up repositories
Reading repository metadata in from local files

acl.i386 2.2.23-5.4.el4 base
apr-util.i386 0.9.4-22.el4 base
at.i386 3.1.8-82.el4 base
audit.i386 1.0.16-3.el4 base

I did a little bit of investigating with Google as to how yum works and discovered a few things..


http://www.mirrorservice.org/sites/mirror.centos.org/Null/updates/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found

The Null in the URl stood out as an obvious case of “there’s something wrong here”, so looking at the yum.conf file ..


baseurl=http://www.mirrorservice.org/sites/mirror.centos.org/$releasever/os/$basearch/

You can see that the NULL is coming from the $releasever variable. So, how is that variable set?

At the top of the yum.conf there is


distroverpkg=centos-release

What yum does is check which version of the package specified by distroverpkg is installed, and assign that to $releasever.

Great, but why does yum think the version is NULL? Well, a search for the first error ..


rpmdb: Lock table is out of available locker entries

resulted in this site which explained everything!

Basically RPM uses a Berkley database and due to too many instances of rpm dieing (the server is an old server) it left locks on the database that were never cleared. In order to fix the problem we have to remove the locks, by unfortunately, deleting the database and recreating it!

Backup the database directory /var/lib/rpm first:


tar cvzf rpmdb-backup.tar.gz /var/lib/rpm

Delete the locked Berkeley databases that rpm uses:


rm /var/lib/rpm/__db.00*

Force rpm to rebuild the databases


rpm --rebuilddb

Now, check the database to make sure it has been rebuilt correctly:


rpm -qa | sort

Problem solved, my database is no longer corrupt and Puppet is installing packages again!

Fedora Linux O’Reilly’s Fedora Linux covers everything you need to know about dealing with Redhat systems and is a recommended read for anyone who wants to seriously get in to Redhat administration.