idimmu . net

A web based memory replacement service.

Python support for SSL and HTTPS is not installed November 11, 2009

Posted by idimmu in mac os x, python.
I wanted to give Google's new Go language a try which requires checking out the source code with Mercurial.

Mercurial was installed from ports on my Mac with


sudo port install mecurial


But checking out the code threw an error.


Chill:~ rus$ hg clone -r release https://go.googlecode.com/hg/ $GOROOT
abort: Python support for SSL and HTTPS is not installed
Exception exceptions.AttributeError: "'httpsrepository' object has no attribute 'urlopener'" in <bound method httpsrepository.__del__ of <mercurial.httprepo.httpsrepository object at 0x3bb910>> ignored


After some hunting I found the answer is to install the python socket ssl library from ports


Chill:~ rus$ sudo port install py25-socket-ssl
---> Fetching py25-socket-ssl
---> Verifying checksum(s) for py25-socket-ssl
---> Extracting py25-socket-ssl
---> Configuring py25-socket-ssl
---> Building py25-socket-ssl
---> Staging py25-socket-ssl into destroot
---> Installing py25-socket-ssl @2.5.4_0
---> Activating py25-socket-ssl @2.5.4_0
---> Cleaning py25-socket-ssl


Which then let me check out the source for Go!


Chill:~ rus$ hg clone -r release https://go.googlecode.com/hg/ $GOROOT
requesting all changes
adding changesets
adding manifests
adding file changes
added 3976 changesets with 16799 changes to 2931 files
updating working directory
1640 files updated, 0 files merged, 0 files removed, 0 files unresolved


Now it's time to compile :D

Error: Could not open command file '/var/log/nagios/rw/nagios.cmd' for update! on RedHat October 30, 2009

Posted by idimmu in linux, redhat.
I'm currently setting up some new clusters under RedHat and each cluster is getting it's own Nagios instance, trying to use the web based management interface however threw and error.

I've seen this error before on Ubuntu and was getting it again under RedHat. Of course I revisited my Ubuntu solution and realised that it didn't help at all, due to using dpkg overrides, also the situation was very different!


root@nagios:/var/log/nagios/rw# ls -al
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:37 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:40 ..


The file didn't exist :(

A quick scan of a working system showed :


prw-rw---- 1 nagios nagcmd 0 2009-10-15 13:32 nagios.cmd


It's a pipe! Woohoo.

So we need to create it


root@nagios:/var/log/nagios/rw# mknod nagios.cmd p
root@nagios:/var/log/nagios/rw# chown nagios:apache nagios.cmd
root@nagios:/var/log/nagios/rw# chmod 660 nagios.cmd
root@nagios:/var/log/nagios/rw# ls -la
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:37 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:43 ..
prw-rw---- 1 nagios apache 0 Oct 30 13:37 nagios.cmd
root@nagios:/var/log/nagios/rw# /etc/init.d/nagios restart


A quick check of the site and .. same error? Still broken? Noooooooooooo

Lets have another look ..


root@mm2su0:/var/log/nagios/rw# ls -al
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:45 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:45 ..
prw-rw---- 1 nagios nagios 0 Oct 30 13:45 nagios.cmd


Nice, nagios changed the permissions for us so apache can't write to it. I'm not setting apache to run as the nagios user :(

A look in the init file for nagios shows that it actually manages the file itself, so we didn't need to actually make it (strange that it wasn't there when I looked then ..) The init file actually handles creation and removal of the file :


root@nagios:/var/log/nagios/rw# /etc/init.d/nagios stop
Stopping nagios: done.
root@nagios:/var/log/nagios/rw# ls
root@nagios:/var/log/nagios/rw# /etc/init.d/nagios start
Starting nagios: done.
root@nagios:/var/log/nagios/rw# ls
nagios.cmd
root@nagios:/var/log/nagios/rw# ls
nagios.cmd
root@nagios:/var/log/nagios/rw# ls -al
total 8
drwxrwxr-x 2 nagios apache 4096 Oct 30 13:52 .
drwxrwxr-x 5 nagios apache 4096 Oct 30 13:52 ..
prw-rw---- 1 nagios nagios 0 Oct 30 13:52 nagios.cmd


To fix this I'm going to stick the apache user in the nagios group.

[/code]
root@nagios:/var/log/nagios/rw# usermod -G nagios apache
root@nagios:/var/log/nagios/rw# /etc/init.d/httpd restart
[/code]

No more error, problem solved!

Creating A Local and HTTP RedHat Yum Repository October 20, 2009

Posted by idimmu in linux, redhat.
We're rolling out a RedHat platform for a major product delivery to a client in the next couple of weeks, which is a great chance for me to build a fresh platform using all the skills I've acquired and actually do things properly (TM).

We're using RedHat Enterprise Linux 5 because this is a very critical deployment and we want an Operating System that is certified to both our hardware and software applications, something that unfortunately Ubuntu does not deliver.

The first stage of the deployment is to create our own RedHat RPM repository on the local network, so we can easily create servers.

I'm going to cover setting up 2 types of repository, initially a file system based repo and then an http based repo so that other servers can talk to the repo and snag RPMs from it.

To start with a repository is just a path with RPMs in it, regardless of file hierarchy, so we need to create this. I'm working with the RHEL 5 DVD so my first task is to simply copy the contents of the DVD on to the server's hard drive.


First we mount the DVD


[rus@redhat ~]$ sudo mkdir /media/cdrom
[rus@redhat ~]$ sudo mount /dev/cdrom /media/cdrom/


And then we copy across the entire of the DVD to where we want the file system based repo to be!


[rus@redhat ~]$ sudo mkdir -p /var/repo/rhel5
[rus@redhat ~]$ sudo cp -r /media/cdrom/* /var/repo/rhel5
[rus@redhat ~]$ find /var/repo/rhel5 -name *.rpm | wc -l
3180


Ok, now we have a lovely directory structure with over 3000 RPMs just gagging to be installed in it ;)

Next up we need to create the Yum repository metadata, to do this we first need to install a tool called createrepo. Fortunately this is on the DVD image we just copied to our hard drive so we can sneakily install it with rpm -i ;)


[rus@redhat Server]$ cd /var/repo/rhel5/Server/
[rus@redhat Server]$ sudo rpm -i createrepo-0.4.11-3.el5.noarch.rpm
warning: createrepo-0.4.11-3.el5.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 37017186


Tada, createrepo is now installed! Do not worry about the NOKEY error, that will be explained and fixed shortly!

To create the metadata we simply run createrepo . in the repo directory.


[rus@redhat Server]$ cd /var/repo/
[rus@redhat repo]$ sudo createrepo .
3180/3180 - rhel5/VT/Virtualization-it-IT-5.2-11.noarch.rpm .rpm3.x86_64.rpm
Saving Primary metadata
Saving file lists metadata
Saving other metadata
[rus@redhat repo]$ ls
repodata rhel5
[rus@redhat repo]$ ls -al repodata/
total 13460
drwxr-xr-x 2 root root 4096 Oct 20 23:00 .
drwxr-xr-x 4 root root 4096 Oct 20 23:00 ..
-rw-r--r-- 1 root root 3178251 Oct 20 23:00 filelists.xml.gz
-rw-r--r-- 1 root root 9431119 Oct 20 23:00 other.xml.gz
-rw-r--r-- 1 root root 1108342 Oct 20 23:00 primary.xml.gz
-rw-r--r-- 1 root root 951 Oct 20 23:00 repomd.xml


As you can see we have lovely metadata now for our 3180 RPMs!

repomd.xml this is the file that describes the other metadata files. It is like an index file to point to the other files. It contains timestamps and checksums for the other files. This lets a client download this one, small file and know if anything else has changed. This also means that cryptographically (ex: gpg) signing this one file can ensure repository integrity.

primary.xml.gz this file stores the primary metadata information. This includes information such as name, epoch, version, release, architecture, file size, file location, description, summary, format, checksums header byte-ranges, dependencies, provides, conflicts, obsoletes, suggests, recommends,file lists for the package for CERTAIN files - specifically files matching: /etc*, *bin/*, /usr/lib/sendmail

filelists.xml.gz this file stores the complete file and directory listings for the packages. The package is identified by: name, epoch, version, release, architecture and package checksum id.

other.xml.gz this file currently only stores the changelog data from packages. However, this file could be used for any other additional metadata that could be useful for clients.

Ok, we now have our file system based repo, so we now need to tell Yum all about it!

Yum repository information is located in /etc/yum.repos.d as a series of files, it is possible to define multiple repositories in a single file, but I prefer to limit each file to one repository as in my opinion this makes it easier to manage, especially when implementing tools like Puppet.

Our repository is a local rhel repo, so create it vi /etc/yum.repo.d/rhel-local.repo


[rhel-local]
gpgcheck=1
name=Red Hat Linux $releasever - $basearch - DVD
baseurl=file:///var/repo


What's going on here is quite simple:

We've created a repo labeled rhel-local. We've told Yum that the RPMs are signed using GPG, we've given it a name, and we've defined the baseurl of the repo to be a local file path!

To test we simply run yum list and you'll see that packages are now available in the rhel-local repo!

That's great, but we have to get on to the real task of creating a HTTP based Yum repository so we can use it to start building servers!! We're going to serve the repo via the Apache HTTPD server so the first logical thing to do is to install it, using our new fancy pants local repo ;)


rus@redhat ~]$ sudo yum install httpd
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.3-31.el5 set to be updated
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd
--> Running transaction check
---> Package apr.x86_64 0:1.2.7-11.el5_3.1 set to be updated
---> Package apr-util.x86_64 0:1.2.7-7.el5_3.2 set to be updated
--> Processing Dependency: libpq.so.4()(64bit) for package: apr-util
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.1.11-1.el5_1.1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================
Package Arch Version Repository Size
=========================================================================
Installing:
httpd x86_64 2.2.3-31.el5 rhel-local 1.2 M
Installing for dependencies:
apr x86_64 1.2.7-11.el5_3.1 rhel-local 119 k
apr-util x86_64 1.2.7-7.el5_3.2 rhel-local 75 k
postgresql-libs x86_64 8.1.11-1.el5_1.1 rhel-local 195 k

Transaction Summary
=========================================================================
Install 4 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
-------------------------------------------------------------------------
Total 324 MB/s | 1.6 MB 00:00
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 37017186

Public key for apr-util-1.2.7-7.el5_3.2.x86_64.rpm is not installed


Oh god no, what is this error? Wait, that looks similar to the rpm NOKEY error we saw earlier! Relax, it's simple!

Remember we set the repository to be GPG signed? Well, it is, but we don't have the GPG key installed that we need to use to compare to the signings! This is easily fixable as the GPG keys came on the DVD, which we copied to the hard drive ;)


[rus@redhat ~]$ sudo rpm --import /var/repo/rhel5/RPM-GPG-KEY-redhat-beta /var/repo/rhel5/RPM-GPG-KEY-redhat-release


Now we can retry!


[rus@redhat ~]$ sudo yum install httpd
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.3-31.el5 set to be updated
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd
--> Running transaction check
---> Package apr.x86_64 0:1.2.7-11.el5_3.1 set to be updated
---> Package apr-util.x86_64 0:1.2.7-7.el5_3.2 set to be updated
--> Processing Dependency: libpq.so.4()(64bit) for package: apr-util
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.1.11-1.el5_1.1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================
Package Arch Version Repository Size
==============================================================================
Installing:
httpd x86_64 2.2.3-31.el5 rhel-local 1.2 M
Installing for dependencies:
apr x86_64 1.2.7-11.el5_3.1 rhel-local 119 k
apr-util x86_64 1.2.7-7.el5_3.2 rhel-local 75 k
postgresql-libs x86_64 8.1.11-1.el5_1.1 rhel-local 195 k

Transaction Summary
==============================================================================
Install 4 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
------------------------------------------------------------------------------
Total 559 MB/s | 1.6 MB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : apr 1/4
Installing : postgresql-libs 2/4
Installing : apr-util 3/4
Installing : httpd 4/4

Installed:
httpd.x86_64 0:2.2.3-31.el5

Dependency Installed:
apr.x86_64 0:1.2.7-11.el5_3.1 apr-util.x86_64 0:1.2.7-7.el5_3.2
postgresql-libs.x86_64 0:8.1.11-1.el5_1.1

Complete!


Disco, all done.

RedHat has a quirk though, a default install of most services means that the service will not install if the box is rebooted, in fact after installing an RPM of a service, like Apache it doesn't even start the daemon! So we must do both, register the service to be started each boot and then start the server up!


[rus@redhat ~]$ sudo /sbin/chkconfig httpd on
[rus@redhat ~]$ sudo /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[ OK ]


Using chkconfig we can tell the system to start the service on boot!

Now we just need to configure Apache to use the repo we've previously set up!

There are an infinite number of ways to configure Apache to serve data, but I'm going to do it the simple and quick way and just use an Alias to do it. Create /etc/httpd/conf.d/repo.conf


Alias /repo/ "/var/repo/"


Now we just need to reload Apache


[rus@redhat ~]$ sudo /etc/init.d/httpd reload
Reloading httpd: [ OK ]


And we should be able to access the all important repomd.xml file with lynx :


[rus@redhat ~]$ lynx http://localhost/repo/repodata/repomd.xml


If Apache throws a 403 forbidden error error this could very well be SELinux getting in the way. If you are using SELinux you will need to change the Security Context of all the repo files so that Apache can read them!


[rus@redhat ~]$ sudo chcon -R -t httpd_sys_content_t /var/repo/


Lynx should now be able to download that file successfully.

The last thing to do now that we can retrieve files is to configure Yum to use the web based repo!

First we're going to disable the existing local file server repo /etc/yum.repos.d/rhel-local.repo with the enabled=0 flag:


[rhel-local]
gpgcheck=1
enabled=0
name=Red Hat Linux $releasever - $basearch - DVD
baseurl=file:///var/repo


And create our new HTTP repo /etc/yum.repos.d/rhel-www.repo


[rhel-www]
gpgcheck=1
name=Red Hat Linux $releasever - $basearch - DVD
baseurl=http://localhost/repo/


To check it's working we then just issue the following Yum commands to first clean up the repo and to then display their status!


[rus@redhat yum.repos.d]$ sudo yum clean all
Loaded plugins: rhnplugin, security
Cleaning up Everything
[rus@redhat yum.repos.d]$ sudo yum repolist all
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
rhel-www | 951 B 00:00
rhel-www/primary | 1.1 MB 00:00
rhel-www 3180/3180
repo id repo name status
rhel-debuginfo Red Hat Enterprise Linux 5Server - x86_64 - Debu disabled
rhel-local Red Hat Linux 5Server - x86_64 - DVD disabled
rhel-www Red Hat Linux 5Server - x86_64 - DVD enabled: 3,180
repolist: 3,180


Voila, one HTTP based RedHat Enterprise Linux RPM repository!

Bash Shortcuts August 20, 2009

Posted by idimmu in linux.
Meta-< -> go to beginning of history file
Meta-> -> go to end of history file
Meta-f -> go to next word in command line
Meta-b -> go to previous word in command line
Meta-d -> delete next word in command line (from the actual position of the prompt)

Ctrl-a -> go to the start of command line
Ctrl-e -> go to the end of command line
Ctrl-p -> previous command in history
Ctrl-n -> next command in history
Ctrl-f -> next character in command line
Ctrl-b -> previous character in command line
Ctrl-r -> reverse search in history file
Ctrl-d -> delete current character
Ctrl-k -> delete from the prompt to the end of command line
Ctrl-_ -> undo (yes, but limited)

Burning an XVid to DVD in Ubuntu Jaunty from the command line July 8, 2009

Posted by idimmu in linux.
I needed to burn an XVid of one of my performances to DVD last night, remotely, as my MythBuntu box was being used by someone else to watch some crappy TV and I wanted to show off! It was a pretty simple process, just involving a bit of a wait during the transcode :)

First you will need to make install the relevent packages:

sudo aptitude install dvdauthor ffmpeg ~nlibav.+-unstripped.+

If you already have dvdauthor and ffmpeg installed, you must install the unstripped libav packages as they contain a tool called mpeg2video and it's this that does the heavy grunt work!

Next we transcode the movie and create a DVD compatible MPEG. This is the part that takes the longest to do.

ffmpeg -i performance.avi -target dvd -aspect 16:9 -sameq performance.mpg

Now we create the DVD structure, the VIDEO_TS etc. etc.


mkdir DVD
dvdauthor --title -f performance.mpg -o DVD
dvdauthor -T -o DVD


Which results in a structure that looks a lot like this:


DVD
DVD/VIDEO_TS
DVD/VIDEO_TS/VTS_01_1.VOB
DVD/VIDEO_TS/VIDEO_TS.BUP
DVD/VIDEO_TS/VTS_01_0.IFO
DVD/VIDEO_TS/VTS_01_3.VOB
DVD/VIDEO_TS/VTS_01_2.VOB
DVD/VIDEO_TS/VIDEO_TS.IFO
DVD/VIDEO_TS/VTS_01_0.BUP
DVD/AUDIO_TS


Then we create an iso image from that structure which we can then use to burn to a blank DVD

mkisofs -dvd-video -o performance.iso DVD

There are a million ways to burn the iso image, I actually SCP'd it to my MacBook and used the Disk Utility tool to burn it, but from the Ubuntu CD DVD Burning documentation you can use a tool called wodim!

You will need to install wodim

apt-get install wodim

list available DVD writers

wodim --devices
wodim: Overview of accessible drives (1 found) :
-------------------------------------------------------------------------
0 dev='/dev/scd0' rwrw-- : 'LITE-ON' 'DVDRW SOHW-1633S'
-------------------------------------------------------------------------

Insert a disk, then burn!

wodim dev=/dev/scd0 driveropts=burnfree -v -data performance.iso