Odd Occurrences In Apache Access Logs

Apache I’ve been watching my Awstats installation this month as idimmu.net is about to peak with over 7000 unique visitors in a month for the first time in it’s history, which is pretty awesome. But there’s been something really weird going on in the results ..

This is kind of ironic as in a recent job interview I was asked

 

What would you look for to ascertain suspicious activity on an instance of Apache serving static image assets?

Obviously I aced the question, and whilst my server isn’t limited to static assets, it does have the GNU tool chain installed 😉

Awstats

6487 views for my Elgg CSS Fix page this month. For the record at the time of writing the site has had 6908 unique views and the next highest viewed page is only at 2446 views. Something is up! I wonder what ..


root@holly /var/log/apache2 # grep elgg idimmu.net.access.log | head
91.198.94.225 - - [24/Mar/2013:06:44:52 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "http://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:44:52 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "http://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:44:53 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "http://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:44:53 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "http://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:54 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "http://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:54 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "http://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:54 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "http://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:55 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "http://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
216.152.249.243 - - [24/Mar/2013:06:49:34 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.1" 200 9931 "http://www.idimmu.net/" "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.0) Opera 7.02 Bork-edition [en]"
91.198.94.225 - - [24/Mar/2013:06:50:55 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37927 "http://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Opera/9.80 (Windows NT 6.2; WOW64) Presto/2.12.388 Version/12.11"

Ok, so my access log rotated out on the 24th March but the data is still meaningful with out having to put together exact logs for the entire month! One IP 91.198.94.225 seems to be retrieving the Elgg page over and over again, several times a minute?!


root@holly /var/log/apache2 # grep 91.198.94.225 idimmu.net.access.log | wc -l
5876
root@holly /var/log/apache2 # grep elgg idimmu.net.access.log | grep 91.198.94.225 | wc -l
5876

And seemingly that same IP address is ONLY requesting my Elgg page, no other pages!


root@holly /var/log/apache2 # grep index.php idimmu.net.access.log | grep 91.198.94.225 | head -n 1
91.198.94.225 - - [24/Mar/2013:06:44:53 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "http://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
root@holly /var/log/apache2 # grep index.php idimmu.net.access.log | grep 91.198.94.225 | wc -l
2938
root@holly /var/log/apache2 # grep -v index.php idimmu.net.access.log | grep 91.198.94.225 | wc -l
2938

Also half of it’s requests are for _/2011/11/21/elgg-1-8-tidypics-group-fix/index.php_ and the other half are for _/2011/11/21/elgg-1-8-tidypics-group-fix_ which is double weird. The index.php page extension just redirects to the extension-less URL due to the incredible intellectual artificial intelligence running the CMS platform I use!

The Elgg page has maybe 1 or 2 back links out there, none are to the index.php extended URL and looking at the Apache logs the referrer is http://www.idimmu.net/index.php which isn’t a valid URL anyway, which suggests whatever is doing this is spoofing the referrer.

Also also, what kind of browser definition is

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11

Some random infosec page I’ve never heard of suggests it’s a valid Chrome user agent running on Windows. You can write lots of interesting plugins for Chrome, including scrapers and bots so this is possibly a valid option. Why it also mentions Safari I’ve no idea though!

The 91.198.94.225 IP appears in Google several times for spamming so possibly that’s the point of the bot. Judging by the rest of the search results there appears to be a LOT of comment spam to old PHP guestbook implementations that look to have seemingly trivial captures, so maybe the bot thinks my site is running one of these? It does appear to be stuck in some kind of infinite loop on one page however 🙁

Whois says it’s an IP located in Poland! Has anyone else seen any ‘interesting’ impressions from this IP address, or anything similar? Fortunately I’m pretty good with computers.


iptables -A INPUT -s 91.198.94.225 -j DROP

In your face Polish!

Linux iptables Pocket Reference For more iptables related nonsense I suggest you try O’Reilly’s Linux iptables Pocket Reference or man iptables if you don’t want to buy anything 😉 Or you know, just keep reading my stuff!

Apache2 with SSL and Tomcat5.5 on Ubuntu

One of the newer features to our site is an access control mechanism to force specific paths to only be delivered over SSL when our customers have particularly sensitive data. We already use Apache2 with mod_jk to talk to the Tomcat5.5 instance running our app so the only part left is to enable SSL!

First make sure mod_ssl is enabled:


root@reltest-tcj0:/var/log/apache2# a2enmod
Which module would you like to enable?
Your choices are: actions asis auth_anon auth_dbm auth_digest auth_ldap cache cern_meta cgid cgi dav_fs dav deflate disk_cache expires ext_filter file_cache headers imap include info jk ldap mem_cache mime_magic proxy_connect proxy_ftp proxy_http proxy rewrite speling ssl suexec unique_id userdir usertrack vhost_alias
Module name? ssl
This module is already enabled!

Then we configure mod_jk to pass it’s SSL environment variables to Tomcat by adding the following to apache2.conf


JkExtractSSL On
JkHTTPSIndicator HTTPS
JkSESSIONIndicator SSL_SESSION_ID
JkCIPHERIndicator SSL_CIPHER
JkCERTSIndicator SSL_CLIENT_CERT

Tell Apache2 to listen on the SSL port by editing ports.conf


Listen 443

We want to make sure we have the latest common CA certificates in order to establish a trusted root for our new shiny signed certificate!


apt-get install ca-certificates

If you have a lovely genuinely signed certificate like we do you might need to then add it’s intermediate certificate to the ca-certificates system. Move the certificate to /usr/share/ca-certificates then add it’s location to /etc/ca-certificates.conf

Now run update-ca-certificates to update the system’s certificate store (located in /etc/ssl/certs/ca-certificates.crt).


root@reltest-tcj0:/etc/apache2/sites-enabled# update-ca-certificates
Updating certificates in /etc/ssl/certs....done.

We want the same site to simply be available over SSL I’m going to duplicate the existing VirtualHost for that site specifying the use of port 80 for the original vhost and port 443 for the new one that uses SSL. The only change that needs to be made to the new vhost are the following SSL directives:


SSLEngine On
SSLCertificateFile /etc/apache2/ssl/domain.com.crt
SSLCertificateKeyFile /etc/apache2/ssl/domain.com.key
SSLCACertificateFile /etc/ssl/certs/ca-certificates.crt

Obviously making sure the keys are in the right place!

And lastly make sure that NameVirtualHost settings exist for both port 80 and port 443!


NameVirtualHost *:80
NameVirtualHost *:443

et voila.

Apache2 ldap auth on Ubuntu Dapper and Feisty

As part of our internal office systems upgrade we have a shiny new LDAP server which we like to use as much as possible. One of the things we use it for is Apache user auth, mainly we control SVN with it so people can only commit to the projects they’re allowed to but we also use it so secure our system’s services from the developers that like to play wannabe sysadmin!

Unfortunately we are running several different flavors of Ubuntu in the office with slightly different Apache2 versions and thus LDAP requirements!

Ubuntu Dapper Drake (Apache 2.0)


AuthType basic
AuthName "BackupPC admin"
AuthLDAPUrl ldap://ldap-server:389/ou=people,dc=domain,dc=com?uid?sub
AuthLDAPGroupAttributeIsDN off
AuthLDAPEnabled on
Require group cn=systems,ou=groups,dc=domain,dc=com
AuthLDAPGroupAttribute memberUid

Ubuntu Feisty Fawn (Apache 2.22)


AuthType Basic
AuthName "SVN Repository"
AuthLDAPUrl ldap://ldap-server:389/ou=people,dc=domain,dc=com?uid?sub
AuthzLDAPAuthoritative On
AuthBasicProvider ldap
AuthLDAPGroupAttribute memberUid
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=developers,ou=groups,dc=domain,dc=com

Obviously you have to make sure you have the right LDAP modules enabled for each version of Apache2 but that’s all that is required to force Apache2 to authenticate against an LDAP group!

Configuring Tomcat 5.5 and Apache 2 with mod_jk

mod_jk is a conduit between a web server and Tomcat, it supports a variety of web servers including IIS. Using mod_jk to put Apache in front of Tomcat lets you use all the power of Apache (caching, gzip, mod_rewrite, etc) whilst at the same time serving content from Tomcat, also with Ubuntu it’s really easy to set up!

First of all install the software, you will need to enable the backports repository on Dapper for this.


apt-get install sun-java6-bin sun-java6-jdk tomcat5.5 libapache2-mod-jk

The Tomcat 5.5 that comes with Ubuntu already has an AJP connector configured on port 8009 so there is no additional configuration to do to it’s server.xml file.

We then need to configure a worker.properties file for Apache2 which tells it about the Tomcat instance, I make mine in /etc/apache2/worker.properties


worker.list=idimmu
worker.idimmu.type=ajp13
worker.idimmu.host=localhost
worker.idimmu.port=8009

Make sure mod_jk is then enabled with a2enmod jk (it probably already is).

And finally we tell Apache2 about the worker instance in /etc/apache2/apache2.conf


JkWorkersFile /etc/apache2/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel info
JkMount /* idimmu

This will direct any requests to the Apache2 server to the Tomcat server