checking whether to enable multibyte string support… no
checking whether to enable multibyte regex support… yes
checking whether to check multibyte regex backtrack… yes
checking for external libmbfl… no
checking for mcrypt support… yes
configure: error: mcrypt.h not found. Please reinstall libmcrypt.

Checking if the package exists in the default repos shows it doesn’t,

# yum search libmcrypt
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
* base: mirrors.versaweb.com
* extras: mirrors.usc.edu
* updates: mirrors.easynews.com
Warning: No matches found for: libmcrypt
No Matches found

You need to obtain this from the epel repos now at http://dl.fedoraproject.org/pub/epel/6/x86_64/ , at the time of writing the version is 2.5.8-9 so you can install this with,

mcryptver=2.5.8-9
arch=x86_64
rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/libmcrypt-$mcryptver.el6.$arch.rpm
rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/libmcrypt-devel-$mcryptver.el6.$arch.rpm

An example,

# mcryptver=2.5.8-9
# arch=x86_64
# rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/libmcrypt-$mcryptver.el6.$arch.rpm
Retrieving http://dl.fedoraproject.org/pub/epel/6/x86_64/libmcrypt-2.5.8-9.el6.x86_64.rpm
warning: /var/tmp/rpm-tmp.BoLykl: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing… ########################################### [100%]
1:libmcrypt ########################################### [100%]
# rpm -Uhv http://dl.fedoraproject.org/pub/epel/6/x86_64/libmcrypt-devel-$mcryptver.el6.$arch.rpm
Retrieving http://dl.fedoraproject.org/pub/epel/6/x86_64/libmcrypt-devel-2.5.8-9.el6.x86_64.rpm
warning: /var/tmp/rpm-tmp.6SW2Vh: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing… ########################################### [100%]
1:libmcrypt-devel ########################################### [100%]

After doing the rewrite for nginx,

location ^~ /common/ {
fastcgi_split_path_info ^(/common)(/?.+)$;
fastcgi_param SCRIPT_FILENAME /vhosts/shared/common$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_pass 127.0.0.1:9000;
include fastcgi_params;
}

nginx was simply returning access denied.

# curl http://localhost/common/file
Access denied.

Which is because the new file has no extension and PHP-FPM now defaults to only allow .php. This can be altered by editing security.limit_extensions in the php-fpm.conf (in this case /etc/php-fpm.d/daemon.conf) to allow the extensions you need. In this case we needed it to support files without extensions so setting security.limit_extensions to FALSE done the job,

security.limit_extensions = FALSE

[initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
[initandlisten] can’t create new thread, closing connection

There is a bug fix in the init.d available per https://jira.mongodb.org/browse/SERVER-6443 to increase the ulimit -u however while working through increasing the available max processes on RHEL I was noting the soft limit would not increase past 1024 when checking /proc/pid/limits even when being increased in the startup script.

Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 256662 processes
Max open files 8192 8192 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256662 256662 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

It turns out the pam package provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes,

# Default limit for number of user’s processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

Changing this allowed the number of processes to be increased.

I have written a check for nagios/icinga to check the varnish backend health. You can get the latest version from github . The version at the time of posting at the bottom of this message.

The defaults are RHEL defaults and ./check_varnishbackends.py should work without any options unless you want to override them. Here is the Nagios/Icinga Setup,

Register the service

define service {
	register			0
	name				check_varnishbackends
	service_description		Varnish Backends
	use				generic-service
}

Assign to host “server”

define service {
	service_description		Varnish Backend
	host_name			server
	use				check_varnishbackends
	check_command			remote_nrpe!check_varnishbackends -a "127.0.0.1" "6082" "/etc/varnish/secret" "/usr/bin/varnishadm"
}

Setup NRPE (Make sure user has permission to varnish secret file or use sudo)

command[check_varnishbackends]=/usr/local/nagios/libexec/check_varnishbackends.py --host $ARG1$ --port $ARG2$ --secret $ARG3$ --path $ARG4$

Version at time of posting,

#!/usr/bin/env python
# Nagios Varnish Backend Check
# v1.1
# URL: www.admingeekz.com
# Contact: sales@admingeekz.com
#
#
# Copyright (c) 2013, AdminGeekZ Ltd
# All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License, version 2, as
# published by the Free Software Foundation.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
#

import sys
import optparse
import subprocess

def runcommand(command, exit_on_fail=True):
    try:
      process = subprocess.Popen(command.split(" "), stdout=subprocess.PIPE)
      output, unused_err = process.communicate()
      retcode = process.poll()
      return output

    except OSError, e:
      print "Error: Executing command failed,  does it exist?"
      sys.exit(2)


def main(argv):
  o = optparse.OptionParser(conflict_handler="resolve", description="Nagios plugin to check varnish backend health.")
  o.add_option('-H', '--host', action='store', type='string', dest='host', default='127.0.0.1', help='The ip varnishadm is listening on')
  o.add_option('-P', '--port', action='store', type='int', dest='port', default=6082, help='The port varnishadm is listening on')
  o.add_option('-s', '--secret', action='store', type='string', dest='secret', default='/etc/varnish/secret', help='The path to the secret file')
  o.add_option('-p', '--path', action='store', type='string', dest='path', default='/usr/bin/varnishadm', help='The path to the varnishadm binary')

  options=o.parse_args()[0]
  command = runcommand("%(path)s -S %(secret)s -T %(host)s:%(port)s debug.health" % options.__dict__)
  backends = command.split("\n")
  backends_healthy, backends_sick = [], []
  for line in backends:
    if line.startswith("Backend") and line.find("test")==-1:
      if line.endswith("Healthy"):
        backends_healthy.append(line.split(" ")[1])
      else:
        backends_sick.append(line.split(" ")[1])
 
  if backends_sick:
    print "%s backends are down.  %s" % (len(backends_sick), "".join(backends_sick))
    sys.exit(2)

  if not backends_sick and not backends_healthy:
    print "No backends detected.  If this is an error, see readme.txt"
    sys.exit(1)

  print "All %s backends are healthy" % (len(backends_healthy))
  sys.exit(0)


if __name__ == "__main__":
    main(sys.argv[1:])

ERROR: Module backupdriver is in use

r1soft-uninstall-buagent -R
This uninstaller will remove the following..

File /etc/init.d/buagent
File /usr/bin/buagent
File /var/lock/buagent.lock
File /var/run/buagent.pid
File /usr/bin/r1key
File /usr/bin/r1soft-cki
File /usr/bin/r1soft-report.sh
File /usr/bin/r1soft-uninstall-buagent
File /usr/share/man/man3/buagent.3.gz
File /usr/share/man/man3/r1key.3.gz
File /usr/share/man/man3/r1soft-uninstall-buagent.3.gz
Directory /etc/buagent
Directory /usr/lib/buagent
Directory /lib/modules/buagent
Directory /var/cache/buagent
Directory /var/db/buagent
User buagent
Group buagent

Would you like to continue? (yes/no):yes
stopping buagent.
removing buagent from services.
removing File /etc/init.d/buagent
removing File /usr/bin/buagent
removing File /var/lock/buagent.lock
removing File /var/run/buagent.pid
removing File /usr/bin/r1key
removing File /usr/bin/r1soft-cki
removing File /usr/bin/r1soft-report.sh
removing File /usr/bin/r1soft-uninstall-buagent
removing File /usr/share/man/man3/buagent.3.gz
removing File /usr/share/man/man3/r1key.3.gz
removing File /usr/share/man/man3/r1soft-uninstall-buagent.3.gz
removing Directory /etc/buagent
removing Directory /usr/lib/buagent
removing Directory /lib/modules/buagent
removing Directory /var/cache/buagent
removing Directory /var/db/buagent
removing buagent user…
unloading backupdriver kernel module…
ERROR: Module backupdriver is in use
done.

Trying to simply rmmod the driver fails because it’s in use but without having forced support there is no way to unload it using this method.


# rmmod backupdriver
ERROR: Module backupdriver is in use


The solution is to reinstall the latest CDP 2 agent and run buagent -u to unload it first then proceed to uninstalling the CDP2 agent and installing CDP3 agent.

Removed ext_addr from “base_features” and commented out the inode_size line. Here is a diff from the default ubuntu e2fsprogs /etc/mke2fs.conf with the necessary changes,

— /etc/mke2fs.orig.conf 2011-11-28 15:20:07.365231775 +0000
+++ /etc/mke2fs.conf 2011-11-28 15:20:48.165231803 +0000
@@ -1,7 +1,7 @@
[defaults]
– base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
+ base_features = sparse_super,filetype,resize_inode,dir_index
blocksize = 4096
– inode_size = 256
+ #inode_size = 256
inode_ratio = 16384

[fs_types]

Then format the partition and this will allow it to be mounted on windows.

I used DiskInternals Linux Reader on windows to mount the drive and copy the files which can be downloaded for free at


height: 120px;
overflow: auto;
font-family: “Consolas”,monospace;
font-size: 9pt;
text-align:left;
background-color: #FCF7EC;
overflow-x: auto; /* Use horizontal scroller if needed; for Firefox 2, not
white-space: pre-wrap; /* css-3 */
white-space: -moz-pre-wrap !important; /* Mozilla, since 1999 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
margin: 0px 0px 0px 0px;
padding:5px 5px 3px 5px;
white-space : normal; /* crucial for IE 6, maybe 7? */


Hopefully others find this useful.

For those of us that don’t have the luxury of using all transactional based tables such as InnoDB and the the ability to use the likes of –single-transaction with –master-data we do have an option of using r1soft to avoid lengthy downtimes in the event we have to setup/resetup a slave. You can turn a potential several hour outage in a 30-60 second READ LOCK and perform the rest in the background.

Firstly a quick explanation of the r1soft mySQL addon, the mySQL addon works by issuing a global lock and then flushes changes from memory to disk to ensure consistent snapshots. As it works with the r1soft device driver the process usually takes less than a second making this the perfect method to take backups and have the master position at the time the backup was taken. At the time of writing r1soft does not support the ability to store the master/slave status data although hopefully in the future it will store and display it to avoid having to use the following trick.

We can however work around this by monitoring the r1soft server-log.txt to determine when it has flushed and unlocked the mySQL instance. As it happens very early on in the process we can manually issue a global read lock, get the current position, monitor the server-log.txt until r1soft has performed the snapshot of mySQL and then release the global lock manually.

I have performed an example of this for this article. I started by logging into the CDP interface, browsing to the system in question and getting to the start backup process, having this all filled in and ready to submit speeds the process up. Once this was ready I logged into the database system and issued a global read lock then obtained the current master position,

mysql> flush tables with read lock;
Query OK, 0 rows affected (0.10 sec)

mysql> show master status;
+—————–+———–+————–+——————+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+—————–+———–+————–+——————+
| mysql-bin.024272 | 120146094 | | |
+—————–+———–+————–+——————+
1 row in set (0.00 sec)

I then clicked ok under the schedule backup window and started a tail on server-log.txt (usually found in /usr/r1soft/buserver/log) and waited until it reached the mySQL lock, flush and unlock.


2011-01-27 02:44:27,743 INFO: (tid:976019) Starting task 'Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
2011-01-27 02:44:27,771 INFO: (tid:976019) Starting Backup Task.
2011-01-27 02:44:27,772 INFO: (tid:976019) Backing up partition tables.
2011-01-27 02:44:27,773 INFO: (tid:976020) Starting task 'Partition Table Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
2011-01-27 02:44:27,790 INFO: (tid:976020) Starting partition table backup for host (xx.xx.xx.xx) device (/dev/sda) Disk Safe (0) recovery point (6025).
2011-01-27 02:44:27,892 INFO: (tid:976020) AGENT: (Righteous Backup Linux Agent) 1.62.0 build 6386
2011-01-27 02:44:28,019 WARN: (tid:976020) AGENT: allowing control from backup server (xx.xx.xx.xx) with valid RSA key
2011-01-27 02:44:28,059 WARN: (tid:976020) AGENT: sending auth challenge for allowed host at (xx.xx.xx.xx) port (41758)
2011-01-27 02:44:28,108 INFO: (tid:976020) AGENT: host (xx.xx.xx.xx) port (41758) authentication successful
2011-01-27 02:44:28,193 INFO: Agent (xx.xx.xx.xx) authenticated successfully
2011-01-27 02:44:28,267 INFO: (tid:976020) AGENT: Partition Backup request accepted. Starting backup.
2011-01-27 02:44:28,269 INFO: (tid:976020) AGENT: Found 4 partition(s), 1 extended partition(s) and 124 extra sector(s)
2011-01-27 02:44:28,270 INFO: (tid:976020) AGENT: Need to back up 126 sectors
2011-01-27 02:44:28,270 INFO: (tid:976020) Sector size (512) bytes.
2011-01-27 02:44:28,271 INFO: (tid:976020) Number of sectors (126).
--SNIP--
2011-01-27 02:44:32,083 INFO: Sending (1) MySQL instances to agent.
2011-01-27 02:44:32,083 INFO: Sending next MySQL instance to agent.
2011-01-27 02:44:32,117 INFO: (tid:976019) AGENT: Backup request accepted. Starting backup.
2011-01-27 02:44:39,340 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,353 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,354 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,355 INFO: (tid:976019) AGENT: Locked 1 MySQL instances.
2011-01-27 02:44:39,355 INFO: (tid:976019) AGENT: Unlocked 1 MySQL instances.
2011-01-27 02:44:39,356 INFO: (tid:976019) AGENT: Snapshot completed in 0.059 seconds.
2011-01-27 02:44:39,357 INFO: (tid:976019) AGENT: Acquired MySQL lock in 0.025 seconds and held lock for 0.085 seconds on 1 instance.
2011-01-27 02:44:39,357 INFO: (tid:976019) Backup pipeline size (256) blocks.
2011-01-27 02:44:39,358 INFO: (tid:976019) Block size (4096).
2011-01-27 02:44:39,358 INFO: (tid:976019) Partition size (34529701) blocks.
2011-01-27 02:44:39,359 INFO: (tid:976019) Mount Point (/).
2011-01-27 02:44:39,803 INFO: (tid:976019) Network Queue Size: 1280 blocks

Once it has completed the stage in bold I simply released the read lock,

mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)

I now had a backup in r1soft and had the master position of the time it was taken. Once the backup completed I simply restored the tables I needed to the slave using the restore to an alternate host and started replication using this data.


INFO 01/27/2011 02:44:27 GMT (tid:976019) Starting task 'Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
INFO 01/27/2011 02:51:10 GMT (tid:976019) Backup Task finished.

I restored the backup to the mysql datadir, as you are backing up the master you can skip all the binlogs. After this was restored I started mySQL, changed the master to the details we got from the master status earlier and started the slave.


mysql> CHANGE MASTER to MASTER_HOST='192.168.1.3', MASTER_USER='slave', MASTER_PASSWORD='xxxxxx', MASTER_LOG_FILE='mysql-bin.024272', MASTER_LOG_POS=120146094;
Query OK, 0 rows affected (0.01 sec)

mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status \G;
Seconds_Behind_Master: 0

This blog post will be about what I have learned in 2010 and some plans for 2011.

So what have I learned, enjoyed and hated in 2010?

– I love puppet

– Adobe flash is the bane of my life. Flash on 64bit linux is nothing short of a pain, it causes the most problems, the most crashes and should rot in hell.

– I am starting to like chrome but have not made the full switch from firefox yet.

– I am quite a lazy programmer, more worrying I actually don’t care.

– I love MongoDB and expect big things from it.

– I am unsure weather to love or hate r1soft. On one hand I like what it brings on the other hand it’s very buggy and I don’t think the release of 3.0 is going to address many of these. This should be an entire blog post in itself later.

– Started using ksplice uptrack, already have this running on a few hundred systems and no problems to date. Only time will tell.

I am looking forward to 2011, I am hoping to continue on with getting large portions of the “todo list” completed. Some of the ones off the top of my head are,

– Make the switch from cacti to collectd+MongoDB. Been working on this on and off for awhile. Primary goals is to make it more flexible, more automated and easier to maintain. Since we maintain so many different environments it has to fit some very specific needs. Hoping to replace the primary cacti install and also some of the other secondary customer-specific ones (such as munin).

– Upgrade to nagios 3.x. We still are running nagios 2.x so an upgrade is needed. At the same time need to rebuild and rewrite the automation systems, templates, hosts and groups. Especially to tie better into collectd, the existing nagios+cacti setup is pretty “hacky”.

– Rebuild the client database, notes and task list systems. This work is already under-way so should be completed in Q1 2011. Better encryption, more functionality, more flexibility, easier to manage.

– Build a new ticket system and drop Kayako. I don’t particularly like the direction Kayako has taken in their latest release so it’s time to stop using it (Very hacked up anyway).

– Better documentation. I, along with many others, really fail in this area and I would love to improve on it. Still toying with the idea of internal wiki’s combined with the existing notes systems and to make better use of them.

– Improve the existing yum repo, make more use of mocks, include some newer RPM’s and update all systems to use this. Replace all custom builds to an RPM from the repo (nginx, memcached, etc).

– Make more use of the nessus systems and plugin feeds. At the moment this is mostly semi-manual , need to work on making it fully automated and easy to maintain.

– Post more blog entries? Goal of atleast 20 more blog posts (Less than 2 per month).

– Rewrite the nagios r1soft/buagent check and release it.

– Make the vbulletin static cache into a plugin. This is just a simple patch we provide to customers who don’t have enough capacity and are receiving a sudden burst of traffic (IE a thread slashdotted/etc). It simply stores the threads for guests in memory for 2 minutes and then displays the cache for guests (supports online counters, multiple themes, browsers, mobile, etc).

That’s my list for the time being. Lets hope I can stick to it. Do you have your list ready? Feel free to share.

So in effort to find the initial cause of this with no real data to work from I had a search around the system for the usual suspects but couldn’t find any particular culprit. Manually inspecting nearly 2 million files was not an option so opted to carry on with the usual setup and enforced posix ACL’s against the apache user and setup some more explicit bandwidth monitoring to obtain data when an attack was occurring.

A few hours later an alert came in that the outbound bandwidth exceeded the threshold so I promptly begin investigating, The process list doesn’t seem to show any obvious usual culprits. After spending a few minutes with iftop and tcpdump I identified the targeted IP and that the traffic was being directed to a DNS server (port 53). I filtered traffic to this IP while investigating the source, as there was no unusual processes I decided to have alook at the apache status and found the GET request containing the destination IP and port (xxx.php?target=xx.xx.xx.xx&port=53).

I got the vhosts path from the httpd.conf and reviewed the file and it looks like a simple php script to perform a UDP flood to the target,

$sock=socket_create(AF_INET,SOCK_DGRAM,SOL_UDP);

if(!$sock) die(“Cant Create Socket!!!”);

$data=”;
for($i=0;$i<1400;$i++)
{
$data.=chr(rand(0,255));
}

while(true)
{
if(!socket_sendto($sock,$data,strlen($data),0,$target,$port)) die(“Error SendTo!!!”);
}

That is a snippet of it. It is very unusual to see these PHP based which is the reason for this blog entry and a definite new addition to the search list.

For those of you interested the customer already had MRTG installed and below is the last week which shows the attack saturating the uplink (100Mbit) when it was occurring but has since been stopped.