Removing munin-node ...
[FAIL] Munin-Node appears to be unconfigured. ... failed!
invoke-rc.d: initscript munin-node, action "stop" failed.
dpkg: error processing munin-node (--purge):
 subprocess installed pre-removal script returned error exit status 6
[FAIL] Munin-Node appears to be unconfigured. ... failed!
invoke-rc.d: initscript munin-node, action "start" failed.
dpkg: error while cleaning up:
 subprocess installed post-installation script returned error exit status 6
Errors were encountered while processing:
 munin-node
E: Sub-process /usr/bin/dpkg returned an error code (1)

The init.d checks if it’s configured

/etc/init.d/munin-node

elif [ ! -e $CONFFILE ]; then
	log_failure_msg "Munin-Node appears to be unconfigured."
	exit 6
fi

Which is what is causing this issue, changing this too,

elif [ ! -e $CONFFILE && $1 != "stop" ]; then
	log_failure_msg "Munin-Node appears to be unconfigured."
	exit 6
fi

Should solve the issue.

The defaults are RHEL defaults and ./check_varnishbackends.py should work without any options unless you want to override them. Here is the Nagios/Icinga Setup,

Register the service

define service {
	register			0
	name				check_varnishbackends
	service_description		Varnish Backends
	use				generic-service
}

Assign to host “server”

define service {
	service_description		Varnish Backend
	host_name			server
	use				check_varnishbackends
	check_command			remote_nrpe!check_varnishbackends -a "127.0.0.1" "6082" "/etc/varnish/secret" "/usr/bin/varnishadm"
}

Setup NRPE (Make sure user has permission to varnish secret file or use sudo)

command[check_varnishbackends]=/usr/local/nagios/libexec/check_varnishbackends.py --host $ARG1$ --port $ARG2$ --secret $ARG3$ --path $ARG4$

Version at time of posting,

#!/usr/bin/env python

#
# check_varnishbackends.py
# Varnish Backend Checker for Icinga/Nagios

# Author
#  - Scott Mcintyre 
#  - http://scott.cm
# Usage
#  -h, --help            	Show help message and exit
#  -H HOST, --host=HOST  	The ip varnishadm is listening on (Default: 127.0.0.1)
#  -P PORT, --port=PORT  	The port varnishadm is listening on (Default: 6082)
#  -s SECRET, --secret=SECRET 	The path to the secret file (Default: /etc/varnish/secret)
#  -p PATH, --path=PATH  	The path to the varnishadm binary (Default: /usr/bin/varnishadm
# Example: ./check_varnishbackends.py -H 127.0.0.1 -P 6082 -S /etc/varnish/secret -p /usr/bin/varnishadm
# 

import sys
import optparse
import subprocess

def runcommand(command, exit_on_fail=True):
    try:
      process = subprocess.Popen(command.split(" "), stdout=subprocess.PIPE)
      output, unused_err = process.communicate()
      retcode = process.poll()
      return output

    except OSError, e:
      print "Error: Executing command failed,  does it exist?"
      sys.exit(2)

def main(argv):
  o = optparse.OptionParser(conflict_handler="resolve", description="Nagios plugin to check varnish backend health.")
  o.add_option('-H', '--host', action='store', type='string', dest='host', default='127.0.0.1', help='The ip varnishadm is listening on')
  o.add_option('-P', '--port', action='store', type='int', dest='port', default=6082, help='The port varnishadm is listening on')
  o.add_option('-s', '--secret', action='store', type='string', dest='secret', default='/etc/varnish/secret', help='The path to the secret file')
  o.add_option('-p', '--path', action='store', type='string', dest='path', default='/usr/bin/varnishadm', help='The path to the varnishadm binary')

  options=o.parse_args()[0]
  command = runcommand("%(path)s -S %(secret)s -T %(host)s:%(port)s debug.health" % options.__dict__)
  backends = command.split("\n")
  backends_healthy, backends_sick = [], []
  for line in backends:
    if line.startswith("Backend") and line.find("test")==-1:
      if line.endswith("Healthy"):
        backends_healthy.append(line.split(" ")[1])
      else:
        backends_sick.append(line.split(" ")[1])

  if backends_sick:
    print "%s backends are down.  %s" % (len(backends_sick), "".join(backends_sick))
    sys.exit(2)

  if not backends_sick and not backends_healthy:
    print "No backends found"
    sys.exit(1)

  print "All %s backends are healthy" % (len(backends_healthy))
  sys.exit(0)

if __name__ == "__main__":
    main(sys.argv[1:])

Luckily a quick google shows a active bug at github which contains a couple of suggestions. I tried simply disabling the auto focus feature bit this ended up making all my panels disappear. This fix by rat4 seems to solve the problem.

Here’s what I done

sudo wget -O /usr/share/cinnamon/js/ui/windowAttentionHandler.js https://raw.github.com/rat4/Cinnamon/21a08f6f9e13f7ba4e3bf633b7ba4b36fd22a291/js/ui/windowAttentionHandler.js
sudo wget -O /usr/share/glib-2.0/schemas/org.cinnamon.gschema.xml https://raw.github.com/rat4/Cinnamon/21a08f6f9e13f7ba4e3bf633b7ba4b36fd22a291/data/org.cinnamon.gschema.xml
sudo apt-get install dconf-tools

Went to System Tools->Dconf Editor and change the value of notification-style to 0

Possible values are,

0 – do nothing here, but other notificators (window list applet) still can take care
1 – bring window to front, like current cinnamon
2 – show banner “app is ready”, like gnome shell

The final step is just to restart Cinnamon.

Now when I get a new message it has the desired effect and simply flashes rather than pops up.

“Unity Sucks” the top recommendation next to the term itself.

In the previous release I was happy because the gnome classic to get gnome 2 worked but as of 11.10 this isn’t possible. It’s either Unity or GNOME 3 (Classic is available). Unity lasted no more than 10 minutes in testing, it’s clearly designed for tablets and is horrendous to use. GNOME 3 I could get used to but would need to tweak more, I really dislike the overly large icons and large notifications.

After a bit of trial and messing around I even managed to make all my panels disappear and couldn’t figure out how to return them and decided to just take the plunge and switch to mint 12 with cinnamon. The migration was rather easy, I simply installed mint and restored my backup of /home to retain all my old settings. Reinstalled skype, thunderbird, chrome, firefox, nagstamon and xchat and that was everything back with the exception of being on Mint 12.

The standard mint brings the gnome 2 feel but I still couldn’t quite take to it so went ahead and installed cinnamon and I have to say I quite like it. With the exception of not having a taskbar on the other screens but there does seem to be a bug issue last updated a few days ago for this at https://github.com/linuxmint/Cinnamon/issues/130 so with any luck this will be introduced soon.

Overall everything seems quite snappy, the UI is what I want and it’s back to business as usual.

It appears twitter don’t allow most of the apps (and there are lots of them) to mass unfollow as part of their TOS.   I found this at http://www.hostnexus.com/blog/how-to-bulk-unfollow-on-twitter and decided to give the steps in there a shot.   I tried a few of the apps with the chrome check all plugin but all of them have daily limits and given up here. This is when I decided instead to just write a quick python script using the API.

It was easy enough to get started with the twitter API.
Simply visit https://dev.twitter.com and login with your account.  
Create a new application,   get your key.
Install the twitter python API at http://code.google.com/p/python-twitter/

You can get the latest version of the script at: https://github.com/Scott-Mc/python/blob/master/twitter-mass-unfollow.py and the version at the time of writing at the bottom of this post.

This was my profile before running the script

This is my profile now after running the script,

This is the code at the time of writing,

#!/usr/bin/env python

#
#Mass unfollow friends on twitter
#

# Author
#  - Scott Mcintyre 
#
# Usage
#
# Get your keys from https://dev.twitter.com/
# Ensure it has write access
# Change the api key values marked ##### below with the twitter keys
# Install http://code.google.com/p/python-twitter/

import twitter
import time

api = twitter.Api(consumer_key='#####', consumer_secret='#####', access_token_key='#####', access_token_secret='#####')

start = time.time()
i = 0
friends=api.GetFriends()

while (1):
 try:
  for u in friends:
   print "Deleting %s" % (u.name)
   api.DestroyFriendship(u.id);
   i = i + 1

 except:
   print "Possibly Rate limited? sleeping 60 seconds"
   time.sleep(60)

elapsed = (time.time() - start)
print "Deleted %s friends in %0.2f seconds" % (str(i), elapsed)

ERROR: Module backupdriver is in use

r1soft-uninstall-buagent -R
This uninstaller will remove the following..

File /etc/init.d/buagent
File /usr/bin/buagent
File /var/lock/buagent.lock
File /var/run/buagent.pid
File /usr/bin/r1key
File /usr/bin/r1soft-cki
File /usr/bin/r1soft-report.sh
File /usr/bin/r1soft-uninstall-buagent
File /usr/share/man/man3/buagent.3.gz
File /usr/share/man/man3/r1key.3.gz
File /usr/share/man/man3/r1soft-uninstall-buagent.3.gz
Directory /etc/buagent
Directory /usr/lib/buagent
Directory /lib/modules/buagent
Directory /var/cache/buagent
Directory /var/db/buagent
User buagent
Group buagent

Would you like to continue? (yes/no):yes
stopping buagent.
removing buagent from services.
removing File /etc/init.d/buagent
removing File /usr/bin/buagent
removing File /var/lock/buagent.lock
removing File /var/run/buagent.pid
removing File /usr/bin/r1key
removing File /usr/bin/r1soft-cki
removing File /usr/bin/r1soft-report.sh
removing File /usr/bin/r1soft-uninstall-buagent
removing File /usr/share/man/man3/buagent.3.gz
removing File /usr/share/man/man3/r1key.3.gz
removing File /usr/share/man/man3/r1soft-uninstall-buagent.3.gz
removing Directory /etc/buagent
removing Directory /usr/lib/buagent
removing Directory /lib/modules/buagent
removing Directory /var/cache/buagent
removing Directory /var/db/buagent
removing buagent user…
unloading backupdriver kernel module…
ERROR: Module backupdriver is in use
done.

Trying to simply rmmod the driver fails because it’s in use but without having forced support there is no way to unload it using this method.


# rmmod backupdriver
ERROR: Module backupdriver is in use


The solution is to reinstall the latest CDP 2 agent and run buagent -u to unload it first then proceed to uninstalling the CDP2 agent and installing CDP3 agent.

Removed ext_addr from “base_features” and commented out the inode_size line. Here is a diff from the default ubuntu e2fsprogs /etc/mke2fs.conf with the necessary changes,

— /etc/mke2fs.orig.conf 2011-11-28 15:20:07.365231775 +0000
+++ /etc/mke2fs.conf 2011-11-28 15:20:48.165231803 +0000
@@ -1,7 +1,7 @@
[defaults]
- base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
+ base_features = sparse_super,filetype,resize_inode,dir_index
blocksize = 4096
- inode_size = 256
+ #inode_size = 256
inode_ratio = 16384

[fs_types]

Then format the partition and this will allow it to be mounted on windows.

I used DiskInternals Linux Reader on windows to mount the drive and copy the files which can be downloaded for free at


height: 120px;
overflow: auto;
font-family: “Consolas”,monospace;
font-size: 9pt;
text-align:left;
background-color: #FCF7EC;
overflow-x: auto; /* Use horizontal scroller if needed; for Firefox 2, not
white-space: pre-wrap; /* css-3 */
white-space: -moz-pre-wrap !important; /* Mozilla, since 1999 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
margin: 0px 0px 0px 0px;
padding:5px 5px 3px 5px;
white-space : normal; /* crucial for IE 6, maybe 7? */


Hopefully others find this useful.

I decided that I would learn python, surprisingly it’s been really easy to pick up and I thought I would share some of the things I made in my first 2 days of python.

The first I written was to phrase the results from yahoo’s football score page(http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html). I was on this page anyway and thought why not!

#!/usr/bin/python
import urllib2
import re
response = urllib2.urlopen('http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html')
html = response.read()
re_channel = re.compile("<td class=\"ko\"><abbr class=\"dtstart\" title=\"\">([0-9a-zA-z ,:]*)</abbr></td>" +
 "\n <td class=\"match.*?\n" +
 " <a href=.*?\">" +
  "\n <span class=\"home\">([a-zA-Z ]*)</span>.*?" +
  "\n <span class=\"score\">([0-9 :-]*)</span>.*?" +
  "\n <span class=\"away\">([a-zA-Z ]*)</span>", re.I | re.S | re.M)
find_result = re_channel.findall(html)
print find_result

Sample output

[('13 Aug, 15:00', 'Wigan Athletic', '1 - 1', 'Norwich City'), ('13 Aug, 15:00', 'Fulham', '0 - 0', 'Aston Villa'), ('13 Aug, 15:00', 'Blackburn Rovers', '1 - 2', 'Wolverhampton Wanderers'), ('13 Aug, 15:00', 'Liverpool', '1 - 1', 'Sunderland'), ('13 Aug, 15:00', 'Queens Park Rangers', '0 - 4', 'Bolton Wanderers'), ('13 Aug, 17:30', 'Newcastle United', '0 - 0', 'Arsenal'), ('14 Aug, 13:30', 'Stoke City', '0 - 0', 'Chelsea'), ('14 Aug, 16:00', 'West Bromwich Albion', '1 - 2', 'Manchester United'), ('15 Aug, 20:00', 'Manchester City', '4 - 0', 'Swansea City'), ('20 Aug, 12:00', 'Sunderland', '0 - 1', 'Newcastle United'), ('20 Aug, 12:45', 'Arsenal', '0 - 2', 'Liverpool'), ('20 Aug, 15:00', 'Swansea City', '0 - 0', 'Wigan Athletic'), ('20 Aug, 15:00', 'Aston Villa', '3 - 1', 'Blackburn Rovers'), ('20 Aug, 15:00', 'Everton', '0 - 1', 'Queens Park Rangers'), ('20 Aug, 17:30', 'Chelsea', '2 - 1', 'West Bromwich Albion'), ('21 Aug, 13:30', 'Norwich City', '1 - 1', 'Stoke City'), ('21 Aug, 14:05', 'Wolverhampton Wanderers', '2 - 0', 'Fulham'), ('21 Aug, 16:00', 'Bolton Wanderers', '2 - 3', 'Manchester City'), ('22 Aug, 20:00', 'Manchester United', '3 - 0', 'Tottenham Hotspur'), ('27 Aug, 12:05', 'Aston Villa', '0 - 0', 'Wolverhampton Wanderers'), ('27 Aug, 12:30', 'Wigan Athletic', '2 - 0', 'Queens Park Rangers'), ('27 Aug, 15:00', 'Swansea City', '0 - 0', 'Sunderland'), ('27 Aug, 15:00', 'Chelsea', '3 - 1', 'Norwich City'), ('27 Aug, 15:00', 'Blackburn Rovers', '0 - 1', 'Everton'), ('27 Aug, 17:30', 'Liverpool', '3 - 1', 'Bolton Wanderers'), ('28 Aug, 13:00', 'Newcastle United', '2 - 1', 'Fulham'), ('28 Aug, 13:30', 'Tottenham Hotspur', '1 - 5', 'Manchester City'), ('28 Aug, 15:00', 'West Bromwich Albion', '0 - 1', 'Stoke City'), ('28 Aug, 16:00', 'Manchester United', '8 - 2', 'Arsenal')]

The next was to obtain a list of the uk charts top 100 and find the youtube video for it.

#!/usr/bin/python
import urllib
import urllib2
import re
import sys
from BeautifulSoup import BeautifulStoneSoup  

def getvideolink(link):
  request = urllib2.Request(link)
  request.add_header('User-Agent', "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Safari/533.3")
  response = urllib2.urlopen(request)
  text = response.read()
  videos = re.findall("<a href=\"/watch\?v=([\w-]+)", text)
  return "http://www.youtube.com/watch?v=%s" % videos[0]

response = urllib2.urlopen('http://localhost/test')
html = response.read()
re_channel = re.compile("<h3>(.*?)</h3>" +
 "\r\n.*?<h4>(.*?)</h4>.*?<h5", re.I | re.S | re.M)
find_result = re_channel.findall(html)
#remove first entry as it's a dup
del find_result[0]
for result in find_result:
 title=unicode(BeautifulStoneSoup(result[0],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
 artist=unicode(BeautifulStoneSoup(result[1],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
 link="http://www.youtube.com/results?search_query=" + urllib.quote_plus(title);
 print getvideolink(link)

Sample output

RIHANNA FT CALVIN HARRIS - WE FOUND LOVE // http://www.youtube.com/watch?v=tg00YEETFzg
FLO RIDA - GOOD FEELING // http://www.youtube.com/watch?v=3OnnDqH6Wj8
ONE DIRECTION - GOTTA BE YOU // http://www.youtube.com/watch?v=nvfejaHz-o0
LABRINTH FT TINIE TEMPAH - EARTHQUAKE // http://www.youtube.com/watch?v=u0fk6syQ7iY
ED SHEERAN - LEGO HOUSE // http://www.youtube.com/watch?v=c4BLVznuWnU
DAVID GUETTA FT USHER - WITHOUT YOU // http://www.youtube.com/watch?v=jUe8uoKdHao
MAROON 5 FT CHRISTINA AGUILERA - MOVES LIKE JAGGER // http://www.youtube.com/watch?v=iEPTlhBmwRg
JLS - TAKE A CHANCE ON ME // http://www.youtube.com/watch?v=djV11Xbc914
PROFESSOR GREEN FT EMELI SANDE - READ ALL ABOUT IT // http://www.youtube.com/watch?v=-_oLfC5Z_Ys
CHRISTINA PERRI - JAR OF HEARTS // http://www.youtube.com/watch?v=8v_4O44sfjM
LMFAO - SEXY AND I KNOW IT // http://www.youtube.com/watch?v=wyx6JDQCslE
DRAKE FT RIHANNA - TAKE CARE // http://www.youtube.com/watch?v=PaXslpx3MWY
COLDPLAY - PARADISE // http://www.youtube.com/watch?v=1G4isv_Fylg
CHARLENE SORAIA - WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
SATURDAYS - MY HEART TAKES OVER // http://www.youtube.com/watch?v=DgmoYgpMNX8
CHER LLOYD FT MIKE POSNER - WITH UR LOVE // http://www.youtube.com/watch?v=axpO86pGHAM
BRUNO MARS - IT WILL RAIN // http://www.youtube.com/watch?v=W-w3WfgpcGg
LADY GAGA - MARRY THE NIGHT // http://www.youtube.com/watch?v=O4IgYxHEAuk
LANA DEL REY - VIDEO GAMES // http://www.youtube.com/watch?v=HO1OV5B_JDw
PIXIE LOTT FT PUSHA T - WHAT DO YOU TAKE ME FOR // http://www.youtube.com/watch?v=OQCcwNMp830
KELLY CLARKSON - MR KNOW IT ALL // http://www.youtube.com/watch?v=0C_oNMH0GTk
LOICK ESSIEN - ME WITHOUT YOU // http://www.youtube.com/watch?v=lKDmJwoZ4RA
FLORENCE & THE MACHINE - SHAKE IT OUT // http://www.youtube.com/watch?v=WbN0nX61rIs
COLLECTIVE - TEARDROP // http://www.youtube.com/watch?v=u7K72X4eo_s
ONE DIRECTION - WHAT MAKES YOU BEAUTIFUL // http://www.youtube.com/watch?v=QJO3ROT-A4E
ED SHEERAN - THE A TEAM // http://www.youtube.com/watch?v=UAWcs5H-qgQ
WANTED - LIGHTNING // http://www.youtube.com/watch?v=MQyHyfLp5NI
JESSIE J - WHO YOU ARE // http://www.youtube.com/watch?v=j2WWrupMBAE
LUCENZO & QWOTE - DANZA KUDURO // http://www.youtube.com/watch?v=rUFgacK8sZ0
GYM CLASS HEROES/ADAM LEVINE - STEREO HEARTS // http://www.youtube.com/watch?v=T3E9Wjbq44E
MAVERICK SABRE - I NEED // http://www.youtube.com/watch?v=VA770wpLX-Q
WESTLIFE - LIGHTHOUSE // http://www.youtube.com/watch?v=Tivph7mTku4
SEAN PAUL FT ALEXIS JORDAN - GOT 2 LUV U // http://www.youtube.com/watch?v=tDq3fNew1rU
LADY GAGA - THE EDGE OF GLORY // http://www.youtube.com/watch?v=QeWBS0JBNzQ
GLEE CAST - RUMOUR HAS IT/SOMEONE LIKE YOU // http://www.youtube.com/watch?v=qb7zjKkLCoQ
SLOW MOVING MILLIE - PLEASE PLEASE PLEASE LET ME GET WHAT I // http://www.youtube.com/watch?v=DMQbzLrvwlE
KATY PERRY - THE ONE THAT GOT AWAY // http://www.youtube.com/watch?v=Ahha3Cqe_fk
ELBOW - ONE DAY LIKE THIS // http://www.youtube.com/watch?v=0NFV8dHrZYM
CHRISTINA PERRI - A THOUSAND YEARS // http://www.youtube.com/watch?v=z5Q8x1wYN4w
SNOW PATROL - THIS ISN'T EVERYTHING YOU ARE // http://www.youtube.com/watch?v=Q-Gljs8Y3Q8
RIZZLE KICKS - WHEN I WAS A YOUNGSTER // http://www.youtube.com/watch?v=Rc2iUwMpb8Y
GOO GOO DOLLS - IRIS // http://www.youtube.com/watch?v=NdYWuo9OFAw
DAPPY - NO REGRETS // http://www.youtube.com/watch?v=WoImizvsj5w
BRUNO MARS - MARRY YOU // http://www.youtube.com/watch?v=xB40cQD677s
ADELE - SET FIRE TO THE RAIN // http://www.youtube.com/watch?v=ss0HAdW1DnY
NICKELBACK - WHEN WE STAND TOGETHER // http://www.youtube.com/watch?v=76RbWuFll0Y
AFROJACK & STEVE AOKI - NO BEEF // http://www.youtube.com/watch?v=ksocjhxX_DQ
ADELE - SOMEONE LIKE YOU // http://www.youtube.com/watch?v=hLQl3WQQoQ0
PIXIE LOTT - ALL ABOUT TONIGHT // http://www.youtube.com/watch?v=swcULf1ATyU
NICKI MINAJ - SUPER BASS // http://www.youtube.com/watch?v=4JipHEz53sU
CALLING - WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
JAMES MORRISON - I WON'T LET YOU GO // http://www.youtube.com/watch?v=sgRb_lfIZ6A
OLLY MURS FT RIZZLE KICKS - HEART SKIPS A BEAT // http://www.youtube.com/watch?v=j5dFe-WKuPs
EXAMPLE - MIDNIGHT RUN // http://www.youtube.com/watch?v=iwYGi7YG4Js
TINCHY STRYDER/CALVIN HARRIS - OFF THE RECORD // http://www.youtube.com/watch?v=UknZiaIC9y8
SAK NOEL - LOCA PEOPLE // http://www.youtube.com/watch?v=-d6b1yn-YhQ
BRUNO MARS - RUNAWAY BABY // http://www.youtube.com/watch?v=UDG_CrqJV-0
JASON DERULO - FIGHT FOR YOU // http://www.youtube.com/watch?v=2aSOQRih6WY
CHER LLOYD - SWAGGER JAGGER // http://www.youtube.com/watch?v=sdbyG2MrBHk
LADY GAGA - BORN THIS WAY // http://www.youtube.com/watch?v=wV1FrqwZyKw
SNOW PATROL - CALLED OUT IN THE DARK // http://www.youtube.com/watch?v=GwTXwJg6_VE
WANTED - WARZONE // http://www.youtube.com/watch?v=yMR382aefmQ
JLS FT DEV - SHE MAKES ME WANNA // http://www.youtube.com/watch?v=FuwTgZOKcf8
NICOLE SCHERZINGER - TRY WITH ME // http://www.youtube.com/watch?v=R7sYiTyBjTY
COBRA STARSHIP FT SABI - YOU MAKE ME FEEL // http://www.youtube.com/watch?v=HpyZEzrDf4c
JESSIE J - WHO'S LAUGHING NOW // http://www.youtube.com/watch?v=KsxSxF3JKeU
NOEL GALLAGHER'S HIGH FLYING - AKA WHAT A LIFE // http://www.youtube.com/watch?v=lwHpLDgWonM
FOSTER THE PEOPLE - PUMPED UP KICKS // http://www.youtube.com/watch?v=SDTZ7iX4vTQ
WRETCH 32 FT JOSH KUMRA - DON'T GO // http://www.youtube.com/watch?v=bj1BMpUnzT8
BIRDY - SKINNY LOVE // http://www.youtube.com/watch?v=aNzCDt2eidg
WANTED - GLAD YOU CAME // http://www.youtube.com/watch?v=2ggzxInyzVE
BIRDY - PEOPLE HELP THE PEOPLE // http://www.youtube.com/watch?v=OmLNs6zQIHo
NICKI MINAJ FT RIHANNA - FLY // http://www.youtube.com/watch?v=3n71KUiWn1I
SKREAM FT SAM FRANK - ANTICIPATE // http://www.youtube.com/watch?v=O3Z1X4MPsqk
WILL YOUNG - JEALOUSY // http://www.youtube.com/watch?v=9MHtrM-jf9o
ADELE - ROLLING IN THE DEEP // http://www.youtube.com/watch?v=rYEDA3JcQqw
JESSIE J FT BOB - PRICE TAG // http://www.youtube.com/watch?v=qMxX-QOV9tI
LMFAO/LAUREN BENNETT/GOONROCK - PARTY ROCK ANTHEM // http://www.youtube.com/watch?v=KQ6zr6kCPj8
JASON DERULO - IT GIRL // http://www.youtube.com/watch?v=4oGUHRXT-wA
BEYONCE - COUNTDOWN // http://www.youtube.com/watch?v=ACkBTqwxcUI
RIZZLE KICKS - DOWN WITH THE TRUMPETS // http://www.youtube.com/watch?v=-aY92XgykhU
COLDPLAY - EVERY TEARDROP IS A WATERFALL // http://www.youtube.com/watch?v=fyMhvkC3A84
CALVIN HARRIS - FEEL SO CLOSE // http://www.youtube.com/watch?v=dGghkjpNCQ8
ADELE - MAKE YOU FEEL MY LOVE // http://www.youtube.com/watch?v=LLoyNxjhTzc
ONE DIRECTION - ANOTHER WORLD // http://www.youtube.com/watch?v=RyZfNBVX1q0
COLDPLAY - VIVA LA VIDA // http://www.youtube.com/watch?v=dvgZkm1xWPE
BEYONCE - LOVE ON TOP // http://www.youtube.com/watch?v=Ob7vObnFUJc
EAGLE-EYE CHERRY - SAVE TONIGHT // http://www.youtube.com/watch?v=dTa2Bzlbjv0
EN VOGUE - DONT LET GO (LOVE) // http://www.youtube.com/watch?v=QUdAT5Fwnvk
KATY PERRY - FIREWORK // http://www.youtube.com/watch?v=QGJuMBdaqIw
MODESTEP - TO THE STARS // http://www.youtube.com/watch?v=UTKSUlMbp9A
DRAKE - HEADLINES // http://www.youtube.com/watch?v=cimoNqiulUE
ED SHEERAN - YOU NEED ME I DON'T NEED YOU // http://www.youtube.com/watch?v=temYymFGSEc
WOODKID - IRON // http://www.youtube.com/watch?v=vSkb0kDacjs
DELILAH - GO // http://www.youtube.com/watch?v=cxNe9jWNuEU
KATY PERRY FT KANYE WEST - ET // http://www.youtube.com/watch?v=t5Sd5c4o9UM
JAMES VINCENT MCMORROW - HIGHER LOVE // http://www.youtube.com/watch?v=9Z-fE1l9SZ4
BRUNO MARS - JUST THE WAY YOU ARE (AMAZING) // http://www.youtube.com/watch?v=LjhCEhWiKXk
PITBULL/NE-YO/AFROJACK/NAYER - GIVE ME EVERYTHING // http://www.youtube.com/watch?v=EPo5wWmKEaI

Then I wanted to test threading so I came up with something to check a list of links and obtain the "Server:" header

#!/usr/bin/env python
import urllib2
import re
import os
import sys
import time
from multiprocessing import Pool

#Accept file name as input or use default
try:
  filename = sys.argv[1]
except IndexError:
  filename = "links"

#Function to load urls from file
def loadurls(filename):
  try:
    inputdata = file(filename).readlines()
    if len(inputdata) > 0:
      return inputdata
  except:
    print "ERROR - Unable to process url list"
    sys.exit()

#Function to get SERVER header
def getversion(url):
  try:
      request=urllib2.Request(url)
      response=urllib2.urlopen(request)
      version=response.info().getheader('Server')
  except:
      return "ERROR - Unable to fetch %s" % (url)

  if version: print "Url: %s Version: %s" % (url, version)

start = time.time()
urls=loadurls(filename)
p = Pool(15)
data = p.map(getversion,urls)

elapsed = (time.time() - start)
print "Processed %s urls in %0.2f seconds" % (len(urls), elapsed)

Sample output

scott@scott:~/python$ ./server-header.py list
Url: http://www.bbc.co.uk
 Version: Apache
Url: http://www.google.co.uk
 Version: gws
Url: http://scottmcintyre.net
 Version: Apache
Processed 4 urls in 1.07 seconds

None of these really serve any purpose and were merely tests I thought I would share.   Overall things have been quite easy and feel reasonably confident after just 2 days.

For those of us that don’t have the luxury of using all transactional based tables such as InnoDB and the the ability to use the likes of –single-transaction with –master-data we do have an option of using r1soft to avoid lengthy downtimes in the event we have to setup/resetup a slave. You can turn a potential several hour outage in a 30-60 second READ LOCK and perform the rest in the background.

Firstly a quick explanation of the r1soft mySQL addon, the mySQL addon works by issuing a global lock and then flushes changes from memory to disk to ensure consistent snapshots. As it works with the r1soft device driver the process usually takes less than a second making this the perfect method to take backups and have the master position at the time the backup was taken. At the time of writing r1soft does not support the ability to store the master/slave status data although hopefully in the future it will store and display it to avoid having to use the following trick.

We can however work around this by monitoring the r1soft server-log.txt to determine when it has flushed and unlocked the mySQL instance. As it happens very early on in the process we can manually issue a global read lock, get the current position, monitor the server-log.txt until r1soft has performed the snapshot of mySQL and then release the global lock manually.

I have performed an example of this for this article. I started by logging into the CDP interface, browsing to the system in question and getting to the start backup process, having this all filled in and ready to submit speeds the process up. Once this was ready I logged into the database system and issued a global read lock then obtained the current master position,

mysql> flush tables with read lock;
Query OK, 0 rows affected (0.10 sec)

mysql> show master status;
+—————–+———–+————–+——————+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+—————–+———–+————–+——————+
| mysql-bin.024272 | 120146094 | | |
+—————–+———–+————–+——————+
1 row in set (0.00 sec)

I then clicked ok under the schedule backup window and started a tail on server-log.txt (usually found in /usr/r1soft/buserver/log) and waited until it reached the mySQL lock, flush and unlock.


2011-01-27 02:44:27,743 INFO: (tid:976019) Starting task 'Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
2011-01-27 02:44:27,771 INFO: (tid:976019) Starting Backup Task.
2011-01-27 02:44:27,772 INFO: (tid:976019) Backing up partition tables.
2011-01-27 02:44:27,773 INFO: (tid:976020) Starting task 'Partition Table Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
2011-01-27 02:44:27,790 INFO: (tid:976020) Starting partition table backup for host (xx.xx.xx.xx) device (/dev/sda) Disk Safe (0) recovery point (6025).
2011-01-27 02:44:27,892 INFO: (tid:976020) AGENT: (Righteous Backup Linux Agent) 1.62.0 build 6386
2011-01-27 02:44:28,019 WARN: (tid:976020) AGENT: allowing control from backup server (xx.xx.xx.xx) with valid RSA key
2011-01-27 02:44:28,059 WARN: (tid:976020) AGENT: sending auth challenge for allowed host at (xx.xx.xx.xx) port (41758)
2011-01-27 02:44:28,108 INFO: (tid:976020) AGENT: host (xx.xx.xx.xx) port (41758) authentication successful
2011-01-27 02:44:28,193 INFO: Agent (xx.xx.xx.xx) authenticated successfully
2011-01-27 02:44:28,267 INFO: (tid:976020) AGENT: Partition Backup request accepted. Starting backup.
2011-01-27 02:44:28,269 INFO: (tid:976020) AGENT: Found 4 partition(s), 1 extended partition(s) and 124 extra sector(s)
2011-01-27 02:44:28,270 INFO: (tid:976020) AGENT: Need to back up 126 sectors
2011-01-27 02:44:28,270 INFO: (tid:976020) Sector size (512) bytes.
2011-01-27 02:44:28,271 INFO: (tid:976020) Number of sectors (126).
--SNIP--
2011-01-27 02:44:32,083 INFO: Sending (1) MySQL instances to agent.
2011-01-27 02:44:32,083 INFO: Sending next MySQL instance to agent.
2011-01-27 02:44:32,117 INFO: (tid:976019) AGENT: Backup request accepted. Starting backup.
2011-01-27 02:44:39,340 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,353 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,354 INFO: (tid:976019) AGENT: Flushed 1 MySQL instances.
2011-01-27 02:44:39,355 INFO: (tid:976019) AGENT: Locked 1 MySQL instances.
2011-01-27 02:44:39,355 INFO: (tid:976019) AGENT: Unlocked 1 MySQL instances.
2011-01-27 02:44:39,356 INFO: (tid:976019) AGENT: Snapshot completed in 0.059 seconds.
2011-01-27 02:44:39,357 INFO: (tid:976019) AGENT: Acquired MySQL lock in 0.025 seconds and held lock for 0.085 seconds on 1 instance.
2011-01-27 02:44:39,357 INFO: (tid:976019) Backup pipeline size (256) blocks.
2011-01-27 02:44:39,358 INFO: (tid:976019) Block size (4096).
2011-01-27 02:44:39,358 INFO: (tid:976019) Partition size (34529701) blocks.
2011-01-27 02:44:39,359 INFO: (tid:976019) Mount Point (/).
2011-01-27 02:44:39,803 INFO: (tid:976019) Network Queue Size: 1280 blocks

Once it has completed the stage in bold I simply released the read lock,

mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)

I now had a backup in r1soft and had the master position of the time it was taken. Once the backup completed I simply restored the tables I needed to the slave using the restore to an alternate host and started replication using this data.


INFO 01/27/2011 02:44:27 GMT (tid:976019) Starting task 'Backup' for host (xx.xx.xx.xx) with host id (xxxxxxxx)
INFO 01/27/2011 02:51:10 GMT (tid:976019) Backup Task finished.

I restored the backup to the mysql datadir, as you are backing up the master you can skip all the binlogs. After this was restored I started mySQL, changed the master to the details we got from the master status earlier and started the slave.


mysql> CHANGE MASTER to MASTER_HOST='192.168.1.3', MASTER_USER='slave', MASTER_PASSWORD='xxxxxx', MASTER_LOG_FILE='mysql-bin.024272', MASTER_LOG_POS=120146094;
Query OK, 0 rows affected (0.01 sec)

mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status \G;
Seconds_Behind_Master: 0