I decided that I would learn python, surprisingly it’s been really easy to pick up and I thought I would share some of the things I made in my first 2 days of python.

The first I written was to phrase the results from yahoo’s football score page(http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html). I was on this page anyway and thought why not!

#!/usr/bin/python
import urllib2
import re
response = urllib2.urlopen(‘http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html’)
html = response.read()
re_channel = re.compile(“<td class=\”ko\”><abbr class=\”dtstart\” title=\”\”>([0-9a-zA-z ,:]*)</abbr></td>” +
“\n <td class=\”match.*?\n” +
” <a href=.*?\”>” +
“\n <span class=\”home\”>([a-zA-Z ]*)</span>.*?” +
“\n <span class=\”score\”>([0-9 :-]*)</span>.*?” +
“\n <span class=\”away\”>([a-zA-Z ]*)</span>”, re.I | re.S | re.M)
find_result = re_channel.findall(html)
print find_result

Sample output

[(’13 Aug, 15:00′, ‘Wigan Athletic’, ‘1 – 1’, ‘Norwich City’), (’13 Aug, 15:00′, ‘Fulham’, ‘0 – 0’, ‘Aston Villa’), (’13 Aug, 15:00′, ‘Blackburn Rovers’, ‘1 – 2’, ‘Wolverhampton Wanderers’), (’13 Aug, 15:00′, ‘Liverpool’, ‘1 – 1’, ‘Sunderland’), (’13 Aug, 15:00′, ‘Queens Park Rangers’, ‘0 – 4’, ‘Bolton Wanderers’), (’13 Aug, 17:30′, ‘Newcastle United’, ‘0 – 0’, ‘Arsenal’), (’14 Aug, 13:30′, ‘Stoke City’, ‘0 – 0’, ‘Chelsea’), (’14 Aug, 16:00′, ‘West Bromwich Albion’, ‘1 – 2’, ‘Manchester United’), (’15 Aug, 20:00′, ‘Manchester City’, ‘4 – 0’, ‘Swansea City’), (’20 Aug, 12:00′, ‘Sunderland’, ‘0 – 1’, ‘Newcastle United’), (’20 Aug, 12:45′, ‘Arsenal’, ‘0 – 2’, ‘Liverpool’), (’20 Aug, 15:00′, ‘Swansea City’, ‘0 – 0’, ‘Wigan Athletic’), (’20 Aug, 15:00′, ‘Aston Villa’, ‘3 – 1’, ‘Blackburn Rovers’), (’20 Aug, 15:00′, ‘Everton’, ‘0 – 1’, ‘Queens Park Rangers’), (’20 Aug, 17:30′, ‘Chelsea’, ‘2 – 1’, ‘West Bromwich Albion’), (’21 Aug, 13:30′, ‘Norwich City’, ‘1 – 1’, ‘Stoke City’), (’21 Aug, 14:05′, ‘Wolverhampton Wanderers’, ‘2 – 0’, ‘Fulham’), (’21 Aug, 16:00′, ‘Bolton Wanderers’, ‘2 – 3’, ‘Manchester City’), (’22 Aug, 20:00′, ‘Manchester United’, ‘3 – 0’, ‘Tottenham Hotspur’), (’27 Aug, 12:05′, ‘Aston Villa’, ‘0 – 0’, ‘Wolverhampton Wanderers’), (’27 Aug, 12:30′, ‘Wigan Athletic’, ‘2 – 0’, ‘Queens Park Rangers’), (’27 Aug, 15:00′, ‘Swansea City’, ‘0 – 0’, ‘Sunderland’), (’27 Aug, 15:00′, ‘Chelsea’, ‘3 – 1’, ‘Norwich City’), (’27 Aug, 15:00′, ‘Blackburn Rovers’, ‘0 – 1’, ‘Everton’), (’27 Aug, 17:30′, ‘Liverpool’, ‘3 – 1’, ‘Bolton Wanderers’), (’28 Aug, 13:00′, ‘Newcastle United’, ‘2 – 1’, ‘Fulham’), (’28 Aug, 13:30′, ‘Tottenham Hotspur’, ‘1 – 5’, ‘Manchester City’), (’28 Aug, 15:00′, ‘West Bromwich Albion’, ‘0 – 1’, ‘Stoke City’), (’28 Aug, 16:00′, ‘Manchester United’, ‘8 – 2’, ‘Arsenal’)]

The next was to obtain a list of the uk charts top 100 and find the youtube video for it.

#!/usr/bin/python
import urllib
import urllib2
import re
import sys
from BeautifulSoup import BeautifulStoneSoup

def getvideolink(link):
request = urllib2.Request(link)
request.add_header(‘User-Agent’, “Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Safari/533.3”)
response = urllib2.urlopen(request)
text = response.read()
videos = re.findall(“<a href=\”/watch\?v=([\w-]+)”, text)
return “http://www.youtube.com/watch?v=%s” % videos[0]

response = urllib2.urlopen(‘http://localhost/test’)
html = response.read()
re_channel = re.compile(“<h3>(.*?)</h3>” +
“\r\n.*?<h4>(.*?)</h4>.*?<h5″, re.I | re.S | re.M)
find_result = re_channel.findall(html)
#remove first entry as it’s a dup
del find_result[0]
for result in find_result:
title=unicode(BeautifulStoneSoup(result[0],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
artist=unicode(BeautifulStoneSoup(result[1],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
link=”http://www.youtube.com/results?search_query=” + urllib.quote_plus(title);
print getvideolink(link)

Sample output

RIHANNA FT CALVIN HARRIS – WE FOUND LOVE // http://www.youtube.com/watch?v=tg00YEETFzg
FLO RIDA – GOOD FEELING // http://www.youtube.com/watch?v=3OnnDqH6Wj8
ONE DIRECTION – GOTTA BE YOU // http://www.youtube.com/watch?v=nvfejaHz-o0
LABRINTH FT TINIE TEMPAH – EARTHQUAKE // http://www.youtube.com/watch?v=u0fk6syQ7iY
ED SHEERAN – LEGO HOUSE // http://www.youtube.com/watch?v=c4BLVznuWnU
DAVID GUETTA FT USHER – WITHOUT YOU // http://www.youtube.com/watch?v=jUe8uoKdHao
MAROON 5 FT CHRISTINA AGUILERA – MOVES LIKE JAGGER // http://www.youtube.com/watch?v=iEPTlhBmwRg
JLS – TAKE A CHANCE ON ME // http://www.youtube.com/watch?v=djV11Xbc914
PROFESSOR GREEN FT EMELI SANDE – READ ALL ABOUT IT // http://www.youtube.com/watch?v=-_oLfC5Z_Ys
CHRISTINA PERRI – JAR OF HEARTS // http://www.youtube.com/watch?v=8v_4O44sfjM
LMFAO – SEXY AND I KNOW IT // http://www.youtube.com/watch?v=wyx6JDQCslE
DRAKE FT RIHANNA – TAKE CARE // http://www.youtube.com/watch?v=PaXslpx3MWY
COLDPLAY – PARADISE // http://www.youtube.com/watch?v=1G4isv_Fylg
CHARLENE SORAIA – WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
SATURDAYS – MY HEART TAKES OVER // http://www.youtube.com/watch?v=DgmoYgpMNX8
CHER LLOYD FT MIKE POSNER – WITH UR LOVE // http://www.youtube.com/watch?v=axpO86pGHAM
BRUNO MARS – IT WILL RAIN // http://www.youtube.com/watch?v=W-w3WfgpcGg
LADY GAGA – MARRY THE NIGHT // http://www.youtube.com/watch?v=O4IgYxHEAuk
LANA DEL REY – VIDEO GAMES // http://www.youtube.com/watch?v=HO1OV5B_JDw
PIXIE LOTT FT PUSHA T – WHAT DO YOU TAKE ME FOR // http://www.youtube.com/watch?v=OQCcwNMp830
KELLY CLARKSON – MR KNOW IT ALL // http://www.youtube.com/watch?v=0C_oNMH0GTk
LOICK ESSIEN – ME WITHOUT YOU // http://www.youtube.com/watch?v=lKDmJwoZ4RA
FLORENCE & THE MACHINE – SHAKE IT OUT // http://www.youtube.com/watch?v=WbN0nX61rIs
COLLECTIVE – TEARDROP // http://www.youtube.com/watch?v=u7K72X4eo_s
ONE DIRECTION – WHAT MAKES YOU BEAUTIFUL // http://www.youtube.com/watch?v=QJO3ROT-A4E
ED SHEERAN – THE A TEAM // http://www.youtube.com/watch?v=UAWcs5H-qgQ
WANTED – LIGHTNING // http://www.youtube.com/watch?v=MQyHyfLp5NI
JESSIE J – WHO YOU ARE // http://www.youtube.com/watch?v=j2WWrupMBAE
LUCENZO & QWOTE – DANZA KUDURO // http://www.youtube.com/watch?v=rUFgacK8sZ0
GYM CLASS HEROES/ADAM LEVINE – STEREO HEARTS // http://www.youtube.com/watch?v=T3E9Wjbq44E
MAVERICK SABRE – I NEED // http://www.youtube.com/watch?v=VA770wpLX-Q
WESTLIFE – LIGHTHOUSE // http://www.youtube.com/watch?v=Tivph7mTku4
SEAN PAUL FT ALEXIS JORDAN – GOT 2 LUV U // http://www.youtube.com/watch?v=tDq3fNew1rU
LADY GAGA – THE EDGE OF GLORY // http://www.youtube.com/watch?v=QeWBS0JBNzQ
GLEE CAST – RUMOUR HAS IT/SOMEONE LIKE YOU // http://www.youtube.com/watch?v=qb7zjKkLCoQ
SLOW MOVING MILLIE – PLEASE PLEASE PLEASE LET ME GET WHAT I // http://www.youtube.com/watch?v=DMQbzLrvwlE
KATY PERRY – THE ONE THAT GOT AWAY // http://www.youtube.com/watch?v=Ahha3Cqe_fk
ELBOW – ONE DAY LIKE THIS // http://www.youtube.com/watch?v=0NFV8dHrZYM
CHRISTINA PERRI – A THOUSAND YEARS // http://www.youtube.com/watch?v=z5Q8x1wYN4w
SNOW PATROL – THIS ISN’T EVERYTHING YOU ARE // http://www.youtube.com/watch?v=Q-Gljs8Y3Q8
RIZZLE KICKS – WHEN I WAS A YOUNGSTER // http://www.youtube.com/watch?v=Rc2iUwMpb8Y
GOO GOO DOLLS – IRIS // http://www.youtube.com/watch?v=NdYWuo9OFAw
DAPPY – NO REGRETS // http://www.youtube.com/watch?v=WoImizvsj5w
BRUNO MARS – MARRY YOU // http://www.youtube.com/watch?v=xB40cQD677s
ADELE – SET FIRE TO THE RAIN // http://www.youtube.com/watch?v=ss0HAdW1DnY
NICKELBACK – WHEN WE STAND TOGETHER // http://www.youtube.com/watch?v=76RbWuFll0Y
AFROJACK & STEVE AOKI – NO BEEF // http://www.youtube.com/watch?v=ksocjhxX_DQ
ADELE – SOMEONE LIKE YOU // http://www.youtube.com/watch?v=hLQl3WQQoQ0
PIXIE LOTT – ALL ABOUT TONIGHT // http://www.youtube.com/watch?v=swcULf1ATyU
NICKI MINAJ – SUPER BASS // http://www.youtube.com/watch?v=4JipHEz53sU
CALLING – WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
JAMES MORRISON – I WON’T LET YOU GO // http://www.youtube.com/watch?v=sgRb_lfIZ6A
OLLY MURS FT RIZZLE KICKS – HEART SKIPS A BEAT // http://www.youtube.com/watch?v=j5dFe-WKuPs
EXAMPLE – MIDNIGHT RUN // http://www.youtube.com/watch?v=iwYGi7YG4Js
TINCHY STRYDER/CALVIN HARRIS – OFF THE RECORD // http://www.youtube.com/watch?v=UknZiaIC9y8
SAK NOEL – LOCA PEOPLE // http://www.youtube.com/watch?v=-d6b1yn-YhQ
BRUNO MARS – RUNAWAY BABY // http://www.youtube.com/watch?v=UDG_CrqJV-0
JASON DERULO – FIGHT FOR YOU // http://www.youtube.com/watch?v=2aSOQRih6WY
CHER LLOYD – SWAGGER JAGGER // http://www.youtube.com/watch?v=sdbyG2MrBHk
LADY GAGA – BORN THIS WAY // http://www.youtube.com/watch?v=wV1FrqwZyKw
SNOW PATROL – CALLED OUT IN THE DARK // http://www.youtube.com/watch?v=GwTXwJg6_VE
WANTED – WARZONE // http://www.youtube.com/watch?v=yMR382aefmQ
JLS FT DEV – SHE MAKES ME WANNA // http://www.youtube.com/watch?v=FuwTgZOKcf8
NICOLE SCHERZINGER – TRY WITH ME // http://www.youtube.com/watch?v=R7sYiTyBjTY
COBRA STARSHIP FT SABI – YOU MAKE ME FEEL // http://www.youtube.com/watch?v=HpyZEzrDf4c
JESSIE J – WHO’S LAUGHING NOW // http://www.youtube.com/watch?v=KsxSxF3JKeU
NOEL GALLAGHER’S HIGH FLYING – AKA WHAT A LIFE // http://www.youtube.com/watch?v=lwHpLDgWonM
FOSTER THE PEOPLE – PUMPED UP KICKS // http://www.youtube.com/watch?v=SDTZ7iX4vTQ
WRETCH 32 FT JOSH KUMRA – DON’T GO // http://www.youtube.com/watch?v=bj1BMpUnzT8
BIRDY – SKINNY LOVE // http://www.youtube.com/watch?v=aNzCDt2eidg
WANTED – GLAD YOU CAME // http://www.youtube.com/watch?v=2ggzxInyzVE
BIRDY – PEOPLE HELP THE PEOPLE // http://www.youtube.com/watch?v=OmLNs6zQIHo
NICKI MINAJ FT RIHANNA – FLY // http://www.youtube.com/watch?v=3n71KUiWn1I
SKREAM FT SAM FRANK – ANTICIPATE // http://www.youtube.com/watch?v=O3Z1X4MPsqk
WILL YOUNG – JEALOUSY // http://www.youtube.com/watch?v=9MHtrM-jf9o
ADELE – ROLLING IN THE DEEP // http://www.youtube.com/watch?v=rYEDA3JcQqw
JESSIE J FT BOB – PRICE TAG // http://www.youtube.com/watch?v=qMxX-QOV9tI
LMFAO/LAUREN BENNETT/GOONROCK – PARTY ROCK ANTHEM // http://www.youtube.com/watch?v=KQ6zr6kCPj8
JASON DERULO – IT GIRL // http://www.youtube.com/watch?v=4oGUHRXT-wA
BEYONCE – COUNTDOWN // http://www.youtube.com/watch?v=ACkBTqwxcUI
RIZZLE KICKS – DOWN WITH THE TRUMPETS // http://www.youtube.com/watch?v=-aY92XgykhU
COLDPLAY – EVERY TEARDROP IS A WATERFALL // http://www.youtube.com/watch?v=fyMhvkC3A84
CALVIN HARRIS – FEEL SO CLOSE // http://www.youtube.com/watch?v=dGghkjpNCQ8
ADELE – MAKE YOU FEEL MY LOVE // http://www.youtube.com/watch?v=LLoyNxjhTzc
ONE DIRECTION – ANOTHER WORLD // http://www.youtube.com/watch?v=RyZfNBVX1q0
COLDPLAY – VIVA LA VIDA // http://www.youtube.com/watch?v=dvgZkm1xWPE
BEYONCE – LOVE ON TOP // http://www.youtube.com/watch?v=Ob7vObnFUJc
EAGLE-EYE CHERRY – SAVE TONIGHT // http://www.youtube.com/watch?v=dTa2Bzlbjv0
EN VOGUE – DONT LET GO (LOVE) // http://www.youtube.com/watch?v=QUdAT5Fwnvk
KATY PERRY – FIREWORK // http://www.youtube.com/watch?v=QGJuMBdaqIw
MODESTEP – TO THE STARS // http://www.youtube.com/watch?v=UTKSUlMbp9A
DRAKE – HEADLINES // http://www.youtube.com/watch?v=cimoNqiulUE
ED SHEERAN – YOU NEED ME I DON’T NEED YOU // http://www.youtube.com/watch?v=temYymFGSEc
WOODKID – IRON // http://www.youtube.com/watch?v=vSkb0kDacjs
DELILAH – GO // http://www.youtube.com/watch?v=cxNe9jWNuEU
KATY PERRY FT KANYE WEST – ET // http://www.youtube.com/watch?v=t5Sd5c4o9UM
JAMES VINCENT MCMORROW – HIGHER LOVE // http://www.youtube.com/watch?v=9Z-fE1l9SZ4
BRUNO MARS – JUST THE WAY YOU ARE (AMAZING) // http://www.youtube.com/watch?v=LjhCEhWiKXk
PITBULL/NE-YO/AFROJACK/NAYER – GIVE ME EVERYTHING // http://www.youtube.com/watch?v=EPo5wWmKEaI

Then I wanted to test threading so I came up with something to check a list of links and obtain the “Server:” header

#!/usr/bin/env python
import urllib2
import re
import os
import sys
import time
from multiprocessing import Pool

#Accept file name as input or use default
try:
filename = sys.argv[1]
except IndexError:
filename = “links”

#Function to load urls from file
def loadurls(filename):
try:
inputdata = file(filename).readlines()
if len(inputdata) > 0:
return inputdata
except:
print “ERROR – Unable to process url list”
sys.exit()

#Function to get SERVER header
def getversion(url):
try:
request=urllib2.Request(url)
response=urllib2.urlopen(request)
version=response.info().getheader(‘Server’)
except:
return “ERROR – Unable to fetch %s” % (url)

if version: print “Url: %s Version: %s” % (url, version)

start = time.time()
urls=loadurls(filename)
p = Pool(15)
data = p.map(getversion,urls)

elapsed = (time.time() – start)
print “Processed %s urls in %0.2f seconds” % (len(urls), elapsed)

Sample output

scott@scott:~/python$ ./server-header.py list
Url: http://www.bbc.co.uk
Version: Apache
Url: http://www.google.co.uk
Version: gws
Url: https://scott.cm
Version: Apache
Processed 4 urls in 1.07 seconds

None of these really serve any purpose and were merely tests I thought I would share.   Overall things have been quite easy and feel reasonably confident after just 2 days.

This blog post will be about what I have learned in 2010 and some plans for 2011.

So what have I learned, enjoyed and hated in 2010?

– I love puppet

– Adobe flash is the bane of my life. Flash on 64bit linux is nothing short of a pain, it causes the most problems, the most crashes and should rot in hell.

– I am starting to like chrome but have not made the full switch from firefox yet.

– I am quite a lazy programmer, more worrying I actually don’t care.

– I love MongoDB and expect big things from it.

– I am unsure weather to love or hate r1soft. On one hand I like what it brings on the other hand it’s very buggy and I don’t think the release of 3.0 is going to address many of these. This should be an entire blog post in itself later.

– Started using ksplice uptrack, already have this running on a few hundred systems and no problems to date. Only time will tell.

I am looking forward to 2011, I am hoping to continue on with getting large portions of the “todo list” completed. Some of the ones off the top of my head are,

– Make the switch from cacti to collectd+MongoDB. Been working on this on and off for awhile. Primary goals is to make it more flexible, more automated and easier to maintain. Since we maintain so many different environments it has to fit some very specific needs. Hoping to replace the primary cacti install and also some of the other secondary customer-specific ones (such as munin).

– Upgrade to nagios 3.x. We still are running nagios 2.x so an upgrade is needed. At the same time need to rebuild and rewrite the automation systems, templates, hosts and groups. Especially to tie better into collectd, the existing nagios+cacti setup is pretty “hacky”.

– Rebuild the client database, notes and task list systems. This work is already under-way so should be completed in Q1 2011. Better encryption, more functionality, more flexibility, easier to manage.

– Build a new ticket system and drop Kayako. I don’t particularly like the direction Kayako has taken in their latest release so it’s time to stop using it (Very hacked up anyway).

– Better documentation. I, along with many others, really fail in this area and I would love to improve on it. Still toying with the idea of internal wiki’s combined with the existing notes systems and to make better use of them.

– Improve the existing yum repo, make more use of mocks, include some newer RPM’s and update all systems to use this. Replace all custom builds to an RPM from the repo (nginx, memcached, etc).

– Make more use of the nessus systems and plugin feeds. At the moment this is mostly semi-manual , need to work on making it fully automated and easy to maintain.

– Post more blog entries? Goal of atleast 20 more blog posts (Less than 2 per month).

– Rewrite the nagios r1soft/buagent check and release it.

– Make the vbulletin static cache into a plugin. This is just a simple patch we provide to customers who don’t have enough capacity and are receiving a sudden burst of traffic (IE a thread slashdotted/etc). It simply stores the threads for guests in memory for 2 minutes and then displays the cache for guests (supports online counters, multiple themes, browsers, mobile, etc).

That’s my list for the time being. Lets hope I can stick to it. Do you have your list ready? Feel free to share.

On Friday I definitely noticed something was not quite right but I was not sure, as the day progressed I felt that the left side of my face was being odd and my eyes were hurting. In a conference call I really noticed it as I started to speak and mentioned in IRC that I was having difficulty speaking clearly.

[17:25] <Scott-Mc> something is definitely wrong ,  even struggling to talk with the mic on the left side of my face:|
—– then 10 minutes later —–
[17:36] <Scott-Mc> god knows what is wrong with me, but I am even struggling to speak

After the conference was finished at about 6PM I started to think about all the things over the past few days and realized something wasn’t quite right and maybe it just wasn’t a bump to the head. So I decided to call the NHS direct 24/7 helpline and after describing my symptoms and answering some questions I got put through to a nurse. She started asking more specific questions because previously I had been unsure exactly what was wrong or how to describe it but within about 5 minutes she had determined what was wrong, told me what it was and had arranged for me to visit the doctor at the A&E.

During the call she asked me to go stand in front of the mirror and smile, it was then I realized that it wasn’t the left side of my face at all – this was perfectly normal, it was the right side. It was paralyzed and it was at this moment I freaked out, especially because she said I should go to the A&E just now.

It was very helpful just to speak to someone as otherwise I would have waited till Monday to go to the doctor if it had not cleared up but speaking to someone made me go immediately. The symptoms did rapidly get worse between about 2PM-6PM as I never noticed it when I looked in the mirror earlier.

I arrived at the A&E about 20 minutes later and waited 3-4 minutes to see a doctor. Again I described my symptoms and he told me I had Bell’s Palsy and explained it alittle. He prescribed me steroids for 1 week and some eye drops. When I asked if he knew how long it would take he said and I quote, “I wont lie, it can take awhile”. I was unsure what to make of this but when I returned home it was straight onto google to read more on it.

As it turns out the vast majority recover from it (85%) and they don’t know what causes it. So far it’s been 8-9 days and I would say that on the fourth day it showed signs of improvement that I definitely noticed (I could smile slightly more) but since then I don’t think it’s improved any which is alittle disappointing. I would really like it to go away ASAP!

At this point I have another appointment with the Doctor on Monday to check the progress, I am hoping it’s cleared up fully in about 2-3 weeks. I hope this is realistic but to be honest it doesn’t really effect me all that much. As I work from home I don’t notice it, the only thing I notice is my right eye can be alittle painful as I have to manually blink it but this has been improving in the past few days also. Overall I have a feeling I should treat this a serious thing but I have not been, still can’t believe it’s been over a week with it already (I was going to blog about it the day after).

What is surprising is the number of number of friends and friends of friends have had this, especially as I had never even heard of it before last week, the good news is that all of them it cleared up just fine, the bads news is that one of them took 6 months.

Not knowing what to write about is generally what made me always not bother and then when there is a subject I am actually not that good at being creative.  The truth is though it is actually just lazyness that has stopped be.

So from today onwards I have decided to have an average of 2 blog posts per month and these can be about anything be it useful information,  a rant about a product/service or just a general meaningless post.