Error: unsupported locale setting

So it looks like it doesn’t have the locale installed (In my case EN_GB) this is easily fixed by running,

sudo dpkg-reconfigure locales

Then selecting the necessary locale, you’ll most likely want to default to UTF8.

You should then be able to run trac and have it accessible, in my case

sudo -u trac tracd -s -p 8000 –pidfile ~trac/dev/tracd.pid -d –basic-auth=”*,/home/trac/dev/htpasswd,dev” ~trac/dev
curl -I localhost:8000
HTTP/1.0 200 OK
Server: tracd/0.11.7 Python/2.6.6
Date: Wed, 16 Jan 2013 07:38:32 GMT
Cache-Control: must-revalidate
Content-Type: text/html;charset=utf-8
Content-Length: 6184
Set-Cookie: trac_form_token=a83b66948209b4197936bd1a; Path=/
Set-Cookie: trac_session=d2dad341a8e80e6fd2f76551; expires=Tue, 16-Apr-2013 07:38:32 GMT; Path=/

CREATE TABLE history (
sports_id int(11) DEFAULT NULL,
event_id int(11) DEFAULT NULL,
country varchar(10) DEFAULT NULL,
scheduled_off varchar(30) DEFAULT NULL,
actual_off varchar(30) DEFAULT NULL,
full_description varchar(150) DEFAULT NULL,
odds double DEFAULT NULL,
settled_date varchar(30) DEFAULT NULL,
first_taken varchar(30) DEFAULT NULL,
latest_taken varchar(30) DEFAULT NULL,
course varchar(150) DEFAULT NULL,
volume_matched double DEFAULT NULL,
number_bets double DEFAULT NULL,
event varchar(50) DEFAULT NULL,
in_play varchar(2) DEFAULT NULL,
selection varchar(50) DEFAULT NULL,
selection_id int(11) DEFAULT NULL,
win_flag varchar(1) DEFAULT NULL,
KEY sports_id (sports_id),
KEY event_id (event_id),
KEY odds (odds),
KEY win_flag (win_flag),
KEY descr (full_description)
) ENGINE=InnoDB DEFAULT

Then use the following script,

#!/usr/bin/python
import csv
import MySQLdb
import re
import os

path=”/path/to/files”

db = MySQLdb.connect(host=’localhost’,
user=’user’,
passwd=’pass’,
db=’betfair’)

cursor = db.cursor()
files = os.listdir(path)

for filename in files:
csv_data = csv.reader(file(path+”/”+filename))
for row in csv_data:
cursor.execute(“INSERT into history(sports_id, event_id, settled_date, country, full_description, course, scheduled_off, event, actual_off, selection_id, selection, odds, number_bets, volume_matched, latest_taken, first_taken, win_flag, in_play) values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)”, row)

db.commit()
cursor.close()

Adjust the path at the to to the location of the raw csv(uncompressed) files that you would like loaded into mySQL.
Edit the database information
Run the script

You can also add filters, for example when processing horse racing data I wanted only the following,

– UK races
– Bets placed before the event (IE not in running)
– Exclude hurdles

This is what I used

for row in csv_data:
if row[1] != “EVENT_ID” and row[3] == “GB” and row[17] == “PE” and “PLACED” not in row[7] and “Hrd” not in row[7]:
if re.search(‘[0-9]f’, row[7]):
cursor.execute(“INSERT into history(sports_id, event_id, settled_date, country, full_description, course, scheduled_off, event, actual_off, selection_id, selection, odds, number_bets, volume_matched, latest_taken, first_taken, win_flag, in_play) values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)”, row)

In case this is useful to anyone here is the regex I have written,

meetings = re.findall(“<h5 id=\”fastfixhead(\d+)\”.*\”>(\w+)<\/a><\/h5>”, info)
for meeting in meetings:
times = re.findall(“<li class=\” \”><a href=\”/card.aspx\?raceid=(\d+)&amp;meetingid=(“+meeting[0]+”)&amp;date=([0-9]{4}-[0-9]{2}-[0-9]{2})&amp.*<strong>([0-9]{2}:[0-9]{2})<\/strong> – ([0-9+][f|m].*)\((\d+) run”, info)

This gathers the course name, raceid, meetingid, date, race time, distance and number of runners. It only gathers UK data (by only matching alphanumeric and underscores it strips non UK codes) so replace “\w+” with “.*” for the meetings match to include other countries.

Sample output,

Wolverhampton

RaceID: 741858
Date: 57674,
Race Time: 2013-01-14
Distance: 13:50,
Number of runners: 7f 32y
RaceID: 741859
Date: 57674,
Race Time: 2013-01-14
Distance: 14:20,
Number of runners: 5f 216y
RaceID: 741860
Date: 57674,
Race Time: 2013-01-14
Distance: 14:50,
Number of runners: 1m 4f 50y
RaceID: 741861
Date: 57674,
Race Time: 2013-01-14
Distance: 15:20,
Number of runners: 1m 141y
RaceID: 741862
Date: 57674,
Race Time: 2013-01-14
Distance: 15:50,
Number of runners: 1m 141y
RaceID: 741863
Date: 57674,
Race Time: 2013-01-14
Distance: 16:20,
Number of runners: 1m 1f 103y
RaceID: 741864
Date: 57674,
Race Time: 2013-01-14
Distance: 16:50,
Number of runners: 1m 1f 103y

It appears twitter don’t allow most of the apps (and there are lots of them) to mass unfollow as part of their TOS.   I found this at http://www.hostnexus.com/blog/how-to-bulk-unfollow-on-twitter and decided to give the steps in there a shot.   I tried a few of the apps with the chrome check all plugin but all of them have daily limits and given up here. This is when I decided instead to just write a quick python script using the API.

It was easy enough to get started with the twitter API.
Simply visit https://dev.twitter.com and login with your account.  
Create a new application,   get your key.
Install the twitter python API at http://code.google.com/p/python-twitter/

You can get the latest version of the script at: https://github.com/Scott-Mc/python/blob/master/twitter-mass-unfollow.py and the version at the time of writing at the bottom of this post.

This was my profile before running the script

This is my profile now after running the script,

This is the code at the time of writing,

#!/usr/bin/env python

#
#Mass unfollow friends on twitter
#

# Author
# – Scott Mcintyre
#
# Usage
#
# Get your keys from https://dev.twitter.com/
# Ensure it has write access
# Change the api key values marked ##### below with the twitter keys
# Install http://code.google.com/p/python-twitter/

import twitter
import time

api = twitter.Api(consumer_key=’#####’, consumer_secret=’#####’, access_token_key=’#####’, access_token_secret=’#####’)

start = time.time()
i = 0
friends=api.GetFriends()

while (1):
try:
for u in friends:
print “Deleting %s” % (u.name)
api.DestroyFriendship(u.id);
i = i + 1

except:
print “Possibly Rate limited? sleeping 60 seconds”
time.sleep(60)

elapsed = (time.time() – start)
print “Deleted %s friends in %0.2f seconds” % (str(i), elapsed)

I decided that I would learn python, surprisingly it’s been really easy to pick up and I thought I would share some of the things I made in my first 2 days of python.

The first I written was to phrase the results from yahoo’s football score page(http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html). I was on this page anyway and thought why not!

#!/usr/bin/python
import urllib2
import re
response = urllib2.urlopen(‘http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html’)
html = response.read()
re_channel = re.compile(“<td class=\”ko\”><abbr class=\”dtstart\” title=\”\”>([0-9a-zA-z ,:]*)</abbr></td>” +
“\n <td class=\”match.*?\n” +
” <a href=.*?\”>” +
“\n <span class=\”home\”>([a-zA-Z ]*)</span>.*?” +
“\n <span class=\”score\”>([0-9 :-]*)</span>.*?” +
“\n <span class=\”away\”>([a-zA-Z ]*)</span>”, re.I | re.S | re.M)
find_result = re_channel.findall(html)
print find_result

Sample output

[(’13 Aug, 15:00′, ‘Wigan Athletic’, ‘1 – 1’, ‘Norwich City’), (’13 Aug, 15:00′, ‘Fulham’, ‘0 – 0’, ‘Aston Villa’), (’13 Aug, 15:00′, ‘Blackburn Rovers’, ‘1 – 2’, ‘Wolverhampton Wanderers’), (’13 Aug, 15:00′, ‘Liverpool’, ‘1 – 1’, ‘Sunderland’), (’13 Aug, 15:00′, ‘Queens Park Rangers’, ‘0 – 4’, ‘Bolton Wanderers’), (’13 Aug, 17:30′, ‘Newcastle United’, ‘0 – 0’, ‘Arsenal’), (’14 Aug, 13:30′, ‘Stoke City’, ‘0 – 0’, ‘Chelsea’), (’14 Aug, 16:00′, ‘West Bromwich Albion’, ‘1 – 2’, ‘Manchester United’), (’15 Aug, 20:00′, ‘Manchester City’, ‘4 – 0’, ‘Swansea City’), (’20 Aug, 12:00′, ‘Sunderland’, ‘0 – 1’, ‘Newcastle United’), (’20 Aug, 12:45′, ‘Arsenal’, ‘0 – 2’, ‘Liverpool’), (’20 Aug, 15:00′, ‘Swansea City’, ‘0 – 0’, ‘Wigan Athletic’), (’20 Aug, 15:00′, ‘Aston Villa’, ‘3 – 1’, ‘Blackburn Rovers’), (’20 Aug, 15:00′, ‘Everton’, ‘0 – 1’, ‘Queens Park Rangers’), (’20 Aug, 17:30′, ‘Chelsea’, ‘2 – 1’, ‘West Bromwich Albion’), (’21 Aug, 13:30′, ‘Norwich City’, ‘1 – 1’, ‘Stoke City’), (’21 Aug, 14:05′, ‘Wolverhampton Wanderers’, ‘2 – 0’, ‘Fulham’), (’21 Aug, 16:00′, ‘Bolton Wanderers’, ‘2 – 3’, ‘Manchester City’), (’22 Aug, 20:00′, ‘Manchester United’, ‘3 – 0’, ‘Tottenham Hotspur’), (’27 Aug, 12:05′, ‘Aston Villa’, ‘0 – 0’, ‘Wolverhampton Wanderers’), (’27 Aug, 12:30′, ‘Wigan Athletic’, ‘2 – 0’, ‘Queens Park Rangers’), (’27 Aug, 15:00′, ‘Swansea City’, ‘0 – 0’, ‘Sunderland’), (’27 Aug, 15:00′, ‘Chelsea’, ‘3 – 1’, ‘Norwich City’), (’27 Aug, 15:00′, ‘Blackburn Rovers’, ‘0 – 1’, ‘Everton’), (’27 Aug, 17:30′, ‘Liverpool’, ‘3 – 1’, ‘Bolton Wanderers’), (’28 Aug, 13:00′, ‘Newcastle United’, ‘2 – 1’, ‘Fulham’), (’28 Aug, 13:30′, ‘Tottenham Hotspur’, ‘1 – 5’, ‘Manchester City’), (’28 Aug, 15:00′, ‘West Bromwich Albion’, ‘0 – 1’, ‘Stoke City’), (’28 Aug, 16:00′, ‘Manchester United’, ‘8 – 2’, ‘Arsenal’)]

The next was to obtain a list of the uk charts top 100 and find the youtube video for it.

#!/usr/bin/python
import urllib
import urllib2
import re
import sys
from BeautifulSoup import BeautifulStoneSoup

def getvideolink(link):
request = urllib2.Request(link)
request.add_header(‘User-Agent’, “Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Safari/533.3”)
response = urllib2.urlopen(request)
text = response.read()
videos = re.findall(“<a href=\”/watch\?v=([\w-]+)”, text)
return “http://www.youtube.com/watch?v=%s” % videos[0]

response = urllib2.urlopen(‘http://localhost/test’)
html = response.read()
re_channel = re.compile(“<h3>(.*?)</h3>” +
“\r\n.*?<h4>(.*?)</h4>.*?<h5″, re.I | re.S | re.M)
find_result = re_channel.findall(html)
#remove first entry as it’s a dup
del find_result[0]
for result in find_result:
title=unicode(BeautifulStoneSoup(result[0],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
artist=unicode(BeautifulStoneSoup(result[1],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
link=”http://www.youtube.com/results?search_query=” + urllib.quote_plus(title);
print getvideolink(link)

Sample output

RIHANNA FT CALVIN HARRIS – WE FOUND LOVE // http://www.youtube.com/watch?v=tg00YEETFzg
FLO RIDA – GOOD FEELING // http://www.youtube.com/watch?v=3OnnDqH6Wj8
ONE DIRECTION – GOTTA BE YOU // http://www.youtube.com/watch?v=nvfejaHz-o0
LABRINTH FT TINIE TEMPAH – EARTHQUAKE // http://www.youtube.com/watch?v=u0fk6syQ7iY
ED SHEERAN – LEGO HOUSE // http://www.youtube.com/watch?v=c4BLVznuWnU
DAVID GUETTA FT USHER – WITHOUT YOU // http://www.youtube.com/watch?v=jUe8uoKdHao
MAROON 5 FT CHRISTINA AGUILERA – MOVES LIKE JAGGER // http://www.youtube.com/watch?v=iEPTlhBmwRg
JLS – TAKE A CHANCE ON ME // http://www.youtube.com/watch?v=djV11Xbc914
PROFESSOR GREEN FT EMELI SANDE – READ ALL ABOUT IT // http://www.youtube.com/watch?v=-_oLfC5Z_Ys
CHRISTINA PERRI – JAR OF HEARTS // http://www.youtube.com/watch?v=8v_4O44sfjM
LMFAO – SEXY AND I KNOW IT // http://www.youtube.com/watch?v=wyx6JDQCslE
DRAKE FT RIHANNA – TAKE CARE // http://www.youtube.com/watch?v=PaXslpx3MWY
COLDPLAY – PARADISE // http://www.youtube.com/watch?v=1G4isv_Fylg
CHARLENE SORAIA – WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
SATURDAYS – MY HEART TAKES OVER // http://www.youtube.com/watch?v=DgmoYgpMNX8
CHER LLOYD FT MIKE POSNER – WITH UR LOVE // http://www.youtube.com/watch?v=axpO86pGHAM
BRUNO MARS – IT WILL RAIN // http://www.youtube.com/watch?v=W-w3WfgpcGg
LADY GAGA – MARRY THE NIGHT // http://www.youtube.com/watch?v=O4IgYxHEAuk
LANA DEL REY – VIDEO GAMES // http://www.youtube.com/watch?v=HO1OV5B_JDw
PIXIE LOTT FT PUSHA T – WHAT DO YOU TAKE ME FOR // http://www.youtube.com/watch?v=OQCcwNMp830
KELLY CLARKSON – MR KNOW IT ALL // http://www.youtube.com/watch?v=0C_oNMH0GTk
LOICK ESSIEN – ME WITHOUT YOU // http://www.youtube.com/watch?v=lKDmJwoZ4RA
FLORENCE & THE MACHINE – SHAKE IT OUT // http://www.youtube.com/watch?v=WbN0nX61rIs
COLLECTIVE – TEARDROP // http://www.youtube.com/watch?v=u7K72X4eo_s
ONE DIRECTION – WHAT MAKES YOU BEAUTIFUL // http://www.youtube.com/watch?v=QJO3ROT-A4E
ED SHEERAN – THE A TEAM // http://www.youtube.com/watch?v=UAWcs5H-qgQ
WANTED – LIGHTNING // http://www.youtube.com/watch?v=MQyHyfLp5NI
JESSIE J – WHO YOU ARE // http://www.youtube.com/watch?v=j2WWrupMBAE
LUCENZO & QWOTE – DANZA KUDURO // http://www.youtube.com/watch?v=rUFgacK8sZ0
GYM CLASS HEROES/ADAM LEVINE – STEREO HEARTS // http://www.youtube.com/watch?v=T3E9Wjbq44E
MAVERICK SABRE – I NEED // http://www.youtube.com/watch?v=VA770wpLX-Q
WESTLIFE – LIGHTHOUSE // http://www.youtube.com/watch?v=Tivph7mTku4
SEAN PAUL FT ALEXIS JORDAN – GOT 2 LUV U // http://www.youtube.com/watch?v=tDq3fNew1rU
LADY GAGA – THE EDGE OF GLORY // http://www.youtube.com/watch?v=QeWBS0JBNzQ
GLEE CAST – RUMOUR HAS IT/SOMEONE LIKE YOU // http://www.youtube.com/watch?v=qb7zjKkLCoQ
SLOW MOVING MILLIE – PLEASE PLEASE PLEASE LET ME GET WHAT I // http://www.youtube.com/watch?v=DMQbzLrvwlE
KATY PERRY – THE ONE THAT GOT AWAY // http://www.youtube.com/watch?v=Ahha3Cqe_fk
ELBOW – ONE DAY LIKE THIS // http://www.youtube.com/watch?v=0NFV8dHrZYM
CHRISTINA PERRI – A THOUSAND YEARS // http://www.youtube.com/watch?v=z5Q8x1wYN4w
SNOW PATROL – THIS ISN’T EVERYTHING YOU ARE // http://www.youtube.com/watch?v=Q-Gljs8Y3Q8
RIZZLE KICKS – WHEN I WAS A YOUNGSTER // http://www.youtube.com/watch?v=Rc2iUwMpb8Y
GOO GOO DOLLS – IRIS // http://www.youtube.com/watch?v=NdYWuo9OFAw
DAPPY – NO REGRETS // http://www.youtube.com/watch?v=WoImizvsj5w
BRUNO MARS – MARRY YOU // http://www.youtube.com/watch?v=xB40cQD677s
ADELE – SET FIRE TO THE RAIN // http://www.youtube.com/watch?v=ss0HAdW1DnY
NICKELBACK – WHEN WE STAND TOGETHER // http://www.youtube.com/watch?v=76RbWuFll0Y
AFROJACK & STEVE AOKI – NO BEEF // http://www.youtube.com/watch?v=ksocjhxX_DQ
ADELE – SOMEONE LIKE YOU // http://www.youtube.com/watch?v=hLQl3WQQoQ0
PIXIE LOTT – ALL ABOUT TONIGHT // http://www.youtube.com/watch?v=swcULf1ATyU
NICKI MINAJ – SUPER BASS // http://www.youtube.com/watch?v=4JipHEz53sU
CALLING – WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
JAMES MORRISON – I WON’T LET YOU GO // http://www.youtube.com/watch?v=sgRb_lfIZ6A
OLLY MURS FT RIZZLE KICKS – HEART SKIPS A BEAT // http://www.youtube.com/watch?v=j5dFe-WKuPs
EXAMPLE – MIDNIGHT RUN // http://www.youtube.com/watch?v=iwYGi7YG4Js
TINCHY STRYDER/CALVIN HARRIS – OFF THE RECORD // http://www.youtube.com/watch?v=UknZiaIC9y8
SAK NOEL – LOCA PEOPLE // http://www.youtube.com/watch?v=-d6b1yn-YhQ
BRUNO MARS – RUNAWAY BABY // http://www.youtube.com/watch?v=UDG_CrqJV-0
JASON DERULO – FIGHT FOR YOU // http://www.youtube.com/watch?v=2aSOQRih6WY
CHER LLOYD – SWAGGER JAGGER // http://www.youtube.com/watch?v=sdbyG2MrBHk
LADY GAGA – BORN THIS WAY // http://www.youtube.com/watch?v=wV1FrqwZyKw
SNOW PATROL – CALLED OUT IN THE DARK // http://www.youtube.com/watch?v=GwTXwJg6_VE
WANTED – WARZONE // http://www.youtube.com/watch?v=yMR382aefmQ
JLS FT DEV – SHE MAKES ME WANNA // http://www.youtube.com/watch?v=FuwTgZOKcf8
NICOLE SCHERZINGER – TRY WITH ME // http://www.youtube.com/watch?v=R7sYiTyBjTY
COBRA STARSHIP FT SABI – YOU MAKE ME FEEL // http://www.youtube.com/watch?v=HpyZEzrDf4c
JESSIE J – WHO’S LAUGHING NOW // http://www.youtube.com/watch?v=KsxSxF3JKeU
NOEL GALLAGHER’S HIGH FLYING – AKA WHAT A LIFE // http://www.youtube.com/watch?v=lwHpLDgWonM
FOSTER THE PEOPLE – PUMPED UP KICKS // http://www.youtube.com/watch?v=SDTZ7iX4vTQ
WRETCH 32 FT JOSH KUMRA – DON’T GO // http://www.youtube.com/watch?v=bj1BMpUnzT8
BIRDY – SKINNY LOVE // http://www.youtube.com/watch?v=aNzCDt2eidg
WANTED – GLAD YOU CAME // http://www.youtube.com/watch?v=2ggzxInyzVE
BIRDY – PEOPLE HELP THE PEOPLE // http://www.youtube.com/watch?v=OmLNs6zQIHo
NICKI MINAJ FT RIHANNA – FLY // http://www.youtube.com/watch?v=3n71KUiWn1I
SKREAM FT SAM FRANK – ANTICIPATE // http://www.youtube.com/watch?v=O3Z1X4MPsqk
WILL YOUNG – JEALOUSY // http://www.youtube.com/watch?v=9MHtrM-jf9o
ADELE – ROLLING IN THE DEEP // http://www.youtube.com/watch?v=rYEDA3JcQqw
JESSIE J FT BOB – PRICE TAG // http://www.youtube.com/watch?v=qMxX-QOV9tI
LMFAO/LAUREN BENNETT/GOONROCK – PARTY ROCK ANTHEM // http://www.youtube.com/watch?v=KQ6zr6kCPj8
JASON DERULO – IT GIRL // http://www.youtube.com/watch?v=4oGUHRXT-wA
BEYONCE – COUNTDOWN // http://www.youtube.com/watch?v=ACkBTqwxcUI
RIZZLE KICKS – DOWN WITH THE TRUMPETS // http://www.youtube.com/watch?v=-aY92XgykhU
COLDPLAY – EVERY TEARDROP IS A WATERFALL // http://www.youtube.com/watch?v=fyMhvkC3A84
CALVIN HARRIS – FEEL SO CLOSE // http://www.youtube.com/watch?v=dGghkjpNCQ8
ADELE – MAKE YOU FEEL MY LOVE // http://www.youtube.com/watch?v=LLoyNxjhTzc
ONE DIRECTION – ANOTHER WORLD // http://www.youtube.com/watch?v=RyZfNBVX1q0
COLDPLAY – VIVA LA VIDA // http://www.youtube.com/watch?v=dvgZkm1xWPE
BEYONCE – LOVE ON TOP // http://www.youtube.com/watch?v=Ob7vObnFUJc
EAGLE-EYE CHERRY – SAVE TONIGHT // http://www.youtube.com/watch?v=dTa2Bzlbjv0
EN VOGUE – DONT LET GO (LOVE) // http://www.youtube.com/watch?v=QUdAT5Fwnvk
KATY PERRY – FIREWORK // http://www.youtube.com/watch?v=QGJuMBdaqIw
MODESTEP – TO THE STARS // http://www.youtube.com/watch?v=UTKSUlMbp9A
DRAKE – HEADLINES // http://www.youtube.com/watch?v=cimoNqiulUE
ED SHEERAN – YOU NEED ME I DON’T NEED YOU // http://www.youtube.com/watch?v=temYymFGSEc
WOODKID – IRON // http://www.youtube.com/watch?v=vSkb0kDacjs
DELILAH – GO // http://www.youtube.com/watch?v=cxNe9jWNuEU
KATY PERRY FT KANYE WEST – ET // http://www.youtube.com/watch?v=t5Sd5c4o9UM
JAMES VINCENT MCMORROW – HIGHER LOVE // http://www.youtube.com/watch?v=9Z-fE1l9SZ4
BRUNO MARS – JUST THE WAY YOU ARE (AMAZING) // http://www.youtube.com/watch?v=LjhCEhWiKXk
PITBULL/NE-YO/AFROJACK/NAYER – GIVE ME EVERYTHING // http://www.youtube.com/watch?v=EPo5wWmKEaI

Then I wanted to test threading so I came up with something to check a list of links and obtain the “Server:” header

#!/usr/bin/env python
import urllib2
import re
import os
import sys
import time
from multiprocessing import Pool

#Accept file name as input or use default
try:
filename = sys.argv[1]
except IndexError:
filename = “links”

#Function to load urls from file
def loadurls(filename):
try:
inputdata = file(filename).readlines()
if len(inputdata) > 0:
return inputdata
except:
print “ERROR – Unable to process url list”
sys.exit()

#Function to get SERVER header
def getversion(url):
try:
request=urllib2.Request(url)
response=urllib2.urlopen(request)
version=response.info().getheader(‘Server’)
except:
return “ERROR – Unable to fetch %s” % (url)

if version: print “Url: %s Version: %s” % (url, version)

start = time.time()
urls=loadurls(filename)
p = Pool(15)
data = p.map(getversion,urls)

elapsed = (time.time() – start)
print “Processed %s urls in %0.2f seconds” % (len(urls), elapsed)

Sample output

scott@scott:~/python$ ./server-header.py list
Url: http://www.bbc.co.uk
Version: Apache
Url: http://www.google.co.uk
Version: gws
Url: https://scott.cm
Version: Apache
Processed 4 urls in 1.07 seconds

None of these really serve any purpose and were merely tests I thought I would share.   Overall things have been quite easy and feel reasonably confident after just 2 days.