I had set myself a goal to make around 20 blog posts per year, it’s now almost December and I still have not bothered. It is safe to say I am not exactly prolific.
I decided that I would learn python, surprisingly it’s been really easy to pick up and I thought I would share some of the things I made in my first 2 days of python.
The first I written was to phrase the results from yahoo’s football score page(http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html). I was on this page anyway and thought why not!
#!/usr/bin/python
import urllib2
import re
response = urllib2.urlopen('http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html')
html = response.read()
re_channel = re.compile("<td class=\"ko\"><abbr class=\"dtstart\" title=\"\">([0-9a-zA-z ,:]*)</abbr></td>" +
"\n <td class=\"match.*?\n" +
" <a href=.*?\">" +
"\n <span class=\"home\">([a-zA-Z ]*)</span>.*?" +
"\n <span class=\"score\">([0-9 :-]*)</span>.*?" +
"\n <span class=\"away\">([a-zA-Z ]*)</span>", re.I | re.S | re.M)
find_result = re_channel.findall(html)
print find_result
Sample output
[('13 Aug, 15:00', 'Wigan Athletic', '1 - 1', 'Norwich City'), ('13 Aug, 15:00', 'Fulham', '0 - 0', 'Aston Villa'), ('13 Aug, 15:00', 'Blackburn Rovers', '1 - 2', 'Wolverhampton Wanderers'), ('13 Aug, 15:00', 'Liverpool', '1 - 1', 'Sunderland'), ('13 Aug, 15:00', 'Queens Park Rangers', '0 - 4', 'Bolton Wanderers'), ('13 Aug, 17:30', 'Newcastle United', '0 - 0', 'Arsenal'), ('14 Aug, 13:30', 'Stoke City', '0 - 0', 'Chelsea'), ('14 Aug, 16:00', 'West Bromwich Albion', '1 - 2', 'Manchester United'), ('15 Aug, 20:00', 'Manchester City', '4 - 0', 'Swansea City'), ('20 Aug, 12:00', 'Sunderland', '0 - 1', 'Newcastle United'), ('20 Aug, 12:45', 'Arsenal', '0 - 2', 'Liverpool'), ('20 Aug, 15:00', 'Swansea City', '0 - 0', 'Wigan Athletic'), ('20 Aug, 15:00', 'Aston Villa', '3 - 1', 'Blackburn Rovers'), ('20 Aug, 15:00', 'Everton', '0 - 1', 'Queens Park Rangers'), ('20 Aug, 17:30', 'Chelsea', '2 - 1', 'West Bromwich Albion'), ('21 Aug, 13:30', 'Norwich City', '1 - 1', 'Stoke City'), ('21 Aug, 14:05', 'Wolverhampton Wanderers', '2 - 0', 'Fulham'), ('21 Aug, 16:00', 'Bolton Wanderers', '2 - 3', 'Manchester City'), ('22 Aug, 20:00', 'Manchester United', '3 - 0', 'Tottenham Hotspur'), ('27 Aug, 12:05', 'Aston Villa', '0 - 0', 'Wolverhampton Wanderers'), ('27 Aug, 12:30', 'Wigan Athletic', '2 - 0', 'Queens Park Rangers'), ('27 Aug, 15:00', 'Swansea City', '0 - 0', 'Sunderland'), ('27 Aug, 15:00', 'Chelsea', '3 - 1', 'Norwich City'), ('27 Aug, 15:00', 'Blackburn Rovers', '0 - 1', 'Everton'), ('27 Aug, 17:30', 'Liverpool', '3 - 1', 'Bolton Wanderers'), ('28 Aug, 13:00', 'Newcastle United', '2 - 1', 'Fulham'), ('28 Aug, 13:30', 'Tottenham Hotspur', '1 - 5', 'Manchester City'), ('28 Aug, 15:00', 'West Bromwich Albion', '0 - 1', 'Stoke City'), ('28 Aug, 16:00', 'Manchester United', '8 - 2', 'Arsenal')]
The next was to obtain a list of the uk charts top 100 and find the youtube video for it.
#!/usr/bin/python
import urllib
import urllib2
import re
import sys
from BeautifulSoup import BeautifulStoneSoup
def getvideolink(link):
request = urllib2.Request(link)
request.add_header('User-Agent', "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Safari/533.3")
response = urllib2.urlopen(request)
text = response.read()
videos = re.findall("<a href=\"/watch\?v=([\w-]+)", text)
return "http://www.youtube.com/watch?v=%s" % videos[0]
response = urllib2.urlopen('http://localhost/test')
html = response.read()
re_channel = re.compile("<h3>(.*?)</h3>" +
"\r\n.*?<h4>(.*?)</h4>.*?<h5", re.I | re.S | re.M)
find_result = re_channel.findall(html)
#remove first entry as it's a dup
del find_result[0]
for result in find_result:
title=unicode(BeautifulStoneSoup(result[0],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
artist=unicode(BeautifulStoneSoup(result[1],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
link="http://www.youtube.com/results?search_query=" + urllib.quote_plus(title);
print getvideolink(link)
Sample output
RIHANNA FT CALVIN HARRIS - WE FOUND LOVE // http://www.youtube.com/watch?v=tg00YEETFzg
FLO RIDA - GOOD FEELING // http://www.youtube.com/watch?v=3OnnDqH6Wj8
ONE DIRECTION - GOTTA BE YOU // http://www.youtube.com/watch?v=nvfejaHz-o0
LABRINTH FT TINIE TEMPAH - EARTHQUAKE // http://www.youtube.com/watch?v=u0fk6syQ7iY
ED SHEERAN - LEGO HOUSE // http://www.youtube.com/watch?v=c4BLVznuWnU
DAVID GUETTA FT USHER - WITHOUT YOU // http://www.youtube.com/watch?v=jUe8uoKdHao
MAROON 5 FT CHRISTINA AGUILERA - MOVES LIKE JAGGER // http://www.youtube.com/watch?v=iEPTlhBmwRg
JLS - TAKE A CHANCE ON ME // http://www.youtube.com/watch?v=djV11Xbc914
PROFESSOR GREEN FT EMELI SANDE - READ ALL ABOUT IT // http://www.youtube.com/watch?v=-_oLfC5Z_Ys
CHRISTINA PERRI - JAR OF HEARTS // http://www.youtube.com/watch?v=8v_4O44sfjM
LMFAO - SEXY AND I KNOW IT // http://www.youtube.com/watch?v=wyx6JDQCslE
DRAKE FT RIHANNA - TAKE CARE // http://www.youtube.com/watch?v=PaXslpx3MWY
COLDPLAY - PARADISE // http://www.youtube.com/watch?v=1G4isv_Fylg
CHARLENE SORAIA - WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
SATURDAYS - MY HEART TAKES OVER // http://www.youtube.com/watch?v=DgmoYgpMNX8
CHER LLOYD FT MIKE POSNER - WITH UR LOVE // http://www.youtube.com/watch?v=axpO86pGHAM
BRUNO MARS - IT WILL RAIN // http://www.youtube.com/watch?v=W-w3WfgpcGg
LADY GAGA - MARRY THE NIGHT // http://www.youtube.com/watch?v=O4IgYxHEAuk
LANA DEL REY - VIDEO GAMES // http://www.youtube.com/watch?v=HO1OV5B_JDw
PIXIE LOTT FT PUSHA T - WHAT DO YOU TAKE ME FOR // http://www.youtube.com/watch?v=OQCcwNMp830
KELLY CLARKSON - MR KNOW IT ALL // http://www.youtube.com/watch?v=0C_oNMH0GTk
LOICK ESSIEN - ME WITHOUT YOU // http://www.youtube.com/watch?v=lKDmJwoZ4RA
FLORENCE & THE MACHINE - SHAKE IT OUT // http://www.youtube.com/watch?v=WbN0nX61rIs
COLLECTIVE - TEARDROP // http://www.youtube.com/watch?v=u7K72X4eo_s
ONE DIRECTION - WHAT MAKES YOU BEAUTIFUL // http://www.youtube.com/watch?v=QJO3ROT-A4E
ED SHEERAN - THE A TEAM // http://www.youtube.com/watch?v=UAWcs5H-qgQ
WANTED - LIGHTNING // http://www.youtube.com/watch?v=MQyHyfLp5NI
JESSIE J - WHO YOU ARE // http://www.youtube.com/watch?v=j2WWrupMBAE
LUCENZO & QWOTE - DANZA KUDURO // http://www.youtube.com/watch?v=rUFgacK8sZ0
GYM CLASS HEROES/ADAM LEVINE - STEREO HEARTS // http://www.youtube.com/watch?v=T3E9Wjbq44E
MAVERICK SABRE - I NEED // http://www.youtube.com/watch?v=VA770wpLX-Q
WESTLIFE - LIGHTHOUSE // http://www.youtube.com/watch?v=Tivph7mTku4
SEAN PAUL FT ALEXIS JORDAN - GOT 2 LUV U // http://www.youtube.com/watch?v=tDq3fNew1rU
LADY GAGA - THE EDGE OF GLORY // http://www.youtube.com/watch?v=QeWBS0JBNzQ
GLEE CAST - RUMOUR HAS IT/SOMEONE LIKE YOU // http://www.youtube.com/watch?v=qb7zjKkLCoQ
SLOW MOVING MILLIE - PLEASE PLEASE PLEASE LET ME GET WHAT I // http://www.youtube.com/watch?v=DMQbzLrvwlE
KATY PERRY - THE ONE THAT GOT AWAY // http://www.youtube.com/watch?v=Ahha3Cqe_fk
ELBOW - ONE DAY LIKE THIS // http://www.youtube.com/watch?v=0NFV8dHrZYM
CHRISTINA PERRI - A THOUSAND YEARS // http://www.youtube.com/watch?v=z5Q8x1wYN4w
SNOW PATROL - THIS ISN'T EVERYTHING YOU ARE // http://www.youtube.com/watch?v=Q-Gljs8Y3Q8
RIZZLE KICKS - WHEN I WAS A YOUNGSTER // http://www.youtube.com/watch?v=Rc2iUwMpb8Y
GOO GOO DOLLS - IRIS // http://www.youtube.com/watch?v=NdYWuo9OFAw
DAPPY - NO REGRETS // http://www.youtube.com/watch?v=WoImizvsj5w
BRUNO MARS - MARRY YOU // http://www.youtube.com/watch?v=xB40cQD677s
ADELE - SET FIRE TO THE RAIN // http://www.youtube.com/watch?v=ss0HAdW1DnY
NICKELBACK - WHEN WE STAND TOGETHER // http://www.youtube.com/watch?v=76RbWuFll0Y
AFROJACK & STEVE AOKI - NO BEEF // http://www.youtube.com/watch?v=ksocjhxX_DQ
ADELE - SOMEONE LIKE YOU // http://www.youtube.com/watch?v=hLQl3WQQoQ0
PIXIE LOTT - ALL ABOUT TONIGHT // http://www.youtube.com/watch?v=swcULf1ATyU
NICKI MINAJ - SUPER BASS // http://www.youtube.com/watch?v=4JipHEz53sU
CALLING - WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
JAMES MORRISON - I WON'T LET YOU GO // http://www.youtube.com/watch?v=sgRb_lfIZ6A
OLLY MURS FT RIZZLE KICKS - HEART SKIPS A BEAT // http://www.youtube.com/watch?v=j5dFe-WKuPs
EXAMPLE - MIDNIGHT RUN // http://www.youtube.com/watch?v=iwYGi7YG4Js
TINCHY STRYDER/CALVIN HARRIS - OFF THE RECORD // http://www.youtube.com/watch?v=UknZiaIC9y8
SAK NOEL - LOCA PEOPLE // http://www.youtube.com/watch?v=-d6b1yn-YhQ
BRUNO MARS - RUNAWAY BABY // http://www.youtube.com/watch?v=UDG_CrqJV-0
JASON DERULO - FIGHT FOR YOU // http://www.youtube.com/watch?v=2aSOQRih6WY
CHER LLOYD - SWAGGER JAGGER // http://www.youtube.com/watch?v=sdbyG2MrBHk
LADY GAGA - BORN THIS WAY // http://www.youtube.com/watch?v=wV1FrqwZyKw
SNOW PATROL - CALLED OUT IN THE DARK // http://www.youtube.com/watch?v=GwTXwJg6_VE
WANTED - WARZONE // http://www.youtube.com/watch?v=yMR382aefmQ
JLS FT DEV - SHE MAKES ME WANNA // http://www.youtube.com/watch?v=FuwTgZOKcf8
NICOLE SCHERZINGER - TRY WITH ME // http://www.youtube.com/watch?v=R7sYiTyBjTY
COBRA STARSHIP FT SABI - YOU MAKE ME FEEL // http://www.youtube.com/watch?v=HpyZEzrDf4c
JESSIE J - WHO'S LAUGHING NOW // http://www.youtube.com/watch?v=KsxSxF3JKeU
NOEL GALLAGHER'S HIGH FLYING - AKA WHAT A LIFE // http://www.youtube.com/watch?v=lwHpLDgWonM
FOSTER THE PEOPLE - PUMPED UP KICKS // http://www.youtube.com/watch?v=SDTZ7iX4vTQ
WRETCH 32 FT JOSH KUMRA - DON'T GO // http://www.youtube.com/watch?v=bj1BMpUnzT8
BIRDY - SKINNY LOVE // http://www.youtube.com/watch?v=aNzCDt2eidg
WANTED - GLAD YOU CAME // http://www.youtube.com/watch?v=2ggzxInyzVE
BIRDY - PEOPLE HELP THE PEOPLE // http://www.youtube.com/watch?v=OmLNs6zQIHo
NICKI MINAJ FT RIHANNA - FLY // http://www.youtube.com/watch?v=3n71KUiWn1I
SKREAM FT SAM FRANK - ANTICIPATE // http://www.youtube.com/watch?v=O3Z1X4MPsqk
WILL YOUNG - JEALOUSY // http://www.youtube.com/watch?v=9MHtrM-jf9o
ADELE - ROLLING IN THE DEEP // http://www.youtube.com/watch?v=rYEDA3JcQqw
JESSIE J FT BOB - PRICE TAG // http://www.youtube.com/watch?v=qMxX-QOV9tI
LMFAO/LAUREN BENNETT/GOONROCK - PARTY ROCK ANTHEM // http://www.youtube.com/watch?v=KQ6zr6kCPj8
JASON DERULO - IT GIRL // http://www.youtube.com/watch?v=4oGUHRXT-wA
BEYONCE - COUNTDOWN // http://www.youtube.com/watch?v=ACkBTqwxcUI
RIZZLE KICKS - DOWN WITH THE TRUMPETS // http://www.youtube.com/watch?v=-aY92XgykhU
COLDPLAY - EVERY TEARDROP IS A WATERFALL // http://www.youtube.com/watch?v=fyMhvkC3A84
CALVIN HARRIS - FEEL SO CLOSE // http://www.youtube.com/watch?v=dGghkjpNCQ8
ADELE - MAKE YOU FEEL MY LOVE // http://www.youtube.com/watch?v=LLoyNxjhTzc
ONE DIRECTION - ANOTHER WORLD // http://www.youtube.com/watch?v=RyZfNBVX1q0
COLDPLAY - VIVA LA VIDA // http://www.youtube.com/watch?v=dvgZkm1xWPE
BEYONCE - LOVE ON TOP // http://www.youtube.com/watch?v=Ob7vObnFUJc
EAGLE-EYE CHERRY - SAVE TONIGHT // http://www.youtube.com/watch?v=dTa2Bzlbjv0
EN VOGUE - DONT LET GO (LOVE) // http://www.youtube.com/watch?v=QUdAT5Fwnvk
KATY PERRY - FIREWORK // http://www.youtube.com/watch?v=QGJuMBdaqIw
MODESTEP - TO THE STARS // http://www.youtube.com/watch?v=UTKSUlMbp9A
DRAKE - HEADLINES // http://www.youtube.com/watch?v=cimoNqiulUE
ED SHEERAN - YOU NEED ME I DON'T NEED YOU // http://www.youtube.com/watch?v=temYymFGSEc
WOODKID - IRON // http://www.youtube.com/watch?v=vSkb0kDacjs
DELILAH - GO // http://www.youtube.com/watch?v=cxNe9jWNuEU
KATY PERRY FT KANYE WEST - ET // http://www.youtube.com/watch?v=t5Sd5c4o9UM
JAMES VINCENT MCMORROW - HIGHER LOVE // http://www.youtube.com/watch?v=9Z-fE1l9SZ4
BRUNO MARS - JUST THE WAY YOU ARE (AMAZING) // http://www.youtube.com/watch?v=LjhCEhWiKXk
PITBULL/NE-YO/AFROJACK/NAYER - GIVE ME EVERYTHING // http://www.youtube.com/watch?v=EPo5wWmKEaI
Then I wanted to test threading so I came up with something to check a list of links and obtain the "Server:" header
#!/usr/bin/env python
import urllib2
import re
import os
import sys
import time
from multiprocessing import Pool
#Accept file name as input or use default
try:
filename = sys.argv[1]
except IndexError:
filename = "links"
#Function to load urls from file
def loadurls(filename):
try:
inputdata = file(filename).readlines()
if len(inputdata) > 0:
return inputdata
except:
print "ERROR - Unable to process url list"
sys.exit()
#Function to get SERVER header
def getversion(url):
try:
request=urllib2.Request(url)
response=urllib2.urlopen(request)
version=response.info().getheader('Server')
except:
return "ERROR - Unable to fetch %s" % (url)
if version: print "Url: %s Version: %s" % (url, version)
start = time.time()
urls=loadurls(filename)
p = Pool(15)
data = p.map(getversion,urls)
elapsed = (time.time() - start)
print "Processed %s urls in %0.2f seconds" % (len(urls), elapsed)
Sample output
scott@scott:~/python$ ./server-header.py list
Url: http://www.bbc.co.uk
Version: Apache
Url: http://www.google.co.uk
Version: gws
Url: http://scottmcintyre.net
Version: Apache
Processed 4 urls in 1.07 seconds
None of these really serve any purpose and were merely tests I thought I would share. Overall things have been quite easy and feel reasonably confident after just 2 days.