I had set myself a goal to make around 20 blog posts per year, it’s now almost December and I still have not bothered. It is safe to say I am not exactly prolific.
I decided that I would learn python, surprisingly it’s been really easy to pick up and I thought I would share some of the things I made in my first 2 days of python.
The first I written was to phrase the results from yahoo’s football score page(http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html). I was on this page anyway and thought why not!
#!/usr/bin/python
import urllib2
import re
response = urllib2.urlopen(‘http://uk.eurosport.yahoo.com/football/premier-league/2011-2012/results/2011_08.html’)
html = response.read()
re_channel = re.compile(“<td class=\”ko\”><abbr class=\”dtstart\” title=\”\”>([0-9a-zA-z ,:]*)</abbr></td>” +
“\n <td class=\”match.*?\n” +
” <a href=.*?\”>” +
“\n <span class=\”home\”>([a-zA-Z ]*)</span>.*?” +
“\n <span class=\”score\”>([0-9 :-]*)</span>.*?” +
“\n <span class=\”away\”>([a-zA-Z ]*)</span>”, re.I | re.S | re.M)
find_result = re_channel.findall(html)
print find_result
Sample output
[(’13 Aug, 15:00′, ‘Wigan Athletic’, ‘1 – 1’, ‘Norwich City’), (’13 Aug, 15:00′, ‘Fulham’, ‘0 – 0’, ‘Aston Villa’), (’13 Aug, 15:00′, ‘Blackburn Rovers’, ‘1 – 2’, ‘Wolverhampton Wanderers’), (’13 Aug, 15:00′, ‘Liverpool’, ‘1 – 1’, ‘Sunderland’), (’13 Aug, 15:00′, ‘Queens Park Rangers’, ‘0 – 4’, ‘Bolton Wanderers’), (’13 Aug, 17:30′, ‘Newcastle United’, ‘0 – 0’, ‘Arsenal’), (’14 Aug, 13:30′, ‘Stoke City’, ‘0 – 0’, ‘Chelsea’), (’14 Aug, 16:00′, ‘West Bromwich Albion’, ‘1 – 2’, ‘Manchester United’), (’15 Aug, 20:00′, ‘Manchester City’, ‘4 – 0’, ‘Swansea City’), (’20 Aug, 12:00′, ‘Sunderland’, ‘0 – 1’, ‘Newcastle United’), (’20 Aug, 12:45′, ‘Arsenal’, ‘0 – 2’, ‘Liverpool’), (’20 Aug, 15:00′, ‘Swansea City’, ‘0 – 0’, ‘Wigan Athletic’), (’20 Aug, 15:00′, ‘Aston Villa’, ‘3 – 1’, ‘Blackburn Rovers’), (’20 Aug, 15:00′, ‘Everton’, ‘0 – 1’, ‘Queens Park Rangers’), (’20 Aug, 17:30′, ‘Chelsea’, ‘2 – 1’, ‘West Bromwich Albion’), (’21 Aug, 13:30′, ‘Norwich City’, ‘1 – 1’, ‘Stoke City’), (’21 Aug, 14:05′, ‘Wolverhampton Wanderers’, ‘2 – 0’, ‘Fulham’), (’21 Aug, 16:00′, ‘Bolton Wanderers’, ‘2 – 3’, ‘Manchester City’), (’22 Aug, 20:00′, ‘Manchester United’, ‘3 – 0’, ‘Tottenham Hotspur’), (’27 Aug, 12:05′, ‘Aston Villa’, ‘0 – 0’, ‘Wolverhampton Wanderers’), (’27 Aug, 12:30′, ‘Wigan Athletic’, ‘2 – 0’, ‘Queens Park Rangers’), (’27 Aug, 15:00′, ‘Swansea City’, ‘0 – 0’, ‘Sunderland’), (’27 Aug, 15:00′, ‘Chelsea’, ‘3 – 1’, ‘Norwich City’), (’27 Aug, 15:00′, ‘Blackburn Rovers’, ‘0 – 1’, ‘Everton’), (’27 Aug, 17:30′, ‘Liverpool’, ‘3 – 1’, ‘Bolton Wanderers’), (’28 Aug, 13:00′, ‘Newcastle United’, ‘2 – 1’, ‘Fulham’), (’28 Aug, 13:30′, ‘Tottenham Hotspur’, ‘1 – 5’, ‘Manchester City’), (’28 Aug, 15:00′, ‘West Bromwich Albion’, ‘0 – 1’, ‘Stoke City’), (’28 Aug, 16:00′, ‘Manchester United’, ‘8 – 2’, ‘Arsenal’)]
The next was to obtain a list of the uk charts top 100 and find the youtube video for it.
#!/usr/bin/python
import urllib
import urllib2
import re
import sys
from BeautifulSoup import BeautifulStoneSoupdef getvideolink(link):
request = urllib2.Request(link)
request.add_header(‘User-Agent’, “Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Safari/533.3”)
response = urllib2.urlopen(request)
text = response.read()
videos = re.findall(“<a href=\”/watch\?v=([\w-]+)”, text)
return “http://www.youtube.com/watch?v=%s” % videos[0]response = urllib2.urlopen(‘http://localhost/test’)
html = response.read()
re_channel = re.compile(“<h3>(.*?)</h3>” +
“\r\n.*?<h4>(.*?)</h4>.*?<h5″, re.I | re.S | re.M)
find_result = re_channel.findall(html)
#remove first entry as it’s a dup
del find_result[0]
for result in find_result:
title=unicode(BeautifulStoneSoup(result[0],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
artist=unicode(BeautifulStoneSoup(result[1],convertEntities=BeautifulStoneSoup.HTML_ENTITIES ))
link=”http://www.youtube.com/results?search_query=” + urllib.quote_plus(title);
print getvideolink(link)
Sample output
RIHANNA FT CALVIN HARRIS – WE FOUND LOVE // http://www.youtube.com/watch?v=tg00YEETFzg
FLO RIDA – GOOD FEELING // http://www.youtube.com/watch?v=3OnnDqH6Wj8
ONE DIRECTION – GOTTA BE YOU // http://www.youtube.com/watch?v=nvfejaHz-o0
LABRINTH FT TINIE TEMPAH – EARTHQUAKE // http://www.youtube.com/watch?v=u0fk6syQ7iY
ED SHEERAN – LEGO HOUSE // http://www.youtube.com/watch?v=c4BLVznuWnU
DAVID GUETTA FT USHER – WITHOUT YOU // http://www.youtube.com/watch?v=jUe8uoKdHao
MAROON 5 FT CHRISTINA AGUILERA – MOVES LIKE JAGGER // http://www.youtube.com/watch?v=iEPTlhBmwRg
JLS – TAKE A CHANCE ON ME // http://www.youtube.com/watch?v=djV11Xbc914
PROFESSOR GREEN FT EMELI SANDE – READ ALL ABOUT IT // http://www.youtube.com/watch?v=-_oLfC5Z_Ys
CHRISTINA PERRI – JAR OF HEARTS // http://www.youtube.com/watch?v=8v_4O44sfjM
LMFAO – SEXY AND I KNOW IT // http://www.youtube.com/watch?v=wyx6JDQCslE
DRAKE FT RIHANNA – TAKE CARE // http://www.youtube.com/watch?v=PaXslpx3MWY
COLDPLAY – PARADISE // http://www.youtube.com/watch?v=1G4isv_Fylg
CHARLENE SORAIA – WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
SATURDAYS – MY HEART TAKES OVER // http://www.youtube.com/watch?v=DgmoYgpMNX8
CHER LLOYD FT MIKE POSNER – WITH UR LOVE // http://www.youtube.com/watch?v=axpO86pGHAM
BRUNO MARS – IT WILL RAIN // http://www.youtube.com/watch?v=W-w3WfgpcGg
LADY GAGA – MARRY THE NIGHT // http://www.youtube.com/watch?v=O4IgYxHEAuk
LANA DEL REY – VIDEO GAMES // http://www.youtube.com/watch?v=HO1OV5B_JDw
PIXIE LOTT FT PUSHA T – WHAT DO YOU TAKE ME FOR // http://www.youtube.com/watch?v=OQCcwNMp830
KELLY CLARKSON – MR KNOW IT ALL // http://www.youtube.com/watch?v=0C_oNMH0GTk
LOICK ESSIEN – ME WITHOUT YOU // http://www.youtube.com/watch?v=lKDmJwoZ4RA
FLORENCE & THE MACHINE – SHAKE IT OUT // http://www.youtube.com/watch?v=WbN0nX61rIs
COLLECTIVE – TEARDROP // http://www.youtube.com/watch?v=u7K72X4eo_s
ONE DIRECTION – WHAT MAKES YOU BEAUTIFUL // http://www.youtube.com/watch?v=QJO3ROT-A4E
ED SHEERAN – THE A TEAM // http://www.youtube.com/watch?v=UAWcs5H-qgQ
WANTED – LIGHTNING // http://www.youtube.com/watch?v=MQyHyfLp5NI
JESSIE J – WHO YOU ARE // http://www.youtube.com/watch?v=j2WWrupMBAE
LUCENZO & QWOTE – DANZA KUDURO // http://www.youtube.com/watch?v=rUFgacK8sZ0
GYM CLASS HEROES/ADAM LEVINE – STEREO HEARTS // http://www.youtube.com/watch?v=T3E9Wjbq44E
MAVERICK SABRE – I NEED // http://www.youtube.com/watch?v=VA770wpLX-Q
WESTLIFE – LIGHTHOUSE // http://www.youtube.com/watch?v=Tivph7mTku4
SEAN PAUL FT ALEXIS JORDAN – GOT 2 LUV U // http://www.youtube.com/watch?v=tDq3fNew1rU
LADY GAGA – THE EDGE OF GLORY // http://www.youtube.com/watch?v=QeWBS0JBNzQ
GLEE CAST – RUMOUR HAS IT/SOMEONE LIKE YOU // http://www.youtube.com/watch?v=qb7zjKkLCoQ
SLOW MOVING MILLIE – PLEASE PLEASE PLEASE LET ME GET WHAT I // http://www.youtube.com/watch?v=DMQbzLrvwlE
KATY PERRY – THE ONE THAT GOT AWAY // http://www.youtube.com/watch?v=Ahha3Cqe_fk
ELBOW – ONE DAY LIKE THIS // http://www.youtube.com/watch?v=0NFV8dHrZYM
CHRISTINA PERRI – A THOUSAND YEARS // http://www.youtube.com/watch?v=z5Q8x1wYN4w
SNOW PATROL – THIS ISN’T EVERYTHING YOU ARE // http://www.youtube.com/watch?v=Q-Gljs8Y3Q8
RIZZLE KICKS – WHEN I WAS A YOUNGSTER // http://www.youtube.com/watch?v=Rc2iUwMpb8Y
GOO GOO DOLLS – IRIS // http://www.youtube.com/watch?v=NdYWuo9OFAw
DAPPY – NO REGRETS // http://www.youtube.com/watch?v=WoImizvsj5w
BRUNO MARS – MARRY YOU // http://www.youtube.com/watch?v=xB40cQD677s
ADELE – SET FIRE TO THE RAIN // http://www.youtube.com/watch?v=ss0HAdW1DnY
NICKELBACK – WHEN WE STAND TOGETHER // http://www.youtube.com/watch?v=76RbWuFll0Y
AFROJACK & STEVE AOKI – NO BEEF // http://www.youtube.com/watch?v=ksocjhxX_DQ
ADELE – SOMEONE LIKE YOU // http://www.youtube.com/watch?v=hLQl3WQQoQ0
PIXIE LOTT – ALL ABOUT TONIGHT // http://www.youtube.com/watch?v=swcULf1ATyU
NICKI MINAJ – SUPER BASS // http://www.youtube.com/watch?v=4JipHEz53sU
CALLING – WHEREVER YOU WILL GO // http://www.youtube.com/watch?v=iAP9AF6DCu4
JAMES MORRISON – I WON’T LET YOU GO // http://www.youtube.com/watch?v=sgRb_lfIZ6A
OLLY MURS FT RIZZLE KICKS – HEART SKIPS A BEAT // http://www.youtube.com/watch?v=j5dFe-WKuPs
EXAMPLE – MIDNIGHT RUN // http://www.youtube.com/watch?v=iwYGi7YG4Js
TINCHY STRYDER/CALVIN HARRIS – OFF THE RECORD // http://www.youtube.com/watch?v=UknZiaIC9y8
SAK NOEL – LOCA PEOPLE // http://www.youtube.com/watch?v=-d6b1yn-YhQ
BRUNO MARS – RUNAWAY BABY // http://www.youtube.com/watch?v=UDG_CrqJV-0
JASON DERULO – FIGHT FOR YOU // http://www.youtube.com/watch?v=2aSOQRih6WY
CHER LLOYD – SWAGGER JAGGER // http://www.youtube.com/watch?v=sdbyG2MrBHk
LADY GAGA – BORN THIS WAY // http://www.youtube.com/watch?v=wV1FrqwZyKw
SNOW PATROL – CALLED OUT IN THE DARK // http://www.youtube.com/watch?v=GwTXwJg6_VE
WANTED – WARZONE // http://www.youtube.com/watch?v=yMR382aefmQ
JLS FT DEV – SHE MAKES ME WANNA // http://www.youtube.com/watch?v=FuwTgZOKcf8
NICOLE SCHERZINGER – TRY WITH ME // http://www.youtube.com/watch?v=R7sYiTyBjTY
COBRA STARSHIP FT SABI – YOU MAKE ME FEEL // http://www.youtube.com/watch?v=HpyZEzrDf4c
JESSIE J – WHO’S LAUGHING NOW // http://www.youtube.com/watch?v=KsxSxF3JKeU
NOEL GALLAGHER’S HIGH FLYING – AKA WHAT A LIFE // http://www.youtube.com/watch?v=lwHpLDgWonM
FOSTER THE PEOPLE – PUMPED UP KICKS // http://www.youtube.com/watch?v=SDTZ7iX4vTQ
WRETCH 32 FT JOSH KUMRA – DON’T GO // http://www.youtube.com/watch?v=bj1BMpUnzT8
BIRDY – SKINNY LOVE // http://www.youtube.com/watch?v=aNzCDt2eidg
WANTED – GLAD YOU CAME // http://www.youtube.com/watch?v=2ggzxInyzVE
BIRDY – PEOPLE HELP THE PEOPLE // http://www.youtube.com/watch?v=OmLNs6zQIHo
NICKI MINAJ FT RIHANNA – FLY // http://www.youtube.com/watch?v=3n71KUiWn1I
SKREAM FT SAM FRANK – ANTICIPATE // http://www.youtube.com/watch?v=O3Z1X4MPsqk
WILL YOUNG – JEALOUSY // http://www.youtube.com/watch?v=9MHtrM-jf9o
ADELE – ROLLING IN THE DEEP // http://www.youtube.com/watch?v=rYEDA3JcQqw
JESSIE J FT BOB – PRICE TAG // http://www.youtube.com/watch?v=qMxX-QOV9tI
LMFAO/LAUREN BENNETT/GOONROCK – PARTY ROCK ANTHEM // http://www.youtube.com/watch?v=KQ6zr6kCPj8
JASON DERULO – IT GIRL // http://www.youtube.com/watch?v=4oGUHRXT-wA
BEYONCE – COUNTDOWN // http://www.youtube.com/watch?v=ACkBTqwxcUI
RIZZLE KICKS – DOWN WITH THE TRUMPETS // http://www.youtube.com/watch?v=-aY92XgykhU
COLDPLAY – EVERY TEARDROP IS A WATERFALL // http://www.youtube.com/watch?v=fyMhvkC3A84
CALVIN HARRIS – FEEL SO CLOSE // http://www.youtube.com/watch?v=dGghkjpNCQ8
ADELE – MAKE YOU FEEL MY LOVE // http://www.youtube.com/watch?v=LLoyNxjhTzc
ONE DIRECTION – ANOTHER WORLD // http://www.youtube.com/watch?v=RyZfNBVX1q0
COLDPLAY – VIVA LA VIDA // http://www.youtube.com/watch?v=dvgZkm1xWPE
BEYONCE – LOVE ON TOP // http://www.youtube.com/watch?v=Ob7vObnFUJc
EAGLE-EYE CHERRY – SAVE TONIGHT // http://www.youtube.com/watch?v=dTa2Bzlbjv0
EN VOGUE – DONT LET GO (LOVE) // http://www.youtube.com/watch?v=QUdAT5Fwnvk
KATY PERRY – FIREWORK // http://www.youtube.com/watch?v=QGJuMBdaqIw
MODESTEP – TO THE STARS // http://www.youtube.com/watch?v=UTKSUlMbp9A
DRAKE – HEADLINES // http://www.youtube.com/watch?v=cimoNqiulUE
ED SHEERAN – YOU NEED ME I DON’T NEED YOU // http://www.youtube.com/watch?v=temYymFGSEc
WOODKID – IRON // http://www.youtube.com/watch?v=vSkb0kDacjs
DELILAH – GO // http://www.youtube.com/watch?v=cxNe9jWNuEU
KATY PERRY FT KANYE WEST – ET // http://www.youtube.com/watch?v=t5Sd5c4o9UM
JAMES VINCENT MCMORROW – HIGHER LOVE // http://www.youtube.com/watch?v=9Z-fE1l9SZ4
BRUNO MARS – JUST THE WAY YOU ARE (AMAZING) // http://www.youtube.com/watch?v=LjhCEhWiKXk
PITBULL/NE-YO/AFROJACK/NAYER – GIVE ME EVERYTHING // http://www.youtube.com/watch?v=EPo5wWmKEaI
Then I wanted to test threading so I came up with something to check a list of links and obtain the “Server:” header
#!/usr/bin/env python
import urllib2
import re
import os
import sys
import time
from multiprocessing import Pool#Accept file name as input or use default
try:
filename = sys.argv[1]
except IndexError:
filename = “links”#Function to load urls from file
def loadurls(filename):
try:
inputdata = file(filename).readlines()
if len(inputdata) > 0:
return inputdata
except:
print “ERROR – Unable to process url list”
sys.exit()#Function to get SERVER header
def getversion(url):
try:
request=urllib2.Request(url)
response=urllib2.urlopen(request)
version=response.info().getheader(‘Server’)
except:
return “ERROR – Unable to fetch %s” % (url)if version: print “Url: %s Version: %s” % (url, version)
start = time.time()
urls=loadurls(filename)
p = Pool(15)
data = p.map(getversion,urls)elapsed = (time.time() – start)
print “Processed %s urls in %0.2f seconds” % (len(urls), elapsed)
Sample output
scott@scott:~/python$ ./server-header.py list
Url: http://www.bbc.co.uk
Version: Apache
Url: http://www.google.co.uk
Version: gws
Url: https://scott.cm
Version: Apache
Processed 4 urls in 1.07 seconds
None of these really serve any purpose and were merely tests I thought I would share. Overall things have been quite easy and feel reasonably confident after just 2 days.
Leave a Reply