Machine Learning, Data Science, R, Python and stuff
My previous post ‘Outlier removal in R using IQR rule‘ has been one of the most visited posts on here. So now lets have a look at it in Python. This time we’ll be using Pandas and NumPy, along with the Titanic dataset. We will also do a little extra thing – log transform the data.
I’ve been having a quick play with Scale-Invariant Feature Transform (SIFT) in Python. I had a few problems installing it (see here). In short SIFT finds the features of an image, a more detailed explanation can be seen here.
In this example we take a picture of the City Hall in Hull and run it through a simple implementation of SIFT to extract the features of the image.
We’ll use this image, mine is called ‘Proj4_img00000021.jpg’. Click on the image to get the full size version
First lets bring in the packages we need. We don’t really need Numpy or Pandas but its just habbit.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
import cv2 import pandas as pd import numpy as np import matplotlib.pyplot as plt from scipy.misc import imread, imresize %matplotlib inline pd.set_option('display.max_rows', 10) |
Next we need to load the image and display it. Remember, the file should be in the same folder as your Python or Notebook script.
|
1 2 3 4 |
image_file = 'Proj4_img00000021.jpg' img = cv2.imread(image_file) plt.imshow(img) |
This should show this…
Now lets converting it to gray scale.
|
1 2 |
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) plt.imshow(gray) |
Giving us this (colors are from the plt.imshow(), trust me its grayscale).
Now the SIFT bits….
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# Create a SIFT Object sift = cv2.xfeatures2d.SIFT_create() # Get the Key Points from the 'gray' image, this returns a numpy array kp = sift.detect(gray, None) # Now we drawn the gray image and overlay the Key Points (kp) img = cv2.drawKeypoints(gray, kp, None, flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS) # Plot it to the screen, looks a little small plt.imshow(img) # Save the image to a file cv2.imwrite('sift_keypoints.jpg',img) |
Notes
Some examples online say cv2.SIFT() instead of cv2.xfeatures2d.SIFT_create(). Apparently SIFT_create() was moved to xfeatures2d because of a patent issue, I don’t know if this is true or not, I just know I had to use cv2.xfeatures2d.SIFT_create(). Also cv2.drawKeypoints(gray, kp, None, ….) insisted on a 3rd parameter, so I just passed it None which in python language is the same as Null.
When you open up the image you should see this…
As you can see, all of the key points have been detected.
Before we start, Yes I know this is not the official image for “Theano: A CPU and GPU Math Expression Compiler”, but they didn’t have a cool logo, in fact they don’t have a logo at all.
I am sure I read somewhere that Anaconda Python (Download) comes with Theano. But I might be mistaken, all I know it wasnt a quick install when I wanted to use it and I was continuously getting this error which suggested it wasn’t installed…
—————————————————————————
ImportError Traceback (most recent call last)
<ipython-input-3-de45f36b45a8> in <module>()
—-> 1 from theano import *ImportError: No module named theano
Well here is how I fixed it.
NOTE: I am using Windows 8 64bit, Python 2.7.10 | Anaconda 2.4.0 (64-bit)
You might need to do some extra things, here is the official link.
EDIT: I had to install Theano on a different Windows PC and didn’t have to do steps 3 and 4.
K Means clustering is an unsupervised machine learning algorithm. An example of a supervised learning algorithm can be seen when looking at Neural Networks where the learning process involved both the inputs (x) and the outputs (y). During the learning process the error between the predicted outcome (predY) and actual outcome (y) is used to train the system. In an unsupervised method such as K Means clustering the outcome (y) variable is not used in the training process.
In this example we look at using the IRIS dataset and cover:
Requirements: I am using Anaconda Python Distribution which has everything you need including Pandas, NumPy, Matplotlib and importantly SciKit-Learn. I am also using iPython Notebook but you can use whatever IDE you want.
Bring in the libraries you need.
|
1 2 3 4 5 6 7 8 9 10 11 |
import matplotlib.pyplot as plt from sklearn import datasets from sklearn.cluster import KMeans import sklearn.metrics as sm import pandas as pd import numpy as np # Only needed if you want to display your plots inline if using Notebook # change inline to auto if you have Spyder installed %matplotlib inline |
Next load the data. Scikit Learn has some sample datasets, by previously importing the datasets we can use the following
|
1 2 |
# import some data to play with iris = datasets.load_iris() |
You can view the data running each line individually.
|
1 2 3 4 |
iris.data iris.feature_names iris.target iris.target_names |
I like to work with Pandas Dataframes, so we will convert the data into that format. Note that we have separated out the inputs (x) and the outputs/labels (y).
|
1 2 3 4 5 6 |
# Store the inputs as a Pandas Dataframe and set the column names x = pd.DataFrame(iris.data) x.columns = ['Sepal_Length','Sepal_Width','Petal_Length','Petal_Width'] y = pd.DataFrame(iris.target) y.columns = ['Targets'] |
It is always important to have a look at the data. We will do this by plotting two scatter plots. One looking at the Sepal values and another looking at Petal. We will also set it to use some colours so it is clearer.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# Set the size of the plot plt.figure(figsize=(14,7)) # Create a colormap colormap = np.array(['red', 'lime', 'black']) # Plot Sepal plt.subplot(1, 2, 1) plt.scatter(x.Sepal_Length, x.Sepal_Width, c=colormap[y.Targets], s=40) plt.title('Sepal') plt.subplot(1, 2, 2) plt.scatter(x.Petal_Length, x.Petal_Width, c=colormap[y.Targets], s=40) plt.title('Petal') |
This is the easy part, providing you have the data in the correct format (which we do). Here we only need two lines. First we create the model and specify the number of clusters the model should find (n_clusters=3) next we fit the model to the data.
|
1 2 3 |
# K Means Cluster model = KMeans(n_clusters=3) model.fit(x) |
Next we can view the results. This is the classes that the model decided, remember this is unsupervised and classified these purely based on the data.
|
1 2 |
# This is what KMeans thought model.labels_ |
You should see something like this…
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 2, 2, 2, 0, 2, 2, 2, 2, 2, 2, 0, 0,
2, 2, 2, 2, 0, 2, 0, 2, 0, 2, 2, 0, 0, 2, 2, 2, 2, 2, 0, 2, 2, 2, 2,
0, 2, 2, 2, 0, 2, 2, 2, 0, 2, 2, 0])
Lets plot the actual classes against the predicted classes from the K Means model.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# View the results # Set the size of the plot plt.figure(figsize=(14,7)) # Create a colormap colormap = np.array(['red', 'lime', 'black']) # Plot the Original Classifications plt.subplot(1, 2, 1) plt.scatter(x.Petal_Length, x.Petal_Width, c=colormap[y.Targets], s=40) plt.title('Real Classification') # Plot the Models Classifications plt.subplot(1, 2, 2) plt.scatter(x.Petal_Length, x.Petal_Width, c=colormap[model.labels_], s=40) plt.title('K Mean Classification') |
Here we are plotting the Petal Length and Width, however each plot changes the colors of the points using either c=colormap[y.Targets] for the original class and c=colormap[model.labels_] for the predicted classess.
The result is….
Ignore the colours (at the moment). Because the the model is unsupervised it did not know which label (class 0, 1 or 2) to assign to each class.
The Fix
Here we are going to change the class labels, we are not changing the any of the classification groups we are simply giving each group the correct number. We need to do this for measuring the performance.
Using this code below we using the np.choose() to assign new values, basically we are changing the 1’s in the predicted values to 0’s and the 0’s to 1’s. Class 2 matched so we can leave. By running the two print functions you can see that all we have done is swap the values.
NOTE: your results might be different to mine, if so you will have to figure out which class matches which and adjust the order of the values in the np.choose() function.
|
1 2 3 4 |
# The fix, we convert all the 1s to 0s and 0s to 1s. predY = np.choose(model.labels_, [1, 0, 2]).astype(np.int64) print (model.labels_) print (predY) |
Re-plot
Now we can re plot the data as before but using predY instead of model.labels_.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# View the results # Set the size of the plot plt.figure(figsize=(14,7)) # Create a colormap colormap = np.array(['red', 'lime', 'black']) # Plot Orginal plt.subplot(1, 2, 1) plt.scatter(x.Petal_Length, x.Petal_Width, c=colormap[y.Targets], s=40) plt.title('Real Classification') # Plot Predicted with corrected values plt.subplot(1, 2, 2) plt.scatter(x.Petal_Length, x.Petal_Width, c=colormap[predY], s=40) plt.title('K Mean Classification') |
No we can see that the K Means classifier has identified one class correctly (red) but some blacks have been classed as greens and vice versa.
There are a number of ways in which we can measure a classifiers performance. Here we will calculate the accuracy and also the confusion matrix.
We need to values y which is the true (original) values and predY which are the models values.
Accuracy
|
1 2 |
# Performance Metrics sm.accuracy_score(y, predY) |
My result was 0.89333333333333331, so we can say that the model has an accuracy of 89.3%. Not base considering the model was unsupervised.
Confusion Matrix
|
1 2 |
# Confusion Matrix sm.confusion_matrix(y, predY) |
My results are…
array([[50, 0, 0],
[ 0, 48, 2],
[ 0, 14, 36]])
Hopefully the table below will render correctly, but we can summaries a the confusion matrix as shown below:
| Real Class | ||||
|---|---|---|---|---|
| 0 | 1 | 2 | ||
| Predicted Class | 0 | 50 | 0 | 0 |
| 1 | 0 | 48 | 2 | |
| 2 | 0 | 14 | 36 | |
The confusion matrix also allows for a wider range of performance metrics to be calculated, see here for more details: https://en.wikipedia.org/wiki/Confusion_matrix.
I was recently asked to guest lecture at De Montfort University to the MSc students studying Intelligent Systems and Robotics, and Business Intelligence Systems and Data Mining. The lecturer was on my MSc project which I completed at the university last year.
The project was around the use of Convolutional Neural Networks with my work focusing on published work from Deepmind, specifically “Playing Atari with Deep Reinforcement Learning” (2013). You can view a Google blog focusing on the 2015 publication here.
Below is a copy of the slides I presented. They are slightly modified from my dissertation viva as I wanted to express to the students the steps taken throughout the project.
If you can’t see the slide below click here.
I’ve just found my slides for a paper I wrote for the UK Computational Intelligence 2014 Conference and thought I’d share them with the world. So I’ve uploaded them to Slideshare, so hopefully you can see them below or find them here.
I did a previous post about this here, along with the videos…
So some time back I started an ongoing post trying to compare R, Matlab and Python. Well my answer is simple, there is no answer. And if you think differently you should ask yourself, “am I just being a fan-boy”?
If you want to read my previous post it is here, if not here is a quick summary of all 3…
Matlab
R / RStudio
Python (more SciKit Learn really)
So that’s it, all 3 are good. It depends on what you want to use it for. My fan-boy opinion is currently R, looks good on the CV, has loads of packages and the graphs look nice. Also, sooo much support for everything you want to do.
So about 2 years ago I reached out to a fellow MSc student through Linkedin who was studying via Distance Learning. It turns out he was a chap based in San Francisco working at a big credit card company. His job title was ‘Data Scientist’ which I thought was novel. It turns out his job was developing methods and algorithms for detecting credit card thefts. We exchanged a couple of emails, general stuff about our plans after the course. He explained how the ‘data scientist’ role was starting to become big over there (USA). So I wondered, what is a data scientist? Why don’t we have any jobs like that around here?
What I found was these types of jobs are popular in London with the big financial companies and the emerging high tech companies with salaries ranging upwards of £60k per annum. Sounds a lot to a northerner but with higher living costs and as stated by Partridge (2002) “I guarantee you’ll either be mugged or not appreciated” it raises the question, is it worth the move?
Slightly further a field, Cambridge also seems to have a high demand for Data Scientists as the city attracts more and more research arms of companies such as Philips and Microsoft. But as we move further up north the demand starts to drop, why?
I started thinking about this and it is true, we don’t have the big financial institutions or the large ‘high tech’ companies. But we do have some big companies. Just thinking in Hull alone we have BP, BAE SYSTEMS, Smith and Nephew, Reckitt Benckiser, ARCO and many more (I got bored thinking of more, sorry). But then we add another question, how many data scientists does one company need?
But then I also thought back to a couple of companies I have worked for, a door manufacturer (£4 million turn over), a ship repair company (now with about a £8 million turn over). But neither of these companies would ever think about hiring a data scientist. But why? The answer is easy, they don’t what what a data scientist is or how they can help their business.
Lets look at me, I have a degree in IT and 15 years experience working in IT, including a couple of times as an IT manager. I’ve spent time managing databases, developing software with the aim of improving the company from a technical perspective. I’ve even taught computing from school to degree level. But never hired, worked as or worked with a Data Scientist.
This leads on to my next question…
So my experience of education has made me question, what the hell are we teaching? Lets look at Level 3 provision in Hull. We have three colleges and now every academy has a 6th form but no-one is teaching skills to become a data scientist. As a city, everything computing seems to be focused on making games, often quoting how much the game industry is worth.
Advertising you teach Game Making is great, but it’s a little like advertising a course in Premiership Football Skills. You get a lot of hype among young people, you get a lot of applications for your course but very few actually end up playing for a premiership team. So the next argument is, “but we teaching people how to make their own games, you know all entrepreneurial and that!”. Good and I fully agree with that, but who is that good for? The chap who sets up the games company and a few of his/her mates? Look at Mojang, sold Minecraft for $2 billion, employs 50 people, Notch takes his $2 billion and buggers of to LA. So is games good for the individual or the local economy?
Lets compare that with the potential of a Data Scientist in Hull. Firstly as shown in the illustration somewhere on this page, a data scientist is multi-disciplinary. You need the data analyst skills, the computer science skills and business/context awareness. Your job, go into a business and make a real change. Look at the data, investigate why things are happening, questioning all the time, looking for statistical relevance. Going a step further with predictive modelling, ‘if we change this, what will happen?’, even further with machine learning. Moving away from just storing data to actually using it. So whats good for the economy? Well for a start companies will be able to understand their businesses in more depth, make them more efficient and possibly reduce costs. Companies in the local area will start catching up with the higher tech companies who are already employing data scientists.
It felt like I ended up ranting, so I’ll finish with a little example I learnt from a school. In this school the teachers fill in a spreadsheet of pupils performance, they pass the spreadsheet to the deputy headteacher who transfers the data to a computer program. The computer system outputs the pupils target for that year, in other words what level they should be at. A couple of points….
Such a simple example but no-one is teaching the skills to be able to do this, whilst others are not seeing the need.
So I’ve had a little problem. I needed to open an Access Database in RStudio. Sounds easy right? Well the problem was that I have MS Office 2007 along with the default OBDC drivers, these are 32bit. I’m also using R and RStudio 64bit. They don’t work well together.
I looked for the 64 bit driver and found it here: Microsoft Access Database Engine 2010 Redistributable.
The next problem is, it doesn’t look like you can install this whilst you have Office 2007 installed. I read you can uninstall Office 2007, install the new driver and then re-install Office 2007.
I didn’t try that, I simply ran the new installer from command prompt with the /passive argument.
Just realised I haven’t updated the blog this month. Which seems odd.
Well I’ve been pretty busy, started the month with a interim project review/update meeting. It felt like a pretty big deal but it was good to have my first paper drafted which showed really good progress. The hardest part is balancing the ideas of multiple experts in the field who all think the project can go in different ways. In this review I have two professors, two senior academics and my very senior (and experienced) industrial supervisor.
Just a quick demo on creating Kaplan Meier estimation using R. Started working with one tutorial that didn’t work but then found something else. More complex example, so here is the basic code you will need. Read more
The PhD thesis is apparently 80,000 words and approximately 10 chapters. Sounds a little daunting at first. I think I did 18,000 for my MSc project. We so far I’ve been plodding on with the literature review mainly looking at Home Telehealth Monitoring trials. At first it started getting messy with notes getting lost and swapping backwards and forwards between papers.
Just thought I’d do a quick post to reflect on the PhD process so far. Everything it going well, I think. I meet my third supervisor next week. I might have to do a presentation, I’ll probably find out about 10 minutes before I do. So I think I’m prepare one just in case. Nervous isn’t the best word to describe it, but I was a little <whatever it’s called> before, then I google’d the chap and now I’m <whatever it’s called> a lot. No only a Professor of Cardiology but also one of the senior cardiology people in the country.
So linear regression seem to be a nice place to start which should lead nicely on to logistic regression. I’ve been given some tutorials/files to work through written for R, well based on my previous post (R vs Matlab vs Python) I decided to have a go at creating a Python version.
You can view the finished file here (iPython Notebook Version).
If anyone is reading the blog, I use Google Analytics so I know 1 person is…!
I am still playing about with the template for the site so if you visit and it looks different don’t worry.
Just got back from a workshop held at the Turing Gateway to Mathematics in Cambridge. The event had a range of fantastic speakers discussing issues around data linkage and data privacy.
Chris Dibben (University of Edinburgh) introduced the idea of linking different sets of data about subjects from multiple sources. He gave the example of how data is collected from pre-birth (pregnancy records) all the way through a persons life until death (or just after). All of this data is stored is different locations with no unique identifier.