Deep Neural Networks

Deep Neural Networks

Training your first artificial neural network

In the first lesson you will learn how to get started with deep learning using Keras and Tensorflow. You get to train your own deep neural network that works wih brain imaging data.

Here is the code for a deep neural network in Keras. Exciting stuff!

# Visualize training history
from keras.models import Sequential
from keras.layers import Dense, BatchNormalization
from keras.layers import Dropout
from keras.optimizers import RMSprop
from keras import optimizers
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import numpy
import keras

# Remove warnings
import warnings
warnings.filterwarnings('ignore')

# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)

dataset = pd.read_csv("https://www.floydhub.com/viewer/data/res7UHjG5WSPgPStBT84xW/ID-OXY-20.csv")
# split into input (X) and output 󰀀 variables

# Remove missing values
dataset = dataset.dropna()

# make a label dataset
dataset["Label"] = dataset["Mark"]

# change rest values to 
# default mode
dataset["Mark"][dataset["Mark"] == "REST"] = 0
# task positive network
dataset["Mark"][dataset["Mark"] == "ADDITION"] = 1
dataset["Mark"][dataset["Mark"] == "PASSTHOUGHT"] = 2
dataset["Mark"][dataset["Mark"] == "JUNK"] = 3

# remove the JUNK data
dataset = dataset[dataset.Mark != 2]
dataset = dataset[dataset.Mark != 3]

# shuffle the data
dataset = dataset.sample(frac=1)

# 52 broadmann areas data
X = np.array(dataset.ix[:, :'CH52'])
# default mode network or task positive network
Y = np.array([[1,0] if i == 0 else [0,1] for i in dataset.Mark])

# Dropout - the number of neurons removed at each layers, who are readded when testing
# Batch size - the number of data points added at each time, affects training time
# Epochs - the number of training/test sessions

# create model
model = Sequential()

# makes the values between 0 and 1
model.add(BatchNormalization(input_shape=(52,)))
model.add(Dropout(0.3))
model.add(Dense(100, init="normal", activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(100, init="normal", activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(100, init="normal", activation='relu'))

model.add(Dense(2, init="normal", activation='softmax'))
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
# Fit the model
history = model.fit(X, Y, validation_split=0.5, nb_epoch=50, batch_size=50, verbose=1)
# list all data in history
print(history.history.keys())
# summarize history for accuracy
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

 

What do I need to do in order to get started with machine learning?

Machine Learning Map

AI is a fast moving discipline

Machine learning and AI is a huge discipline that is always growing. However it’s easy to get started and it’s moving really fast so you can just grab on and stay on the bleeding edge of technology.

Tensorflow

Google open sourcing Tensorflow https://www.tensorflow.org/ made it much easier to get started in machine learning. What would have taken months to do ten years ago can now be accomplished in days.

Keras

Even better, Keras came along as an abstraction on top of Tensorflow for deep learning, one of the most advanced parts of deep learning.

Types of machine learning

There are many different types of machine learning. The most advances in machine learning have been done using artificial neural networks, so if you want to start out doing just one thing, start out with artificial neural networks.

Here is a list of different types of machine learning.

Decision tree learning Association rule learning Artificial neural networks Deep learning Inductive logic programming Support vector machines Clustering Bayesian networks Reinforcement learning Representation learning Similarity and metric learning Sparse dictionary learning Genetic algorithms Rule-based machine learning Learning classifier systems

What you need to know about data and machine learning

Machine learning is a way to make advanced statistical models using math. It’s a way to make a computer guess. For machine learning to work you need good data, the more data the better. So before you go further it might be good to take a moment and think about data. The reason why machine learning models are becoming so good is that we have so much data. However machine learning models can’t perform magic (even thought it might seem like it).

Garbage in = Garbage out Take a moment and think about what problem you are trying to solve? What data do you need for solving your problem?

Classic machine learning datasets MNIST MNIST is a dataset of handwritten digits. It is the hello world of machine learning. The goal with MNIST is to classify (guess) which handwritten digit is displayed by use of a machine learning model.

http://yann.lecun.com/exdb/mnist/

IRIS IRIS is another equally famous machine learning dataset. Iris is a dataset for classifying different types of flowers.

https://archive.ics.uci.edu/ml/datasets/iris

Open Datasets from Kaggle Here are places you can find ready made datasets for machine learning http://kaggle.com/

Here is a list of all the datasets available at Kaggle with a link on where to download them.

Dataset Link IMDB

https://www.kaggle.com/deepmatrix/imdb-5000-movie-dataset Soccer scores dataset https://www.kaggle.com/hugomathien/soccer Credit card fraud detection

https://www.kaggle.com/dalpozz/creditcardfraud Human resources analytics

https://www.kaggle.com/ludobenistant/hr-analytics Food facts

https://www.kaggle.com/openfoodfacts/world-food-facts Climate change

https://www.kaggle.com/berkeleyearth/climate-change-earth-surface-temperature-data Daily news for stock market predictions https://www.kaggle.com/aaron7sun/stocknews New York Stock Exchange https://www.kaggle.com/dgawlik/nyseUS Stocks Fundamentals

https://www.kaggle.com/usfundamentals/us-stocks-fundamentals Bitcoin historical data https://www.kaggle.com/mczielinski/bitcoin-historical-data Adult census income, predict earning based on census https://www.kaggle.com/uciml/adult-census-income House sales in King County https://www.kaggle.com/harlfoxem/housesalesprediction EEG brain waves https://www.kaggle.com/wanghaohan/eeg-brain-wave-for-confusion Synchronized brain wave dataset https://www.kaggle.com/berkeley-biosense/synchronized-brainwave-dataset CT medical image analysis https://www.kaggle.com/kmader/siim-medical-image-analysis-tutorial Structural MRI Dataset https://www.kaggle.com/ilknuricke/neurohackinginrimages Circadian rhythm in the brain https://www.kaggle.com/kmader/circadian-rhythm-in-the-brain Complete genome dataset https://www.kaggle.com/zusmani/mygenome Question answers dataset 100 000 + questions https://www.kaggle.com/stanfordu/stanford-question-answering-dataset What is a note https://www.kaggle.com/juliancienfuegos/what-is-a-note Health nutrition and population across the world https://www.kaggle.com/theworldbank/health-nutrition-and-population-statistics

Scrape data for machine learning Scraping is another way to get data. Scraping means to request data from websites and store it offline for later analysis.

Here you can learn more about scraping.

https://www.dataquest.io/blog/web-scraping-tutorial-python/

What is an artificial neural network? Machine Learning Map

The part of machine learning that has been making the most progress in the last years are artificial neural networks, a form of machine learning that is loosely modeled on how the neurons in our brain works. Artificual neural network has been proved really effective at making progress in many different domains.

For getting started with artificial neural networks, Keras is an excellent library built on top of the Tensorflow API. https://keras.io/

Here is a good video explaining artificial neural networks https://www.youtube.com/watch?v=ILsA4nyG7I0

The math of machine learning Image of Activation function The truth is that math is a big part of machine learning but you don’t need to know any math to get started with machine learning. It’s an excellent idea to pick up the math along the way and to expand the field of machine learning you definitely need to learn math.

If you are curious, here are some resources for looking at the math behind deep learning.

The different math sources AI draws math from three different sources, linear algebra, calculus and statistics.

Linear algebra http://machinelearningmastery.com/linear-algebra-machine-learning/

Calculus https://www.umiacs.umd.edu/~hal/courses/2013S_ML/math4ml.pdf

Statistics http://machinelearningmastery.com/crash-course-statistics-machine-learning/

Neural network from scratch Building a simple neural network from scratch. Walkthrough and experiments. http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/

Lets train our first neural network! To train a neural network we need two things.

A dataset with a classifier to train on A mathematical neural network Task default dataset from my brain To make it fun, lets use a dataset that takes input data from my brain measured with laser sensors to try to determine what brain region is active, the default mode network or the task positive network.

The data takes measurement from two probes placed on my forehead and returns eight values of oxygenated and deoxygenated blood.