Recognizing Hand written Digits with deep learning
I trained the model with models written from scratch as well as pre trained models
Importing FAST AI
Downloading MNIST Dataset
path = untar_data(URLs.MNIST)
Path.BASE_PATH = path
Checking the list of files and directories in the path
(path/'training').ls()
Getting the files from the directory
zeroes = (path/'training'/'0').ls().sorted()
ones = (path/'training'/'1').ls().sorted()
twos = (path/'training'/'2').ls().sorted()
threes = (path/'training'/'3').ls().sorted()
fours = (path/'training'/'4').ls().sorted()
fives = (path/'training'/'5').ls().sorted()
sixes = (path/'training'/'6').ls().sorted()
sevens = (path/'training'/'7').ls().sorted()
eights = (path/'training'/'8').ls().sorted()
nines = (path/'training'/'9').ls().sorted()
Displaying an image
im3_path = ones[1]
im3 = Image.open(im3_path)
im3
Converting each image to a tensor
zero_tensors = [tensor(Image.open(o)) for o in zeroes]
one_tensors = [tensor(Image.open(o)) for o in ones]
two_tensors = [tensor(Image.open(o)) for o in twos]
three_tensors = [tensor(Image.open(o)) for o in threes]
four_tensors = [tensor(Image.open(o)) for o in fours]
five_tensors = [tensor(Image.open(o)) for o in fives]
six_tensors = [tensor(Image.open(o)) for o in sixes]
seven_tensors = [tensor(Image.open(o)) for o in sevens]
eight_tensors = [tensor(Image.open(o)) for o in eights]
nine_tensors = [tensor(Image.open(o)) for o in nines]
len(three_tensors),len(seven_tensors)
Stacking each image on top of image to make it a stack of images
stacked_ones = torch.stack(one_tensors).float()/255
stacked_twos = torch.stack(two_tensors).float()/255
stacked_threes = torch.stack(three_tensors).float()/255
stacked_fours = torch.stack(four_tensors).float()/255
stacked_fives = torch.stack(five_tensors).float()/255
stacked_sixes = torch.stack(six_tensors).float()/255
stacked_sevens = torch.stack(seven_tensors).float()/255
stacked_eights = torch.stack(eight_tensors).float()/255
stacked_nines = torch.stack(nine_tensors).float()/255
stacked_zeroes = torch.stack(zero_tensors).float()/255
Same for validation set
valid_0_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'0').ls()])
valid_0_tens = valid_0_tens.float()/255
valid_1_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'1').ls()])
valid_1_tens = valid_1_tens.float()/255
valid_2_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'2').ls()])
valid_2_tens = valid_2_tens.float()/255
valid_3_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'3').ls()])
valid_3_tens = valid_3_tens.float()/255
valid_4_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'4').ls()])
valid_4_tens = valid_4_tens.float()/255
valid_5_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'5').ls()])
valid_5_tens = valid_5_tens.float()/255
valid_6_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'6').ls()])
valid_6_tens = valid_6_tens.float()/255
valid_7_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'7').ls()])
valid_7_tens = valid_7_tens.float()/255
valid_8_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'8').ls()])
valid_8_tens = valid_8_tens.float()/255
valid_9_tens = torch.stack([tensor(Image.open(o))
for o in (path/'testing'/'9').ls()])
valid_9_tens = valid_9_tens.float()/255
valid_9_tens.shape,valid_0_tens.shape
stacked_fives.shape
We already have our independent variables x —these are the images themselves. We'll concatenate them allinto a single tensor, and also change them from a list of matrices (a rank-3 tensor) to a list of vectors (a rank-2tensor). We can do this using view , which is a PyTorch method that changes the shape of a tensor withoutchanging its contents. -1 is a special parameter to view that means "make this axis as big as necessary tofit all the data":
train_x = torch.cat([stacked_zeroes,stacked_ones,stacked_twos,stacked_threes,stacked_fours,stacked_fives,
stacked_sixes,stacked_sevens,stacked_eights,stacked_nines]).view(-1, 28*28)
Labels for the image
train_y = tensor([0]*len(zeroes) + [1]*len(ones)+[2]*len(twos)+[3]*len(threes)+[4]*len(fours)+[5]*len(fives)+
[6]*len(sixes)+[7]*len(sevens)+[8]*len(eights)+[9]*len(nines)).unsqueeze(1)
train_x.shape,train_y.shape
A Dataset in PyTorch is required to return a tuple of (x,y) when indexed. Python provides a zip functionwhich, when combined with list , provides a simple way to get this functionality:
dset = list(zip(train_x,train_y))
x,y = dset[0]
x.shape,y
valid_x = torch.cat([valid_0_tens,valid_1_tens,valid_2_tens,valid_3_tens,valid_4_tens,valid_5_tens,
valid_6_tens,valid_7_tens,valid_8_tens,valid_9_tens]).view(-1, 28*28)
valid_y = tensor([0]*len(valid_0_tens) + [1]*len(valid_1_tens)+[2]*len(valid_2_tens)+[3]*len(valid_3_tens)+
[4]*len(valid_4_tens)+[5]*len(valid_5_tens)+
[6]*len(valid_6_tens)+[7]*len(valid_7_tens)+[8]*len(valid_8_tens)+[9]*len(valid_9_tens)).unsqueeze(1)
valid_dset = list(zip(valid_x,valid_y))
Getting initial weights
def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()
Using sigmoid function for loss so that the values remain between 0 and 1
def sigmoid(x): return 1/(1+torch.exp(-x))
Calculating Loss
def mnist_loss(predictions, targets):
predictions = predictions.sigmoid()
return torch.where(targets==1, 1-predictions, predictions).mean()
Calculating accuracy
def batch_accuracy(xb, yb):
preds = xb.sigmoid()
correct = (preds>0.5) == yb
return correct.float().mean()
Using a batch to train data instead of using all images or one image
dl = DataLoader(dset, batch_size=256)
xb,yb = first(dl)
xb.shape,yb.shape
valid_dl = DataLoader(valid_dset, batch_size=256)
dls = DataLoaders(dl, valid_dl)
nn.Linear does the same thing as our init_params and linear together. It contains both the weights and biases in a single class. Here's how we replicate our model from the previous section:
learn = Learner(dls, nn.Linear(28*28,1), opt_func=SGD,
loss_func=mnist_loss, metrics=batch_accuracy)
Fitting our model with a learning rate of 0.1
learn.fit(10, lr=0.1)
Making a simple neuronet by adding non linearity and adding layers
def simple_net(xb):
res = xb@w1 + b1
res = res.max(tensor(0.0))
res = res@w2 + b2
return res
w1 = init_params((28*28,30))
b1 = init_params(30)
w2 = init_params((30,1))
b2 = init_params(1)
simple_net = nn.Sequential(
nn.Linear(28*28,30),
nn.ReLU(),
nn.Linear(30,1),
)
learn = Learner(dls, simple_net, opt_func=SGD,
loss_func=mnist_loss, metrics=batch_accuracy)
learn.fit(40,0.1)
plt.plot(L(learn.recorder.values).itemgot(2));
learn.recorder.values[-1][2]
As you can see after appying neural net the accuracy is around 20% which is not satisfactory.
We already know that a single nonlinearity with two linear layers is enough to approximate any function. Sowhy would we use deeper models? The reason is performance. With a deeper model (that is, one with morelayers) we do not need to use as many parameters; it turns out that we can use smaller matrices with morelayers, and get better results than we would get with larger matrices, and few layers. That means that we can train the model more quickly, and it will take up less memory. In the 1990s researcherswere so focused on the universal approximation theorem that very few were experimenting with more than onenonlinearity. This theoretical but not practical foundation held back the field for years. Some researchers,however, did experiment with deep models, and eventually were able to show that these models could performmuch better in practice. Eventually, theoretical results were developed which showed why this happens. Today,it is extremely unusual to find anybody using a neural network with just one nonlinearity.
db = ImageDataLoaders.from_folder(path=path, train='training', test='testing', valid_pct=0.2);
db
learn = cnn_learner(db, resnet18, pretrained=False,
loss_func=F.cross_entropy, metrics=accuracy)
learn.fit_one_cycle(1, 0.01)