How to Detect Dog Breed with Neural Network using Flask in AWS

Complete guided project using Neural Network algorithm and AWS.

Image from Unsplash

Objective

Let me tell you why I started this.

In the place I stay, I see different kinds of dogs every day. They come in a variety of colours and sizes. I often wonder which breed they belong to. It’s not easy to know or remember their breed. So, this idea came to me –

“What if I could build an app for mobile which will take a photo of a dog as input and tell me which breed the dog belongs to.

So I started exploring. I had some background in web development, some knowledge of machine learning and a very basic idea of how apps work on mobile.

Materials needed

I figured that below are the basic things I’ll need to do –

 Building the model

Ø  Data preparation: I needed dog images with their names to train a model.

Ø  Model training: To train a model, a CNN (Convolutional Neural Network) model is right choice. I should be able to train the model with the computational resources available to me.

Ø  Model evaluation: Because I was creating this app for my own, I was okay with the validation done while training. Just the usual metrics would be fine.

Building an application

To build a simple application which can be run in mobile, I planned to use web application. I knew that I had to decide a framework and web server which would work without much effort with my model. So, a simple web server would suffice.

Ø  Web pages: I needed simple pages – one to capture or upload a photo and submit it, and another page to tell me the possible breed of the dog.

Ø  Detecting the category with the model: So, the captured image needed to be fed into the model to predict the breed.

Building the model

Data preparation

When I started searching for dog images, I realized that there was a good amount of work already done on dog breeds. Though I was not going to reinvent the wheel, I looked at it as a good learning experience. I came across two repositories of dog images.

When I started searching for dog images, I realized that there was a good amount of work already done on dog breeds. Though I was not going to reinvent the wheel, I looked at it as a good learning experience. I came across two repositories of dog images.

–          First dataset one was from a Kaggle playground Dog Breed Identification. This dataset has 120 different breeds.

–          Second dataset was related to the Udacity dog project. You can download the dataset from here. This dataset had 133 categories of images. I chose this one for it had more categories.

The Udacity project had provided bottleneck features for the dataset. Although I did build the model and web application with the bottleneck features, for this discussion, I’ll focus on creating my own model and training it. One reason is I wanted this exercise to be able to work on any such dataset in future (maybe flower dataset or any good image categorisation).

I’ll start placing some code snippets for reference.

from google.colab import drive
drive.mount('/content/drive')

As I was working on the Goggle Collab platform, the below code mounts the drive content with the runtime.

!cp -r "/content/drive/My Drive/Colab Notebooks/dog_breeds/images/" "/home/"

It’s a good idea to copy the mounted data to the runtime storage. This speeds up the training. However, if the data being copied is large, this step may take significant time.

import os
import glob 

# path to your dataset
DATASET_PATH = '/home/images' 

# get the list of paths of the dog category folders
paths = os.path.join(DATASET_PATH, '*')
paths = glob.glob(paths)

# arrange the folder names as the list of dog names.
dogs_cls = [path.split('/')[len(path.split('/'))-1] for path in sorted(paths)]

# number of classes
nb_classes = len(dogs_cls)
print(dogs_cls)

The data folder is read into paths (i.e. individual dog categories) and a list of dog names is prepared. I have used a data generator for feeding into the training. But I’ll cover that while explaining the model preparation and training.

  Model training

Before reaching the model, let me go over few approaches I have used for training my model –

Decay learning rate: I have used a decay function which will decay the learning rate as we run more epochs. This helps to avoid over fitting on training data.

import keras

# learning rate decay
class DecayLR(keras.callbacks.Callback):
    def __init__(self, base_lr=0.001, decay_epoch=10):
        super(DecayLR, self).__init__()
        self.base_lr = base_lr
        self.decay_epoch = decay_epoch 
        self.lr_history = []

    

    # set lr on_train_begin
    def on_train_begin(self, logs={}):
        # setting lr with keras backend
        K.set_value(self.model.optimizer.lr, self.base_lr) 

    # change learning rate at the end of epoch
    def on_epoch_end(self, epoch, logs={}):
        new_lr = self.base_lr * (0.8 ** (epoch // self.decay_epoch))
        self.lr_history.append(K.get_value(self.model.optimizer.lr))
        K.set_value(self.model.optimizer.lr, new_lr)

Data generator: Data generator is very useful for feeding data to the model training. Instead of loading a huge amount of data on memory, the data generator just reads the number of images required for one batch. I won’t do into detail about how to use the generator. I guess the code comments will help understand what is happening.

Ablation: Ablation is a further control to say how many images I want to use. It helps to fine-tune the model and parameters with a lesser number of epochs before running the complete set for lots of epochs.

import numpy as np
from keras.applications.resnet50 import ResNet50, preprocess_input

# data generator with augmentation
class DataGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    
    def __init__(self, mode='train', ablation=None, dogs_cls=[],

                 batch_size=32, dim=(224, 224), n_channels=3, shuffle=True):

        'Initialization'
        self.dim = dim
        self.batch_size = batch_size
        self.labels = {}
        self.list_IDs = []
        self.mode = mode        

        for i, cls in enumerate(dogs_cls):
            paths = glob.glob(os.path.join(DATASET_PATH, cls, '*'))
            brk_point = int(len(paths)*0.8)
            if self.mode == 'train':
                paths = paths[:brk_point]
            else:
                paths = paths[brk_point:]
            if ablation is not None:
                paths = paths[:ablation]
            self.list_IDs += paths
            self.labels.update({p:i for p in paths})            

        self.n_channels = n_channels
        self.n_classes = len(dogs_cls)
        self.shuffle = shuffle
        self.on_epoch_end()


    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(len(self.list_IDs) / self.batch_size)) 

    def __getitem__(self, index):
        'Generate one batch of data'
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size] 

        # Find list of IDs
        list_IDs_temp = [self.list_IDs[k] for k in indexes]

        # Generate data

        X, y = self.__data_generation(list_IDs_temp)

        return X, y

 
    def on_epoch_end(self):
        'Updates indexes after each epoch'
        self.indexes = np.arange(len(self.list_IDs))
        if self.shuffle == True:
            np.random.shuffle(self.indexes) 

    def __data_generation(self, list_IDs_temp):
        'Generates data containing batch_size samples'
        # Initialization
        X = np.empty((self.batch_size, *self.dim, self.n_channels))
        y = np.empty((self.batch_size), dtype=int)
        
        # Generate data

        for i, ID in enumerate(list_IDs_temp):
            # read the image
            img = kimage.load_img(ID, target_size=(224, 224))
            x = kimage.img_to_array(img)
            # create tensor that can be input to the model
            img_tensor = np.expand_dims(x, axis=0)
            img_tensor = preprocess_input(img_tensor)         

            X[i,] = img_tensor         

            # Store class
            y[i] = self.labels[ID]

        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

As can be seen, I have used “preprocess_input” from ResNet50 before assigning the input. Next, we’ll see the model creation.

Compile model: I had tried few options for creating the model.

a)      Own model – I had created my own CNN model with around 15 layers. I never got past 20% accuracy. The more depth I created, the more time it took to train the model. Even though I used the GPU mode in the Collab, accuracy was not improving with my own model structure.

b)      Pre-trained model – I first tried the pre-trained bottleneck features available from the Udacity project. It gave good accuracy. But as I said, I wanted a more flexible approach so I can train on other datasets later. I came across few good references which explained using pre-trained Xception, ResNet50, InveptionV3, VGG16 models trained with the Imagenet dataset. I understood that using any of these would give me decent accuracy. However, I finally chose ResNet50 because it was relatively lightweight. The model size was important because I have to run it in a small web application, small enough to run peacefully in AWS EC2 free-tier instance (more of this later).

Anyway, I was going to train my model with the ResNet50 pre-trained model.

from keras import optimizers
from keras.applications.resnet50 import ResNet50, preprocess_input
from keras import Model, Sequential, regularizers
from keras.layers import GlobalAveragePooling2D
from keras.layers import Dropout, Flatten, Dense

# model
model_whole = Sequential()

# get the ResNet50 model without the top
resnet_model = ResNet50(weights='imagenet', include_top=False, input_shape=(224,224,3)) 

# create the top layers
# need not be exactly this, keep may be two dense at minimum and the softmax at end
model_top = Sequential()
model_top.add(GlobalAveragePooling2D(input_shape = resnet_model.output_shape[1:]))
model_top.add(Dense(1024, activation='relu'))
model_top.add(Dropout(0.2))
model_top.add(Dense(512, activation='relu'))
model_top.add(Dropout(0.2))
model_top.add(Dense(256, activation='relu', kernel_regularizer=regularizers.l2(.01)))
model_top.add(Dropout(0.2))
model_top.add(Dense(nb_classes, activation='softmax')) 

# add the ResNet50 and the top layers
model_whole.add(resnet_model)
model_whole.add(model_top) 

# set the layers in ResNet50 as non trainable
# this is important as I am going to use the trained weights as is
for layer in model_whole.layers[0].layers:
  layer.trainable = False 

# set an optimizer
sgd = optimizers.SGD()
model_whole.compile(loss='categorical_crossentropy',optimizer= sgd,
              metrics=['accuracy'])

Train the model: Next is training the model by feeding the images from generator.

#training_generator = DataGenerator('train', ablation=20, dogs_cls=dogs_cls, batch_size=10,shuffle=True)
#validation_generator = DataGenerator('val', ablation=20, dogs_cls=dogs_cls, batch_size=10,shuffle=True)
training_generator = DataGenerator('train', dogs_cls=dogs_cls, batch_size=10,shuffle=True)
validation_generator = DataGenerator('val', dogs_cls=dogs_cls, batch_size=10,shuffle=True)

Create the generators for training and validation set. You can use with ablation (commented)

filepath = '/home/best_dog_model_xfer.hdf5'
checkpoint = ModelCheckpoint(filepath, monitor='val_accuracy', verbose=1, save_best_only=True, mode='max') 

# start with lr=0.1
decay = DecayLR(base_lr=0.1, decay_epoch=20)

Using “val_accuracy” score with the checkpoint. Set the decay object.

Fit the model

# fit the model
model_whole.fit_generator(generator=training_generator,
                    validation_data=validation_generator,
                    epochs=50, callbacks=[decay, checkpoint])

As the epochs run, the fit method continued to write the model file. After it is completed, I have the model I need to use with the web application.

I did some sample detection with the created model.

from keras.models import load_model
# read the model file
loaded_model = load_model('/home/best_dog_model_xfer.hdf5')

Loaded the model from storage.

def predict_breed(img_path):
    # loading RGB image as PIL.Image.Image type
    img = kimage.load_img(img_path, target_size=(224, 224))
    # converting PIL.Image.Image type to 3D tensor with shape (224, 224, 3)
    x = kimage.img_to_array(img)
    # converting 3D tensor to 4D tensor with shape (1, 224, 224, 3) and return 4D tensor
    img_tensor = np.expand_dims(x, axis=0)
    predicted_vector = loaded_model.predict(preprocess_input(img_tensor))
    # returning the dog breed that is predicted by the model
    return predicted_vector

A method to return the prediction vector for the image path.

import pandas as pd
from skimage import io

# take a dog to predict from the data set
dog_index = 10
# the photo within the folder for the selected dog
photo_index = 10
 
# get the path for image
dog_path_new = os.path.join(DATASET_PATH, dogs_cls[dog_index], '*')
dog_path_new = glob.glob(dog_path_new)
image_path = dog_path_new[photo_index]


# read the image and show it
image_new = io.imread(image_path)
plt.imshow(image_new)


# print the original and predicted dogs list
print('Original dog ->'+dogs_cls[dog_index])
predictions = predict_breed(image_path)
dogs = {'Names':dogs_cls,'Prediction':predictions[0]}
df = pd.DataFrame(dogs).sort_values(by=['Prediction'], ascending=False).head()
df['Prediction'] = df['Prediction'].astype(float).map("{:.2%}".format)
print(df.to_string(index=False))

Print the list of prediction for a particular dog.

Original dog ->Border_terrier
                       Names Prediction
              Border_terrier    100.00%
                    Shih-Tzu      0.00%
 soft-coated_wheaten_terrier      0.00%
                       cairn      0.00%
           Brabancon_griffon      0.00%

This concludes the part of preparing my model. Now, I can proceed with using this model in my application.

Building the application

My objective was to have a very basic lightweight application. The reason being I was going to host this application free of cost on an Amazon AWS EC2 instance. So, in my search for such an application framework, I zeroed down to Flask. Flask is python based. It’s super easy to build a small app on Flask.

EC2 instance setup: I won’t go into details of creating an EC2 instance and installing things. I’ll probably create a separate page for this later.

a)      Created a t2.micro instance with Ubuntu 18.04 image.

b)      Installed Python 3.7, Keras, TensorFlow and the usual like pandas etc.

c)        Created an environment. I named the environment “webapp”

Web application:

I created below folder structure –

/home--
                |- webapp --
                                         |-image
                                         |-model
                                         |-data
                                         |-templates
                                         -app.py

“model”

The “model” folder just contains the model file I exported previously.

“data”

The “data” folder contains a JSON file with the dog names.

“image”

This is the temporary storage where the submitted image will be stored. However, after the image is read, I deleted the temporary file. I’ll show the way to delete in code later.

“templates”

This folder contains my html page. The page is used as a template from the app.py file.

“templates à fileinput.html”

Headers and few variables

<!doctype html>

<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
<meta http-equiv="Cache-control" content="No-Cache">
<title>Select input image</title>
<script type="text/javascript">
var fileReader = new FileReader();
var filterType = /^(?:image\/bmp|image\/cis\-cod|image\/gif|image\/ief|image\/jpeg|image\/jpeg|image\/jpeg|image\/pipeg|image\/png|image\/svg\+xml|image\/tiff|image\/x\-cmu\-raster|image\/x\-cmx|image\/x\-icon|image\/x\-portable\-anymap|image\/x\-portable\-bitmap|image\/x\-portable\-graymap|image\/x\-portable\-pixmap|image\/x\-rgb|image\/x\-xbitmap|image\/x\-xpixmap|image\/x\-xwindowdump)$/i;

var image = new Image();
var canvas=document.createElement("canvas");

I need to explain a problem I faced while trying to submit the image with the normal form submission.Small images were fine. However, images clicked by mobile camera or high-resolution images were crashing my Flask runtime. The free tier EC2 instance did not have the resource to load and process the large image.

The solution I adopted was to pre-process the image client-side (browser) before it is sent to my server. So, the below code is populating a canvas element with the image. Once the canvas is ready, I can apply the required changes to the image. I resized the image to the point my instance can handle.

fileReader.onload = function (event) {
  image.onload=function(){
        var context=canvas.getContext("2d");
        // Resizing image to 600x400 or similar resolution
       var MAX_WIDTH = 600;
        var MAX_HEIGHT = 400;
        var width = image.width;
        var height = image.height;


        if (width > height) {
          if (width > MAX_WIDTH) {
            height *= MAX_WIDTH / width;
            width = MAX_WIDTH;
          }

        } else {
          if (height > MAX_HEIGHT) {
            width *= MAX_HEIGHT / height;
            height = MAX_HEIGHT;
          }

        }       

        canvas.width=width;

        canvas.height=height;
              // drawing the canvas with the image
        context.drawImage(image,
          0,
          0,
          image.width,
          image.height,
          0,
          0,
          canvas.width,
          canvas.height
        );

        // showing a preview image with the resized image
        document.getElementById("upload-Preview").src = canvas.toDataURL();
  }
  image.src=event.target.result;
};

Some checks when the file is selected.

var loadImageFile = function () {
  var uploadImage = document.getElementById("upload-Image");
   // check the length of uploaded file.

  if (uploadImage.files.length === 0) {
    return;
  }


  // validate if a valid file is selected
  var uploadFile = document.getElementById("upload-Image").files[0];
  if (!filterType.test(uploadFile.type)) {
    alert("Please select a valid image.");

    return;

  }

  fileReader.readAsDataURL(uploadFile);
}

Lastly one utility method and the submit method to post the image data.

// method to convert Base64 image to Unit8
function b64ToUint8Array(b64Image) {
   var img = atob(b64Image.split(',')[1]);
   var img_buffer = [];
   var i = 0;
   while (i < img.length) {
      img_buffer.push(img.charCodeAt(i));

      i++;
   }
   return new Uint8Array(img_buffer);
} 

function submitform() {
    var imageform = document.getElementById('imageform');

    // convert the image to Unit8
    var b64Image = canvas.toDataURL('image/jpeg');
    var u8Image  = b64ToUint8Array(b64Image);

    // create a form for submitting with POST
    var formData = new FormData();

    // create a blob with the image date and append to form object
    var blob = new Blob([ u8Image ], {type: 'image/jpg'});
    blob.name = 'test.jpg';
    blob.filename='test.jpg';

    // server will always receive the image as ‘test.jpg’
    formData.append('file', blob, 'test.jpg');



    // create new XML Http request
    var xhr = new XMLHttpRequest();
    xhr.onreadystatechange = function() {
        if (this.status == 200) {
    // show the results in a div if successful
            document.getElementById("result").innerHTML =
            this.responseText;

        }
    };

    xhr.open("POST", "/", true);
    // send the form data
    xhr.send(formData);
}
</script>
</head>

Now that the script part is done, below are the HTML elements (I am not very good at making nice pages. You are welcome to enhance the display).

<body>
<h1 style="color: #5e9ca0; text-align: center;">Dog Breed Identifier</h1>
<h3 style="color: #2e6c80;">How to use the tool:</h3>
<p>1. Select an image of a dog. On mobile browser, you can capture image after clicking the select button.</p>
<p>2. The selected image will show on screen.</p>
<p>3. Click on <span style="background-color: #2b2301; color: #fff; display: inline-block; padding: 3px 10px; font-weight: bold; border-radius: 5px;">Upload image</span> button to submit the image for processing.</p>
<p>&nbsp;</p>
<form id="imageform" name="uploadForm">
<h3 style="color: #2e6c80; text-align: center;">Select and upload image</h3>
<div style="text-align: center;"><input id="upload-Image" style="text-align: center;" type="file" onchange="loadImageFile();"/>
<input type="button" value="Upload image"  onclick="submitform();"/></div>
<p>
<div id="result" style="color: #2e6c80; text-align: center;"><h3 style="color: #2e6c80; text-align: center;">List of predicted dog breeds will appear here</h3></div> 
</p>
</form>
<h3 style="color: #2e6c80; text-align: center;">Image preview will appear below.</h3>
<p>
<img id="upload-Preview" />
</p>
<p>&nbsp;</p>
</body>
</html>

This looks like below.

Image by Author

Now that the initial display part is over, I’ll go into the processing part in the app.py file.

“app.py”

Usual imports –

import flask
import pandas as pd
import numpy as np
from skimage import io
from skimage import transform
import os
import sys
import glob
from PIL import Image
from keras.models import load_model
from flask import Flask, request, redirect, url_for, Response, make_response
from werkzeug import secure_filename
from keras.applications.resnet50 import ResNet50, preprocess_input
from keras import Model
from keras.preprocessing import image as kimage

# load the model
loaded_model = load_model('./ best_dog_model_xfer.hdf5')

Load the dog names from the JSON file. You can as well have in in file e.g. [‘dog1’, ‘dog2’,..].

import json
dogs =[]
with open(‘data/dog_names.json’) as json_file:
dogs = json.load(json_file)

or

dogs = ['Affenpinscher','Afghan_hound','Airedale_terrier','Akita',….]

I used the same method used for predicting.

def predict_breed(img_path):
    # loading RGB image as PIL.Image.Image type
    img = kimage.load_img(img_path, target_size=(224, 224))
    # converting PIL.Image.Image type to 3D tensor with shape (224, 224, 3)
    x = kimage.img_to_array(img)
    # converting 3D tensor to 4D tensor with shape (1, 224, 224, 3) and return 4D tensor
    img_tensor = np.expand_dims(x, axis=0)
 
    predicted_vector = loaded_model.predict(preprocess_input(img_tensor))
    # returning the dog breed that is predicted by the model
    return predicted_vector
UPLOAD_FOLDER = './image'
ALLOWED_EXTENSIONS = set(['jpg', 'jpeg', 'txt', 'JPG', 'JPEG']) 

app = flask.Flask(__name__, template_folder='templates')
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER 

def allowed_file(filename):
    return '.' in filename and \
           filename.rsplit('.', 1)[1] in ALLOWED_EXTENSIONS

The next part is divided into two routes –

a)      Handling the POST request and saving the temporary image

def allowed_file(filename):
    return '.' in filename and \
           filename.rsplit('.', 1)[1] in ALLOWED_EXTENSIONS 

@app.route('/', methods=['GET', 'POST'])
def upload_file():
    if request.method == 'POST':
        # check if the post request has the file part
        if 'file' not in request.files:
            flash('No file part')
            return redirect(request.url)
        file = request.files['file']
        # if user does not select file and submit, browser can also
        # submit an empty file without filename
        if file.filename == '':
            flash('No selected file')
            return redirect(request.url)
        if file and allowed_file(file.filename):
            filename = secure_filename(file.filename)
            file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
            return redirect(url_for('read_uploaded_file',
                                    filename=filename))
    return flask.render_template('fileinput.html')

b)      Reading the temporary image file, checking image size and returning the prediction

@app.route('/read_file', methods=['GET'])
def read_uploaded_file():
    filename = secure_filename(request.args.get('filename'))
    try:
        if filename and allowed_file(filename):
            filepath = os.path.join(app.config['UPLOAD_FOLDER'],filename)    

            predictions = bottle_predict_breed(filepath)
            dogs_n_preds = {'Names':dogs,'Prediction':predictions[0]}
            df = pd.DataFrame(dogs_n_preds).sort_values(by=['Prediction'], ascending=False).head()
            df['Prediction'] = df['Prediction'].astype(float).map("{:.2%}".format)
            output = df.to_html(index=False)
            os.remove(filepath)
            headers = {'Content-Type': 'text/html'}
            return make_response(output,200,headers)
        else:
            return "Unable to read file"
    except IOError:
        pass
    return "Unable to read file"

Notice the use of “headers” and the “make_response” call.

Since I am returning the result data frame as an HTML (with “df.to_html()”), not using the “headers” and “make_response” would display the HTML tags in the browser like “<table><tbody>…”

That’s all needed as part of the app.py file.

Run the Flask app:

Assuming that you have logged into the EC2 instance console. The next steps are simple.

Activate the “webapp” python environment.

$ workon webapp

Go into the “webapp” folder

$ cd webapp

Run the Flask app. This starts the Flask service on port 5000 by default.

$ flask run --host=0.0.0.0

“Host=0.0.0.0” makes it available to any IP. However, your AWS security group has to allow access to the instance.

For Example,

If my EC2 instance IP is 10.11.12.13, I can now access now the webpage from the browser by http://10.11.12.13:5000

Application screens

Image by Author

Choose an image. The preview is shown at the bottom.

Inage by Author

Upload the image. The prediction table will be shown.

Image by Author

This concludes the story. I hope you get equally excited as me after successfully running.

CODE AWAY 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: