Tensorflow Model Optimization

Dec 20 2019

Picture here

This is all taken from sentdex's excellent Tensorflow tutorial series. Check it all out here:
https://www.youtube.com/playlist?list=PLQVvvaa0QuDfhTox0AjmQ6tvTgMBZBEXN\


In the previous post we described a model that was somewhat effective. However, if possible, we wish to arrive at a model that is optimal, thus we must look to optimize some parameters. The simplest way to start is with the simplest variables. E.g. number of dense layers, number of convolution layers and amount of nodes per layer. A lot of this will be trial and error.


The below is one approach that can be used:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D
from tensorflow.keras.callbacks import TensorBoard
import pickle
import time
import numpy as np

x = np.asarray(pickle.load(open("x.pickle", "rb")))
y = np.asarray(pickle.load(open("y.pickle", "rb")))

x = x/255.0

dense_layers = [0, 1]
layer_sizes = [32, 64, 128]
conv_layers = [2, 3]
dense_layer_size = 512

for dense_layer in dense_layers:
for layer_size in layer_sizes:
for conv_layer in conv_layers:
NAME = f"{conv_layer}-conv -{layer_size}-nodes-{dense_layer}-dense-{int(time.time())}"
tensorboard = TensorBoard(log_dir=f"logs\\{NAME}")

model = Sequential()
model.add(Conv2D(layer_size, (3,3), input_shape=x.shape[1:]))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))

for l in range(conv_layer - 1):
model.add(Conv2D(layer_size, (3,3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Flatten()) # This converts our 3D feature maps to 1D feature vectors

for l in range(dense_layer):
model.add(Dense(dense_layer_size))
model.add(Activation("relu"))

model.add(Dense(1))
model.add(Activation("sigmoid"))

model.compile(loss="binary_crossentropy",
optimizer="adam",
metrics=["accuracy"])

model.fit(x, y, batch_size=32, epochs=10, validation_split=0.15, callbacks=[tensorboard])


The first convolution layer is a requirement, hence the for l in range(conv_layer - 1):, also note that this can take a very long time to compute if you're testing a lot of different variables. As a result of this optimization, a 3 conv, 0 dense and 128 conv nodes seemed to work best. The number of epochs would be the next thing I test to see when the model starts to over-fit. A Dense() with Dropout() would also be worth testing.


In order to check what's going on I'm using Tensorboard, which provides graphical feedback. This is done via the logs: tensorboard = TensorBoard(log_dir=f"logs\\{NAME}") and then callbacks=[tensorboard] in model.fit(). You access the link via the terminal command: tensorboard --logdir logs/.