﻿ Middlemind Games

## Tutorial 4: Tensorflow Logistic Regression: Training and Validation

• ### Author: Brian A. Ree

#### 0: Tensorflow Logistic Regression: LoadTensorData

##### One small change we had to make to our LoadTensorData class, which won't affect older regression tutorials, is the structure of the answers tensor object. The new code is shown below. The extra use of the list variable forces a tensor object shape of (X, 1), which is necessary for performing the sigmoid_cross_entropy_with_logits call.

```def generateData(self, lColumnMap=[]):
print ("")
print ("")
print("Generating Tensor Data:")

self.resetRows()
self.columnMap = lColumnMap
self.dataModelColCount = len(self.columnMap)
self.dataModelAnsColCount = 2  # category yes, category no

# Prep training date

# Convert base data
val2 = []
val3 = []
rowcnt = 0
val = []
for col in self.columnMap:
val.append(float(row.getMemberByName(col)))
# efl
val2.append(val)

val4 = []
val3.append(val4)

rowcnt += 1
# efl

self.rows = tf.to_float(val2)
self.rowCount = rowcnt

print("TensorRow Data Shape: %s" % self.rows.get_shape())
print('TensorRow Count: %i' % (self.rowCount))

# Convert train data
val2 = []
val3 = []
rowcnt = 0
rc = 0
if rowcnt < tp:
val = []
for col in self.columnMap:
val.append(float(row.getMemberByName(col)))
# efl
val2.append(val)

val4 = []
val3.append(val4)
rc += 1
# eif
rowcnt += 1
# efl

self.train = tf.to_float(val2)
self.trainCount = rc

print("TensorTrain Data Shape: %s" % self.train.get_shape())
print('TensorTrain Count: %i' % (self.trainCount))

# Convert validate data
val2 = []
val3 = []
rowcnt = 0
rc = 0
if rowcnt > tp and rowcnt < (tp + vp):
val = []
for col in self.columnMap:
val.append(float(row.getMemberByName(col)))
# efl
val2.append(val)

val4 = []
val3.append(val4)
rc += 1
# eif
rowcnt += 1
# efl

self.validate = tf.to_float(val2)
self.validateCount = rc

print("TensorValidate Data Shape: %s" % self.validate.get_shape())
print('TensorValidate Count: %i' % (self.validateCount))

# edef
```

##### The lines... val4 = [] val4.append(float(row.getMemberByName('Answer'))) val3.append(val4) Force the extra structure we need. It's a small change we've made to each of the tensor loading loops in this class. Next up we'll review the code change to allow our logistic regression model to run in our Main executable.

```elif model_type == 'logistic_regression':
tfModel = RegModelLogistic.RegModelLogistic(verbose, tData, lin_reg_positive_result, randomSeed, trainStepsMultiplier, logPrint, learning_rate, evalType)
tfModel.startTraining()
```

#### 1: Tensorflow Linear Regression: Model Details

##### You can see that our new LoadTensorData class takes a trainPrct and a validatePrct argument as well as our LoadFeatureData class instance. Now let's take a look at the RegModelLogistic class which stands for regression model logistic. It takes our LoadTensorData instance as an argument as well as a few other arguments. Let's list them here.

• verbose: Boolean flag that indicates if we should use verbose logging for extra debugging information.
• tData: This is an instance of our LoadTensorData class and will be used to access the tensor data we've prepared.
• log_reg_positive_result: This is determines the threshold to trigger a yes inference.
• randomSeed: This is a boolean flag that indicates if we should try to prep our weights with a ramdom seed value.
• trainStepsMultiplier: This is an integer value that multiplies the training steps by itself to set the total number of training steps. Training steps are calculated from the size of the training set.
• logPrint: This is a value controlling how often the error is printed to the console as the model is trained.
• learning_rate: This is an important argument, it controls how quickly the model learns. However this should be a very small incremental value like 0.000001 or so depending on your data.
• evalType: This is a string representing a specific evaluation to run for the given model. This allows us to specify special custom checks in our model's code.

##### So let's take a look at our RegModelLogistic implementation. One quick note is that we're going to overlook the checkpoint code for now. This feature allows the code to save it's trained model at a checkpoint so that we can save any precious time spent training our model. I've played with the code and it does seem to do it's job but I will leave that as an exploration exercise for you.

```import tensorflow as tf
import os

class RegModelLogistic:
""" A general implementation of a TensorFlow logistic regression model. """

verbose = False

w = None
b = None

dataModelColCount = 0
dataModelAnsColCount = 0
totalTrainingSteps = 1000
trainStepsMultiplier = 1
checkpoint = False
randomSeed = False
learning_rate = 0.0000001
positiveResult = 0.55

logPrint = 100
evalType = ''

def __init__(self, lVerbose=False, lLoadTensorData=None, lPositiveResult=0.50, lRandomSeed=False, lTrainStepsMultiplier=1.0, lLogPrint=100, lLearningRate=0.0000001, lEvalType=''):
self.verbose = lVerbose
self.learning_rate = lLearningRate
self.positiveResult = lPositiveResult
self.randomSeed = lRandomSeed
self.trainStepsMultiplier = lTrainStepsMultiplier
self.logPrint = lLogPrint
self.evalType = lEvalType
# edef

def combine_inputs(self, x):
return tf.matmul(x, self.w) + self.b
# edef

def inference(self, x):
# Compute inference model over data x and return the result.
return tf.nn.softmax(self.combine_inputs(x))
# edef

def loss(self, x, y):
# Compute loss over training data x and expected outputs y.
return tf.reduce_mean(-tf.reduce_sum(y * tf.log(self.inference(x)), reduction_indices=1))
# edef

def inputs(self):
# Read/generate input training data x and expected outputs y.
# edef

def train(self, totalLoss):
# edef

def evaluate(self, sess, test_x, test_y):
Y_predicted = tf.equal(tf.argmax(self.inference(test_x), 1), tf.argmax(test_y, 1))
mse = tf.reduce_mean(tf.cast(Y_predicted, tf.float32))
print('Accuracy: %.4f' % sess.run(mse))
# edef

def startTraining(self):

print('Found training steps: %i' % self.totalTrainingSteps)

with tf.Session() as sess:
print('Found tensor dimension: %ix%i' % (self.dataModelColCount, self.dataModelAnsColCount))
if self.randomSeed == True:
self.w = tf.Variable(tf.random_normal([self.dataModelColCount, self.dataModelAnsColCount], stddev=0.5), name='weights')
self.b = tf.Variable(tf.random_normal([self.dataModelAnsColCount], stddev=0.5), name='bias')
else:
self.w = tf.Variable(tf.zeros([self.dataModelColCount, self.dataModelAnsColCount]), name='weights')
self.b = tf.Variable(tf.zeros([self.dataModelAnsColCount]), name='bias')
# eif

print("Weight: %s" % self.w.get_shape())
print("Bias: %s" % self.b.get_shape())

# Model setup
tf.global_variables_initializer().run()

# Create a saver
if self.checkpoint == True:
saver = tf.train.Saver()
# eif

x, y = self.inputs()
total_loss = self.loss(x, y)
train_op = self.train(total_loss)

coord = tf.train.Coordinator()
training_steps = self.totalTrainingSteps

initial_step = 0

if self.checkpoint == True:
# Verify we don't have a checkpoint saved already
ckpt = tf.train.get_checkpoint_state(os.path.dirname(__file__) + "/checkpoints/")

if ckpt and ckpt.model_checkpoint_path:
# Restores from checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
initial_step = int(ckpt.model_checkpoint_path.rsplit('-', 1))
# eif
# eif

# Training loop
if self.checkpoint == True:
for step in range(initial_step, training_steps):
sess.run([train_op])

if step % self.logPrint == 0:
print ("Loss: ", sess.run([total_loss]))
# eif

if step % 1000 == 0:
saver.save(sess, './checkpoints/eod-model', global_step=step)
# eif

# efl
else:
for step in range(training_steps):
sess.run([train_op])

if step % self.logPrint == 0:
print ("Loss: ", sess.run([total_loss]))
# eif

# efl
# eif

# ewith
# edef

# eclass
```

##### Not too bad for a logistic regression model. As usual let's go over the class variables first. We'll ignore the variables that were covered in our review of the constructor arguments.

• w: The tensor containing our weights.
• b: The tensor containing our bias.
• dataModelColCount: The number of columns in our data model. Based on the data in DataRow2Tensor entry used by this class.
• totalTrainingSteps: The total number of training steps is set to the number of rows in the training set times the trainStepsMultiplier

##### Nothing too crazy right. All the arguments make sense we're defining some data driven values that control how our class operates its logistic regression model. Next up let's review the methods in this class and then start diving into some code.

• combine_inputs: Combines the input tensor in the same way that an inference was performed in out linear regression model. The output of this method will be the input of a sigmoid method to determine our yes/no inference.
• inference: Defines the operations necessary to create our inference formula, this is the formula that defines our guess.
• loss: Defines the operations necessary to calulate the loss associated with the current weights and bias.
• inputs: This is a simple method that returns our input data, this will be the training and training answer data stored in our local instance of LoadTensorData class.
• train: Defines the operations needed to train the model.
• evaluate: Defines the operations needed to evaluate the accuracy of the model.

##### Let's take a look at the inference method. This method contains the set of operations to create our inference formula. That is given an input X and our current value of weights and biases we generate an output, Y_predicted.

```def combine_inputs(self, x):
return tf.matmul(x, self.w) + self.b
# edef

def inference(self, x):
# Compute inference model over data x and return the result.
return tf.nn.softmax(self.combine_inputs(x))
# edef
```

##### Let's take a look at how inference is used, it'll make more sense that way. Next up the loss function where we compare our inferred answer to our actual answer.

```def loss(self, x, y):
# Compute loss over training data x and expected outputs y.
return tf.reduce_mean(-tf.reduce_sum(y * tf.log(self.inference(x)), reduction_indices=1))
# edef
```

##### The loss method for a logistic regression model works a little differently than our linear regression model. Our combine_inputs method essentially provides the value we are trying to optimize, the inference method in this network is one that provides the sigmoid result for the combined inputs. You'll also notice that we're using a different loss method. In this model we use sigmoid_cross_entropy_with_logits to compute the loss. Next up inputs and train methods will be reviewed. Let's take a look at them.

```def inputs(self):
# Read/generate input training data x and expected outputs y.
# edef

def train(self, totalLoss):
# edef
```

##### We're going to take a look at our evaluate method next. This method is used after training to compare the results of our neural network to the known answers we loaded in our LoadTensorData class.

```def evaluate(self, sess, test_x, test_y):
Y_predicted = tf.equal(tf.argmax(self.inference(test_x), 1), tf.argmax(test_y, 1))
mse = tf.reduce_mean(tf.cast(Y_predicted, tf.float32))
print('Accuracy: %.4f' % sess.run(mse))
# edef
```

##### Alright now, we're almost done. Soon we'll be doing some test runs to check our model's performance. But first we need to review the startTraining method. This is the most complex part of this class but you've seen all the supporting methods so you have an idea what we're doing. Let's look at the code.

```def startTraining(self):

print('Found training steps: %i' % self.totalTrainingSteps)

with tf.Session() as sess:
print('Found tensor dimension: %ix%i' % (self.dataModelColCount, self.dataModelAnsColCount))
if self.randomSeed == True:
self.w = tf.Variable(tf.random_normal([self.dataModelColCount, self.dataModelAnsColCount], stddev=0.5), name='weights')
self.b = tf.Variable(tf.random_normal([self.dataModelAnsColCount], stddev=0.5), name='bias')
else:
self.w = tf.Variable(tf.zeros([self.dataModelColCount, self.dataModelAnsColCount]), name='weights')
self.b = tf.Variable(tf.zeros([self.dataModelAnsColCount]), name='bias')
# eif

print("Weight: %s" % self.w.get_shape())
print("Bias: %s" % self.b.get_shape())

# Model setup
tf.global_variables_initializer().run()

# Create a saver
if self.checkpoint == True:
saver = tf.train.Saver()
# eif

x, y = self.inputs()
total_loss = self.loss(x, y)
train_op = self.train(total_loss)

coord = tf.train.Coordinator()
training_steps = self.totalTrainingSteps

initial_step = 0

if self.checkpoint == True:
# Verify we don't have a checkpoint saved already
ckpt = tf.train.get_checkpoint_state(os.path.dirname(__file__) + "/checkpoints/")

if ckpt and ckpt.model_checkpoint_path:
# Restores from checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
initial_step = int(ckpt.model_checkpoint_path.rsplit('-', 1))
# eif
# eif

# Training loop
if self.checkpoint == True:
for step in range(initial_step, training_steps):
sess.run([train_op])

if step % self.logPrint == 0:
print ("Loss: ", sess.run([total_loss]))
# eif

if step % 1000 == 0:
saver.save(sess, './checkpoints/eod-model', global_step=step)
# eif

# efl
else:
for step in range(training_steps):
sess.run([train_op])

if step % self.logPrint == 0:
print ("Loss: ", sess.run([total_loss]))
# eif

# efl
# eif

# ewith
# edef
```

##### The method start by calculating the desired training step count. We use the size of the data set and then multiply that by a factor to create a larger or smaller total number of training steps to run. We set the tensor flow session to use during our training with this line of code, with tf.Session() as sess. The ensures that all tensor flow operations are run on the same session and that the session is released once we exit the with block. Let's take a look at the first few lines of our training method.

```print('Found tensor dimension: %ix%i' % (self.dataModelColCount, self.dataModelAnsColCount))
if self.randomSeed == True:
self.w = tf.Variable(tf.random_normal([self.dataModelColCount, self.dataModelAnsColCount], stddev=0.5), name='weights')
self.b = tf.Variable(tf.random_normal([self.dataModelAnsColCount], stddev=0.5), name='bias')
else:
self.w = tf.Variable(tf.zeros([self.dataModelColCount, self.dataModelAnsColCount]), name='weights')
self.b = tf.Variable(tf.zeros([self.dataModelAnsColCount]), name='bias')
# eif

print("Weight: %s" % self.w.get_shape())
print("Bias: %s" % self.b.get_shape())

# Model setup
tf.global_variables_initializer().run()

# Create a saver
if self.checkpoint == True:
saver = tf.train.Saver()
# eif
```

##### As a quick sanity check we print out the dimension of our training tensor by printing out the number of columns loaded from the column data map. The next piece of code determines if we intialize our weight and bias tensors with initial values or just zeros. Notice that our weight tensor shape is determined by the columns we're including in our data, training, and validation tensors. The bias tensor is a single dimension and has a value for each column in our data. After we setup the shape of our weight and bias tensors we run the tensor flow variable initialization call. If checkpoints are enabled we initialize a new instance of the saver class.

```x, y = self.inputs()
total_loss = self.loss(x, y)
train_op = self.train(total_loss)

coord = tf.train.Coordinator()
training_steps = self.totalTrainingSteps

initial_step = 0

if self.checkpoint == True:
# Verify we don't have a checkpoint saved already
ckpt = tf.train.get_checkpoint_state(os.path.dirname(__file__) + "/checkpoints/")

if ckpt and ckpt.model_checkpoint_path:
# Restores from checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
initial_step = int(ckpt.model_checkpoint_path.rsplit('-', 1))
# eif
# eif
```

##### Next up if the checkpoint boolean is set we setup our checkpoint tracker by setting the directory where the checkpoint files will be stored. If there are checkpoint files in the target checkpoint directory then we load the latest training checkpoint and we also set the current training step. The next block of code will run our actual training loop and then call our evaluate method.

```# Training loop
if self.checkpoint == True:
for step in range(initial_step, training_steps):
sess.run([train_op])

if step % self.logPrint == 0:
print ("Loss: ", sess.run([total_loss]))
# eif

if step % 1000 == 0:
saver.save(sess, './checkpoints/eod-model', global_step=step)
# eif

# efl
else:
for step in range(training_steps):
sess.run([train_op])

if step % self.logPrint == 0:
print ("Loss: ", sess.run([total_loss]))
# eif

# efl
# eif

```

##### So now that we have covered all of the code running our linear regression tensorflow model let's actually run it! Uncomment the following line run(exes["goog_lin_reg_avg100day"]);, and comment this one run(exes["weight_age_lin_reg"]);. Now run Main.py and you should see output similar to the output depicted below. This execution loads all the stock price data we have for the SPY exchange traded fund, a fund that tracks the S&P 500. We're trying to predict the closing price buy looking at the patterns created by the closing price, the opening price, and the simple 100 day moving average.

```Application Version: 0.4.0.6
Loaded 3130 rows from this data file.
CleanCount: 0 RowCount: 3129 RowsFound: 3129
Found feature type: goog_log_reg
Generating Feature Data: Type: goog_log_reg
Loaded 3129 rows from this data file.
Cleaning row data...
CleanCount: 18 RowCount: 3111 RowsFound: 3111
Generating Tensor Data:
TensorRow Data Shape: (3111, 2)
TensorRow Count: 3111
TensorTrain Data Shape: (2488, 2)
TensorTrain Count: 2488
2017-08-10 18:37:13.222037: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
TensorValidate Data Shape: (621, 2)
TensorValidate Count: 621
2017-08-10 18:37:13.222052: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
Found training steps: 2488
Found tensor dimension: 2x2
2017-08-10 18:37:13.222058: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Weight: (2, 2)
Bias: (2,)
('Loss: ', [2.0792916])
('Loss: ', [2.0784445])
('Loss: ', [2.0784395])
('Loss: ', [2.0784333])
('Loss: ', [2.0784285])
('Loss: ', [2.0784237])
('Loss: ', [2.0784178])
('Loss: ', [2.0784118])
('Loss: ', [2.0784073])
('Loss: ', [2.0784023])
('Loss: ', [2.0783966])
('Loss: ', [2.0783913])
('Loss: ', [2.0783861])
('Loss: ', [2.0783808])
('Loss: ', [2.0783753])
('Loss: ', [2.0783701])
('Loss: ', [2.0783644])
('Loss: ', [2.0783589])
('Loss: ', [2.0783539])
('Loss: ', [2.0783484])
('Loss: ', [2.0783429])
('Loss: ', [2.0783372])
('Loss: ', [2.0783319])
('Loss: ', [2.0783272])
('Loss: ', [2.078321])
('Loss: ', [2.0783157])
('Loss: ', [2.0783095])
('Loss: ', [2.078305])
('Loss: ', [2.0783])
('Loss: ', [2.078295])
('Loss: ', [2.0782897])
('Loss: ', [2.078284])
('Loss: ', [2.0782781])
('Loss: ', [2.0782731])
('Loss: ', [2.0782676])
('Loss: ', [2.0782623])
('Loss: ', [2.0782566])
('Loss: ', [2.0782521])
('Loss: ', [2.0782464])
('Loss: ', [2.0782411])
('Loss: ', [2.0782354])
('Loss: ', [2.0782299])
('Loss: ', [2.0782251])
('Loss: ', [2.0782197])
('Loss: ', [2.0782151])
('Loss: ', [2.0782101])
('Loss: ', [2.0782032])
('Loss: ', [2.0781982])
('Loss: ', [2.0781929])
('Loss: ', [2.0781877])
Accuracy: 0.5459
```