Theano

A language in a language

Dealing with weights matrices and gradients can be tricky and sometimes not trivial. Theano is a great framework for handling vectors, matrices and high dimensional tensor algebra. Most of this tutorial will refer to Theano however TensorFlow is another great framework capable of providing an incredible abstraction for complex algebra. More on TensorFlow in the next chapters.

import theano
import theano.tensor as T

Symbolic variables

Theano has it's own variables and functions, defined the following

x = T.scalar()
x
<TensorType(float64, scalar)>

Variables can be used in expressions

y = 3*(x**2) + 1

y is an expression now

Result is symbolic as well

type(y)
y.shape
Shape.0
printing

As we are about to see, normal printing isn't the best when it comes to theano

print(y)
Elemwise{add,no_inplace}.0
theano.pprint(y)
'((TensorConstant{3} * (<TensorType(float64, scalar)> ** TensorConstant{2})) + TensorConstant{1})'
theano.printing.debugprint(y)
Elemwise{add,no_inplace} [id A] ''   
 |Elemwise{mul,no_inplace} [id B] ''   
 | |TensorConstant{3} [id C]
 | |Elemwise{pow,no_inplace} [id D] ''   
 |   |<TensorType(float64, scalar)> [id E]
 |   |TensorConstant{2} [id F]
 |TensorConstant{1} [id G]

Evaluating expressions

Supply a dict mapping variables to values

y.eval({x: 2})
array(13.0)

Or compile a function

f = theano.function([x], y)
f(2)
array(13.0)

Other tensor types

X = T.vector()
X = T.matrix()
X = T.tensor3()
X = T.tensor4()

Automatic differention

  • Gradients are free!
x = T.scalar()
y = T.log(x)
gradient = T.grad(y, x)
print(gradient)
print(gradient.eval({x: 2}))
print((2 * gradient))
Elemwise{true_div}.0
0.5
Elemwise{mul,no_inplace}.0

Shared Variables

  • Symbolic + Storage
import numpy as np
x = theano.shared(np.zeros((2, 3), dtype=theano.config.floatX))
x
<TensorType(float64, matrix)>

We can get and set the variable's value

values = x.get_value()
print(values.shape)
print(values)
(2, 3)
[[ 0.  0.  0.]
 [ 0.  0.  0.]]
x.set_value(values)

Shared variables can be used in expressions as well

(x + 2) ** 2
Elemwise{pow,no_inplace}.0

Their value is used as input when evaluating

((x + 2) ** 2).eval()
array([[ 4.,  4.,  4.],
       [ 4.,  4.,  4.]])
theano.function([], (x + 2) ** 2)()
array([[ 4.,  4.,  4.],
       [ 4.,  4.,  4.]])

Updates

  • Store results of function evalution
  • dict mapping shared variables to new values
count = theano.shared(0)
new_count = count + 1
updates = {count: new_count}

f = theano.function([], count, updates=updates)
f()
array(0)
f()
array(1)
f()
array(2)

results matching ""

    No results matching ""