Day #161 - Tensorflow RNN APIs

Key Summary

• Encoder (Original Text)
• Decoder (Translated Text)

Learning's

• Read Sequence Data
• SequenceExample proto to store sequence data + context
• Efficient Storage, Efficient parser
• tf.parse_single_sequence_example

Batch Sequence Data

• Static Padding - Pick Max Sequence length, Convert input to that length
• tf.train.batch(...)
• Dynamic Padding tf.train.batch(...,dynamic_pad=true)
• Bucketing - tf.contrib.training.bucket_by_sequence_length(...,dynamic_pad=True)

State Saver - Truncated Back Propagation to Time

• tf.contrib.training.batch_sequences_with_states(...)

RNN API

• RNN is unit of computation repeated over and over
• Inputs / Previous State / Intermediate calculation / Updated State
• LSTM (Long Short Term Memory)
• RNNCell (Provide Knowledge about specific RNN Architecture)
• Represent a time step as a layer
• Tensor Cells

  #BasicRNNCell, BasicLSTMCell, MultiRNNCell, GRUCell, LSTMCell, GridLSTMCell

Fully Dynamic Computation

• tf.while_loop (dynamic loops + gradients)
• tf.TensorArray (dynamic Tensor slice access, gradients)
• Encoder (8 Stacked LSTM Layers)

Fused RNN Cells

• XLA fused time steps
• Manually fused time steps
• Works everywhere fast on XOR (GPU, Android)

Dynamic Decoding

• tf.contrib.seq2seq

Dynamic Decoder Goal

• Pick Sampling method
• Pick RNN Cells
• Pick attention, in-grah beam search

	#Example1
	#simple RNN
	def rnn(cell, input_list, initial_state):
	state = initial_state
	outputs = []
	for i, input in enumerate(input_list):
	output, state = cell(input,state)
	outputs.append(output)
	return(outputs,state)
	#LSTM State (Two Sensors)
	#Cell function and initial state are coupled
	#Example2
	#Basic LSTM Cell
	class BasicLSTMCell(RNNCell):
	@property
	def state_size(self):
	return(self._num_units,self._num_units)
	@property
	#Each tensor has num units column
	def output_size(self):
	return self._num_units
	#Accepts input tensor, decompose state to two tensors
	def __call__(self,input,state):
	(c,h) = state
	...
	return new_h, (new_c, new_h)
	#Tensor Cells
	#BasicRNNCell, BasicLSTMCell, MultiRNNCell, GRUCell, LSTMCell, GridLSTMCell
	#Example 3 Dynamic Computation
	tf.while_loop #Dynamic Computation
	while_loop(cond, body, loop_vars, parallel_iterations=10,swap_memory=False)
	#Example loop
	ijk_0 = (0,(1,2))
	#condition function
	condition = lambda i, -: i < 10
	#Actual computation
	body = lambda i, jk: (i+1,(jk[0]+jk[1], jk[0]-jk[1]))
	(i_final,jk_final) = tf.while_loop(condition, body, ijk_0)
	#tf.TensorArray - Dynamic Tensor Access, Gradients
	matrix_rows = tf.shape(matrix)[0]
	#Example Unpack + Read
	ta = tf.TensorArray(tf.float32,size=matrix_rows)
	ta = ta.unstack(matrix)
	read_row = ta.read(row_ix_t)
	#Example Writes + Pack
	ta = tf.TensorArray(tf.float32,size=matrix_rows)
	ta = ta.write(row_ix_t,vector)
	...
	new_matrix = ta.stack()
	#Bulding Blocks of Tensorflow
	tf.map_fn
	tf.scan
	tf.flodl
	tf.foldr
	tf.nn.dynamic_rnn
	tf.nn.birectional_dynamic_rnn
	tf.contrib.seq2seq.dynamic_decode
	#RNN Fully Dynamic RNN
	#8 layer LSTM with residual connections
	#Each layer is on seperate GRU
	cell = MultiRNNCell([DeviceWrapper(ResidualWrapper(LSTMCell(num_units=512)),device='/gpu%d'%i)
	for i in range(8)]))
	encoder_outputs, encoder_final_state = dynamic_rnn(cell,inputs, sequence_length, parallel_iterations=32,swap_memory=true)
	#XLA fused time steps
	#Before
	MultiRNNCell([LSTMCell(num_units=512) for i in range(8)]
	#Fuse Each Layer Individually
	MultiRNNCell([CompiledWrapper(LSTMCell(num_units=512)) for i in range(8)])
	#Pick a sampling method
	#For training read inputs from dense ground truth vectors
	helper = TrainingHelper(decoder_inputs,sequence_length)
	#Scheduled Sampling
	helper = ScheduledEmbeddingTrainingHelper(decoder_inputs,sequence_length,embedding,sampling_probability)
	#For inference
	helper = GreedyEmbeddingHelper(embeddings,start_tokens,end_token)
	#Pick a Decoder
	decoder = BasicDecoder(cell=cell,helper=helper,initial_state=encoder_final_state)
	#With Attention
	decoder = AttentionDecoder(cell=cell,helper=helper,attention=...,initial_state=encoder_final_state)
	#Decode
	decoder_outputs, final_decoder_state = dynamic_decode(decoder)
	decoder_logits = decoder_outputs.rnn_output
	decoder_sample_ids = decoder_output,sample_id

view raw RNNNotes.py hosted with ❤ by GitHub

Happy Mastering DL!!!