add background for TensorArray #4564

Superjomn · 2017-10-03T00:10:25Z

…_tensor_array_design

wangkuiyi · 2017-10-03T03:16:14Z

doc/design/tensor_array.md

@@ -1,9 +1,50 @@
 # Design for TensorArray
+## Background
+Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call `states[step_id]` will get the state in `step_id`th time step.


I'd suggest changing the first paragraph as:

This design doc presents the necessity of a new C++ class TensorArray. In addition to the very simple C++ implementation

class TensorArray : public std::vector<LoDTensor> { public: explicit TensorArray(const LoDTensor&); explicit TensorArray(int size);

we also need to expose it to PaddlePaddle's Python API, because users would want to use it with our very flexible operators WhileOp. An example for your reference:

I agree with @wangkuiyi . We should introduce the TensorArray and then start describing its use case in the RNN.

abhinavarora · 2017-10-03T16:37:24Z

doc/design/tensor_array.md

@@ -1,9 +1,50 @@
 # Design for TensorArray
+## Background
+Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call `states[step_id]` will get the state in `step_id`th time step.


I agree with @wangkuiyi . We should introduce the TensorArray and then start describing its use case in the RNN.

abhinavarora · 2017-10-03T16:37:58Z

doc/design/tensor_array.md

+## Background
+Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call `states[step_id]` will get the state in `step_id`th time step.
+
+An RNN could be implemented with the following pseudo codes


An RNN can be implemented with the following pseudocode

abhinavarora · 2017-10-03T16:39:25Z

doc/design/tensor_array.md

+   step++;
+}
+```
+According to the [RNN roadmap](https://github.com/PaddlePaddle/Paddle/issues/4561), there are several different RNNs to support.


several different RNNs that Paddle will eventually support.

abhinavarora · 2017-10-03T16:42:02Z

doc/design/tensor_array.md

+```
+According to the [RNN roadmap](https://github.com/PaddlePaddle/Paddle/issues/4561), there are several different RNNs to support.
+
+Currently, we have an RNN implementation called `recurrent_op` which takes tensor as input; it splits the input tensors into `input_segments`. 


Currently, the basic RNN implementation supported by Paddle is the recurrent_op which takes tensors as input and splits them into input_segments

abhinavarora · 2017-10-03T16:45:32Z

doc/design/tensor_array.md

+
+Currently, we have an RNN implementation called `recurrent_op` which takes tensor as input; it splits the input tensors into `input_segments`. 
+
+Considering a tensor can't store variable-length sequences directly, we proposed the tensor with the level of details (`LoDTensor` for short). Segmenting the `LoDTensor` is much more complicated than splitting a tensor, that makes it necessary to refactor the `recurrent_op` with `LoDTensor` segmenting support.


The first line can be changes to Since a tensor cannot store variable-length sequences directly, Paddle implements the tensor with level of details (LoDTensor for short).

abhinavarora · 2017-10-03T16:52:15Z

doc/design/tensor_array.md

+In the second stage, `dynamic_recurrent_op` should be introduced to handle inputs with variable-length sequences. 
+The implementation is the same with `recurrent_op` except that **how to split the original input `LoDTensors` and outputs to get the `input_segments` and `output_segments`** . 
+
+In the next stage, a dynamic RNN model based on dynamic operators would be supported. Though it can't be built on `recurrent_op` or `dynamic_recurrent_op` directly, the logic about how to split a tensor or a LoD tensor and get `input_segments` is the same.


The second sentence should be:
Though it can't be built over recurrent_op or dynamic_recurrent_op directly, the logic behind splitting a tensor or a LoD tensor into input_segments remains the same.

abhinavarora · 2017-10-03T16:53:16Z

doc/design/tensor_array.md

+In the next stage, a dynamic RNN model based on dynamic operators would be supported. Though it can't be built on `recurrent_op` or `dynamic_recurrent_op` directly, the logic about how to split a tensor or a LoD tensor and get `input_segments` is the same.
+
+## Why `TensorArray`
+In the three different RNNs, the logic of how to split the inputs to segments, states and outputs are similar and could be shared as a separate module.


the logic behind splitting the inputs to segments, states and outputs is similar and can be shared in a separate module.

abhinavarora · 2017-10-03T16:54:28Z

doc/design/tensor_array.md

+
+The array of `states`, `input_segments` and `output_segments` would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes. 
+
+So there should be an array-like container which might store the segments of a tensor or LoD tensor.


So there should be an array-like container, which can store the segments of a tensor or LoD tensor.

abhinavarora · 2017-10-03T16:55:02Z

doc/design/tensor_array.md

+The array of `states`, `input_segments` and `output_segments` would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes. 
+
+So there should be an array-like container which might store the segments of a tensor or LoD tensor.
+**This container could store an array of tensor and provides several methods to split a tensor or a LoD tensor** ,


This container could store an array of tensors and provides several methods to split a tensor or a LoD tensor

abhinavarora · 2017-10-03T16:55:50Z

doc/design/tensor_array.md

+
+So there should be an array-like container which might store the segments of a tensor or LoD tensor.
+**This container could store an array of tensor and provides several methods to split a tensor or a LoD tensor** ,
+that's where the notion `TensorArray` comes from.


This is where the notion of TensorArray comes from.

mkliegl

I think the idea and design are clean and elegant for LoD tensors of level=1.

Below are some thoughts on generalizing the idea - though this may be beyond the scope of this design.

I think the general problem can be stated as this: We want to convert an LoD tensor to a sequence of minibatches for efficient computation and then restore the computation results to an LoD tensor matching the original levels structure.

What we are allowed to batch together depends on which levels have sequential dependencies.

To use the example in other documents: Suppose an LoD tensor represents a document: It contains several paragraphs, each paragraph contains several sentences, each sentence contains several words.

If we treat the sentences within a paragraph as independent, but the paragraphs as having a sequential dependency, then we can batch together the sentences. So something like pack(batch_levels=[2]).

If we treat the paragraphs as independent, too, then we can batch all sentences together. Something like: pack(batch_levels=[1, 2]).

If the paragraphs and sentences both have sequential dependency, we have no choice but to run each sentence one by one. This would be pack(batch_levels=[]).

Finally, if we treat the paragraphs as independent, but the sentences as having a sequential dependency, then we can batch together the first sentences of all paragraphs, then batch together the second sentences of all paragraphs, and so on. This would be something like pack(batch_levels=[1]).

I think one can come up with real-life use cases for all these scenarios, so it would be nice to have that flexibility.

Finally, two questions I haven't thought much about yet but was curious if you could clarify:

Is there an elegant way to handle reverse sequential dependency (so we can implement bidirectional RNN's)?
Where is the best place to handle "max batch size" issues? Some options seem:
a. TensorArray::pack
b. No max batch size (i.e., creator of the LoD tensor is responsible for ensuring the pack operation does not create too large batches).
c. The computation code like rnnstep takes care of splitting into smaller batches when necessary.

Superjomn · 2017-10-03T18:14:59Z

Good suggestions, I will add some real-life use cases of pack and unpack latter.

Reply to those two questions:

the first
For a reversed RNN, the RNN operators will have a reversed attribute, and that will control the RNN to traverse the sequence from tail to head.
There is another way to do this, add a reverse() function to TensorArray.
the second
TensorArray's unpack splits a LoDTensor into batches, for example, a LoD tensor's content might have 3 sequences at some level, the sequence length is 4,3,2 respectively.

xxxx
xxx
xx

after unpack operation, it will be split into 4 batches:

0      1      2     3
x      x      x     x
x      x      x
x      x

The batches have 3, 3, 2, 1 instances respectively.

So the max batch size might be the first batch's size 3, the number of batches is 4.

The pack operation will concatenate the batches to the original LoD-formated Tensor.

@mkliegl

abhinavarora

LGTM! This is only for the grammatical mistakes. Make sure somebody who has more context on TensorArray verifies the correctness of the information.

…_tensor_array_design

wangkuiyi

LGTM

Superjomn added 2 commits October 2, 2017 17:09

add background

46d19e8

fix

cc69165

Superjomn assigned abhinavarora and jacquesqiao Oct 3, 2017

update

64a88cc

Superjomn unassigned abhinavarora and jacquesqiao Oct 3, 2017

Superjomn changed the title ~~add background~~ add background for TensorArray Oct 3, 2017

Superjomn added 2 commits October 2, 2017 18:02

update

70deb4a

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into bug/fix…

6092f49

…_tensor_array_design

Superjomn requested review from abhinavarora, wangkuiyi, jacquesqiao and mkliegl October 3, 2017 01:06

wangkuiyi reviewed Oct 3, 2017

View reviewed changes

abhinavarora suggested changes Oct 3, 2017

View reviewed changes

mkliegl reviewed Oct 3, 2017

View reviewed changes

fix grammers

4d7bed3

abhinavarora previously approved these changes Oct 4, 2017

View reviewed changes

refactor interfaces

93021b0

Superjomn dismissed abhinavarora’s stale review via 93021b0 October 5, 2017 19:09

Superjomn added 2 commits October 5, 2017 15:11

fix grammer

dfe77e0

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into bug/fix…

26d3a31

…_tensor_array_design

wangkuiyi approved these changes Oct 7, 2017

View reviewed changes

Superjomn merged commit 3419384 into PaddlePaddle:develop Oct 7, 2017

Superjomn deleted the bug/fix_tensor_array_design branch October 7, 2017 01:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add background for TensorArray #4564

add background for TensorArray #4564

Superjomn commented Oct 3, 2017 •

edited

Loading

wangkuiyi Oct 3, 2017 •

edited by Superjomn

Loading

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

abhinavarora Oct 3, 2017

mkliegl left a comment

Superjomn commented Oct 3, 2017 •

edited

Loading

abhinavarora left a comment

wangkuiyi left a comment


		Currently, we have an RNN implementation called `recurrent_op` which takes tensor as input; it splits the input tensors into `input_segments`.

		Considering a tensor can't store variable-length sequences directly, we proposed the tensor with the level of details (`LoDTensor` for short). Segmenting the `LoDTensor` is much more complicated than splitting a tensor, that makes it necessary to refactor the `recurrent_op` with `LoDTensor` segmenting support.


		The array of `states`, `input_segments` and `output_segments` would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes.

		So there should be an array-like container which might store the segments of a tensor or LoD tensor.

add background for TensorArray #4564

add background for TensorArray #4564

Conversation

Superjomn commented Oct 3, 2017 • edited Loading

wangkuiyi Oct 3, 2017 • edited by Superjomn Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkliegl left a comment

Choose a reason for hiding this comment

Superjomn commented Oct 3, 2017 • edited Loading

abhinavarora left a comment

Choose a reason for hiding this comment

wangkuiyi left a comment

Choose a reason for hiding this comment

Superjomn commented Oct 3, 2017 •

edited

Loading

wangkuiyi Oct 3, 2017 •

edited by Superjomn

Loading

Superjomn commented Oct 3, 2017 •

edited

Loading