What is the role of TimeDistributed layer in Keras?
I am trying to grasp what TimeDistributed wrapper does in Keras.
I get that TimeDistributed "applies a layer to every temporal slice of an input."
But I did some experiment and got the results that I cannot understand.
In short, in connection to LSTM layer, TimeDistributed and just Dense layer bear same results.
model = Sequential()
model.add(LSTM(5, input_shape = (10, 20), return_sequences = True))
model.add(TimeDistributed(Dense(1)))
print(model.output_shape)
model = Sequential()
model.add(LSTM(5, input_shape = (10, 20), return_sequences = True))
model.add((Dense(1)))
print(model.output_shape)
For both models, I got output shape of (None, 10, 1).
Can anyone explain the difference between TimeDistributed and Dense layer after an RNN layer?
Dense layer flattening the input and then reshaping, hence connecting different time steps and having more parameters, and TimeDistributed keeping the time steps separated (hence having less parameters). In your case Dense should have had 500 paramters, TimeDistributed only 50
In keras - while building a sequential model - usually the second dimension (one after sample dimension) - is related to a time dimension. This means that if for example, your data is 5-dim with (sample, time, width, length, channel) you could apply a convolutional layer using TimeDistributed (which is applicable to 4-dim with (sample, width, length, channel)) along a time dimension (applying the same layer to each time slice) in order to obtain 5-d output.
The case with Dense is that in keras from version 2.0 Dense is by default applied to only last dimension (e.g. if you apply Dense(10) to input with shape (n, m, o, p) you'll get output with shape (n, m, o, 10)) so in your case Dense and TimeDistributed(Dense) are equivalent.
Follow WeChat
Success story sharing
Want to stay one step ahead of the latest teleworks?
Subscribe Now相似问题
- What are metaclasses in Python?
- Finding the index of an item in a list
- What does the "yield" keyword do?
- What is the difference between Python's list methods append and extend?
- What does if __name__ == "__main__": do?
- What is __init__.py for?
- Accessing the index in 'for' loops
- What is the difference between __str__ and __repr__?
- How to use return_sequences option and TimeDistributed layer in Keras?
Inputtensor, is there any difference from this compared to just doing amapof the model applied to a list that contains each slice of theInput?