alternative
  • Home (current)
  • About
  • Tutorial
    Technologies
    C#
    Deep Learning
    Statistics for AIML
    Natural Language Processing
    Machine Learning
    SQL -Structured Query Language
    Python
    Ethical Hacking
    Placement Preparation
    Quantitative Aptitude
    View All Tutorial
  • Quiz
    C#
    SQL -Structured Query Language
    Quantitative Aptitude
    Java
    View All Quiz Course
  • Q & A
    C#
    Quantitative Aptitude
    Java
    View All Q & A course
  • Programs
  • Articles
    Identity And Access Management
    Artificial Intelligence & Machine Learning Project
    How to publish your local website on github pages with a custom domain name?
    How to download and install Xampp on Window Operating System ?
    How To Download And Install MySql Workbench
    How to install Pycharm ?
    How to install Python ?
    How to download and install Visual Studio IDE taking an example of C# (C Sharp)
    View All Post
  • Tools
    Program Compiler
    Sql Compiler
    Replace Multiple Text
    Meta Data From Multiple Url
  • Contact
  • User
    Login
    Register

Deep Learning - RNN - Recurrent Neural Network - Problems with RNN Tutorial

RNN works well with Sequential Data like textual data and time series data.

It is not mostly used because of 2 major problems i.e.

  • Problem of Long-Term Dependency
  • Unstable/Stagnated Training

the main cause of these two problems is an Unstable Gradient.

Problem 1 of Long-Term Dependency

In RNN we have sequential data where current data depend on previous data. but if the length of sequential data increases then current data will not be able to remember the very old data and hence long-term dependency gets reduced.

Example 1- the iPhone is the top-selling mobile.

here, the word 'mobile' is dependent on the 'iPhone' and is also known as short-term dependency due to short sentences.

 

Example 2- the iPhone is a superb mobile. I purchased it last year. But, I found it very difficult to understand its features.

here, the word 'feature' is dependent on the word 'iPhone', due to such long sentences case RNN will fail and this problem is known as the long-term dependency problem.

This problem arises because of the Vanishing Gradient Problem.

For 100 Steps the new value(100th) will face difficulty in remembering the old value(2 nd)

 \(\frac{\delta L}{\delta W_{h}} = \frac{\delta L}{\delta Y'} \frac{\delta Y'}{\delta O_{3}} \frac{\delta O_{3}}{\delta W_{h}} + \frac{\delta L}{\delta Y'} \frac{\delta Y'}{\delta O_{3}} \frac{\delta O_{3}}{\delta O_{2}}\frac{\delta O_{2}}{\delta W_{h}} + \frac{\delta L}{\delta Y'} \frac{\delta Y'}{\delta O_{3}} \frac{\delta O_{3}}{\delta O_{2}}\frac{\delta O_{2}}{\delta O_{1}}\frac{\delta O_{1}}{\delta W_{h}}+.........+\frac{\delta L}{\delta Y'} \frac{\delta Y'}{\delta O_{100}} \frac{\delta O_{100}}{\delta O_{99}} ....\frac{\delta O_{3}}{\delta O_{2}}\frac{\delta O_{2}}{\delta O_{1}}\frac{\delta O_{1}}{\delta W_{h}}\)

 

Solution -

1] To use different activation functions like relu or leaky relu

2] Better weight initialization

3] Skip RNNs

4] Use LSTM architecture

Problem 2 of Unstable Training

What if long-term dependency is so big that it will dominate the short-term dependency. Due to this gradient update will be infinite, the weight infinite, and the model will not get trained.

Solution-

1] Gradient Clipping

2] To have a Controlled Learning Rate

3] Use LSTM architecture

 

Deep Learning

Deep Learning

  • Introduction
  • LSTM - Long Short Term Memory
    • Introduction
  • ANN - Artificial Neural Network
    • Perceptron
    • Multilayer Perceptron (Notation & Memoization)
    • Forward Propagation
    • Backward Propagation
    • Perceptron Loss Function
    • Loss Function
    • Gradient Descent | Batch, Stochastics, Mini Batch
    • Vanishing & Exploding Gradient Problem
    • Early Stopping, Dropout. Weight Decay
    • Data Scaling & Feature Scaling
    • Regularization
    • Activation Function
    • Weight Initialization Techniques
    • Optimizer
    • Keras Tuner | Hyperparameter Tuning
  • CNN - Convolutional Neural Network
    • Introduction
    • Padding & Strides
    • Pooling Layer
    • CNN Architecture
    • Backpropagation in CNN
    • Data Augmentation
    • Pretrained Model & Transfer Learning
    • Keras Functional Model
  • RNN - Recurrent Neural Network
    • RNN Architecture & Forward Propagation
    • Types Of RNN
    • Backpropagation in RNN
    • Problems with RNN

About Fresherbell

Best learning portal that provides you great learning experience of various technologies with modern compilation tools and technique

Important Links

Don't hesitate to give us a call or send us a contact form message

Terms & Conditions
Privacy Policy
Contact Us

Social Media

© Untitled. All rights reserved. Demo Images: Unsplash. Design: HTML5 UP.