Difference between Batch Gradient Descent and Stochastic Gradient Descent

 

Difference between Batch Gradient Descent and Stochastic Gradient Descent

In order to train a Linear Regression model, we have to learn some model parameters such as feature weights and bias terms. An approach to do the same is Gradient Descent which is an iterative optimization algorithm capable of tweaking the model parameters by minimizing the cost function over the train data. It is a complete algorithm i.e it is guaranteed to find the global minimum (optimal solution) given there is enough time and the learning rate is not very high. Two Important variants of Gradient Descent which are widely used in Linear Regression as well as Neural networks are Batch Gradient Descent and Stochastic Gradient Descent(SGD). Batch Gradient Descent: Batch Gradient Descent involves calculations over the full training set at each step as a result of which it is very slow on very large training data. Thus, it becomes very computationally expensive to do Batch GD. However, this is great for convex or relatively smooth error manifolds. Also, Batch GD scales well with the number of features. eq1 eq2 Stochastic Gradient Descent: SGD tries to solve the main problem in Batch Gradient descent which is the usage of whole training data to calculate gradients at each step. SGD is stochastic in nature i.e. it picks up a “random” instance of training data at each step and then computes the gradient, making it much faster as there is much fewer data to manipulate at a single time, unlike Batch GD. eq3 There is a downside of the Stochastic nature of SGD i.e. once it reaches close to the minimum value then it doesn’t settle down, instead bounces around which gives us a good value for model parameters but not optimal which can be solved by reducing the learning rate at each step which can reduce the bouncing and SGD might settle down at global minimum after some time.

Difference between Batch Gradient Descent and Stochastic Gradient Descent

S.NO.Batch Gradient DescentStochastic Gradient Descent
1.Computes gradient using the whole Training sampleComputes gradient using a single Training sample
2.Slow and computationally expensive algorithmFaster and less computationally expensive than Batch GD
3.Not suggested for huge training samples.Can be used for large training samples.
4.Deterministic in nature.Stochastic in nature.
5.Gives optimal solution given sufficient time to converge.Gives good solution but not optimal.
6.No random shuffling of points are required.The data sample should be in a random order, and this is why we want to shuffle the training set for every epoch.
7.Can’t escape shallow local minima easily.SGD can escape shallow local minima more easily.
8.Convergence is slow.Reaches the convergence much faster.

Comments

Post a Comment

Popular posts from this blog

General Statistical Models

  General Statistical Models The time has come to learn some theory. This is a preview of STAT 5101–5102. We don’t need to learn much theory. We will proceed with the general strategy of all introductory statistics: don’t derive anything, just tell you stuff. 3.1  Probability Models 3.1.1  Kinds of Probability Theory There are two kinds of probability theory. There is the kind you will learn in STAT 5101-5102 (or 4101–4102, which is more or less the same except for leaving out the multivariable stuff). The material covered goes back hundreds of years, some of it discovered in the early 1600’s. And there is the kind you would learn in MATH 8651–8652 if you could take it, which no undergraduate does (it is a very hard Ph. D. level math course). The material covered goes back to 1933. It is all “new math”. These two kinds of probability can be called  classical  and  measure-theoretic , respectively. The old kind (classical) is still very useful. For every 1000 people who know a lot of pr
Read more

Solution of a linear system

Image
  Solution of a linear system [ edit ] The steepest descent algorithm applied to the  Wiener filter [11] Gradient descent can be used to solve a system of linear equations � � − � = 0 reformulated as a quadratic minimization problem. If the system matrix  �  is real  symmetric  and  positive-definite , an objective function is defined as the quadratic function, with minimization of � ( � ) = � � � � − 2 � � � , so that ∇ � ( � ) = 2 ( � � − � ) . For a general real matrix  � ,  linear least squares  define � ( � ) = ‖ � � − � ‖ 2 . In traditional linear least squares for real  �  and  �  the  Euclidean norm  is used, in which case ∇ � ( � ) = 2 � � ( � � − � ) . The  line search  minimization, finding the locally optimal step size  �  on every iteration, can be performed analytically for quadratic functions, and explicit formulas for the locally optimal  �  are known. [5] [12] For example, for real  symmetric  and  positive-definite  matrix  � , a simple algorithm can be as follows, [5
Read more