K.CHANDRAN

  Gradient Descent With Momentum The problem with gradient descent is that the weight update at a moment (t) is governed by the learning rate and gradient at that moment only. It doesn’t take into account the past steps taken while traversing the cost space. Image by author It leads to the following problems. The gradient of the cost function at saddle points( plateau) is negligible or zero, which in turn leads to small or no weight updates. Hence, the network becomes stagnant, and learning stops The path followed by Gradient Descent is very jittery even when operating with mini-batch mode Consider the below cost surface. Image by author Let’s assume the initial weights of the network under consideration correspond to point A. With gradient descent, the Loss function decreases rapidly along the slope AB as the gradient along this slope is high. But as soon as it reaches point B the gradient becomes very low. The weight updates around B is very small. Even after many iterations, the...

 A RANDOM GRAPH MODEL  LEMMA:1.1 A random graph G_(n,p), given that its number of edges is m, is equally likely to be one of the (((n¦2))¦m) graph that have m edges. Proof: Let G_0  be any labelled graph with m edges. Then since {G_(n,p)= G_0  }⊆{|E_(n,p) |=m} We have  P(G_(n,p)= G_0 |E_(n,p) |=m)=  (P(G_(n,p)= G_0 |E_(n,p) |=m))/(P(|E_(n,p) |=m))   =  P(G_(n,p)= G_0 )/((|E_(n,p) |=m) ) =  (p^m (1-p)^((n¦2)-m ))/(〖(((n¦2))¦m)  p〗^m (1-p)^((n¦2)-m ) ) = (((n¦2))¦m)^(-1). Thus G_(n,p) conditional on the event {G_(n,p) has m edges} is equal in distribution to G_(n,m), the graph chosen uniformly at random from all graphs with m edges. Obviously, the main difference between those two models of random graph is that in G_(n,m.)we choose its number of edges, while in the case of G_(n,p) the number of edges is the binomial random variable with the parameters and p. intuitively, for large n random graphs G_(n,m) and G_(n,p)  should beh...

  Solution of a linear system [ edit ] The steepest descent algorithm applied to the  Wiener filter [11] Gradient descent can be used to solve a system of linear equations � � − � = 0 reformulated as a quadratic minimization problem. If the system matrix  �  is real  symmetric  and  positive-definite , an objective function is defined as the quadratic function, with minimization of � ( � ) = � � � � − 2 � � � , so that ∇ � ( � ) = 2 ( � � − � ) . For a general real matrix  � ,  linear least squares  define � ( � ) = ‖ � � − � ‖ 2 . In traditional linear least squares for real  �  and  �  the  Euclidean norm  is used, in which case ∇ � ( � ) = 2 � � ( � � − � ) . The  line search  minimization, finding the locally optimal step size  �  on every iteration, can be performed analytically for quadratic functions, and explicit formulas for the locally optimal  �  are known. [5] [12] F...