Lecture 2 - Deep Feedforward Networks PDF

Lecture 2: Deep Feedforward Networks Deep Learning 1 @ UvA Yuki M. Asano DEEP LEARNING ONE - 1 Lecture Overview o Modularity in deep learning o Deep learning nonlinearities o Gradient-based learning o Chain rule o Backpropagation UVA DEEP LEARNING COURSE – EFS...

Lecture 2: Deep Feedforward Networks Deep Learning 1 @ UvA Yuki M. Asano DEEP LEARNING ONE - 1 Lecture Overview o Modularity in deep learning o Deep learning nonlinearities o Gradient-based learning o Chain rule o Backpropagation UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 2 COURSE LEARNING COURSE – ‹#›– 2 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 2 Last time o We went from here: o To here: o So how do deep neural networks do it? UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 3 COURSE LEARNING COURSE – ‹#›– 3 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 3 From linear functions to nonlinear = from shallow to deep o Consider linear function f = ReLU(Ax), A in ℝ!×# , x in ℝ#×$, ReLU(x) = max(0, x) new ReLU ReLU ReLU −1 0 𝑅𝑒𝐿𝑈 = 3 3 new ReLU ReLU o 𝑅𝑒𝐿𝑈 3 = 3 but we want something non-linear! −1 0 UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 4 COURSE LEARNING COURSE – ‹#›– 4 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 4 From linear functions to nonlinear = from shallow to deep o Consider linear function f = ReLU(Ax), A in ℝ!×# , x in ℝ#×$ , ReLU(x) = max(0, x) o What about y = ReLU(Bf) = ReLU(B ReLU(Ax)) ? Inputs that end up non-zero A B ReLU ReLU Original x space Original x space f = ReLU(Ax) y = ReLU(B ReLU(Ax)) UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 5 COURSE LEARNING COURSE – ‹#›– 5 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 5 We’ve learned XOR. Original x space In practice: (5 layer MLP) https://arxiv.org/abs/1906.00904 UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 6 COURSE LEARNING COURSE – ‹#›– 6 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 6  Deep Feedforward Networks Deep feedforward networks o Feedforward neural networks Deep feedforward networks, also often called feedforward neur ◦ Also called multi-layer perceptrons (MLPs) or multilayer perceptrons (MLPs), are the quintessential deep lea ◦ The goal is to approximate some function f The goal of a feedforward network is to approximate some function f ∗ ◦ A feedforward network defines∗a mapping for a classifier, y = f (x) maps an input x to a category y. A feedfor defines a mapping y = f (x; θ) and learns the value of the parameters in the best function approximation. ◦ Learns the value of the parameters θ that result in the best function approximation.These models are called feedforward because information flow function being evaluated from x, through the intermediate computa o No feedback defineconnections f , and finally to the output y. There are no feedback connec ◦ When including outputs feedback connections, of the model are fedwe obtain back intorecurrent neuralfeedforward itself. When networks. ne ◦ Nb: brainsare have many feedback extended connections to include feedback connections, they are called recu networks, presented in chapter 10. UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES DEEPER INTO DEEP VISLab Feedforward networks are of extreme importance EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE to LEARNING machine - 7 COURSE LEARNING COURSE – ‹#›– 7 VISLab LEARNING AND OPTIMIZATIONS - 7 le Deep feedforward networks o In a formula of a composite of functions: y = 𝑓 𝑥; 𝜃 = 𝑎. 𝑥; 𝜃$,…,1 = ℎ. (ℎ.2$ … (ℎ$ 𝑥, θ$ , … ), θ.2$ , θ. ) where 𝜃3 is the parameter in the l-th layer o We can simplify the notation by 𝑎. = 𝑓 𝑥; 𝜃 = ℎ. ∘ ℎ.2$ ∘ ⋯ ∘ ℎ$ ∘ 𝑥 where each functions ℎ3 is parameterized by the parameter 𝜃3 UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 8 COURSE LEARNING COURSE – ‹#›– 8 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 8 Neural networks in blocks o We can visualize 𝑎. = ℎ. ∘ ℎ.2$ ∘ ⋯ ∘ ℎ$ ∘ 𝑥 as a cascade of blocks Forward connections (Feedforward architecture) Input ℎ! ℎ" ℎ# ℎ$ ℎ% Output hidden layers UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVES EFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 9 COURSE LEARNING COURSE – ‹#›– 9 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 9 What is a module? o Module ⇔ Building block ⇔ Transformation ⇔ Function o A module receives as input either data 𝑥 or another module’s output o A module returns an output 𝑎 based on its activation function ℎ … CHAPTER 6. DEEP FEEDFORWARD NETWORKS CHAPTER 6. DEEP FEEDFORWARD NETWORKS o A module may or may not have trainable parameters 𝑤 o Examples: f = Ax, f= exp(x) y y y y w w h1 h2 h1 h h2 h W W x1 x2 x1 x x2 x Figure 6.2: An example of 6.2: Figure a feedforward An example network, drawn in twonetwork, of a feedforward different styles. drawn Specifically, in two different styles. Specifically, this is the feedforward network we use to solve the XOR example. It has this is the feedforward network we use to solve the XOR example. a single hidden It has a single hidden layer containing two units. (Left)In this style, we draw every unit as a node in the graph. UVA DEEP LEARNING COURSE – EFSTRATIOS layer GAVVES containing EFSTRATIOS two –LEARNING GAVVES DEEP units. UVA (Left)In DEEP ONE this COURSE LEARNING style, we - 10 COURSE draw every unit as a node in theVISLab DEEPER graph. INTO DEEP This style is very explicit EFSTRATIOSandGAVVES unambiguous – UVA but DEEP for networks LEARNING – ‹#›–than larger 10 this example VISLab This style is very explicit and unambiguous but for networksLEARNING larger than ANDthis example - 10 OPTIMIZATIONS Requirements (1) The activation functions must be 1st-order differentiable (almost) everywhere (2) Take special care when there are cycles in the architecture of blocks o No other requirements o We can build as complex hierarchies as we want UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 11 COURSE LEARNING COURSE – ‹#›– 11 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 11 Feedforward model o The vast majority of models o Almost all CNNs/Transformers o As simple as it gets Feedforward architecture Input ℎ! ℎ" ℎ# ℎ$ ℎ% Output UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 12 COURSE LEARNING COURSE – ‹#›– 12 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 12 Non-linear feature learning perspective o Linear models ◦ logistic regression, linear regression ◦ convex, with closed-form solution ◦ can be fit efficiently and reliably ◦ limited capacity o Extend to nonlinear models ◦ apply the linear model not to x itself but to a transformed input φ(x), where φ is a nonlinear transformation ◦ kernel trick, e.g., RBF kernel ◦ nonlinear dimension reduction UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 13 COURSE LEARNING COURSE – ‹#›– 13 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 13 Non-linear feature learning perspective o The strategy of deep learning is to learn φ: y = f(x; θ, w) = φ(x; θ)Tw ◦ φ defines a hidden layer ◦ find the θ that corresponds to a good* representation. ◦ no longer a convex training problem o We design families φ(x; θ) rather than the right function ◦ Encode human knowledge to help generalization * good = linearly separable (in the case of classification) UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 14 COURSE LEARNING COURSE – ‹#›– 14 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 14  Non-linear feature learning perspective o Learning XOR Original x space Learned h space 1 1 h2 x2 0 0 Numbers in nodes = thresholds 0 1 0 1 2 x1 h1 o In the transformed space represented by the features extracted by a neural igure 6.1: Solving the XOR problem by learning a representation. The bold numbers rinted on the network, a linear plot indicate the model value that canfunction the learned now solve the problem. must output at each point. Left)A linear model applied directly to the original input cannot implement the XOR unction. When x1 = 0, the model’s output must increase as x2 increases. When x1 = 1, he model’s output UVA must decrease DEEP LEARNING as x2 increases. COURSE – EFSTRATIOS GAVVES A linear EFSTRATIOS EFSTRATIOS model GAVVES DEEP GAVVES must –LEARNING – UVAUVA DEEP DEEP apply ONE LEARNING aCOURSE fixed - 15 COURSE LEARNING – ‹#›– 15 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 15 Directed acyclic graph models o We can mix up our hierarchies o Makes sense when we have good knowledge of problem domain o Makes sense when combining multiple inputs & modalities ◦ E.g., RGB & LIDAR Interweaved & skip connections Eg combining images + text Loss Input ℎ! ℎ" ℎ# ℎ$ ℎ% ℎ& ℎ' UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 16 COURSE LEARNING COURSE – ‹#›– 16 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 16 Hierarchies of modules o Data efficient modules and hierarchies ◦ Trade-off between model complexity and efficiency ◦ Often, more training iterations with a “weaker” model better than fewer with a “stronger” one* ◦ ReLUs are basically half-linear functions, but give SoTA (also) because they train faster o Not too complex modules, better complex hierarchies ◦ Again, ReLUs are basically half-linear functions, but give SoTA o Use parameters smartly ◦ Often, the real constraint is GPU memory. o Compute modules in the right order to feed next modules * Not for extremely large models UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 17 COURSE LEARNING COURSE – ‹#›– 17 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 17 Loopy connections o Module’s past output is module’s future input o We must take care of cycles, i.e., unfold the graph (“Recurrent Neural Networks”) o Mostly not used (anymore) Loopy connections (must be unfolded) Input ℎ! ℎ" ℎ# ℎ$ ℎ% Output UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 18 COURSE LEARNING COURSE – ‹#›– 18 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 18 How to get w? gradient-based learning o The nonlinearity causes the loss function to be nonconvex ◦ no linear equation solution o We need to train the network with iterative, gradient based optimizers ◦ Stochastic gradient descent o No convergence guarantee and sensitive to the initialization of the parameters UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 19 COURSE LEARNING COURSE – ‹#›– 19 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 19 How to get w? gradient-based learning To use the gradient to adjust weights, we need some measuring stick (a “loss” or “cost” function) UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 20 COURSE LEARNING COURSE – ‹#›– 20 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 20 Cost function o Usually, maximum likelihood on the training set w ∗ = arg max 8 𝑝#>?@3 (𝑦|𝑥; 𝑤) ; ?@3 (𝑦|𝑥) is the output from the last layer ◦ The idea of maximum likelihood Estimation is to find the parameters of the model that can best explain the data. o Taking the logarithm, the maximum likelihood is to minimizing the negative log-likelihood: ℒ 𝑤 = −𝔼?@3 (𝑦|𝑥; 𝑤) ◦ which is equivalently described as the cross-entropy between training data and model distribution; 𝑝A?DED is the empirical data distribution UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 21 COURSE LEARNING COURSE – ‹#›– 21 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 21  as the cross-entropy between the training data and the model distribution. This −Efunction J(θ ) =cost x,y∼p̂dataislog by (y | x). p model (6.12) Cost functions given m of the cost function changes J( −Ex,yto θ ) =model from model, ∼p̂data model (y | x). log pdepending (6.12) m of logopIf we model specify. The the expansion model of the as above equation typically The specific that do not depend formmodel on the of the parameters cost functionand changes mayfrom model to model, depending be dis- on the le, as we saw specific5.5.1 in section if plog form, of pmodel model (y.x | The ) = expansion N (y ; ofθ), f(x; theI above ), equation typically yields some terms that do not depend on the model parameters and may be dis- e mean squared error cost, carded. For example, as we saw in section 5.5.1, if pmodel (y | x ) = N (y ; f(x; θ), I ), o 1We thencan recover we recover the the mean mean squared squared error error cost cost, 2 J (θ ) = E x,y∼p̂data ||y − f (x; θ )|| + const, (6.13) 2 1 J (θ ) = E x,y∼p̂data ||y − f (x; θ )||2 + const, (6.13) tor of 12 2 and a term that does not depend on θ. The discarded The ◦up n the varianceto constant of is based the Gaussian a scaling factor 1on the variance ofdistribution, of in which thethis Gaussian case distribution, which is 2 and a term that does not depend on θ. The discarded arametrize. not parameterized, Previously, constant is basedweonsaw and the thattherefore can be discarded. theofequivalence variance the Gaussianbetween distribution, which in this case o The od estimation equivalence with we chose an output not holds regardless distribution to parametrize. of we the saw function and minimization Previously, that of used to predict the equivalence between the r holds for a mean maximum of the Gaussian. linearlikelihood model, estimation but in fact,with the an output distribution equivalence holds and minimization of (x; θ ) usedmean squaredthe to predict error holds mean of for theaGaussian. linear model, but in fact, the equivalence holds regardless of the f (x; θ ) used to predict the mean of the Gaussian. of this UVA approach of deriving the EFSTRATIOScost DEEP LEARNING COURSE – EFSTRATIOS GAVVES function EFSTRATIOS GAVVES DEEP GAVVES from –LEARNING – UVAUVA DEEP DEEP maximum ONE LEARNING - 22 COURSE LEARNING COURSE VISLab – ‹#›– 22 VISLab DEEPER INTO DEEP An advantage of this approach of deriving the cost function from maximum LEARNING AND OPTIMIZATIONS - 22 Cost functions o Euclidean loss ℎ 𝑥, 𝑦 = 0.5 𝑦 − 𝑥 F o Suitable for regression problems o Sensitive to outliers ◦ Magnifies errors quadratically o Other cost functions: cross-entropy, KL-divergence (see also ML 1) UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 23 COURSE LEARNING COURSE – ‹#›– 23 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 23 Cost functions o Main point: cost functions describe what the model should do o The gradient of the cost function must be large and predictable enough to serve as a good guide for learning algorithms o Functions that saturate (become very flat) undermine this objective. o In many cases, this is due to the activation functions saturation. o The negative log-likelihood help to avoid this problem for many models because it can undo the exponentiation of the output (eg see softmax definition later) UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 24 COURSE LEARNING COURSE – ‹#›– 24 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 24 Deep learning modules DEEP LEARNING ONE – 25 Activation functions o Defined how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the network. o If output range limited, then called a “squashing function.” o The choice of activation function has a large impact on the capability and performance of the neural network. o Different activation functions may be combined, but rare o All hidden layers typically use the same activation function o Need to be differentiable at most points UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 26 COURSE LEARNING COURSE – ‹#›– 26 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 26 Linear Units 𝒙 ∈ ℝ!×+ , 𝒘 ∈ ℝ,×+ ℎ 𝑥; 𝑤 = 𝒙 ⋅ 𝒘- + 𝑏 𝑑ℎ =𝒘 𝑑𝒙 o Identity activation function o No activation saturation o Hence, strong & stable gradients ◦ Reliable learning with linear modules UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 27 COURSE LEARNING COURSE – ‹#›– 27 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 27 Rectified Linear Unit (ReLU) ReLU ℎ 𝑥 = max 0, 𝑥 𝜕ℎ 1 when 𝑥 > 0 =F 𝜕𝑤 0, when 𝑥 ≤ 0 UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 28 COURSE LEARNING COURSE – ‹#›– 28 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 28 Rectified Linear Unit (ReLU) o Advantages ◦ Sparse activation: In randomly initialized network, ~50% active ◦ Better gradient propagation: Fewer vanishing gradient problems compared to sigmoidal activation functions that saturate in both directions. ◦ Eg for sin(x), x 0 ℎ 𝑥 =F 𝑎𝑥, when x ≤ 0 𝜕ℎ 1, when 𝑥 > 0 =F 𝜕𝑥 𝑎, when x ≤ 0 o Leaky ReLUs allow a small, positive gradient when the unit is not active. o Parametric ReLUs, or PReLU, treat 𝑎 as learnable parameter UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 31 COURSE LEARNING COURSE – ‹#›– 31 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 31 Exponential Linear Unit (ELU) ELU 𝑥, when 𝑥 > 0 ℎ 𝑥 =F exp 𝑥 − 1 , x ≤ 0 𝜕ℎ 1, when 𝑥 > 0 =F 𝜕𝑥 exp(𝑥) , x ≤ 0 o ELU is a smooth approximation to the rectifier. o It has a non-monotonic “bump” when x < 0. o It serves as the default activation for models such as BERT. UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 32 COURSE LEARNING COURSE – ‹#›– 32 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 32 Gaussian Error Linear Unit GELU o Similar to ELU, but non-monotonic (change in gradient sign) o Default for Vision Transformers & state of the art (see Lect 4) https://arxiv.org/pdf/1710.05941 UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 33 COURSE LEARNING COURSE – ‹#›– 33 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 33 Sigmoid and Tanh o 𝑡𝑎𝑛ℎ 𝑥 has better output range [−1, +1] ◦ Data centered around 0 (not 0.5) → stronger gradients ◦ Less “positive” bias for next layers (mean 0, not 0.5) o Both saturate at the extreme → 0 gradients ◦ Easily become “overconfident” (0 or 1 decisions) ◦ Undesirable for middle layers ◦ Gradients ≪ 1 with chain multiplication o 𝑡𝑎𝑛ℎ 𝑥 better for middle layers o Sigmoids for outputs to emulate probabilities ◦ Still tend to be overcofident UVA DEEP LEARNING COURSE – EFSTRATIOS GAVVESEFSTRATIOS EFSTRATIOS GAVVES DEEP GAVVES –LEARNING – UVAUVA DEEP DEEP ONE LEARNING - 34 COURSE LEARNING COURSE – ‹#›– 34 DEEPER INTO DEEP VISLab VISLab LEARNING AND OPTIMIZATIONS - 34 Sigmoid and Tanh Sigmoid Tanh 1 𝑒< − 𝑒 2< ℎ 𝑥 = ℎ 𝑥 = < 1 + 𝑒 2< 𝑒 + 𝑒 2< 𝜕ℎ 𝜕ℎ = G(