History

• 1958: Perceptron
• 1980s: Multi-layer perceptron
  • Not significantly different from DNN today
• 1986: Backpropagation
  • Usually not work with more than 3 hidden layers
• 2006: RBM (Restricted Boltzmann machine) initialization
• 2009: GPU
• 2011: speech recognition
• 2012: image recognition

Step 1: function set - neural network

Different connection leads to differnt network structures.

Given a network structure, define a function set (with unknown parameters: weight & bias).

• connection, # layers, # neurons on each layer

Fully Connected Feedforward network (as an example)

• structure
  • recall feature transformation in logistic regression:juejin.cn/post/722996…

• examples

• matrix operation -> can speed up using parallel computing techniques

• design of network structure

  • 相较传统机器学习，难点从feature engineering变成了network structure design

Step 2: goodness of a function

Same as multi-class classification

Step 3: find the best function - gradient descent

Toolkit for backpropagation:

Why deep network