MindMap Gallery Neural Networks and Deep Learning Convolutional Neural Networks
The main content of convolutional neural networks is summarized, such as basic concepts, convolution operations, basic structures, parameter learning methods and some convolutional neural network example structures.
Edited at 2023-02-26 23:13:29One Hundred Years of Solitude is the masterpiece of Gabriel Garcia Marquez. Reading this book begins with making sense of the characters' relationships, which are centered on the Buendía family and tells the story of the family's prosperity and decline, internal relationships and political struggles, self-mixing and rebirth over the course of a hundred years.
One Hundred Years of Solitude is the masterpiece of Gabriel Garcia Marquez. Reading this book begins with making sense of the characters' relationships, which are centered on the Buendía family and tells the story of the family's prosperity and decline, internal relationships and political struggles, self-mixing and rebirth over the course of a hundred years.
Project management is the process of applying specialized knowledge, skills, tools, and methods to project activities so that the project can achieve or exceed the set needs and expectations within the constraints of limited resources. This diagram provides a comprehensive overview of the 8 components of the project management process and can be used as a generic template for direct application.
One Hundred Years of Solitude is the masterpiece of Gabriel Garcia Marquez. Reading this book begins with making sense of the characters' relationships, which are centered on the Buendía family and tells the story of the family's prosperity and decline, internal relationships and political struggles, self-mixing and rebirth over the course of a hundred years.
One Hundred Years of Solitude is the masterpiece of Gabriel Garcia Marquez. Reading this book begins with making sense of the characters' relationships, which are centered on the Buendía family and tells the story of the family's prosperity and decline, internal relationships and political struggles, self-mixing and rebirth over the course of a hundred years.
Project management is the process of applying specialized knowledge, skills, tools, and methods to project activities so that the project can achieve or exceed the set needs and expectations within the constraints of limited resources. This diagram provides a comprehensive overview of the 8 components of the project management process and can be used as a generic template for direct application.
Neural Networks and Deep Learning convolutional neural network
Introduction to CNN
Typical CNN structure preview
basic properties
sparse connection
Compared with the fully connected network FC, CNN is a local connection, that is, the output of a neuron in the previous layer is only connected to the input of several adjacent neurons in the next layer, and the input of a neuron in the next layer only receives the input of the previous layer. The output of several neighboring neurons
Parameter sharing
Receptive field (field of view)
The input of a neuron in the current layer is the output of several neurons near the previous layer, and what is felt is the output of the neighboring neurons in the previous layer. This input area is called the receptive field of the current neuron.
Convolution kernel
The signals within the receptive field are weighted to form the activation of the current neuron. Adjacent neurons have different but equal receptive fields (regardless of boundaries).
The activation of each neuron is generated by the weighted summation of the signals in their respective sensory fields using the same set of weight coefficients, that is, each neuron uses the same weight coefficient vector. This set of shared weight coefficients is called a convolution kernel.
Approximate translation invariance
A translation of the input signal has the same translation of the output signal
The properties of the convolution operation itself, the basic properties of linear time-invariant systems
By properly designing the pooling unit and selecting the activation function, CNN can approximately maintain translation invariance.
Example
Identify a dog in an image. It is still a dog after translation.
Convolution operation and its physical meaning
Convolution operation
Input signal x(t)
System unit impulse response h(t) (CNN convolution kernel)
Output signal y(t)
Convolution properties
Interchangeability
translation invariance
full convolution length
N K-1
Effective convolution length
N-K 1
physical meaning
filter
low pass filter
Extract the slowly changing low-frequency components of the signal
h1[n]={1/2,1/2}
high pass filter
Extract rapidly changing high-frequency components of signals
h2[n]={1/2,-1/2}
bandpass filter
Extract moderately varied ingredients
Convolution filter function
For a complex signal that contains various frequency components, different filters implemented by different convolution kernels can obtain components of different varying scales in the signal.
adaptive filtering
The error between the output of the network output layer and the expected response is used to train the output layer network
The BP algorithm back-propagates the output layer error to each previous layer, and trains the convolution kernels of each layer in turn using the back-propagation error.
The structure of basic CNN
One-dimensional convolution
Neuron activation value
neuron output
Rectified linear activation function ReLU
z=max{0,a}
convolution channel
Convolution operation between input and convolution kernel Activation function operation
Compare with fully connected network
Few shared parameters
Divide and conquer input features of different natures
2D convolution
formula
Image understanding
The two-dimensional convolution operation is equivalent to sliding hij in the Xij data array. When amn needs to be calculated, h00 slides to align with Xmn, and then the product term Xm in j hij is calculated and added.
The effective convolution output size is (D1-K1 1)×(D2-K2 1)
detection level
Calculate activation function, neuron output
multi-channel convolution
Convolution channel/convolution plane
The matrix generated by each convolution kernel h through convolution operation
Example
enter
32×32 image, 3 channels representing RGB three primary colors
Convolution kernel
6 5×5 convolution kernels, two for each input channel
output
Generates 6 28×28 convolution channels
Pooling
max pooling
Pooling that takes the maximum value of the window, that is, selects the maximum value within a small window as the pooling result
average pooling
Average within window as pooling result
decimation pooling
Fixed point value within window as pooling result
Window properties
size
M1×M2
pooling stride
S
Equal-length zero-padding convolution
K is an odd number
Add (K-1)/2 zeros to both ends of the input
K is an even number
Add K/2 zeros to one side and (K/2)-1 zeros to the other side.
Constituting CNN
Composition of convolutional layers
Convolution operation stage
Detection level (ReLU function)
Pooling (optional)
Typical CNN network structure
Some extended structures of convolution
tensor convolution
3D data volume
tensor convolution kernel
convolution plane
Channel-dimensional convolution
Extract different features of the channel dimension
1×1 convolution kernel
S-stride convolution
CNN parameter learning
CNN’s BP algorithm idea
forward propagation
Convolution layer convolution calculation
FC layer fully connected calculation activation output
Pooling layer performs pooling
Backpropagation
The FC layer is calculated according to the standard BP backpropagation algorithm.
Convolutional layer and pooling layer backpropagation algorithm
Backpropagation formula for convolutional layers
Backpropagation formula for pooling layer
2D expansion
CNN example introduction
LeNet-5 network
AlexNet network and VGGNet network
activation function
The ReLU activation function trains 6 times faster than the tanh activation function
AlexNet structure
VGGNet structure
Use deeper layers, smaller convolution kernels, and multiple convolution layers corresponding to one pooling layer.
Ideas for improving training effects
Obtain better training results by increasing the depth of CNN
A direct increase in the number of layers will bring negative effects
Easy to overfit
gradient disappears
gradient explosion
GoogLeNetNetwork
Macro construction module Inception
4 parallel branches
Generate output by branching and merging modules
Each branch contains a 1×1 convolution
The purpose is to divide and conquer to reduce parameters and computational complexity
structure
Residual networks and dense networks
residual network
network degradation problem
The accuracy on the training set is saturated or even drops.
Residual network characteristics
Easy to optimize and can improve accuracy by adding considerable depth
The residual block inside the residual network uses skip connections, which alleviates the vanishing gradient problem caused by increasing depth in the deep neural network.
residual building block
residual grid structure
dense network
Dense network characteristics
Maintain the structure of the feedforward network, connecting the output from the input layer or from the current layer to the input of each subsequent layer
For L-layer networks, there can be L(L-1)/2 connections
dense network structure