The data flow from the original input to the output target passes through multiple linear or nonlinear components. Each component processes information and in turn affects subsequent components.

When we finally get the output, we don't know exactly how much each component contributes. This question is called contribution Allocation problem.

The contribution allocation problem is a very critical issue, which is related to how to learn the parameters in each component.

smart concept

Artificial intelligence concept

The subject position of artificial intelligence

Interdisciplinary research with brain science and cognitive science

Research on methods and technologies of intelligent simulation

Perception: that is, simulating human perception ability to perceive and process external stimulus information (visual and speech, etc.). Main research areas include speech information processing and computer vision.

Learning: Simulating human learning ability, mainly studying how to learn from examples or interacting with the environment. Main research areas include supervised learning, unsupervised learning and reinforcement learning.

Cognition: Simulates human cognitive abilities. The main research areas include knowledge representation, natural language understanding, reasoning, planning, decision-making, etc.

symbolism

connectionism

Behaviorism

Model proposed

ice age

The renaissance caused by the backpropagation algorithm

Decline in popularity

The rise of deep learning

After data preprocessing, such as removing noise, etc. For example, in text classification, removing stop words, etc.

Extract some effective features from raw data. For example, in image classification, extract edges, scale invariant feature transform (SIFT) features, etc.

Perform certain processing on the features, such as dimensionality reduction and dimensionality enhancement. Dimensionality reduction includes two approaches: Feature Extraction and Feature Selection. Commonly used feature transformation methods include principal component analysis (PCA), linear discriminant analysis (Linear Discriminant Analysis), etc.

The core part of machine learning, making predictions through a function

local representation

distributed representation

Different from "shallow learning", the key problem that deep learning needs to solve is the distribution of contribution

In a sense, deep learning can also be regarded as a kind of reinforcement learning (RL). Each internal component cannot directly obtain supervision information, but needs to obtain it through the final supervision information (reward) of the entire model, and there is Certain delay.

The neural network model can use the error back propagation algorithm, which can better solve the contribution distribution problem.

traditional learning style

New way of learning

Theano: A Python toolkit from the University of Montreal, used to efficiently define, optimize and execute Theano project is currently out of maintenance. Multidimensional array data corresponds to mathematical expressions. Theano can transparently use GPUs and efficient symbols differential.

Caffe: The full name is Convolutional Architecture for Fast Feature Embedding. It is a computing framework for convolutional network models. The network structure to be implemented can be specified in the configuration file and does not require coding. Caffe is implemented in C and Python and is mainly used for computer vision.

TensorFlow: A Python toolkit developed by Google that can run on any device with a CPU or GPU. TensorFlow's calculation process is represented using data flow graphs. Tensor Flow's name comes from the fact that the operation object in its calculation process is a multi-dimensional array, that is, a tensor.

Chainer: One of the earliest neural network frameworks that uses dynamic computing graphs. Its core development team is Preferred Networks, a machine learning startup from Japan. Compared with static calculation graphs used by Tensorflow, Theano, Caffe and other frameworks, dynamic calculation graphs can dynamically construct calculation graphs at runtime, so they are very suitable for some complex decision-making or reasoning tasks.

PyTorch5: A deep learning framework developed and maintained by Facebook, NVIDIA, Twitter and other companies. Its predecessor is Torch6 of Lua language. PyTorch is also a framework based on dynamic computing graphs, which has obvious advantages in tasks that require dynamically changing the structure of neural networks.

Neural networks can be implemented efficiently by using matrix operations.

Affine layer

Identity function

softmax function

Classification problem

Enter multiple sets of data at once

When looking for optimal parameters (weights and biases), you are looking for parameters that make the value of the loss function as small as possible, so you need to calculate the derivatives of the parameters (the gradient to be precise)

Why not directly use recognition accuracy as an indicator?

mean square error

cross entropy error

Extract some test data

Manual setting

training gained

The gradient of the loss function with respect to the weight parameters

Number of cycles/minibatch size

The amount of yellow is the value obtained during backpropagation

The green quantity is a known quantity

Network based on fully connected layer (Affine layer)

CNN based network

Convolution[convolution layer]-ReLU-(Pooling[pooling layer])

The layer close to the output uses the previous Affine [affine transformation] - ReLU combination

The final output layer uses the previous Affine-Softmax combination

Problems with the fully connected layer

Convolution operation

Correlation

Variants of convolution

Mathematical properties of convolution

Convolution operation on 3D data

Multiple convolution operations

Batch processing

Local connection: Each neuron in the convolutional layer (assumed to be the l-th layer) is only connected to the neurons in a local window in the next layer (l-1 layer), forming a local connection network. The number of connections between the convolutional layer and the next layer is greatly reduced, from the original n(l) × n(l - 1) connections to n(l) × m connections. m is the filter size.

Weight sharing: The filter w(l) as a parameter is the same for all neurons in layer l.

Due to local connections and weight sharing, the parameters of the convolutional layer only have an m-dimensional weight w(l) and a 1-dimensional bias b(l), with a total of m 1 parameters.

The number of neurons in layer l is not chosen arbitrarily, but satisfies n(l) = n(l−1) − m 1.

Maximum: Generally, the maximum value of all neurons in a region is taken.

Average aggregation (Mean): Generally, the average value of all neurons in the area is taken.

where Y(′d) is the output of the pooling layer, f(·) is the nonlinear activation function, w(d) and b(d) are learnable scalar weights and biases.

There are no parameters to learn

The number of channels does not change

Robust to small position changes (robustness)

The filter before learning is randomly initialized, so there is no pattern in the shades of black and white, but the filter after learning becomes a regular image. We found that through learning, the filters are updated into regular filters, such as filters that gradient from white to black, filters that contain blocky areas (called blobs), etc. Filter responsive to horizontal and vertical edges

It can be seen that the filters of the convolutional layer will extract original information such as edges or patches. The CNN just implemented will pass this raw information to subsequent layers.

Information extracted from the convolutional layers of CNN. The neurons in layer 1 respond to edges or patches, layer 3 responds to texture, layer 5 responds to object parts, and the final fully connected layer responds to the category of the object (dog or car).

If multiple convolutional layers are stacked, as the layers deepen, the extracted information becomes more complex and abstract. This is a very interesting part of deep learning. As the layers deepen, neurons change from simple shapes to "high-level" information. In other words, just as we understand the "meaning" of things, the objects of response gradually change.

LeNet was proposed in 1998 as a network for handwritten digit recognition. It has consecutive convolutional layers and pooling layers, and finally outputs the results through a fully connected layer.

Excluding the input layer, LeNet-5 has a total of 7 layers.

join table

It was proposed in 2012 and uses many technical methods of modern deep convolutional networks.

Inception module: A convolutional layer contains multiple convolution operations of different sizes

The Inception network is stacked by multiple inception modules and a small number of aggregation layers.

V1 version

The earliest v1 version of the Inception network is the very famous GoogLeNet [Szegedy et al., 2015], and won the 2014 ImageNet image classification competition.

The information propagation efficiency is improved by adding direct edges to the nonlinear convolutional layer.

nonlinear elements

Heading towards a deeper network

Further improve recognition accuracy

Deeper motivation

ILSVRC competition

VGG

GoogLeNet

ResNet

Problems that need to be solved

GPU-based speedup

Distributed learning

Digit reduction of arithmetic precision

Object detection

Image segmentation

Image caption generation

Image style transformation

Image generation

Autopilot

reinforcement learning

Thesaurus-based approach

count-based approach

Inference-based approach (word2vec)

A diagram based on the hypernym-hyponym relationship according to the meaning of each word

The most famous synonym dictionary

effect

Used via the NLTK module

New words continue to appear, making it difficult to adapt to changes in the times

High labor cost

Unable to express subtle differences in words

A corpus is a large amount of text data

Corpora used in the field of natural language processing sometimes add additional information to text data. For example, each word of the text data can be marked with a part-of-speech. Here, it is assumed that the corpus we use has no tags added.

famous corpora

preprocessing

Construct compact and reasonable vector representations in the word domain

The meaning of a word is formed by the words surrounding it

Context refers to the words surrounding a centered word

The size of the context is called the window size

The simplest way to use a vector is to count how many times a word appears around it.

text = 'You say goodbye and I say hello.'

Set window size to 1

cosine similarity

Get the word vector of the query word

Obtain the cosine similarity between the word vector of the query word and all other word vectors respectively.

Results based on cosine similarity, showing their values ​​in descending order

In the co-occurrence matrix, common words like the will be considered to have a strong correlation with nouns like car

PMI

PMI based on co-occurrence matrix

insufficient

Get the PPMI matrix based on the co-occurrence matrix

We need to observe the distribution of data and find important "axes"

Singular value decomposition (SVD)

The PTB corpus is often used as a benchmark for evaluating proposed methods

Preprocessing of PTB corpus

The preprocessing I did

Hyperparameter assignment

For the query word you, you can see that personal pronouns such as i and we are ranked first. These are words with the same usage in grammar.

The query word year has synonyms such as month and quarter.

The query word car has synonyms such as auto and vehicle.

When using toyota as the query term, car manufacturer names or brand names such as nissan, honda and lexus appeared.

Use the corpus to calculate the number of words in the context, convert them into a PPMI matrix, and then obtain good word vectors based on SVD dimensionality reduction.

In the real world, corpora deal with a very large number of words. For example, it is said that the English vocabulary has over 1 million words. If the vocabulary size exceeds 1 million, then using the count-based method requires generating a huge matrix of 1 million × 1 million, but it is obviously unrealistic to perform SVD on such a large matrix.

Inference-based approaches using neural networks

Target

reasoning method

Convert words to vectors

Neural Networks

structure

Features

Convert scores into probabilities using the Softmax function

Find the cross-entropy error between these probabilities and the supervised labels

Learn it as a loss

W(in) weight is the distributed representation of the word we want

The probability that wt occurs after wt−1 and wt 1 occur.

Loss function L (negative log likelihood) of CBOW model

word2vec has two models

Skip-gram is a model that inverts the context and target words processed by the CBOW model.

skip-gram network structure diagram

skip-gram models and probability

Loss function comparison

Judging from the accuracy of the distributed representation of words, the skip-grm model gives better results in most cases.

Scenarios where new words need to be added to the vocabulary and the distributed representation of the words updated

Properties of distributed representations of words

Accuracy of distributed representations of words

The distributed representation of words obtained using word2vec can be used to find approximate words

transfer learning

Using distributed representations of words, it is also possible to convert documents (sequences of words) into fixed-length vectors.

artificially created word similarity evaluation set to evaluate

Compare the scores given by people and the cosine similarity given by word2vec to examine the correlation between them

in conclusion

Can the original purpose of the CBOW model "predict target words from context" be used for something? Can P(wt|wt−2, wt−1) play a role in some practical scenarios?

The windows we considered before are all symmetrical, and then we only consider the left window.

Use probability to evaluate the likelihood that a sequence of words will occur, that is, the extent to which a sequence of words is natural.

Probability representation

Markov chain

insufficient

RNN has a mechanism that can remember context information no matter how long the context is. Therefore, time series data of arbitrary length can be processed using RNN.

word2vec is a method aimed at obtaining distributed representation of words, and is generally not used in language models.

The structure of the RNN layer

The input at time t is xt, which implies that time series data (x0, x1, ··· , xt, ···) will be input into the layer. Then, in the form corresponding to the input, output (h0, h1, ··· , ht, ···)

The output has two forks, which means the same thing was copied. A fork in the output will become its own input.

We use the word "moment" to refer to the index of time series data (that is, the input data at time t is xt). Expressions such as "the t-th word" and "the t-th RNN layer" are used, as well as expressions such as "the word at time t" or "the RNN layer at time t".

The RNN layer at each moment receives two values, which are the input passed to this layer and the output of the previous RNN layer.

RNN has two weights, namely the weight Wx that converts the input x into the output h and the weight Wh that converts the output of the previous RNN layer into the output at the current moment. Additionally, there is bias b.

From another perspective, RNN has a state h, which is continuously updated through the above formula. So h can be called a hidden state or a hidden state vector

The two schematic drawing methods are equivalent

time-based backpropagation

To find the gradient based on BPTT, the intermediate data of the RNN layer at each moment must be saved in memory. Therefore, as the time series data gets longer, the computer's memory usage (not just calculations) also increases.

In order to solve the above problem, when processing long time series data, the common practice is to cut the network connection into an appropriate length.

Networks that are too long in the direction of the time axis are truncated at appropriate locations to create multiple small networks, and then the error backpropagation method is performed on the cut-out small networks. This method is called Truncated BPTT (truncated BPTT).

In Truncated BPTT, the backward propagation connection of the network is cut off, but the forward propagation connection is still maintained.

Processing order

At the beginning of the input data, an "offset" needs to be made within individual batches.

Notice

Call a layer that processes T steps at a time a "Time RNN layer"

The layer that performs the single-step processing in the Time RNN layer is called the "RNN layer"

Like Time RNN, layers that process time series data holistically are named starting with the word "Time", which is the naming convention laid out in this book. After that, we will also implement the Time Affine layer, Time Embedding layer, etc.

forward propagation

Backpropagation

forward propagation

Backpropagation

RNN-based language models are called RNNLM

structure

Sample

Target neural network structure

Time Affine

Time Softmax

The input data is 1

The input data is multiple

The bigger the probability, the better, and the smaller the confusion, the better.

Perplexity represents the number of options that can be chosen next. If the confusion is 1.25, it means that the number of candidates for the next word is about 1.

Encapsulate the above operations into classes

There are three hidden layers h1, h2 and h3, where h1 is calculated from two inputs x1 and x2, h2 is calculated from two other input layers x3 and x4, and h3 is calculated from two hidden layers h1 and h2.

activation function

MatMul (Matrix Product) Node

There is an established method to solve gradient explosion, which is called gradient clipping.

It is assumed here that the gradients of all parameters used by the neural network can be integrated into a variable and represented by the symbol g. Then set the threshold to threshold. At this time, if the L2 norm g of the gradient is greater than or equal to the threshold, the gradient is corrected as described above.

LSTM is the abbreviation of Long Short-Term Memory, which means that it can maintain short-term memory (Short-Term Memory) for a long time.

First express the calculation of tanh(h(t−1)*Wh xt*Wx b) as a rectangular node tanh (ht−1 and xt are row vectors)

Let’s compare the interface (input and output) of LSTM and RNN

The difference between the interface of LSTM and RNN is that LSTM also has path c. This c is called a memory unit (or simply "unit"), which is equivalent to the dedicated memory department of LSTM.

The characteristic of the memory unit is that it only receives and transmits data within the LSTM layer. That is to say, from the side receiving the output of the LSTM, the output of the LSTM only has the hidden state vector h. Memory unit c is invisible to the outside world, and we don't even need to consider its existence.

ct stores the memory of the LSTM at time t, which can be considered to contain all necessary information from the past to time t. Then based on the ct of this carrier memory, the hidden state ht is output.

calculate

Gate

The hidden state ht only applies the tanh function to the memory unit ct, and we consider applying gates to tanh(ct). Since this gate manages the output of the next hidden state ht, it is called an output gate.

The output gate is calculated as follows. The sigmoid function is represented by σ()

ht can be calculated from the product of o and tanh(ct). The calculation method is the element-wise product, which is the product of the corresponding elements. It is also called the Hadamard product.

Now, we add a gate to forget unnecessary memories on the memory unit c(t−1), which is called the forget gate here.

The calculation of the forget gate is as follows

ct is obtained by the product of this f and the corresponding element of the previous memory unit ct−1

Now we also want to add some new information to this memory unit that should be remembered, for this we add a new tanh node

The result calculated based on the tanh node is added to the memory unit ct−1 at the previous moment.

We add a gate to g in Figure 6-17. This newly added gate is called the input gate here.

The input gate determines the value of each element of the new information g. Input gates do not add new information without consideration; rather, they make choices about which information to add. In other words, the input gate adds weighted new information.

The input gate is calculated as follows

Backpropagation of memory cells only flows through the " " and "×" nodes. The " " node flows out the gradient from the upstream as it is, so the gradient does not change (degenerates). The calculation of the "×" node is not a matrix product, but the product of the corresponding elements (Hadama product), which will not cause gradient changes.

Deepening the LSTM layer (stacking multiple LSTM layers) often works well.

How many layers are appropriate?

By deepening the depth, more expressive models can be created, but such models often suffer from overfitting. To make matters worse, RNNs are more prone to overfitting than conventional feedforward neural networks, so overfitting countermeasures for RNNs are very important.

Countermeasures

Dropout

Dropout layer insertion position

Weight tying can be literally translated as "weight binding".

The trick to binding (sharing) the weights of the Embedding layer and the Affine layer is weight sharing. By sharing the weights between these two layers, the number of parameters learned can be greatly reduced. In addition to this, it improves accuracy.

Select the word with the highest probability, the result is uniquely determined

Words with high probability are easy to be selected, words with low probability are difficult to be selected.

Use better language models

This model has two modules - Encoder and Decoder. The encoder encodes the input data and the decoder decodes the encoded data.

seq2seq consists of two LSTM layers, the encoder LSTM and the decoder LSTM.

The hidden state h of LSTM is a fixed-length vector. The difference between it and the model in the previous section is that the LSTM layer receives the vector h. This single, small change allowed ordinary language models to evolve into decoders that could handle translation.

Trying to get seq2seq to do addition calculations

filling

In many cases, learning progresses faster and final accuracy improves after using this technique.

Intuitively, the propagation of gradients can be smoother and more effective after inverting the data.

The encoder using the helmet is called Peeky Decoder, and the seq2seq using Peeky Decoder is called Peeky seq2seq.

The encoder converts the input sentence into a fixed-length vector h, which concentrates all the information required by the decoder. It is the only source of information for the decoder.

The output h of the encoder, which concentrates important information, can be assigned to other layers of the decoder

Two vectors are input to the LSTM layer and the Affine layer at the same time, which actually represents the concatenation of the two vectors.

An encoder is used in seq2seq to encode the time series data, and then the encoded information is passed to the decoder. At this point, the output of the encoder is a fixed-length vector.

Fixed-length vectors mean that whatever the length of the input statement (no matter how long) is, will be converted into a vector of the same length.

The length of the encoder's output should change accordingly based on the length of the input text

Because the encoder processes from left to right, strictly speaking, the "cat" vector just contains the information of the three words "我的人", "は" and "猫". Considering the overall balance, it is best to include information around the word "cat" evenly. In this case, bidirectional RNN (or bidirectional LSTM) that processes time series data from both directions is more effective.

Previously we put the "last" hidden state of the encoder's LSTM layer into the "initial" hidden state of the decoder's LSTM layer.

The decoder in the previous chapter only used the last hidden state of the encoder's LSTM layer. If using hs, only the last row is extracted and passed to the decoder. Next we improve the decoder to be able to use all hs.

We focus on a certain word (or set of words) and convert this word at any time. This allows seq2seq to learn the correspondence between "which words in the input and output are related to which words"

structural changes

How to calculate

Learning of weight a

Integrate

If we consider the overall balance, we want the vector to contain information around the word "cat" more evenly.

Bidirectional LSTM adds an LSTM layer processing in the opposite direction on top of the previous LSTM layer.

Splice the hidden states of the two LSTM layers at each moment and use it as the final hidden state vector (in addition to splicing, you can also "sum" or "average", etc.)

The output of the attention layer (context vector) is connected to the input of the LSTM layer at the next moment. Through this structure, the LSTM layer is able to use the information of the context vector. In contrast, the model we implemented uses context vectors in the Affine layer.

Deepen the RNN layer

residual connection

The history of machine translation

Since 2016, Google Translate has been using neural machine translation for actual services. Machine translation system called GNMT

GNMT requires large amounts of data and computing resources. GNMT uses a large amount of training data, (1 model) learned on nearly 100 GPUs for 6 days. In addition, GNMT is also trying to further improve accuracy based on technologies such as ensemble learning and reinforcement learning that can learn 8 models in parallel.

RNN can handle variable-length time series data well. However, RNN also has shortcomings, such as parallel processing problems.

RNN needs to be calculated step by step based on the calculation results of the previous moment, so it is (basically) impossible to calculate RNN in parallel in the time direction. This will become a big bottleneck when performing deep learning in a parallel computing environment using GPUs, so we have the motivation to avoid RNN.

Transformer does not use RNN, but uses Attention for processing. Let's take a brief look at this Transformer.

Self-Attention

Use a fully connected neural network with one hidden layer and activation function ReLU. In addition, Nx in the figure means that the elements surrounded by the gray background are stacked N times.

NTM (Neural Turing Machine)

Neural networks can also gain additional capabilities using external storage devices.

Based on Attention, encoders and decoders implement "memory operations" in computers. In other words, this can be interpreted as, the encoder writes the necessary information to memory and the decoder reads from memory Get the necessary information.

In order to imitate the computer's memory operation, NTM's memory operation uses two Attentions,

NTM has successfully solved long-term memory problems, sorting problems (arranging numbers from large to small), etc.

Difficult to optimize and computationally intensive

The fitting ability is too strong and it is easy to overfit.

Network structure diversity

Difficulties with low-dimensional spaces

Difficulties with high-dimensional spaces

Gradient descent method type

learning rate decay

Gradient direction optimization

The Gaussian initialization method is the simplest initialization method. The parameters are randomly initialized from a Gaussian distribution with a fixed mean (such as 0) and a fixed variance (such as 0.01).

When the number of input connections of a neuron is n(in), its input connection weight can be set to be initialized with the Gaussian distribution of N(0,sqrt(1/nin)).

If the number of output connections nout is also considered, it can be initialized according to the Gaussian distribution of N(0,sqrt(2/(nin nout)))

Uniform distribution initialization uses uniform distribution to initialize parameters within a given interval [−r, r]. The setting of the hyperparameter r can also be adjusted adaptively according to the number of connections of neurons.

Activation function type

We tried using the weight initial values ​​recommended in the paper by Xavier Glorot et al.

If the number of nodes in the previous layer is n, the initial value uses a Gaussian distribution with a standard deviation of (1/sqrt(n))

When the activation function uses ReLU, it is generally recommended to use the initial value dedicated to ReLU, which is the initial value recommended by Kaiming He et al., also known as the "He initial value".

When the number of nodes in the current layer is n, the initial value of He uses a Gaussian distribution with a standard deviation of (2/sqrt(n))

The different sources and measurement units of each dimensional feature will cause the distribution range of these feature values ​​to be very different. When we calculate the Euclidean distance between different samples, features with a large value range will play a dominant role.

scaling normalization

standard normalization

After the input data is whitened, the correlation between features is low and all features have the same variance.

One of the main ways to achieve whitening is to use principal component analysis to remove the correlation between components.

Also called Batch Normalization, BN method

In order to make each layer have the appropriate breadth, the distribution of activation values ​​​​is "forced" to be adjusted.

Perform regularization so that the mean of the data distribution is 0 and the variance is 1.

Any intermediate layer in the neural network can be normalized.

advantage

Batch Norm layer

Limitations of batch normalization

Layer normalization normalizes all neurons in an intermediate layer.

Network structure

Optimization parameters

regularization coefficient

Hyperparameter performance cannot be evaluated using test data

If the test data is used to confirm the "goodness" of the hyperparameter value, it will cause the hyperparameter value to be adjusted to only fit the test data.

The training data is used for parameter learning (weights and biases), and the validation data is used for performance evaluation of hyperparameters. In order to confirm the generalization ability, the test data should be used at the end (ideally only once)

grid search

random search

Bayesian optimization

Dynamic resource allocation

Weight decay is a method that has been frequently used to suppress overfitting. This method penalizes large weights during the learning process.

Simply put, the loss function becomes

λ is a hyperparameter that controls the strength of regularization

Dropout Method

If the network model becomes very complex, it will be difficult to deal with it using only weight decay.

The method of randomly deleting neurons during the learning process chooses to discard neurons randomly each time. The simplest way is to set a fixed probability p. For each neuron, there is a probability p to determine whether to retain it.

data augmentation

label smoothing

Self-training is to first use labeled data to train a model, and use this model to predict the labels of unlabeled samples, add samples with relatively high prediction confidence and their predicted pseudo labels to the training set, and then retrain the new model, and Keep repeating this process.

Co-training is an improved method of self-training

Two classifiers based on different views promote each other. A lot of data has different perspectives that are relatively independent.

Due to the conditional independence of different perspectives, models trained on different perspectives are equivalent to understanding the problem from different perspectives and have certain complementarity. Collaborative training is a method that uses this complementarity to perform self-training. First, two models f1 and f2 are trained on the training set according to different perspectives, and then f1 and f2 are used to predict on the unlabeled data set. Samples with relatively high prediction confidence are selected to be added to the training set, and two different perspectives are retrained. model and repeat this process.

inductive transfer learning

Deriving transfer learning

Once training is completed, the model remains fixed and is no longer iteratively updated.

It is still very difficult for a model to be successful on many different tasks at the same time.

Optimizer-based meta-learning

Model-agnostic meta-learning