mirror of
https://github.com/microsoft/ai-edu.git
synced 2026-09-01 14:50:53 +08:00
More English Translations (#665)
* Create 14.4-A real application of the multi-classification task.md * Rename 14.4-A real application of the multi-classification task.md to 14.5-A real application of the multi-classification task.md * Update 14.5-A real application of the multi-classification task.md * Rename 14.5-A real application of the multi-classification task.md to 14.6-A real application of the multi-classification task.md * Create 15.0-Network optimization.md * Update 15.0-Network optimization.md * Create 15.1-Weight matrix initialization.md * Update 15.1-Weight matrix initialization.md
This commit is contained in:
Mark Chen
+90
@@ -0,0 +1,90 @@
|
||||
|
||||
<!--Copyright © Microsoft Corporation. All rights reserved.
|
||||
适用于[License](https://github.com/Microsoft/ai-edu/blob/master/LICENSE.md)版权许可-->
|
||||
|
||||
## 14.6 Multi-classification task - MNIST handwriting recognition
|
||||
|
||||
### 14.6.1 Reading Data
|
||||
|
||||
MNIST data is in the format of images, so we use `mode="vector"` to read it and convert it into the format of vectors.
|
||||
|
||||
```Python
|
||||
def LoadData():
|
||||
print("reading data...")
|
||||
dr = MnistImageDataReader(mode="vector")
|
||||
......
|
||||
```
|
||||
|
||||
### 14.6.2 Build a model
|
||||
|
||||
A total of 4 hidden layers are connected with the ReLU activation function, and the last output layer is connected with the Softmax classification function.
|
||||
|
||||
<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images/Images/14/mnist_net.png" />
|
||||
|
||||
Figure 14-18 The abstract model for completing the MNIST classification task
|
||||
|
||||
The following are the main parameter settings:
|
||||
|
||||
```Python
|
||||
if __name__ == '__main__':
|
||||
dataReader = LoadData()
|
||||
num_feature = dataReader.num_feature
|
||||
num_example = dataReader.num_example
|
||||
num_input = num_feature
|
||||
num_hidden1 = 128
|
||||
num_hidden2 = 64
|
||||
num_hidden3 = 32
|
||||
num_hidden4 = 16
|
||||
num_output = 10
|
||||
max_epoch = 10
|
||||
batch_size = 64
|
||||
learning_rate = 0.1
|
||||
|
||||
params = HyperParameters_4_0(
|
||||
learning_rate, max_epoch, batch_size,
|
||||
net_type=NetType.MultipleClassifier,
|
||||
init_method=InitialMethod.MSRA,
|
||||
stopper=Stopper(StopCondition.StopLoss, 0.12))
|
||||
|
||||
net = NeuralNet_4_0(params, "MNIST")
|
||||
|
||||
fc1 = FcLayer_1_0(num_input, num_hidden1, params)
|
||||
net.add_layer(fc1, "fc1")
|
||||
r1 = ActivationLayer(Relu())
|
||||
net.add_layer(r1, "r1")
|
||||
......
|
||||
fc5 = FcLayer_1_0(num_hidden4, num_output, params)
|
||||
net.add_layer(fc5, "fc5")
|
||||
softmax = ClassificationLayer(Softmax())
|
||||
net.add_layer(softmax, "softmax")
|
||||
|
||||
net.train(dataReader, checkpoint=0.05, need_test=True)
|
||||
net.ShowLossHistory(xcoord=XCoordinate.Iteration)
|
||||
```
|
||||
|
||||
### 14.6.3 Results of execution
|
||||
|
||||
The stop condition we chose is when the absolute loss value reaches 0.12. When the process reaches 6 epochs, the loss value of 0.119 is reached, and the training stops.
|
||||
|
||||
<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images/Images/14/mnist_loss.png" />
|
||||
|
||||
Figure 14-19 Changes in loss function value and accuracy during training
|
||||
|
||||
Figure 14-19 is an illustration of the training process. Below is the output of the last few lines.
|
||||
|
||||
```
|
||||
......
|
||||
epoch=6, total_iteration=5763
|
||||
loss_train=0.005559, accuracy_train=1.000000
|
||||
loss_valid=0.119701, accuracy_valid=0.971667
|
||||
time used: 17.500738859176636
|
||||
save parameters
|
||||
testing...
|
||||
0.9697
|
||||
```
|
||||
|
||||
Finally, the accuracy rate obtained with the test set is 96.97%.
|
||||
|
||||
### Code Location
|
||||
|
||||
ch14, Level6
|
||||
@@ -0,0 +1,20 @@
|
||||
<!--Copyright © Microsoft Corporation. All rights reserved.
|
||||
Applicable to [License](https://github.com/Microsoft/ai-edu/blob/master/LICENSE.md) copyright license-->
|
||||
|
||||
# Chapter 15 Network Optimization
|
||||
|
||||
As networks become more complex, training becomes more and more difficult, and the time it takes becomes longer and longer. The reasons may be:
|
||||
|
||||
- Many parameters
|
||||
- Large amounts of data
|
||||
- The gradient disappears
|
||||
- The loss function has a gentle slope
|
||||
|
||||
In order to solve the problems listed above, scientists have studied the performance of networks in depth and found methods which can more or less improve the training of DNNs:
|
||||
|
||||
- Weight matrix initialization
|
||||
- Batch normalization
|
||||
- Gradient descent optimization algorithm
|
||||
- Adaptive learning rate algorithm
|
||||
|
||||
Of course, there are more methods which are not listed here, so only these common methods are given for your reference.
|
||||
@@ -0,0 +1,190 @@
|
||||
<!--Copyright © Microsoft Corporation. All rights reserved.
|
||||
Applicable to [License](https://github.com/Microsoft/ai-edu/blob/master/LICENSE.md) copyright license-->
|
||||
|
||||
## 15.1 Weight matrix initialization
|
||||
|
||||
The initialization of the weight matrix is a very important concept, and it is the first step in training the neural network. Choosing the correct initialization method will bring more results with less effort. This is like climbing the Himalayas. If you choose to climb from the southern slope, it will be much easier than from the northern slope. The initialization of the weight matrix is equivalent to choosing a different path when going down the mountain. You don't know the difficulty of this path before you choose, but know that it can reach the bottom of the mountain. This choice is random. Even if you use the correct initialization algorithm, each reinitialization will have a large impact on the training results.
|
||||
|
||||
For example, the weight value obtained during the first initialization is (0.12847, 0.36453), and the second initialization obtained (0.23334, 0.24352). After experimentation, the first initialization took 3000 iterations to reach a model with an accuracy of 96%. It only took 2000 iterations for the other initialization to reach the same accuracy. This exercise is common in practice.
|
||||
|
||||
### 15.1.1 Zero initialization
|
||||
|
||||
That is, the initial value of `W` in all layers is set to 0.
|
||||
|
||||
$$
|
||||
W = 0
|
||||
$$
|
||||
|
||||
But for multi-layer networks, it must not be initialized with zero. Otherwise, the weight value cannot learn and become a reasonable result. Look at the following printout of a zero-initialized weight matrix's values:
|
||||
```
|
||||
W1= [[-0.82452497 -0.82452497 -0.82452497]]
|
||||
B1= [[-0.01143752 -0.01143752 -0.01143752]]
|
||||
W2= [[-0.68583865]
|
||||
[-0.68583865]
|
||||
[-0.68583865]]
|
||||
B2= [[0.68359678]]
|
||||
```
|
||||
|
||||
The values of the three elements inside `W1`, `B1`, and `W2` are all the same. This is because the initial values are all 0. So, the gradient is returned uniformly, causing all the values of `W` to be updated synchronously. In this case, no matter how many rounds, the final result will not be correct.
|
||||
|
||||
### 15.1.2 Standard initialization
|
||||
|
||||
The standard/normal initialization method guarantees that the input mean of the activation function is 0 and the variance is 1. Initialize `W` according to the following formula:
|
||||
|
||||
$$
|
||||
W \sim N \begin{bmatrix} 0, 1 \end{bmatrix}
|
||||
$$
|
||||
|
||||
`W` is the weight matrix, `N` represents the Gaussian distribution. The Gaussian distribution is also called the normal distribution, so this initialization is also called normal initialization in some places.
|
||||
|
||||
The details of initialization are generally determined according to the number of inputs and outputs of the fully connected layer:
|
||||
|
||||
$$
|
||||
W \sim N
|
||||
\begin{pmatrix}
|
||||
0, \frac{1}{\sqrt{n_{in}}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
$$
|
||||
W \sim U
|
||||
\begin{pmatrix}
|
||||
-\frac{1}{\sqrt{n_{in}}}, \frac{1}{\sqrt{n_{in}}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
When the target problem is relatively simple, the network depth is not large, so standard initialization is sufficient. However, when using DNNs, you will encounter the problem shown in Figure 15-1.
|
||||
|
||||
<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images/Images/15/init_normal_sigmoid.png" ch="500" />
|
||||
|
||||
Figure 15-1 The performance of standard initialization on the Sigmoid activation function
|
||||
|
||||
Figure 15-1 is a 6-layer DNN, using a fully connected layer and Sigmoid activation function. The figure shows the histogram of the activation function of each layer. Note that the activation values of each layer are close to [0,1] on both sides. From the curve of the Sigmoid function, the derivative of these values is close to 0, and the gradient of the back propagation gradually disappears. The value in the middle section is relatively small, which is not favorable for parameter learning.
|
||||
|
||||
### 15.1.3 Xavier initialization method
|
||||
|
||||
Based on the above observations, Xavier Glorot et al. developed the following Xavier$^{[1]}$ initialization method.
|
||||
|
||||
Condition: During forward propagation, the variance of the activation value remains unchanged; during back propagation, the variance of the gradient of the state value remains unchanged.
|
||||
|
||||
$$
|
||||
W \sim N
|
||||
\begin{pmatrix}
|
||||
0, \sqrt{\frac{2}{n_{in} + n_{out}}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
$$
|
||||
W \sim U
|
||||
\begin{pmatrix}
|
||||
-\sqrt{\frac{6}{n_{in} + n_{out}}}, \sqrt{\frac{6}{n_{in} + n_{out}}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
`W` is the weight matrix, `N` represents the normal distribution, and `U` represents the uniform distribution.
|
||||
|
||||
It is assumed that the activation function is symmetrical about 0 and is mainly aimed at fully connected neural networks - suitable for tanh and softsign.
|
||||
|
||||
That is, the weight matrix parameters should satisfy the uniform distribution in this domain.
|
||||
|
||||
Abstract of the paper: The neural network can't work as desired before 2006, and the main reason is the weight matrix initialization method. The Sigmoid function is not suitable for deep learning because it will cause gradient saturation. Based on the above reasons, we propose a parameter initialization method that can converge quickly.
|
||||
|
||||
The Xavier initialization method has the advantage of initializing W directly with Gaussian distribution:
|
||||
|
||||
In general neural networks, the variance of neuron output values will continue to increase during forward propagation, and the use of Xavier and other methods can theoretically ensure that the input and output variances of each layer of neurons are consistent.
|
||||
|
||||
Figure 15-2 shows the performance of the network with a depth of 6 layers. Note that the distribution of the activation function output values of the following layers conform to the normal distribution, which is conducive to the learning of the neural network.
|
||||
|
||||
<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images/Images/15/init_xavier_sigmoid.png" ch="500" />
|
||||
|
||||
Figure 15-2 The performance of Xavier initialization on the Sigmoid activation function
|
||||
|
||||
Table 15-1 Comparison of the activation values of each layer of random initialization and Xavier initialization with the back propagation gradient
|
||||
|
||||
| |Activation value of each layer|Back propagation gradient of each layer|
|
||||
|---|---|---|
|
||||
| Random initialization|<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images\Images\15\forward_activation1.png"><br/>The activation value distribution gradually concentrates|<img src= "https://aiedugithub4a2.blob.core.windows.net/a2-images\Images\15\backward_activation1.png"><br/>The strength of backpropagation declines layer by layer|
|
||||
| Xavier initialization|<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images\Images\15\forward_activation2.png"><br/>The activation value is evenly distributed|<img src=" https://aiedugithub4a2.blob.core.windows.net/a2-images\Images\15\backward_activation2.png"><br/>The strength of backpropagation remains unchanged|
|
||||
|
||||
However, with the advancement of deep learning, people felt that the reverse power of Sigmoid was limited. So, the ReLU activation function was invented. Figure 15-3 shows the performance of Xavier initialization on the ReLU activation function.
|
||||
|
||||
<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images/Images/15/init_xavier_relu.png" ch="500" />
|
||||
|
||||
Figure 15-3 The performance of Xavier initialization on the ReLU activation function
|
||||
|
||||
Note that as the layer deepens, the activation value is gradually biased towards 0 when using ReLU, which will also cause the problem of gradient disappearance. So He Kaiming et al. developed the MSRA initialization method, also called He initialization method.
|
||||
|
||||
### 15.1.4 MSRA initialization method
|
||||
|
||||
The MSRA initialization method $^{[2]}$ is also called the He method because the author's last name is He.
|
||||
|
||||
Condition: During forward propagation, the variance of the state value remains unchanged; during back propagation, the variance of the gradient of the activation value remains unchanged.
|
||||
|
||||
Network initialization is a very important thing. However, the traditional initialization of Gaussian distribution with fixed variance makes it difficult for the model to converge when the network becomes deeper. The VGG team handled the initialization problem in this way: they first trained an 8-layer network, and then used this network to initialize a deeper network.
|
||||
|
||||
"Xavier" is a relatively good initialization method. However, when Xavier derived it, it is assumed that the activation function is linear near the zero point. Obviously, ReLU and PReLU that we currently use do not have this condition. Therefore, with MSRA initialization, the variance will change after using the ReLU activation function, so the method of initializing the weights should also be changed.
|
||||
|
||||
When only considering the number of inputs, the MSRA initialization is a Gaussian distribution with a mean value of 0 and a variance of 2/n, which is suitable for the ReLU activation function:
|
||||
|
||||
$$
|
||||
W \sim N
|
||||
\begin{pmatrix}
|
||||
0, \sqrt{\frac{2}{n}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
$$
|
||||
W \sim U
|
||||
\begin{pmatrix}
|
||||
-\sqrt{\frac{6}{n_{in}}}, \sqrt{\frac{6}{n_{out}}}
|
||||
\end{pmatrix}
|
||||
$$
|
||||
|
||||
The distribution of activation values from 0 to 1 in Figure 15-4 is very uniform in each layer, and the gradient will not disappear due to the deepening of the layer. Therefore, when using ReLU, it is recommended to use the MSRA method for initialization.
|
||||
|
||||
<img src="https://aiedugithub4a2.blob.core.windows.net/a2-images/Images/15/init_msra_relu.png" ch="500" />
|
||||
|
||||
Figure 15-4 The performance of MSRA initialization on the ReLU activation function
|
||||
|
||||
For Leaky ReLU:
|
||||
|
||||
$$
|
||||
W \sim N \begin{bmatrix} 0, \sqrt{\frac{2}{(1+\alpha^2) \hat n_i}} \end{bmatrix}
|
||||
\\\\ \hat n_i = h_i \cdot w_i \cdot d_i
|
||||
\\\\ h_i: height of convolution kernel, w_i: width of convolution kernel, d_i: number of convolution kernel
|
||||
$$
|
||||
|
||||
### 15.1.5 Summary
|
||||
|
||||
Table 15-2 Application scenarios of several initialization methods
|
||||
|
||||
|ID|Network Depth|Initialization Method|Activation Function|Description|
|
||||
|---|---|---|---|---|
|
||||
|1|Single layer|Zero initialization|None|Yes|
|
||||
|2|Double Layer|Zero Initialization|Sigmoid|Error, unable to carry out correct backpropagation|
|
||||
|3|Double Layer|Random Initialization|Sigmoid|Yes|
|
||||
|4|Multilayer|Random initialization|Sigmoid|The activation value is distributed in a concave shape, which is not conducive to back propagation|
|
||||
|5|Multilayer|Xavier initialization|Tanh|Correct|
|
||||
|6|Multilayer|Xavier initialization|ReLU|The activation value distribution is biased towards 0, which is not conducive to back propagation|
|
||||
|7|Multilayer|MSRA Initialization|ReLU|Correct|
|
||||
|
||||
From Table 15-2, due to changes in network depth and activation functions, people continue to study new initialization methods to adapt and finally get combinations 1, 3, 5, and 7.
|
||||
|
||||
### Code location
|
||||
|
||||
ch15, Level1
|
||||
|
||||
### Thinking and Practice
|
||||
|
||||
1. When there are multiple layers, initialization cannot be with zeros. But if all the values of the weight matrix are initialized to 0.1, is it okay?
|
||||
2. Use the example in 14.6 to compare the training effects of Xavier and MSRA initialization.
|
||||
|
||||
### Reference
|
||||
|
||||
[1] Understanding the difficulty of training deep feedforward neural networks. link: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
|
||||
|
||||
by Xavier Glorot, Yoshua Bengio in AISTATS 2010.
|
||||
|
||||
[This is the Chinese version] (https://blog.csdn.net/victoriaw/article/details/73000632), thanks to the translator.
|
||||
|
||||
[2] He Kaiming, Microsoft Research Asia, 2015. https://arxiv.org/pdf/1502.01852.pdf
|
||||
Reference in New Issue
Block a user