tensflow2.0系列-反向训练参数问题

在2.0中进行TNN训练反向设置参数时，由于是自定义训练步骤，且每步输入的batch_size = 1 的数据时，在参数设计神经网络中不出现tf.keras.layers.BatchNormalization()时训练效果会更好，如下图所示

1
def inverse_network(input_dimension, model_name, filling_up_limit, filling_low_limit, p_c_up_limit, p_c_low_limit):
2
    inputs = tf.keras.Input(shape=(input_dimension, ))
3

4
    x = tf.keras.layers.Dense(16)(inputs)
5
    # x = tf.keras.layers.BatchNormalization()(x)
6
    x = tf.keras.layers.Activation('tanh')(x)
7

8
    # f_output = tf.keras.layers.Dense(1)(x)
9
    # filling = tf.keras.layers.BatchNormalization()(f_output)
10
    filling = tf.keras.layers.Dense(1)(x)
11
    filling_concrete = ((tf.math.sin(filling)+1)/2*
12
                        (filling_up_limit - filling_low_limit) + filling_low_limit)
13
    filling_soil = 1 - filling_concrete
14
    filling_rate = tf.concat([filling_concrete, filling_soil], axis=1)
15

16
    # p_c_output = tf.keras.layers.Dense(1)(x)
17
    # p_c = tf.keras.layers.BatchNormalization()(p_c_output)
18
    p_c = tf.keras.layers.Dense(1)(x)
19
    constant = (tf.math.sin(p_c)+1)/2*(p_c_up_limit - p_c_low_limit) + p_c_low_limit
20

21
    return tf.keras.Model(
22
        inputs=inputs,
23
        outputs=[filling_rate, constant],
24
        name=model_name
25
    )

并且在训练时应直接在步骤中输出神经网络的输出，不应再此调用inverse_net.predict(input_bg)否者相当于再次训练了一轮。如下所示：：

1
    for i in range(bandgaps.shape[0]):
2
        optimizer = tf.keras.optimizers.Adam(1e-3)
3
        input_bg = bandgaps[i:i+1, :]
4
        input_pm = parameters[i:i+1, :]
5
        flag = False
6
        for epoch in range(epochs):
7
            loss, grads,  filling_rate, constant = train_step(input_bg, inverse_net, model, bg_maxs, bg_mins, soil_maxs, soil_mins,
8
                              filling_rate_maxs, filling_rate_mins, input_pm)
9
            optimizer.apply_gradients(zip(grads, inverse_net.trainable_variables))
10
            loss = float(loss.numpy())
11
            if epoch > 100 and loss < 1e-2:
12
                flag =True
13
                print(f"Sample {i:02d} | Epochs: {epoch + 1:04d} | Loss: {np.round(loss, 6)}")
14
                break
15

16
        # filling_rate, constant = inverse_net.predict(input_bg)
17
        d_filling_rate[i] = filling_rate
18
        d_p_c[i] = constant
19
        d_soil[i] = input_pm
20
        d_bg[i] = input_bg
21
        d_loss[i] = loss

这样设置后效果会提升很多：

带tf.keras.layers.BatchNormalization()的效果：

1
1/1 [==============================] - 0s 83ms/step
2
Sample 00 | Epochs: 2000 | Loss: 5.281638e+02
3
1/1 [==============================] - 0s 64ms/step
4
Sample 01 | Epochs: 2000 | Loss: 3.181857e+02
5
1/1 [==============================] - 0s 77ms/step
6
Sample 02 | Epochs: 2000 | Loss: 4.975657e-01
7
1/1 [==============================] - 0s 65ms/step
8
Sample 03 | Epochs: 2000 | Loss: 4.303170e+02
9
Sample 04 | Epochs: 1524 | Loss: 0.009939
10
1/1 [==============================] - 0s 67ms/step
11
WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x00000205D7B2B0D0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
12
1/1 [==============================] - 0s 75ms/step
13
Sample 05 | Epochs: 2000 | Loss: 5.356166e+02
14
WARNING:tensorflow:6 out of the last 6 calls to <function Model.make_predict_function.<locals>.predict_function at 0x00000205D7E7EC10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
15
1/1 [==============================] - 0s 65ms/step
16
Sample 06 | Epochs: 2000 | Loss: 2.695414e+02
17
Sample 07 | Epochs: 1119 | Loss: 0.009881

不带tf.keras.layers.BatchNormalization()的效果：

1
Sample 00 | Epochs: 2000 | Loss: 5.281638e+02
2
Sample 01 | Epochs: 2000 | Loss: 3.181929e+02
3
Sample 02 | Epochs: 2000 | Loss: 3.027072e-01
4
Sample 03 | Epochs: 2000 | Loss: 7.938098e-01
5
Sample 04 | Epochs: 0303 | Loss: 0.009693
6
Sample 05 | Epochs: 0263 | Loss: 0.00985
7
Sample 06 | Epochs: 0256 | Loss: 0.009734
8
Sample 07 | Epochs: 0210 | Loss: 0.009648
9
Sample 08 | Epochs: 0126 | Loss: 0.006857
10
Sample 09 | Epochs: 0129 | Loss: 0.007794
11
Sample 10 | Epochs: 0334 | Loss: 0.009692
12
Sample 11 | Epochs: 0102 | Loss: 0.002452
13
Sample 12 | Epochs: 0274 | Loss: 0.009767

ISAJ