keras重载继续训练的问题

colab时间有限。中断后，应重新连接，加载模型应继续训练。问题是，每次重新加载模型后，训练开始loss比中断前多loss大大的，训练几个batch后，loss它会慢慢下降。

重新加载的代码有问题，模型优化器状态重新初始化。

• 定义和编译模型
• 加载权重（load_weights）
• 训练
• 保存模型和权重
• 连接中断
• 重新连接，回到第一步

参考stackoverflow,

When to use? If you’re using compile, surely it must be after load_model(). After all, you need a model to compile. (PS: load_model automatically compiles the model with the optimizer that was saved along with the model) What does compile do? Compile defines the loss function, the optimizer and the metrics. That’s all. It has nothing to do with the weights and you can compile a model as many times as you want without causing any problem to pretrained weights. You need a compiled model to train (because training uses the loss function and the optimizer). But it’s not necessary to compile a model for predicting. Do you need to use compile more than once? Only if:

• You want to change one of these: - Loss function - Optimizer / Learning rate - Metrics - The trainable property of some layer
• You loaded (or created) a model that is not compiled yet. Or your load/save method didn’t consider the previous compilation.

Consequences of compiling again: If you compile a model again, you will lose the optimizer states. This means that your training will suffer a little at the beginning until it adjusts the learning rate, the momentums, etc. But there is absolutely no damage to the weights (unless, of course, your initial learning rate is so big that the first training step wildly changes the fine tuned weights).

错误是：每次重新编译模型，然后加载权重。模型优化器的初始状态发生了变化。

• 定义模型(第一次训练需要编译模型)
• 加载整个模型（keras.models.load_model）
• 训练
• 保存模型和权重
• 连接中断
• 重新连接，回到第一步

这个问题比较另类，只是自己记录。

模型定义中的代码，

def r_square(y_true, y_pred):  from keras import backend as K  SS_res =  K.sum(K.square(y_true - y_pred))   SS_tot = K.sum(K.square(y_true - K.mean(y_true)))   return (1 - SS_res/(SS_tot   K.epsilon()))  model.compile(loss = "mse", optimizer = opt, metrics = [r_square]) 

加载模型报错，

ValueError: Unknown metric function: r_square. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.

原因是使用了自定义的metric函数，参看stackoverflow，和tensorflow官方文档 需要使用custome_objects参数，解决方法如下，

model = keras.models.load_model(model_file, custom_objects={ 
        "r_square":r_square})