Pytorch training进阶

Pytorch training进阶知识

模型保存/读取(单卡)

保存有两种方式

#保存整个模型
torch.save(model,save_dir)
#保存模型权重
torch.save(model.state_dict,save_dir)

读取方式

## 加载权重
load_dict = torch.load(svae_dir)
model = Mymodel()
model.state_dict = load_dict
## 在进行正常训练或者validation

修改模型

改某层

import torchvision.models as models
net = models.resnet50()
##resnet last layer:fc(2048,1000)
##修改时直接覆盖原有层
classifier = nn.Sequential(OrderedDict([('fc1', nn.Linear(2048, 128)),
                          ('relu1', nn.ReLU()), 
                          ('dropout1',nn.Dropout(0.5)),
                          ('fc2', nn.Linear(128, 10)),
                          ('output', nn.Softmax(dim=1))
                          ]))
    
net.fc = classifier
##到此为止就将resnet改成了一个图片分类器

添加额外输入,以及额外输出

class NT(nn.Module):
    def __init__(self,net):
        self.net = net()
        self.fc1 = nn.Linear(1000,10)
        self.a = nn.softmax(dim=1)
    def forward(self,x,addtion):
		x1 = self.net(x)
        x2 = self.fc1(x1)
        x3 = self.a(x2)
        return x1,x3
net = models.resnet50()
model = NT(net)

自定义层

## 方式一
net = nn.Sequential(
        nn.Linear(784, 256),
        nn.ReLU(),
        nn.Linear(256, 10), 
        )
##带名字
import collections
net2 = nn.Sequential(collections.OrderedDict([
          ('fc1', nn.Linear(784, 256)),
          ('relu1', nn.ReLU()),
          ('fc2', nn.Linear(256, 10))
          ]))
##model_list列表
net = nn.ModuleList([nn.Linear(784, 256), nn.ReLU()])
net.appen()
##model_dict字典
net = nn.ModuleDict({
    'linear': nn.Linear(784, 256),
    'act': nn.ReLU(),
})
net['output'] = nn.Linear(256, 10)

损失函数自定

def my_loss(output,target):
	loss = torch.mean((output - target)**2)
    return loss

个人习惯写在model中(直接调用),有label返回输出和loss,无label返回预测值

大名鼎鼎“Scheduler”

神经网络中,学习率的调整往往比较重要。

Scheduler就能根据迭代次数来调整L_R,点击查看官方文档

###代码------代码###
optimizer = torch.optim.Adam(lr=config.l_r)
scheduler = torch.optim.lr_scheduler.LambdaLR
##training
for epoch in range(config.num_epoches):
    optimizer.zero_grad()
    ....
    ....
    loss.backward()
    optimizer.step()
    scheduler.step()

finetune

加载pytorch中附带的计算机视觉模型。如果将pretrained=True,就会自动下载模型权重,并且加载。然后你再来直接使用或者微调。

import torchvision.models as models
resnet18 = models.resnet18()
# resnet18 = models.resnet18(pretrained=False)  等价于与上面的表达式
alexnet = models.alexnet()
vgg16 = models.vgg16()
squeezenet = models.squeezenet1_0()
densenet = models.densenet161()
inception = models.inception_v3()
googlenet = models.googlenet()
shufflenet = models.shufflenet_v2_x1_0()
mobilenet_v2 = models.mobilenet_v2()
mobilenet_v3_large = models.mobilenet_v3_large()
mobilenet_v3_small = models.mobilenet_v3_small()
resnext50_32x4d = models.resnext50_32x4d()
wide_resnet50_2 = models.wide_resnet50_2()
mnasnet = models.mnasnet1_0()

训练特定层

###冻结model里面的神经网络层
def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False
import torchvision.models as models###计算机视觉预训练库还有timm
# 冻结参数的梯度
feature_extract = True
model = models.resnet18(pretrained=True)
set_parameter_requires_grad(model, feature_extract)
# 修改模型
num_ftrs = model.fc.in_features##获取hidden_size
model.fc = nn.Linear(in_features=num_ftrs, out_features=4, bias=True)

半精度训练

使用有限的GPU创造更多的价值

Pytorch默认的浮点数存储格式为float32,绝大多数场景不需要,所以往往float16就够了。

from torch.cuda.amp import autocast
@autocast()
def forwoard(self,x):
    ...
    return x

training time

for i in train_loader:
    x = x.to(device)
    with autocast():
        output = model(x)
    .....

参数类

大佬们写参数定义往往喜欢用argprase来保存参数,个人比较懒,喜欢这么写

class CFG():
    num_epoch = 10
    data_dir = './input/data_train'
    .........
    lr = 1e-1

写完了直接在model,trainloop里面调用,不用传参了(传来传去很麻烦)

本图文内容来源于网友网络收集整理提供,作为学习参考使用,版权属于原作者。
THE END
分享
二维码
< <上一篇
下一篇>>