1.激活函数

1.1 Sigmoid / Logistic

δ

(

x

)

=

1

1

+

e

x

δ

(

x

)

=

δ

(

1

δ

)

delta(x)=frac{1}{1+e^{-x}}\delta'(x)=delta(1-delta)

δ(x)=1+ex1δ(x)=δ(1δ)

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.sigmoid(x)
plt.plot(x,y)
plt.show()


1.2 Tanh

t

a

n

h

(

x

)

=

e

x

e

x

e

x

+

e

x

t

a

n

h

(

x

)

x

=

1

t

a

n

h

2

(

x

)

tanh(x)=frac{e^x-e^{-x}}{e^x+e^{-x}}\frac{partial tanh(x)}{partial x}=1-tanh^2(x)

tanh(x)=ex+exexexxtanh(x)=1tanh2(x)

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.tanh(x)
plt.plot(x,y)
plt.show()


1.3 ReLU

f

(

x

)

=

m

a

x

(

0

,

x

)

f(x)=max(0,x)

f(x)=max(0,x)

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.relu(x)
plt.plot(x,y)
plt.show()


1.4 Softmax

p

i

=

e

a

i

k

=

1

N

e

a

k

p

i

a

j

=

{

p

i

(

1

p

j

)

i

=

j

p

i

p

j

i

j

p_i=frac{e^{a_i}}{sum_{k=1}^N{e^{a_k}}}\ frac{partial p_i}{partial a_j}=left{ begin{array}{lc} p_i(1-p_j) & i=j \ -p_ip_j&ineq j\ end{array} right.

pi=k=1Neakeaiajpi={pi(1pj)pipji=ji=j

import torch.nn.functional as F
logits = torch.rand(10)
prob = F.softmax(logits,dim=0)
print(prob)

tensor([0.1024, 0.0617, 0.1133, 0.1544, 0.1184, 0.0735, 0.0590, 0.1036, 0.0861,
0.1275])


2.损失函数

2.1 MSE

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
mse = F.mse_loss(y,x@w)
print(mse)

tensor(238.5115)


import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,10)
y = torch.randint(0,9,[100])
entropy = F.cross_entropy(x@w,y)
print(entropy)

tensor(3.6413)


3. 求导和反向传播

3.1 求导

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
mse = F.mse_loss(x@w,y)

torch.Size([64, 1])


3.2 反向传播

• Tensor.backward()
import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,10)
y = torch.randint(0,9,[100,])
entropy = F.cross_entropy(x@w,y)
entropy.backward()

torch.Size([64, 10])


by CyrusMay 2022 06 28

——————五月天（因为你 所以我）——————

THE END