# 朴素贝叶斯法

x

x

x,利用贝叶斯定理求出后验概率最大的输出

y

y

y,朴素贝叶斯实现简单，学习和预测效率都很高，是一种常用的方法。

# 朴素贝叶斯的学习与分类

## 基本方法

x

R

n

x subseteq R^n

xRn为n维度向量的集合，输出空间为类标记集合

y

y

y = {

c

1

,

c

2

,

.

.

.

,

c

k

c_1,c_2,...,c_k

c1,c2,...,ck}。

x

X

x in X

xX,输出为类标记

y

Y

y in Y

yY

X

X

X是定义在输入空间

X

X

X上的随机向量，

Y

Y

Y是定义在输入空间

y

y

y上的随机变量，

P

(

X

,

Y

)

P(X,Y)

P(X,Y)

X

X

X

Y

Y

Y的联合概率分布，训练数据集为

T

T

T = {

(

x

1

,

y

1

)

,

(

x

2

,

y

2

)

,

.

.

.

.

,

(

x

N

,

y

N

)

(x_1,y_1),(x_2,y_2),....,(x_N,y_N)

(x1,y1),(x2,y2),....,(xN,yN)}

P

(

X

,

Y

)

P(X,Y)

P(X,Y)独立同分布产生。

P

(

X

,

Y

)

P(X,Y)

P(X,Y),具体地，学习以下先验概率分布及条件概率分布，先验概率分布：

P

(

Y

=

c

k

)

k

=

1

,

2

,

3

,

.

.

.

.

,

K

P(Y = c_k) k = 1,2,3,....,K

P(Y=ck)k=1,2,3,....,K

P

(

X

=

x

Y

=

c

k

)

=

P

(

X

(

1

)

=

x

(

1

)

,

,

X

(

n

)

=

x

(

n

)

Y

=

c

k

)

P(X = x|Y = c_k) \ = P(X^{(1)} = x^{(1)},cdots,X^{(n)} = x^{(n)}|Y = c_k)

P(X=xY=ck)=P(X(1)=x(1),,X(n)=x(n)Y=ck)

k

=

1

,

2

,

3

,

.

.

.

,

K

k = 1,2,3,...,K

k=1,2,3,...,K

P

(

X

,

Y

)

P(X,Y)

P(X,Y)

P

(

X

=

x

Y

=

c

k

)

P(X = x|Y = c_k)

P(X=xY=ck) 有指数级数量的参数，其估计实际是不可行的，事实上，假设

x

(

j

)

x^{(j)}

x(j)可能取值有

S

j

S_j

Sj个，

j

=

1

,

2

,

3...

,

n

j = 1,2,3...,n

j=1,2,3...,n
Y可能取值有K个，那么总共的参数个数有：

K

j

=

1

n

S

j

K prod^n_{j = 1}S_j

Kj=1nSj

P

(

X

=

x

Y

=

c

k

)

=

P

(

X

(

1

)

=

x

(

1

)

,

,

X

(

n

)

=

x

(

n

)

Y

=

c

k

)

=

j

=

1

n

P

(

X

(

j

)

=

x

(

j

)

Y

=

c

k

)

P(X = x|Y = c_k) \ = P(X^{(1)} = x^{(1)},cdots,X^{(n)} = x^{(n)}|Y = c_k) \ = prod^n_{j = 1}P(X^{(j)} = x^{(j)}|Y = c_k)

P(X=xY=ck)=P(X(1)=x(1),,X(n)=x(n)Y=ck)=j=1nP(X(j)=x(j)Y=ck)

x

x

x,通过学习的模型计算后验概率分布，

P

(

Y

=

c

k

X

=

x

)

P(Y = c_k|X = x)

P(Y=ckX=x)

x

x

x类的输出，后验概率计算根据贝叶斯定理进行：

P

(

Y

=

c

k

X

=

x

)

=

P

(

X

=

x

Y

=

c

k

)

P

(

Y

=

c

k

)

k

P

(

X

=

x

Y

=

c

k

)

P

(

Y

=

c

k

)

P(Y = c_k|X = x) = frac{P(X = x|Y = c_k)P(Y = c_k)}{sum_kP(X = x|Y = c_k)P(Y =c_k)}

P(Y=ckX=x)=kP(X=xY=ck)P(Y=ck)P(X=xY=ck)P(Y=ck)

P

(

Y

=

c

k

X

=

x

)

=

P

(

Y

=

c

k

)

j

P

(

X

(

j

)

=

x

(

j

)

Y

=

c

k

)

k

P

(

Y

=

c

k

)

j

P

(

X

(

j

)

=

x

(

j

)

Y

=

c

k

)

P(Y = c_k|X = x) = frac{P(Y =c_k)prod_jP(X^{(j)} = x^{(j)}| Y = c_k)}{sum_kP(Y = c_k)prod_jP(X^{(j)} =x^{(j)} | Y = c_k)}

P(Y=ckX=x)=kP(Y=ck)jP(X(j)=x(j)Y=ck)P(Y=ck)jP(X(j)=x(j)Y=ck)

y

=

f

(

x

)

=

a

r

g

m

a

x

c

k

P

(

Y

=

c

k

)

j

P

(

X

(

j

)

=

x

(

j

)

Y

=

c

k

)

k

P

(

Y

=

c

k

)

j

P

(

X

(

j

)

=

x

(

j

)

Y

=

c

k

)

y = f(x) = argmax_{c_k}frac{P(Y =c_k)prod_jP(X^{(j)} = x^{(j)}| Y = c_k)}{sum_kP(Y = c_k)prod_jP(X^{(j)} =x^{(j)} | Y = c_k)}

y=f(x)=argmaxckkP(Y=ck)jP(X(j)=x(j)Y=ck)P(Y=ck)jP(X(j)=x(j)Y=ck)

c

k

c_k

ck都是相同的。

y

=

a

r

g

m

a

x

c

k

P

(

Y

=

c

k

)

j

P

(

X

(

j

)

=

x

(

j

)

Y

=

c

k

)

y = argmax_{c_k}P(Y = c_k)prod_jP(X^{(j)} = x^{(j)}|Y = c_k)

y=argmaxckP(Y=ck)jP(X(j)=x(j)Y=ck)

# 后验概率最大化含义

L

(

Y

,

f

(

X

)

)

=

{

1

Y

f

(

X

)

0

Y

=

f

(

X

)

L(Y,f(X))=left{begin{matrix}1，Y not = f(X)\0，Y =f(X)end{matrix}right.

L(Y,f(X))={1Y=f(X)0Y=f(X)

f

(

X

)

f(X)

f(X)是分类决策函数，这是期望风险函数为：

R

e

x

p

(

f

)

=

E

[

L

(

Y

,

f

(

X

)

)

]

R_{exp}(f) = E[L(Y,f(X))]

Rexp(f)=E[L(Y,f(X))]

P

(

X

,

Y

)

P(X,Y)

P(X,Y)取的，由此取条件期望：

R

e

x

p

(

f

)

=

E

x

k

=

1

K

[

L

(

c

k

,

f

(

X

)

)

]

P

(

c

k

X

)

R_{exp}(f) = E_xsum^K_{k = 1}[L(c_k,f(X))]P(c_k|X)

Rexp(f)=Exk=1K[L(ck,f(X))]P(ckX)

f

(

x

)

=

a

r

g

m

i

n

y

y

k

=

1

k

L

(

c

k

,

y

)

P

(

c

k

X

=

x

)

=

a

r

g

m

i

n

y

y

k

=

1

K

P

(

y

c

k

X

=

x

)

=

a

r

g

m

i

n

y

y

(

1

P

(

y

=

c

k

X

=

x

)

)

=

a

r

g

m

a

x

y

y

P

(

y

c

k

X

=

x

)

f(x) = argmin_{y in y}sum^k_{k = 1}L(c_k,y)P(c_k|X = x) \= argmin_{y in y}sum^K_{k = 1}P(y not=c_k|X = x) \= argmin_{y in y} (1 - P(y = c_k|X = x)) \= argmax_{yin y}P(y_{c_k}|X = x)

f(x)=argminyyk=1kL(ck,y)P(ckX=x)=argminyyk=1KP(y=ckX=x)=argminyy(1P(y=ckX=x))=argmaxyyP(yckX=x)

f

(

x

)

=

a

r

g

m

a

x

c

k

P

(

c

k

X

=

x

)

f(x) = argmax_{c_k}P(c_k|X = x)

f(x)=argmaxckP(ckX=x)

# 朴素贝叶斯参数估计

## 极大似然估计

P

(

Y

=

c

k

)

P(Y = c_k)

P(Y=ck)

P

(

X

(

j

)

=

x

(

j

)

Y

=

c

k

)

P(X^{(j)} = x^{(j)} | Y = c_k)

P(X(j)=x(j)Y=ck).可以应用极大似然估计法去估计相应的概率：

P

(

Y

=

c

k

)

P(Y = c_k)

P(Y=ck)的极大似然估计是：

P

(

Y

=

c

k

)

=

i

=

1

N

I

(

y

i

=

c

k

)

N

k

=

1

,

2

,

3...

,

K

P(Y = c_k) = frac{sum^N_{i = 1}I(y_i = c_k)}{N} k = 1,2,3...,K

P(Y=ck)=Ni=1NI(yi=ck)k=1,2,3...,K

j

j

j个特征，

x

(

j

)

x^{(j)}

x(j)可能取值集合为{

a

j

1

,

a

j

2

,

.

.

.

.

,

a

j

S

j

a_{j1},a_{j2},....,a_{jS_j}

aj1,aj2,....,ajSj} 条件概率

P

(

X

(

j

)

=

a

j

l

Y

=

c

k

)

P(X^{(j)} = a_{jl} | Y = c_k)

P(X(j)=ajlY=ck)的极大似然估计是：

P

(

X

(

j

)

=

a

j

l

Y

=

c

k

)

=

i

=

1

N

I

(

x

i

(

j

)

=

a

j

l

,

y

i

=

c

k

)

i

=

1

N

I

(

y

i

=

c

k

)

j

=

1

,

2

,

.

.

.

,

n

;

l

=

1

,

2

,

.

.

.

.

,

S

j

k

=

1

,

2

,

3....

,

K

P(X^{(j)} = a_{jl} | Y = c_k) =frac{sum^N_{i = 1}I(x^{(j)}_i =a_{jl},y_i = c_k)}{sum^N_{i = 1}I(y_i = c_k)} \ j = 1,2,...,n; l = 1,2,....,S_j k = 1,2,3....,K

P(X(j)=ajlY=ck)=i=1NI(yi=ck)i=1NI(xi(j)=ajl,yi=ck)j=1,2,...,n;l=1,2,....,Sjk=1,2,3....,K

x

i

j

i

j

a

j

l

j

l

I

x^{j}_i第i个样本和第j个特征，\ a_{jl} 第j个特征可能取第l个值，I为指示函数

xijijajljlI

# 条件概率公式

P

(

Y

X

)

=

P

(

X

,

Y

)

P

(

X

)

P(Y|X) = frac{P(X,Y)}{P(X)}

P(YX)=P(X)P(X,Y)

## 如何求解联合概率分布/

P

(

X

,

Y

)

=

P

(

X

Y

)

P

(

Y

)

=

P

(

x

1

,

x

2

,

.

.

.

,

x

n

Y

)

P

(

Y

)

P(X,Y) = P(X|Y)*P(Y) = P(x_1,x_2,...,x_n|Y)*P(Y)

P(X,Y)=P(XY)P(Y)=P(x1,x2,...,xnY)P(Y)

## 变量独立性假设

P

(

x

1

,

x

2

,

.

.

.

.

,

x

n

Y

)

=

i

=

1

n

P

(

x

i

Y

)

P(x_1,x_2,....,x_n|Y) = prod^n_{i = 1}P(x_i|Y)

P(x1,x2,....,xnY)=i=1nP(xiY)

THE END

)">