15.2.2 Term weighting

Next: 15.3 Term Distribution Model Up: 15.2 The Vector Space Previous: 15.2.1 Vector similarity

vector space $B$N3FMWAG$r$I$N$h$&$JCM$K$9$l$P$$$$$N$+(B?
3$B$D$N(B frequency (Table 15.3)
- term frequency $\rightarrow$ t_i,j (d_j$BCf$N(Bw_i$B$N?t(B)
- document frequency $\rightarrow$ df_i (w_i$B$,4^$^$l$k(B document$B?t(B)
- collection frequency $\rightarrow$ cf_i (w_i$B$NAm?t(B)
$\begin{eqnarray*}$\sum_{j} tf_{i,j} = cf_i$ \end{eqnarray*}$
$B4K>W4X?t(B
$B?M4V$N463P$O(B log $B$K6a$$(B, $B=EMWEY$O(B $B4K>W4X?t(B f(tf) $B$GI=8=$5$l$k$3(B $B$H$,B?$$(B

$\begin{eqnarray*}f(tf) &=& \sqrt{tf}\\ f(tf) &=& 1+\log(tf) \end{eqnarray*}$
document frequency
- $B0UL#E*$K=EMW$J8l(B(keyword)$B$O(B1$B$D$N(B document $B=8Cf$7$d$9$$(B $\rightarrow$ df_i $B>.(B
- $B0UL#E*$K=EMW$G$J$$8l(B(function word $B$J$I(B)$B$O6Q0l$KJ,I[(B $\rightarrow$ df_i $BBg(B
- Table 15.4 $\rightarrow$ insurance $B$N$[$&$,(B try $B$h$j=EMW8l(B
tf idf
- inversed document frequency (idf)
  $idf = \log(\frac{N}{df_i})$ , (N $B$O(B document $B?t(B)
  w_i $B$,$9$Y$F$N(B 1 $B$D$N(B document $B$K4^$^$l$k(B $\rightarrow$ $idf = \log N$
  w_i $B$,$9$Y$F$N(B document $B$K4^$^$l$k(B $\rightarrow$ idf =0
- term frequency $B$H(B document frequency $B$rAH$_9g$o$;$k(B
  
  $\begin{eqnarray*}weigth(i,j) = \left\{ \begin{array}{rll} (1+\log(tf_{i.j}))... ...\geq 1\\ 0 & &if\ tf_{i,j} = 0 \end{array} \right. \nonumber \end{eqnarray*}$
$B$$$m$$$m$J(B tf $\cdot$ idf $\rightarrow$ Table 15.5

1999-08-03