Deep Learning Study Note : 4 Numerical Computation

*original text : Deep Learning,  Ian Goodfellow Yoshua Bengio Aaron Courville, 2016
http://www.deeplearningbook.org/



Chapter 4 Numerical Computaion

4.1 Overflow and Underflow
we incur some approximation error when we represent the number in the computer(rounding error).

1) underflow
when numbers near zero are rounded to zero
2) overflow
when numbers with large magnitude are approximated as ∞ or −∞

example : softmax function
when all of the xi are equal to some constant c.
Analytically, outpus is 1/n
If c is very negative, exp(c) will underflow => undefined
If c is very positive, exp(c) will overflow => undefined

Both of these difficulties can be resolved by instead evaluating softmax(z) where z = x − maxi xi.
Subtracting maxi xi, the largest argument to exp being 0 => rules out overflow
at least one term in the denominator has a value of 1 => rules out underflow