Calculating Absolute Error

Error Measures

• The absolute error between x and e is denoted by ∥x - e∥ := √(xk - ek)^2, where k=0 to n-1
• The relative error of x to x ≠ 0 is defined by ∥x - e∥ / ∥x∥
• The number of significant digits in x̃ is the number of leading digits that are correct relative to the true value x
• x̃ has m significant digits with respect to x if |x - x̃| ≤ 5 · 10^(-m-1)

Machine Representation of Numbers

• A base β ≥ 2 is a positive integer, and Zβ = {0, 1,..., β - 1} are the digits
• The β-positional representation of x ∈ R is xβ = (−1)^s (xn xn-1 ... x1 x0.x-1 x-2 ... x-m)β
• Examples of number systems include binary (β = 2, Z2 = {0, 1}), decimal (β = 10, Z10 = {0, 1,..., 9}), and hexadecimal (β = 16, Z16 = {0, 1,..., 9, A, B, C, D, E, F})

Fixed-Point Representation

• The set of real numbers with at most D total number of digits in β-positional representation is denoted by Rβ,D
• Zβ,D is the set of integers in Rβ,D with at most D total number of digits
• A fixed-point number system is defined by FI(β, N, k) := {(−1)^s (xN-2 ... xk.xk-1 ... x0)β : s ∈ {0, 1}, {xk}N-2 k=0 ⊂ Zβ}
• Theorem 2 states that |FI(β, N, k)| = 2^β^(N-1) - 1

Floating-Point Representation

• The IEEE 754 single precision format is FL(β = 2, N = 32, p = 23, emin = -126, emax = 127)
• The IEEE 754 double precision format is FL(β = 2, N = 64, p = 52, emin = -1022, emax = 1023)
• The IEEE 754 quadruple precision format is FL(β = 2, N = 128, p = 112, emin = -16382, emax = 16383)
• IEEE 754 representations of transcendental numbers π and e are shown in 64-bit format