roundoff error

views updated

roundoff error The error caused by truncating numbers in a calculation, usually necessitated because registers in a computer can only hold numbers of a fixed length, say t binary digits. Arithmetical operations on such numbers often give results requiring more than t digits for their representation, which must then be reduced to t digits for further calculation. The nearest t-digit approximation may be used (rounding) or digits after the tth may be dropped (chopping or truncation). The repeated reduction to t digits can cause systematic buildup of error in certain types of calculation.

More From encyclopedia.com