Another day, another component. Or some of them, in a bundle. Math libraries.

3.1.4. created the "magic math libraries" that rebuild themselves if a FPU becomes available later in the livetime of the system, .e.g. through setpatch, though there was still something odd about them: The rounding convention.

The math libraries (singbas, doubbas and the corresponding "trans" libs) all follow a "round towards zero" strategy in software, and similarly, set the FPU - if present - to round in the same direction. This gives consistent results, though it is - as far as the math is concerned - not ideal as it will loose precision.

3.2 changes that and follows a strict "round to even" rounding policy, which is a better and more precise rounding mode. It means that it tries to round to the nearest number if rounding is required, and in case of a tie, it rounds to the next number with the LSB set as zero. It can then be shown that under such conditions the output of the computation is identical to the output that had been obtained by computing with infinite precision, followed by round-to-even, which is a nice result. IOWs, you cannot possibly round better.

There is one exception, and that is the "convert float to int" function of the math libraries. They continue to round to zero. There are two reasons for that: First, because the C standard says so, and second, because some programs depend on it, for example the (really naive) floating point to text conversion (i.e. "printf %g") of the Aztec compiler.

Now, it seems easy just to configure the FPU such that it follows a different rounding convention. While the math libraries certainly do that, all the CPU only implementation *also* has to change, and this was a tremendous task. So every function in the "bas" libraries was touched, and the code to adjust the digits for rounding was modified to implement a proper "round to even" strategy. Multiply, add, subtract, division, square root.

The libraries were then verified for correctness by the "Paranoia test", which is a numerical unit test that checks for corner cases of the implementation, and checks for proper rounding. Again, running a test sounds harmless, but bear in mind that the test is in C, is compiled with SAS/C, and SAS/C does not support IEEE single precision numbers. So, the first step was to get the test working on the ieeesingbas library, and then to get the libraries working correctly, which took another two weeks of work.

Final result now: While the old libraries returned an "acceptable result" with some "numerical flaws" due to improper rounding, the new libraries pass all numerical tests with "excellent". So, no numerical flaws left in the math department, and the CPU output is identical to the FPU output.

mathffp (math.library and mathtrans.library) remain in the sad state they have been before. Unfortunately, there is little we can fix here because the number format is just broken. There are no infinities, there are no NaN ("not a number"), there is no gradual underflow (no "denormalized numbers"), so for any kind of numerics, stay away from this math library.

The only reason why it is still in ROM despite its sad state is that some software may depend on it early on booting (still to be available), but mathffp is one of my prime candidates to be thrown out in the future. It's bad numerical quality, beyond fixing. The code quality is fine - hand optimized assembler, coming directly from Motorola - but that doesn't help if it doesn't compute right.