Development updates

[4pLaY]

AmigaOS 3.2 is currently in development and there is no set release date. Here I will, with the permission of Thomas Richter, post some of his personal updates on the development.

[4pLaY]

Time to start with a series of posts describing the changes for 3.2. Exec is first.

There is not so much about exec, really. 3.1.4 brought one new bug into exec, namely the Alert AN_BadFreeAdr, which was triggered on a bad memory release, forgot to restore one register and hence caused another crash afterwards. This was fixed by SetPatch of 3.1.4.1, and now the fix was migrated into the code.

There is another series of fixes, though of old bugs. Alert() in general had a series of problems. First, it did not save all registers. Thus, in case ROMWack was entered later on, the register file was partially overwritten and thus made a post-mortem debug with ROMWack rather pointless. Also, ROMWack tried to detect the CPU type - by the same function exec uses during bootstrap. This sounds fine, but it is customary these days to move the vector base register out of the zero-page, and this is something the mentioned CPU-detection algorithm did not expect - it rather moved it back. Sometimes with fatal side effects. The code is now more careful in not touching VBR if it does not have to. The problem of the lost registers must have been introduced somewhere in the v37 series because the 1.2 kickstart did better.

Unfortunately, this is not sufficient to make ROMWack happy because there is another way how to enter it - namely through the dos.library "Alert" requester. ("Software error - task held..."). Unfortunately, this requester *also* overwrote the registers, and it neither saved the stackframe of the crash. The latter is, however, important to be able to debug a crash with the ROMWack. Thus, the dos.library exception vector - aka the error requester - was improved by saving all registers, and by also making a copy of the exception stack frame. If the user presses "cancel", the stack frame and the registers are restored, and the code enters the default exec error handler, which again points by default to the ROMWack.

Thus, ROMWack should have been become a lot more useful than before as it has access to the important registers and the stack frame causing the problem.
Or, use a debugger in first place...

There is much more to be said about the dos.library, but this is for a later episode.

[4pLaY]

More progress: Expansion.

There were still some interesting issues in expansion. In case a RAM-board was unsized and requested auto-sizing by the Os, the Os overwrote the last byte behind the indicated slot size and thus might have trashed another expansion sitting behind it. This was still an old bug from 3.x times of which I do not know when exactly it was made.

Another old bug was the A3000 superkickstart support. What this does is that the code attempts to check whether the MMU is active, and maps the upper 512K RAM to a kickstart image. If so, the A3000 expansion skips this memory region, and does not insert it into the memory list. Or at least, it attempted to do so. Due to a logic error, expansion tested the wrong bit in the MMU configuration register and thus tested the bit whether there is a supervisor MMU table present, and only performed the mapping *if not so*. Instead, it should have tested the MMU-enable bit, and perform the superkickstart logic only if this bit is *set* (not clear). This stole 512K for A3000 users using the mmu.library which does enable the MMU, and uses "supervisor" MMU tables. The 3.1.4.1 SetPatch installs a workaround to merge the upper 512K into the memory list in such cases, though 3.2 will fix the problem at its root. Again, an old bug that survived.

Then, another issue with file systems. Expansion configures file systems with 600 bytes of stack, which is a bit short. This increases now to 1024 bytes. Actually, it is more complicated that this: The stack size in the DosList structure is measured in bytes for C-style handlers, and in longwords for BCPL-style handlers. Thus, if expansion leaves there a "256", this is fine for BCPL style handlers (256*4=1024), but way, way, way too short for C-style handlers (256 bytes!). This problem was addressed in the dos.library and system-startup, though.

[4pLaY]

The next ROM component: Utility.

The utility.library contains several utility functions for tag lists, arithmetics, ID generation and so on. We haven't found any old or new bug in it, but extended its capabilities. One of the things that happened is that the long division functions (32/32 division) for the 68K processor were not very adequate as they used a rather slow "egyptian" division algorithm that generates its results by shifting and subtraction. Well, it works, but it is slow, so it was replaced by a better performing "Algorithm D" implementation by Knuth.

It is also weird that we do have a 32x32->64 multiplication, but no 64/32 division function in utility. The latter is, in fact, required by multiple Os components that handle large disk drives, so two functions for that were added, a signed and a unsigned quad-word division. For the 68020 to 68040, it uses the long division of the processors, for the 68000, a long Algorithm D implementation is used, and for the 68060, a similar algorithm D implementation as well as this processor lacks a 64 divide.

Utility also includes a set of new string functions. Strncpy() which is a length-limited string copy receiving a target buffer and a target buffer size. Unlike its ANSI counterpart strncpy, it always NUL-terminates its destination, and returns in case of success the pointer to the terminating NUL character for easy string concatenation, or NULL for buffer overflow, with a partial string then having been copied and terminated.

Strncat() concatenates strings into a length-limited buffer, using the same type of interface, and Snprintf() is a string-formatting function into a lenght-limited buffer on top of the exec RawDoFmt() function, quite similar to ANSI snprintf(), just that it returns the required buffer size, not the size of the result string. This helps to prevent the usual "off-by-one" error one typically runs into by confusing buffer sizes and string lengths.

[4pLaY]

More components: dos.

The dos.library was actually Tripos, an operating system of its own until CBM bought it, and integrated it into AmigaOs. For Os 2.0, the BCPL source was completely replaced by an assembler/C implementation that came from "arp", the "Amiga Replacement Project", available independently for Kickstart 1.3.

For 3.2, we made more modifications to this antique core. First, we have a new dos.library function that acts as a "hook" whenever the dos.library attempts to bring up the "Please insert..." requester. An external program can hook in there, and can improve the requester, or offer more choices. In fact, Os 3.2 ships with such a program, namely "AssignWedge". The AssignWedge offers the functionality to also mount a handler, or deny a request for a specific volume from that point on. The "Assign" command can then be used to remove such a permanent denial.

Then we have an increased stack size for built-in handlers. In 3.1.4 and before, SetPatch took care of increasing their stack, but the defaults have never been touched. In 3.2, these defaults grew. It is, however, no longer the dos.library that mounts them, see below

The System() call received new tags: SYS_Error specifies an error output stream, which is mirrored by the CreateNewProc tags NP_Error and NP_CloseError that never worked before. This error ouptut stream defaults to the standard output if not defined, but may be given to redirect error messages to a separate stream. The shell offered support for error redirection since ever by means of the "*>" redirection token, but it wasn't very useful since many commands did not use it.

This changes for 3.2: First, the dos.library internal function PrintFault() uses it now, and we have also new calls to receive it (ErrorOutput()), to write a string through it (PutErrStr()) or to change it (SelectError()). All internal commands and commands in C: have been extended to use this new error stream, allowing you to separate error from regular output. Note well: If there is no error output redirection, the error output goes into the standard output for compatibility reasons, but if "*>" is explicitly given, stderr goes to some separate file.

Then, we have SYS_CmdStream as a new tag for System(), which tells the call that instead of taking the commands to be executed from a file rather than its first argument. This sounds fairly harmless, but provides an important new feature. Before that, let me also say that SYS_Asynch now also works for System() in case the command string is empty, and thus if SYS_CmdStream is used instead.

Now, what is all the fuzz about it? With SYS_CMdStream = Open("S:Shell-Startpup"), SYS_Output=Open("CON:") and SYS_ASync=TRUE, you can now create a new CLI, similar to "NewShell". Actually, the "NewShell" command and SYS:System/CLI do now exactly that, and the archaic and undocumented method by which NewShell used to operate is opsolete and has been removed. One piece of BCPL junk less, yay!

This said, the whole interface towards the shell has been simplified. We used to have three (actually, four!) methods how a shell can be launched: One at system startup, one when the user used the "Run" command or the "System()" dos.library function, and one for "NewShell", and the shell had to fiddle out what the user wanted to do, and then had to call into three also mostly or completely undocumented functions, CliInit(), CliInitNewCli() and CliInitRun() to get initialized.

This is of course utterly complex and error prone. Now that "NewCli" goes through "System()", "CliInitNewCli()" is obsolete and no longer needed. And, due to another change, "CliInit()" is also obsolete. I come to that in a minute.

So, what happens if the system boots up: Well, the dos.library is initialized, then the dos.library used to initalize the shell, then starts the shell, the shell is supposed to understand what the system wants from it through an archaic and mysterious interface whose documenation is hard to find and to digest, then calls into CliInit(). CliInit() then mounts all the handlers, opens the startup-sequence, and returns to the shell, which then runs the startup-sequence. Thus, this used to be like "ping-poing":

Dos->(CreateProc())->Shell->(CliInit()->)Dos->Shell

Yummie. All this changed in 3.2. The system startup procedure which was part of the mysterious and undocumented CliInit() function was removed from the dos.library whatsoever, and moved into a separate system component, "System-Startup". It resides in ROM, or in L: for those that go for a software-only upgrade, and is loaded there by LoadModule. So the procedure looks like this now:

Dos->(CreateProc())-System-Startup->Shell.

No ping-pong. What System-Startup does is what CliInit() used to do, and a lot more. So it mounts all the handlers, all the volumes, creates the default assigns like S:, DEVS:, L: and C:, and then creates the boot shell, by the mechanism discussed above: System(), with SYS_CMDStream=Open("S:Startup-Sequence") and SYS_OUtput=Open("CON:...").

Thus, CliInitNewCli() done for good, CliInit() done for good, and the only init mechanism the shell is left with is CliInitRun(), everything else was removed. Thus, no longer any guesswork as what to call, and a very simple mechanism to get the shell going.

We have a couple of other improvements as well:
- ExAll() had a bug by forgetting to pass an argument to a potential error requester and thus could crash or show nonsense on errors.

- All the BCPL inherited I/O functions such as FWrite or FRead were terribly slow. They made single-byte-I/O, going through three function layers to get data, and thus delayed all operations down that depended on them, such as the icon.library which uses FRead to read icons. FRead() and FWrite() have been rewritten to make fast block I/O, and "burst" directly into the user buffer if possible without going through an internal buffer, hopefully also speeding up the icon.library and many other system components that use them.

- System() ignored the NP_Name tag. Instead, a new process created by it was always named "Background CLI". Now you can name it as you like. For example, "Initial CLI", the name used by System-Startup (obviously!).

- System() had more problems in erraneously releasing file handles in case of failure. The interface specification say that the file handles remain open and untouched in case of failure, but did not always do so. This might have caused hard to trace bugs in low-memory situations.

- There was some confusion about the stack size of file handlers. Unfortunately, the default stack size of a handler is measured in bytes for C-style handlers, and in long words for BCPL-style handlers, thus caused some mismatch if the dos.library adjusted them. In fact, the 3.1 SetPatch increased the stack size of the internal handlers to something like 16K instead of 4K due to this mismatch in units. We fixed that.

- Speaking of stack sizes, handlers may now contain a magic "$STACK:" token that indicates a minimum stack size they want to have. If such a stack token is present, the default stack size is increased to this minimum size. It is always measured in bytes, regardless of the handler type. Thus, there is no longer any guessing necessary as what to put into the "Stack" mount line as the handler can communicate this now itself.

- WFPrintf() has some issues with invalid templates for its arguments.

- Speaking about the ROMWack, I already mentioned in exec that the exec default exception handler preserves now registers for the system debugger. Unfortunately, the dos.library default Alert requester ("Software error, Task held...") did not. Now, 3.2 will fix that and the Alert requester will store the stack frame away and restore it when the system debugger kicks in. Thus, post-mortem debugging with ROMWack should finally work again. This was broken somewhere on the road from Kickstart 1.2 to 1.3 (yes, really that old!).

- Mounting BCPL handlers leaked 20 bytes of memory. Another fix of an ancient bug.

- Finally, the CLI structure was extended. It now contains also the history. The new ExtendedCommandLineInterface structure is created by AllocDosObject(), and you better use that function to create CLIs as otherwise the "history" command of the shell does not work. Of course, all system functions use it, and as of 3.2, ARexx was updated to go through AllcoDosObject() instead of rolling its own CLI. As said, legacy code that does not will continue to work, but then "history" will print an error as the CLI is unextended. The shell will continue to work, though. I haven't observed this case in the wild, yet, as most programs create CLIs through either System() or "NewCLI" and both are of course fine.

There is more that needs to be said about System-Startup as this new system module does now a lot more than CliInit(), but I leave this for a later episode. This one is long enough.

[4pLaY]

Another episode: System-Startup

This is a new ROM component. It can be roughly compared to the init process on Unix machines as it is the first process created by the dos.library. Formerly, the dos.library created the shell, and the shell had to fiddle out what to do with a rather mysterious and undocumented process, and then had to call CliInit(). Now, System-Startup is created right away, and everything that was in CliInit() is now part of System-Startup. At the end of its doing, System-Startup creates the "Initial CLI" and thus boots the system.

So what does Sytem-Startup do?

It first creates and initializes intuition (but see below) and places its library vector in the dos.library and the GlobVec (yuck, we still have it). It then initializes some structures of the dos.library and creates the initial task table (the list of CLIs, none at this point), and then creates the device nodes from the mounted devices that left boot nodes in expansion, i.e. those expansion boards the system can boot from.

Then, it initializes the filesystem and places it in the dos.library. Here is the first difference: If a newer file system has been loaded from RDB, it will be used instead and replace the ROM file system. Thus, if you have an FFS 47 on harddisk, FFS 47 will also be used for your floppy.

It then creaets mount entries for PRT:, PAR: and SER: which go to "L:Port-Handler", as ever. As a new activity, it *also* creates a mount entry for "PIPE:" which goes to "L:Queue-Handler", so you don't have to mount the queue handler manually and can use the "|" (pipe symbol) already in  the initial CLI without mounting PIPE: first.

It then starts the file system of the boot volume and creates all the standard system assigns, L:, FONTS:; DEVS:, LIBS:, S:, C:, SYS: and ENVARC:, and then starts all other volumes that have been recorded in expansion.

After that, it mounts CON:, the con-handler, and creates the mount entries for CON: and RAW:. A new feature is that it checks whether the boot volume or any other volume contains a newer version of L:con-handler, and if so, uses that instead. Thus, the con-handler for the initial CLI may come from disk.

The same goes for RAM: and RAM-Handler, which may also reside on disk as a newer version and then replaces the ROM version, and for the Shell, which is either taken from ROM, or from L:Shell-Seg if one of the volumes mounted so far contains a newer version of it.

For the shell, it then creates the resident-entries "Shell", "BootShell" and "CLI", corresponding to the default user shell with C entry, the boot shell C entry, and the BCPL entry of the shell. They all point to the same shell at this point.

Then, the system checks for available updates on disks, which is a new feature. Thus, for every ROM module not yet initialized System-Startup now checks whether there is a corresponding module on disk that could replace it because it has a newer version. I.e. "DEVS:audio.device" would replace the ROM-based audio.device if it is newer.

For intuition, we have to use a special trick because it is already initialized. 3.2 will come with intuition 47 in ROM. For those that need to use CyberGfx, System-Startup checks the boot volume - and only there - for intuition v40 in LIBS:, or a newer version of intuition everywhere else. If a newer intuition or intuition v40 is found, the exiting copy of intuition is expunged from memory - intuition V47 can be made shutdown, a new feature - and a new intuition is build and installed.

If the boot monitor was DBLPAL or DBLNTSC, System-Startup "enlightens" now the system to use AGA modes, such that the boot shell can also use these new resolutions. This was done by SetPatch before.

Finally, all remaining ROM modules flagged as "AutoInit" are initialized.

The code then opens the CON: window for the boot shell, and S:Startup-Sequence. If flagged in the boot menu, System-Startup initiates another new module, "syslog". I will talk about "syslog" later.  Last but not least, it creates the boot shell with System(), the new functionality I talked about yesterday:

Code: [Select]

SystemTags(NULL,       
      SYS_Input,consolestream, /* CON:///-1/AmigaDOS/Auto/NoClose/Smart */
      SYS_Output,NULL,
      SYS_CmdStream,cmdstream, /* :S/Startup-Sequence */
      SYS_Asynch,TRUE,    /* this is the trick */
      SYS_UserShell,TRUE, /* use "Shell" as shell */
      NP_Name,"Initial CLI",
      NP_CopyVars,TRUE,
      TAG_DONE);


and thus boots the system, and then dies away. The above "System()" call is the 3.2 equivalent of "newcli", just less mysterious.

[4pLaY]

So, another component: The shell.

The major news is that the shell has now a history, and TAB expansion, a feature long waited for. By "the shell has a history" I mean exactly that: The history function moved from CON: into the shell. In fact, what happens here is that the shell places the console into a new console mode within which the line editing of CON: remains unaltered, just if TAB or any history related keys are pressed, the console reports to the shell by means of a special CSI (control) sequence that includes the search pattern and the key. It is then up to the shell to parse the pattern, scan the path or its history, and bring it back to the screen. The latter works by the already existing ACTION_FORCE packet of the console that was documented before, but did not work quite right in 3.1 according to its specifications as it should insert data into the keyboard buffer. For 3.1, it inserted data into the output buffer, which was not quite intended.

For the records, ACTION_FORCE comes from the WShell, and the new console mode comes from ViNCEd, so ViNCEd supports the new shell right away along with the v47 CON: handler.

The TAB expansion and history functions can be turned off by a shell variable, "set simpleshell on" resets the shell back to 3.1, but then you do not have any history at all because CON: doesn't have it anymore.

Then we have a couple of bug fixes as well: The $? operator that checks for the presence of a variable only worked for environment variables, but not for local variables.

We also have now "&&" as a new shell operator which sequences commands, but does not concatenate outputs. That is, || is a "concatenating pipe", and "&&" is sequential execution of commands.

As the dos.library has now a stderr stream, error output redirection with *> makes now sense, and all the shell internal commands and the commands in C: have been reworked to make use of it.

If you had two variables near each other such as $foo$foo, the shell failed to expand the second variable if there was no space between them. This was due to a mixup in the parser state machine that has been fixed.

The Execute command became a shell internal. There is no longer the need to make it resident to allow recursive execution as it is already part of the shell. An old version of execute as stand-alone will ship on the Install3.2: disk.

The input stream of shell scripts can now be redirected with "<filename" as well, i.e. if "S:foo" is a script "S:foo <bla" pipes the file "bla" into the stdin of the commands in S:foo. This never worked before. In fact, input and output redirection of scripts never worked before.

Input redirection with << did not work for some shell commands due to a mixup in the BCPL buffered IO.

The shell accepts a new shell variable,  "debug". If "debug" is set to "on", the shell prints all the commands it executes through the serial port, at 9600 8N1, for debugging purposes. Usually, however, then the new kickstart component "syslog" sits there and picks them up, and redirects them to RAM: so you can see what happened. There will be more to say about "syslog" and the new boot menu in another episode.

[4pLaY]

More components: The bootmenu.

Actually, we haven't found any new or old bugs here, just added features. The "boot options" page gets now three new checkbox gadgets. We already had "Disable CPU Caches" as workaround feature for some broken software.

A new gadget is "Update ROM Modules", which is normally checked. If this is not checked, System-Startup will not look for new modules and will operate with the ones in ROM. This may help in case the system modules got somewhat trashed.

We also have "Trace startup-sequence". If this button is checked, the "Initial CLI" gets the "Interactive" variable set to "on", so you can trace through the startup-sequence and check every command executed.

Finally, "Enable System log". This loads and initiates a new module, "syslog", which either sits in ROM, or in LIBS:modules/syslog. If it is enabled, it grabs all data going through the serial port and redirects it to RAM:syslog. It also sets the Shell variable "debug" to "on". If so, the shell will log each executed command through syslog. Thus, you will be able to see what the shell is actually executing, and the debug output of any other program that would normally go through the serial port.

[4pLaY]

Next episode: trackdisk.device.

Actually, one probably wouldn't expect much news from such an old device, but yet there is. The trackdisk.device suffered from two bugs which remained undetected in 3.1.4.

One is that with a properly crafted sector layout, the trackdisk device will overwrite memory beyond the track buffer because it does not check the sector index fully.

Another is that the device uses a busy-loop to time the step motor frequency, though this loop is not throttled by any type of hardware register access. This means that on fast hardware, floppy seeking becomes unreliable because the stepping frequency becomes too high. Now the loop is throttled by a hardware register access, so the timing should be (more) reliable.

[4pLaY]

Another component: The con-handler.

The con-handler is providing the window the shell runs in, but also other applications can open a CON: and request text lines, or - as a variant - keyboard presses from RAW:, the "non-cooked" version of CON:

There is surprisingly much about CON:, its improvements and defects. Probably some bugs first:

*) ACTION_WAIT_CHAR aka WaitForChar() was buggy, if several tasks inititated ACTION_WAIT_FOR_CHAR simultaneously, the handler might have done strange things. It neither counted the number of lines correctly to make ARExx happy.
*) Errors returning from the output device, i.e. typically the console.device, were not forwarded to the caller.
*) The parser of the incoming CSI sequences from the console.device improved. Now cursor movements by more than one position are supported - and also necessary as part of ACTION_FORCE and the tab expansion. ESC[ was not handled as substitute for CSI, though it is necessary to do so for 7-bit connections (see also below).
*) ACTION_STACK and ACTION_QUEUE (both for ARexx) had some issues not collecting data correctly. ACTION_FORCE was implemented outright incorrectly and did not offload its data into the keyboard input buffer, but the output buffer.
*) CON: forgot to re-enable the close-window event when switching from RAW mode to CON mode, thus leaving the user sit with a function-less close gadget.
*) ACTION_DISK_INFO disabled the auto-ability of the window without a chance to re-enable it. CON: now implements ACTION_UNDISK_INFO, which is already in use by some programs (Ed, More, Dir and some more).
*) An Off-by-one error may have read uninitialized memory when pasting into the console.
*) Any pending read request was not answered when closing the window, potentially causing a crash.
*) The con window is now opened in "new style" colors. While it makes typically no difference, it implies that the "Ed" window is now black on white, not yellow on black.
*) ACTION_GET_VARS was never documented, and allowed access to the (ever changing) global variables of CON: This is an interface nightmare that was just dropped for sanity.
*) ACTION_REPLACE, ACTION_PEEK, ACTION_SIZE_HIST, ACTION_SET_HIST and ACTION_GET_HIST were documented, but never implemented. As their purpose was unclear, their support was dropped.
*) Actually, the entire history functionality was dropped. This is done by the shell in a much better way with a much saner interface.
*) The console supports now ViNCEd's "medium mode" which forwards history requests and TAB expansions as special CSI sequences to the shell. Which makes use of it. So we finally have TAB expansion in the console, by the right party, which is the shell.
*) The con-handler had a very strange bug where, if you left the cursor sitting at the start of a physical line that is part of a long logical line going through more than one physical line, the console.device broke the line implicitly. Thus, when resizing the window, the cursor was not positioned correctly and the logical line was split. This was very hairy to debug and to fix as it depends on some pecularities of the console.device.
*) If concurrent input and output was happening, results may have been strange as the con-handler attempts to buffer them, and delay its execution, though not always with the correct end result as input and output cursor/replace/insertion activities are not commutatitive and cannot be interchanged freely.
*) The con-handler was extended not only to operate on top of the console.device, but any other device as well. As such, it replaces now the AUX-Handler completely. In fact, all of AUX what is left is a small stub function that initiates the con-handler with the right parameters. Which implies that you can get TAB-Expansion on a shell running over a serial line. Yay!
*) A forward slash in the window title can now be escaped by a backwards slash, and the backwards slash by a backwards slash itself. This also allows complex window titles.
*) The console window can now also optionally be equipped with an iconification gadget, all provided conclip is running to take care of the events. This allows iconification of shell windows.
*) Also, the console window allows dropping icons into it which are then expanded to the corresponding file names as if typed on the keyboard, including escaping of special characters. Again, this requires the help of conclip.

So, we are moving towards a much smarter console. Unfortunately, there is not yet a scroll gadget because this would require support from the console.device - which I'm currently not trying to touch. But this will come, sooner or later.

[4pLaY]

Another new component: The ram-handler

The most important new feature of the v47 ram-handler are "external links", i.e. links from the ram disk to outside file systems that are resolved as soon as a file or directory is accessed. You would typically use this to point "RAM:ENV" to "ENVARC:" such that any file that is accessed in ENV: is automatically copied from ENVARC: to ENV: when needed. There is no need for HappyEnv or any other dedicated env-handler as RAM: does it all itself.

There are some improvements in memory management, though. Regularly sized blocks (i.e. full size) are taken directly from exec instead from the pool as there is no benefit going through the indirection. RAM is, however, now smarter saving space by only allocating what is needed for the file. Whereas the previous versions of RAM: allocated always entire blocks, except for very small files, the v47 version ensures that the last block of a file does not take an entire block, but only the fraction that is needed.

Finally, we had one interesting new/old bug, in (sigh) ExNext() (how unsurprising). If you deleted a file pointed to by fib_Key in the FileInfoBlock, and the memory taken for this file was overwritten or released, ExNext() on the (then stale) key could have caused a MuGA hit. The hit itself was harmless, but the code that caused the hit was non-sensical, so was repaired and fixed. That was not a new bug, but an old one that might have been there all the way from the ancient days.

This said, did I mention that ExNext() is misdesigned?

[4pLaY]

Next component: The scsi.device.

This device is one of the "bad surprises" of the operating system over and over again. It started as an SCSI device for the A3000, and then was drilled up to support IDE, then again ATAPI, so the code quality is... quite moderate, to put it this way. It consists of many files that are conditionally used depending on the compiler option and the build target, and it is not easy to read and follow.

"Of course" this caused problems during development, and the infamous "maxtransfer" bug where data losses were caused if more than 255 sectors are transported in one go was, actually, in multiple places, not just one. One of them was a higher level bug we fixed, namely at the level where 'trackdisk-like' read and write commands are translated into 'scsi-like' commands. But what we forgot is that there is also *the very same bug again* on a lower level where the scsi-level commands are translated to IDE commands. Thus, if the command was not broken up into 255 sector pairs at 'trackdisk level', for example because the user selected "SCSI direct" options, then the same bug appeared again at the 'scsi to IDE' transfer level. Yuck.

So I hope we have get this bugger done for good.

There are a couple of other minor changes. For example 'TD_SEEK' was not implemented correctly. Actually, I wonder why anyone would want to issue this command - 'move the read/write head to the indicated sector' - on a harddisk. But, at least, the harddisk should not attempt to write to the surface if it is issued. This was an old bug that got fixed.

Then, the code used its own implementation for the "Gary IDE detection logic", but code for that is already present in the exec.library, and the code there also depends on the motherboard model by a compiler switch. Thus, instead of repeating the whole logic again in the scsi.device, the function now falls back to the exec.library.

Finally, some words of warning:

Concerning maxtransfer: While the scsi.device breaks now sector counts > 255 down into transfers of 255 bytes each, it *seems* that there are at least some buggy implementations of IDE around - typical flash chips - that consider the byte count as "signed byte" due to an oversight, and that do stupid things for counts > 127. So for *some arrangements*, users still have to use a maxtransfer value, though then of 0x0000ffe0. This is generally not needed, but only for buggy drives beyond our control.

And another warning: Some people still use IDEFix. First, this is unnecessary as the scsi.device implements ATAPI now fine, so a second level patch is not necessary. Second, IDEfix does not issue LBA48 commands as the scsi.device does, and hence limits the maximum drive capacity to (IIRC) 128MB. Any larger drive will appear in the HDToolBox only as 128MB drive. Then people come along, partition their drive/card under UAE, and insert it into the Amiga, and then all hell breaks loose because IDEfix cannot address any blocks beyond the 128MB limit, even though the partition appears to be the right size.

Solution is ideally not to use idefix, or simply "partition the drive on the system you want to use it on" because you then become aware of the limits of the software you depend on. Well, hopefully...

[4pLaY]

Another component: Gadtools.

Gadtools is a rather simple GUI toolset that can be used to build buttons and menus. As far as bugs are concerned, gadtools had a defect when including a boopsi as replacement for an image in a menu to be constructed and created nonsense if you tried. This had been fixed, boopsis are now acceptable for menu items.

DrawBevelBox() supports now a new box type, or rather forwards a new box type to a new flavour of the intuition frameiclass. This frame type is a double-lined black/white frame with an optional text on top called a "context frame". It is exactly the type of frame the preferences editors draw around contextual gadgets. The code has been factored out from the preferences editors and put into intuition as it is used so often, and made accessible by a simple call through gadtools.

The most important change is that gadtools allows now building of font-sensitive GUIs. A special flag in the "NewGadget" structure instructs gadtools that gadget coordinates and positions are relative to a font that is recorded in the VisualInfo, and which can also be changed there. If this flag is active, the coordinate/sizes are given in quarters of the font width and height, with an additional -8..+7 pixel offset in the low order bits. Gadtools then automatically scales the coordinates according to the font dimension. As result, all system GUIs such as preferences, Format, DiskCopy and so on scale now according to the system default font and are no longer bound to topaz.8 as they used to be.

It is a very simple method and only works well for fixed-width fonts. Otherwise, gadtools would have to go through a complete window layout engine that moves gadgets around for an optimal look. We have that, too, it is part of the layout.class of reaction. While the latter GUI layout is certainly much more flexible and allows arbitrary fonts, it is also a lot slower and causes a lot of stress for the lower-end machines, which is one of the reasons why we did not pick it for the 3.2 system programs at this stage. So pick your poison: Simple, with some restrictions, or flexible, but slower.

Anyhow, developers have now freedom: Either choose the simple gadtools "fixed width fonts raster based" gadtools layout, or the flexible reaction "layout.class" engine.

[4pLaY]

Next component: The FastFileSystem.

There is not really much to say as we haven't found any bugs. There are two slight improvements:

First, the FFS stores now all the bitmap blocks contiguous instead of interleaved with the bitmap list blocks. This has the (potential) advantage that a future FFS implementation may read them in one single go. We currently read them one by another. That's ok, but not quite as fast as we could. Still, it's faster than the previous version of the FFS as the code no longer allocates the memory for each block individually, reads it, and then releases it again, so AllocMem() and FreeMem() were stressed quite a lot. Instead, the FFS makes one allocation, and then keeps it.

Second, the FFS supports now ACTION_DIE, a new dos packet, that indicates to the file system that it should shut down if possible, releasing all its resources. It remains mounted, but with no handler task attached. This is one of the core ingredients to the new "Mounter" tool which mounts partitions from the RDB, and - as part of its job - also unmounts previous file systems on the same physical layer. With ACTION_DIE, it can release many more resources and does not have to keep copies of the file system lie around in memory, doing nothing.

Actually, ACTION_DIE was already integrated into CrossDos and the CDFS in 3.1.4, so we just completed a job that was started long before. It was a bit more tricky for the FFS because the FFS is heavily multi-threaded, and its code flow is not exactly easy to follow.

To trigger an ACTION_DIE manually, the "mount" command received a new option: "SHUTDOWN". Thus, "mount df0: SHUTDOWN" un-mounts df0: All provided, of course, that there are no locks open on DF0:, no files are open, and the FFS is idle as you would otherwise "pull the rug under its feet".

"Assign Dismount DF0:" is quite different. All it does is that it makes the mounted file system invisible to the dos.library and hence to programs that want to open files on it, but it keeps running in the background, potentially modifying the disk, and potentially (still) reacting on programs that talk to it. Thus, if a program still accesses the disk, "Assign Dismount" will not stop such programs, but the program will continue to operate. If you mounted a different disk layout on the same drive, this will cause quite some desaster.....

So the old rule still applies: RDB is not for removable media. Mount them with "SuperFloppy = 1", and no RDB on them, and the FFS will take care about the disk geometry itself and will size itself accordingly.

Thus, a safer way to un-mount a partition is first to attempt a "mount shutdown", and if that worked, kick the mount out of the system with "Assign dismount". Unfortunately, the latter is still not quite clean as it cannot reclaim the memory for the actual DosNode (i.e. the information on the disk in the dos.library) as there are multiple ways how it could have been allocated, but at least the major part of the memory and resouces is then cleanly released.

[4pLaY]

Another day, another component: Graphics.

There is not much new about graphics. There are two old bugs we found, namely when attempting to render VSprites on a NULL ViewPort which caused MuForce hits. It seems unlikely that this happens, but there are a couple of games that load their own view by LoadView(NULL). If, at that point, you are still holding an icon on the workbench - happens quite easily by an erratic mouse button - and then quit the game, the workbench is already continuing to move the icon even though the graphics view is not yet loaded. The gels functions have been updated to handle this case now gracefully.

There is also some news on AllocBitMap() and RTG support. AllocBitMap() has been enhanced a little bit to use its "friend bitmap" pointer differently in case a specific "flag" combination is set. In such a case, the "friend bitmap" becomes a taglist, and carries information such on the display ID. Previous versions of the operating system act gracefully on this flag combination and return NULL, i.e. fail the allocation.

Now, why is this important? Consider an RTG graphics scheme where a software needs to open a screen in a non-native graphics mode. At this point, there is no "friend bitmap" because the screen is not yet open, yet the system calls into AllocBitmap() for the screen. AllocBitMap() is now patched over by the RTG software, and is supposed to take the bitmap to the graphics RAM of the RTG board. Unfortunately, it cannot really know as there has not been an indicator on AllocBitMap() where the screen is supposed to go, and which graphics mode it is going to be.

P96, at least, "went fishing": It looked at the stack frame from the call, used a heuristics, and if the heuristics said that the call is likely coming from intuition, it checked the stack frame for the display ID where it was laying around - by pure chance, not by design - and then used the display ID from there.

Of course, this is not a robust algorithm, and as the stack layout changed with intuition V45 as it was recompiled by another compiler, P96 would have been broken. So V45 of intuition contains a special "kludge", a short piece of assembler stub, that prepares a stack frame within OpenScreen() that looks like the one from V40, and then called into AllocBitMap() to make P96 happy.

Needless to say, this is all a fragile design and need to go, and so it does now. Intuition v47 (more to be said about this) uses now the new AllocBitMap() interface with tags prepared to list the display ID, and the latest P96 versions (2.3 and up) grab the displayID now through the new interface. Actually, this interface was already put into P96 years ago (around ~2000 or so), but wasn't activated as its counterpart in intuition and graphics was missing. Not any more.

It is likely that cgfx requires a similar kludge, probably for additional functions, but we don't know what exactly, and Frank hasn't reacted on the request to document them, so users of CGfx wil have to downgrade intuition to v40. Just put it in LIBS: and System-Startup will pick it up there. No need for reboot, but no new intuition functions either.