#ctran — Public Fediverse posts

My attempt to tidy #CTRAN's code is making it worse, not better. In my attempts to remove global variables, I'm ending up passing 4-7 arguments for each function, sometimes just to get information to a sub-function three levels down.

I've avoided it so far, but I'm close to creating a class for each file type, shoving the variables/tables (SetDependencyList(), SetExternalClassList()) in that and just calling each function/method in turn to generate each part of the file. No private methods, just a class to neaten each stage of building a file.

Another option is to redesign the current globals so that relevant data is packaged together. But I don't know how that looks yet, and I don't think I will until I've done an initial round of refactoring.

I know I don't have to make CTRAN any faster. On a modern machine it runs almost instantly. In DOSBox it's under a second - not as fast as the original, but not terrible. The aim here is to make the code more understandable, and to not do anything unnecessary or confusing (e.g. looking up a class name to look up the class again, or looking up a class to find one bit of information that could have been cached ahead of time).

I can safely say I've painted myself into a corner with this code.

It's going to be so much better when I'm finished. But the road there...

#CTRAN #SIBOSDK

I got up this morning, grabbed the laptop and worked out what I needed to do to create OMF files with CTRAN.

I now know how to generate everything... EXCEPT ONE THING! TopSpeed generates a "Type Index" for every class with one (possibly two) bytes of debug information. I have no idea what it uses to generate this.

The OMF docs imply that it's ignored by most linkers. But that's not the point.

Damn, I thought this was going to be so easy.

#CTRAN #ECOBJ #SIBOSDK

CTRAN to OBJ File Pipeline

I've just thought of this one. Bear with me.

• #CTRAN generates TASM code. Badly, but still.
• It also generates C files that are compiled by TopSpeed C and then processed by #ECOBJ.
• The TopSpeed OBJ Disassembler (tsda) disassembles OMF files compiled by TSC.

First, why aren't I using tsda to make at least some of the TASM code generation in CTRAN more accurate?

But more interestingly, why not add the ECOBJ OMFMaker library to CTRAN and have it just generate OBJ files directly, already in the correct layout? No need for a compiler or a separate app to manipulate the OBJ file. Just ctran -m (-o is already taken) to directly make the object file with class descriptor in the CODE segment.

Small bugfix release for #CTRAN to deal with the "no EXTERNALs" issue.

https://codeberg.org/thelastpsion/ctran/releases/tag/0.1.1

#SIBOSDK #Psion

Like #CTRAN, #ECOBJ will eventually be rolled into a compiler that's aware of the SIBO/EPOC16 ecosystem. CTRAN will probably end up as something to convert category files from legacy projects to whatever OO C I end up using. And a compiler will just generate OMF files in the right format without conversion.

But just because these tools are stop-gaps, doesn't mean they're useless. First, all of this is prep-work and learning for me. Second, a lot of the code I'm writing will be re-used in different places further down the line. But also, at the rate I'm going with the project, that "stop-gap" could last years.

#SIBOSDK

One of the aims of the new SDK is to create drop-in replacements for the old tools. This was pretty easy with #CTRAN - switches in the original enabled specific outputs. No switches, no output files.

#ECOBJ is a little different. As its simplest, it takes one parameter - the name of the OMF file to process. It then overwrites that file with the re-jigged output. Fine as a quick tool that's never called by a human, not so fine for... everything else. You can specify a separate output file, but I've never seen a Makefile, make.bat or TopSpeed project file that does.

I wonder if I need to create two executables. One would be ecobj/ECOBJ.EXE, which is the drop-in replacement for the original tool. And then another that will dump and analyse... maybe tsda, to replace the tool that comes with the TopSpeed compiler? They'd mostly use the same functions/classes, but the default behaviours would be different.

I need to create the bulk of the latter in order to build the former anyway.

#SIBOSDK

Finally, after months of eager anticipation from all corners of the internet, #CTRAN 0.1.0 is out.

Please, everyone, contain your excitement.

https://codeberg.org/thelastpsion/ctran/releases/tag/0.1.0

Seriously, though, I'm pleased with this release. It's working at least as well as the original, and better in quite a few ways. Plus, I've learned a lot about making my code "right", not just "work".

There's a bit more to do to get the user experience to where I want it to be. But the big challenge for me is to get automated tests working so that I can easily check a raft of edge cases. That's going to require quite a bit of refactoring, but it'll be worth it.

#Psion #SIBOSDK

Modifications made and a bit of tidying done.

I've switched from -u to -r as it affects REQUIRE statements in category files. I've also improved the external category file path processing.

I've made a PR for me to check over in the morning. If I don't notice any glaring issues I'll squash-and-merge. Then I'll do some final tweaks to the README, add any comments that Future Me might find useful, bump the version to 0.1.0 and make a release.

#CTRAN

Well that works nicely!

-u overrides CWD. Because I've re-used the external class path parser, it'll skip nicely through a list of paths too.

Now I need to do some tidying.

#CTRAN

Currently, #CTRAN looks for external category files in the CWD unless -e is specified. This means Psion CTRAN can only look in one place for .EXT files - the CWD or wherever -e points. New CTRAN allows path lists, so -e ./:~/sibosdk/inc/ will work.

After writing this all out in previous toots, a separate switch for subcategory file paths feels appropriate in order to maintain backwards compatibility while adding new features for new devs. So, CWD first unless a path is specified.

Now I just need to decide on a switch letter. Maybe -u for sUbcategory path?

I've ditched the "enforce semicolons" idea for path separation. If you have to have two separate lines in your Makefile for DOS/Windows and POSIX, you might as well just use different path separators.

So, native path separators now work in #CTRAN. A simple fix, but worth doing.

Part of the reason for checking long file name support when generating "skeleton" files - boilerplate for classes, one file per class.

Let's take the game Solipeg, an EPOC16 game using Psion OO. It has two classes: soldlg_open and soldlg_option. Psion #CTRAN will create SOLDLG_O.C for soldlg_open, and then overwrite it for soldlg_option. There are no checks.

Now, I don't think many people used the skeleton file generator, and if they did it was probably only once. After all, it's pretty destructive - Psion CTRAN overwrites any files it finds, even if you've edited the files already. But it's in the original, so...

At least with a check for LFS or a "DOS Mode" switch I can decide what to do - use long names, or add a number and a warning. Probably also a "force overwrite" switch.

I've decided to add a "DOS Mode" switch to #CTRAN. That way I can have an explicit switch for 8.3 vs long file names, and forcing semicolon path separators vs native ones.

DOS Mode would be automatically enabled when running on DOS (via go32v2). Running on other OSes will allow long file names and use native path separators unless the DOS Mode switch is used.

I'm not sure what to do about #FreeDOS's long file name support. I don't know if #FreePascal has the ability to detect file system features like case sensitivity and long file name support without trial-and-error?

The SDK of Theseus.

Or, for UK viewers, Trigger's Toolchain.

#SIBOSDK #CTRAN #ECOBJ

Slightly confusingly, there are two three types of "include" in #CTRAN:

• INCLUDE points to C header files. They directly convert to C as #include statements. They're not important here.
• REQUIRE adds "sub-category files," essentially broken-out parts of the main category file. I guess these are "child" files.
• EXTERNAL refers to "external category files", which are summaries of pre-compiled (or at least pre-processed by CTRAN) libraries. These could be seen as "parent" files.

EXTERNAL files already have a path switch (-e). This is the one that I've already modified with the path separator.

At the moment, REQUIRE files only look at the working folder. This didn't matter in Psion CTRAN as your main category file could only be in the working folder.

This line wouldn't work in Psion CTRAN. It would strip out the path and just look for file.cat in the working directory.

ctran path\to\file.cat

However, it does work in new CTRAN. Thing is, it doesn't use that path for any REQUIRE files. So, say path/to/file.cat includes subcat.cl, it won't look in path/to/, but in ./.

What's the right way to handle this? Should it try the CWD first, then path/to/, then give up? Do I use a different switch to specify search paths for REQUIRE files? Do I extend the -e EXTERNAL path switch to cover REQUIRE as well?

EDIT: Got INCLUDE and REQUIRE mixed up.

At the moment I'm only allowing multi-path strings to be separated by ;. That's how it's done in DOS, and my aim was to maintain backward compatibility with the original tools.

This obviously causes issues in Linux and other POSIX-adjacent OSes, as a semicolon means something different to a *NIX shell. POSIX generally uses a colon instead of a semicolon as a path separator.

However... multi-path strings weren't a thing with Psion's original CTRAN.EXE. This is a new feature. In fact, this was the final straw that made me write the new #CTRAN. So there's actually nothing to be compatible with, because no existing code will use it.

I guess my question is, should I just use Free Pascal's native PathSeparator constant instead of fixing the separator as ;? As long as it's documented, a new developer would just need to make different build scripts for DOS/Windows and POSIX.

Another option would be to add a "DOS compatibility" switch that forces the use of semicolons as path separators.

Paths in #CTRAN need some thought.

There are two parts to this. First, multi-platform compatibility (i.e. DOS/Windows vs POSIX), and how to search for files included with the REQUIRE keyword (sub-category files).

🧵

I think I'm going to try to improve path management in #CTRAN, make a release and then take a break from it. After that, most of the other TODOs are pretty small anyway.

I'm tempted to go back to my rewrite of ECOBJ.EXE, another tool in the #Psion SIBO C SDK. This one modifies 8086 Object Model Format (.OBJ) files for classes on EPOC16. I don't think it'll take me more than a few weeks to have something that works, even if it's not perfect.

#SIBOSDK #ECOBJ

BLOG: Make It Right - Next Steps With CTRAN

Rather than go straight into some of the more technical issues I've been facing, I've just written a quick recap on my rewrite of Psion's weird OO C preprocessor, and where I'm heading next with it and the rest of the SIBO C SDK.

https://thelastpsion.com/posts/make-it-right-next-steps-with-ctran/

#CTRAN #Psion #SIBOSDK

A few more quick wins this evening on #CTRAN:

• A little IsConcrete() function to check if the class is concrete or abstract (according to Psion's rules, that is).
• The code for getting external category file IDs is a little tidier. Plus it can now be used in more places, including IsConcrete().
• Method inheritance is now checked when each class is added to the big DependencyList (a specialised TDictionary), rather than checking them after all the classes have been added and having to work out which file the class is in. Should save quite a few cycles, although it'll be barely noticeable on any processor in the last 20 years.

I've still got a lot of TODOs and FIXes to go through, but I feel like I've made some steady progress over the past few days, untangling the mess I've created for myself over the past 2.5 years of CTRAN's existence.

For now, though, I'm putting the laptop away. Might go read a book.

https://codeberg.org/thelastpsion/ctran/commit/7864fe44b6dcf448a0ceb1b51b7f417dd14d4217

Well, after all that, we have a wee refactor. Somehow there's only 4 lines of code difference, but I'm putting it down to extra comments in the code and better readability.

https://codeberg.org/thelastpsion/ctran/commit/1f88d329c693aefc2647fea0b17df3dd86759a3f

#CTRAN #SIBOSDK

I think I've found another bug in #Psion #CTRAN's TASM generator. (NOTE: This post looks long, but just needs a bit of scrolling.)

If I'm right, my code already replicates the bug, but I'm almost certain it's wrong and I have no way to test it.

Let's take a look at this example class, snipped from a category file in VECTOR:

CLASS   vecdw   win
{
    REPLACE wn_init
    REPLACE wn_draw
    REPLACE wn_key
    ADD wn_beg_draw
    ADD wn_beg_draw_clear
    ADD wn_end_draw
    ADD dw_set_size
! ...snip from here...
}

When inheriting methods that aren't REPLACED, CTRAN adds in a NULL to indicate that this method can be passed through. It also adds an offset to show where the first REPLACEd method is (or just the number of inherited methods if none are being replaced).

From the above it will generate this C code:

{2,(P_CLASS *)ERC_WIN,sizeof(PR_VECDW),6,0x6b,19,0},
{
vecdw_wn_key,
NULL,
NULL,
NULL,
NULL,
NULL,
vecdw_wn_draw,
vecdw_wn_init,
vecdw_wn_beg_draw,
vecdw_wn_beg_draw_clear,
vecdw_wn_end_draw,
vecdw_dw_set_size
}

The five NULL entries are for five inherited methods between wn_key and wn_draw.

The TASM generator, however, does this:

 _TEXT segment byte public 'CODE'
GLDEF_C c_vecdw
 dw   2
 dw   ERC_WIN
 dw   (SIZE PR_VECDW)
 db   6
 db   06bh
 db   19
 db   0
 dw   vecdw_wn_key
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   vecdw_wn_draw
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   vecdw_wn_init
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   vecdw_wn_beg_draw
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   vecdw_wn_beg_draw_clear
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   vecdw_wn_end_draw
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   vecdw_dw_set_size
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0
 dw   0

Anyone else think that someone forgot to reset i to zero?

Sigh.

On the other hand, my aim for #CTRAN 1.0.0 is feature-parity with "classic" CTRAN 5.02. As long as it works at least as well as the original, I don't have to perfect it. The new TASM generator is clearly better than the original, even with possible bugs.

If my 8086 knowledge was better, I'd probably pivot the generator to create NASM code. That would be more useful for modern developers.

Also, improvements to the code, TASM or NASM, wouldn't be a complete waste. I'd need to write a mini C lexer/parser, with some customisations to be compatible with Psion's category file spec (e.g. ; signifies EOL). That would also put my code leaps and bounds above classic CTRAN, allowing me to syntax check the C embedded in category files rather than blindly copying the code.

One other point. My version of #CTRAN is really a stop-gap. At some point in the distant future, I want to write a C compiler that targets Psion's SIBO/EPOC16 architecture. Eventually I want to remove the need for category files, implementing some sort of OO. Writing a C lexer/parser would be good practice for the real thing.

My rewrite of #CTRAN improves on the original. For C code generation, it's a drop-in replacement for the original DOS tool, but also runs on modern OSes.

I've also replicated TASM output as best I can. In places, I'm sure my TASM generator is better than Psion's. For one, it doesn't crash.

But because "classic" CTRAN's TASM generator is buggy as hell and my 8086 knowledge is poor, I don't always know what Psion were trying to achieve. Even worse, I'm not sure they even cared that much about it.

So... should I care?

New issue, still #TASM-related. Almost certainly a bug in Psion's #CTRAN that I feel I need to resolve.

This line in C struct UNDOMAN:

VOID * pEngine;

Gets translated into TASM code like this:

Undoman pEngine dw ?

But this:

VOID* pEngine;

Turns into this:

UndomanPengine VOID* <>

And this:

VOID *pEngine;

Turns into this:

UndomanPengine dw ?

I think the last one is correct, but I could do with the opinion of someone more knowledgeable than me!

#SIBOSDK #ASM #x86

I was thinking about removing TASM support in #CTRAN because I couldn't get it to work properly, matching the output of #Psion's original version. But last weekend I realised what I needed to do* to get it working.

So now it works as well as Psion CTRAN. In fact, it probably works better than the original, because the output is neater, doesn't miss out class names, and it doesn't crash.

Which is great! Except for the fact that no one is going to use CTRAN to generate TASM code today. I don't think anyone actually did it back in the 90s. I only added it to give it feature parity with the original.

But I know that the code that generates the TASM files is janky as hell. The right way to fix the jank is to write a custom mini C lexer/parser to only parse a typedef struct. Otherwise, I leave the jank as-is or go back to removing TASM support.

* Pointers are always word values. I was ignoring the * and wondering why CTRAN was outputting completely the wrong type sizes.

#SIBOSDK

A few fixes and changes in #CTRAN yesterday and today.

• Concrete class detection needed refining to match Psion's CTRAN.
• Class property that are pointers weren't being handled properly in a few places. This fixes a long-standing bug in TASM include files.
• I've also spun off a few chunks of code into their own functions. Should make things easier for the next stage.

The last bit will hopefully help me with my next step - the Big Refactor. I want to start moving functions out of the main ctran.pp file and into new, appropriate units. My plan is to just hack into the code, rip bits out and move them into new files, trying various layouts until something sticks.

I think I'm going to make a release before I do this, just to draw a line under the current state of the code.

#SIBOSDK

And that's the .ASM and .ING files done!

https://codeberg.org/thelastpsion/ctran/commit/dbc50dbdab0e14cef1ecc4b1391dcada0f54b59e

There are other issues with #TASM file generation in my code, such as class property generation. Thing is, I think there are bugs in the original #Psion #CTRAN TASM support, and as I'm not fluent in 8086 assembly (yet) I can't tell what's right.

I'm sorely tempted to remove TASM support completely. I can't see anyone using it.

My new CTRAN is only meant to be a stop-gap until I eventually get around to writing a C compiler with built-in class support. But that could take me years, so it needs to exist for now.

I wish Colly Myers was still alive so that I could ask him questions about CTRAN. About the whole of the SDK, to be honest.