Pushing Matrix Brandy to its limits

[Richard Russell] · February 2022

As I understand it, there are two key 'enabling' technologies in Matrix Brandy which ultimately determine what it can and can't do:

The ability to call arbitrary code, using SYS "Brandy_dlcall" ...
The ability to display arbitrary output, using SYS "Brandy_GetVideoDriver" ...

The question that arises, for me, is does this mean that in principle Matrix Brandy can do literally anything that BBC BASIC for SDL 2.0 can do?

For example, normally Matrix Brandy doesn't support outputting proportional-spaced text, but presumably by passing the SDL surface handle returned from SYS "Brandy_GetVideoDriver" to some custom code called using SYS "Brandy_dlcall" it ought to be possible to achieve this.

So, given enough skill and effort, can Matrix Brandy do anything or are there still some limitations?

Soruk · February 2022

That is basically the theory - I did expand SYS (on non-RISC OS platforms) to allow for up to 16 parameters (from 10), but some X calls seem to require a lot, and if the function requires more than (edit) 15 parameters, then it isn't going to work. Also, I can imagine setting up the memory buffer blocks to pass parameters would be tricky at best along with intimate knowledge of the shape of the structs used, which can vary from platform to platform. All parameters are passed to the function as size_t values, so if a function is expecting a 32-bit int on a 64-bit platform, I'm not entirely sure how that's going to work out, and the results probably won't be pretty! Pointer parameters should be okay. (I might build a little .so file that does pointless small things to try to figure this one out.) Upstream, SYS filled an array of int32's (before deciding on non-RISC OS platforms to just fail as unsupported), which to be able to support 64-bit systems properly and to use real memory locations rather than offsets from the bottom of workspace memory I changed to an array of size_t's.

I did succeed in getting some simple libxcb code working, but I subsequently noticed that libxcb is linked anyway by virtue of libSDL-1.2.

Drawing to the SDL surface by SDL calls should probably be best done with the cursor switched off and in *REFRESH OFF mode, as all output from Brandy itself writes to its own internal frame buffer(s) and is scaled according to the display mode as it's copied to SDL's pixel buffer.

There's one thing Matrix Brandy can not do in any shape or form is be an assembler.

[Richard Russell] · February 2022

Soruk wrote: »

I did expand SYS (on non-RISC OS platforms) to allow for up to 16 parameters (from 10)

BBCSDL's parameter limits are 12 integer plus 8 float, which have been enough for everything I've wanted to do. How do Matrix Brandy's 16 parameters split between integer and float (or isn't it a hard split in the same way)?

if a function is expecting a 32-bit int on a 64-bit platform, I'm not entirely sure how that's going to work

No common platform has an ABI that distinguishes between a 32-bit integer and a 64-bit integer, does it? That would break BBCSDL, which has no concept of different integer types when passing parameters to SYS functions. There are only two different types, integer and float.

Drawing to the SDL surface by SDL calls should probably be best done with the cursor switched off and in *REFRESH OFF mode, as all output from Brandy itself writes to its own internal frame buffer(s) and is scaled according to the display mode as it's copied to SDL's pixel buffer.

Presumably it's still possible to mix native output with directly-drawn output, for example if one wanted to draw a graph using BBC BASIC graphics statements and then label the axes using proportional-spaced text. But I can understand that one might need to be careful with the order in which the various components are output.

There's one thing Matrix Brandy can not do in any shape or form is be an assembler.

Why not? If the assembler is callable using SYS "Brandy_dlcall" what's the problem?

Soruk · February 2022

Richard_Russell wrote: »

Soruk wrote: »

I did expand SYS (on non-RISC OS platforms) to allow for up to 16 parameters (from 10)

BBCSDL's parameter limits are 12 integer plus 8 float, which have been enough for everything I've wanted to do. How do Matrix Brandy's 16 parameters split between integer and float (or isn't it a hard split in the same way)?

All the parameters are handled as size_t's, which allow for 32 bit ints, 64-bit ints on 64-bit platforms and pointers. I did try bit-stuffing a double, with something daft like:

DIM m%% 8
|m%%=12.34
SYS"Brandy_dlcall", "floattest",]m%%

(like I said, it was a daft notion, to lay out the 64 bits of a double in memory, then read them back as if a 64-bit int, and see if it could be bit-stuffed into the call. But that didn't work.) If the called function is expecting a pointer to double then passing m%% itself will work.

Richard_Russell wrote: »

if a function is expecting a 32-bit int on a 64-bit platform, I'm not entirely sure how that's going to work

No common platform has an ABI that distinguishes between a 32-bit integer and a 64-bit integer, does it? That would break BBCSDL, which has no concept of different integer types when passing parameters to SYS functions. There are only two different types, integer and float.

Actually, from some quick tests, this actually seems OK at least under 64-bit Linux. I was worried the parameters would get out of alignment if it was expecting a 32-bit int and a 64-bit int is being forced at it, but that doesn't appear to upset it.

Richard_Russell wrote: »

Drawing to the SDL surface by SDL calls should probably be best done with the cursor switched off and in *REFRESH OFF mode, as all output from Brandy itself writes to its own internal frame buffer(s) and is scaled according to the display mode as it's copied to SDL's pixel buffer.

Presumably it's still possible to mix native output with directly-drawn output, for example if one wanted to draw a graph using BBC BASIC graphics statements and then label the axes using proportional-spaced text. But I can understand that one might need to be careful with the order in which the various components are output.

Some calls to update the screen will specifically only update the bit being modified, others will redraw the entire screen - especially scrolling. And if you try to do this in MODE 3 or 6, then the black bars will also get painted over whatever you're doing.

Richard_Russell wrote: »

There's one thing Matrix Brandy can not do in any shape or form is be an assembler.

Why not? If the assembler is callable using SYS "Brandy_dlcall" what's the problem?

Fair point - but it wouldn't be in a "compatible" fashion, as in doing

[
LDA #0
JSR&FFF4
RTS
]

(6502 used purely for illustration!)

[Richard Russell] · February 2022

Soruk wrote: »

All the parameters are handled as size_t's, which allow for 32 bit ints, 64-bit ints on 64-bit platforms and pointers.

So are you saying you can't pass float parameters using SYS? That would be a very serious limitation, from my perspective, and is the sort of thing I was trying to establish in terms of probing what is and isn't possible.

For example to draw a rotated and/or scaled image onto your output canvas I would have expected to call rotozoomSurface() (in SDL_gfx) which has the following signature:

SDL_Surface *rotozoomSurface(SDL_Surface *src, double angle, double zoom, int smooth)

Fair point - but it wouldn't be in a "compatible" fashion

But nothing achievable using SYS "Brandy_dlcall" is going to work in a compatible fashion! That's the whole point of my query: to establish what features that aren't available natively in Matrix Brandy can be implemented using the SYS statement as a workaround. But clearly not in a compatible way.

I am sure you can guess where this is going. I do not expect to be able to maintain BBC BASIC for SDL 2.0 for much longer, and nobody else has expressed an interest in taking it on, so I'm having to 'think the unthinkable' which is to recommend to all my users that they switch to using Matrix Brandy instead (as the closest alternative available). But that means offering them workarounds for the many things that BBCSDL can do natively that Matrix Brandy cannot.

Soruk · February 2022

Richard_Russell wrote: »
Soruk wrote: »

All the parameters are handled as size_t's, which allow for 32 bit ints, 64-bit ints on 64-bit platforms and pointers.

So are you saying you can't pass float parameters using SYS? That would be a very serious limitation, from my perspective, and is the sort of thing I was trying to establish in terms of probing what is and isn't possible.

For example to draw a rotated and/or scaled image onto your output canvas I would have expected to call rotozoomSurface() (in SDL_gfx) which has the following signature:
SDL_Surface *rotozoomSurface(SDL_Surface *src, double angle, double zoom, int smooth)

Yes, that is the thing, right now that I can't figure out how to resolve this. The SYS call handler (exec_sys, mainstate.c line 2340) which was originally written to collect the arguments and pass on to the RISC OS SWI handler (which only expects int32s and pointers (for strings), and complain on non-RISC OS platforms. The heavy lifting of actually calling the external plugins is in mos_sys.c line 585 (case SWI_Brandy_dlcall) and line 659 (case SWI_Brandy_dlcalladdr).

It's worth noting that if you have dlopen()ed SDL2 that likely has different functions with the same name as an SDL1.2 function then it would be best to capture the library handle with something like SYS "Brandy_dlopen","libSDL.so.2" TO sdl2h%%, then use SYS"Brandy_dlgetaddr", "SDL_function", sdl2h%% TO sdl_functionaddr%% to ensure that you have the function address from the correct library, then use SYS"Brandy_dlcalladdr", sdl_functionaddr%%, parm%%, parm2%%...

Of course, you can also do all this from a text-mode build, just make sure you get SDL2 to open and manage the window for you.

Richard_Russell wrote: »

Fair point - but it wouldn't be in a "compatible" fashion

But nothing achievable using SYS "Brandy_dlcall" is going to work in a compatible fashion! That's the whole point of my query: to establish what features that aren't available natively in Matrix Brandy can be implemented using the SYS statement as a workaround. But clearly not in a compatible way.

I am sure you can guess where this is going. I do not expect to be able to maintain BBC BASIC for SDL 2.0 for much longer, and nobody else has expressed an interest in taking it on, so I'm having to 'think the unthinkable' which is to recommend to all my users that they switch to using Matrix Brandy instead (as the closest alternative available). But that means offering them workarounds for the many things that BBCSDL can do natively that Matrix Brandy cannot.

I figured that might be the case, and that would be a very sad day when it comes and the entire BBC BASIC community will be all the poorer for it. I hope I am not speaking out of turn when I suggest it might be worthwhile documenting the C build of BBCSDL / BB4C, how it hangs together, and perhaps why specific things have been done the way they have been, as that may assist anyone who considers taking it on. I have to confess one of my big struggle with Matrix Brandy is figuring out how all the internals actually hang together, much of it has been by trial and error (in some cases, an awful lot of error!)

[Richard Russell] · February 2022

Soruk wrote: »

Yes, that is the thing, right now that I can't figure out how to resolve this.

I sympathise. To make it work in a cross-platform fashion in BBCSDL I found I had to use 'nested functions' (which are a non-standard GCC extension) and disabling optimisation (#pragma GCC optimize ("O0")). And even then there's no guarantee that it will work, because it's relying on undefined behaviour.

It's the 64-bit Windows ABI which really screws everything, because of the strange way that integer parameters 'shadow' float parameters. For example in the case of the function signature I listed previously, which was basically (size_t, double, double, size_t) the parameters get passed not as you would expect, in rcx, xmm0, xmm1 and rdx but in rcx, xmm1, xmm2 and r9.

The 64-bit Linux (System V) ABI doesn't behave like this, the integer parameters are considered entirely independent of the float parameters and follow their own separate register-allocation rules. I've no doubt that Microsoft had some good reason for the 'shadowing' behaviour but it makes SYS in BBCSDL much more complicated.

I hope I am not speaking out of turn when I suggest it might be worthwhile documenting the C build of BBCSDL / BB4C, how it hangs together, and perhaps why specific things have been done the way they have been

That's a lot more difficult than you might imagine, because the BBCSDL C source code was created by 'translating' the assembly-language BB4W code (partially mechanically, partially by hand) and in many cases I had already forgotten then - let alone now - how the assembly language code worked! If that wasn't bad enough, because of this translation process variable names are often meaningless, because they simply correspond to the register names in the original code.

Here's an example, plucked out of the C source, of part of the DIM statement (probably the most complex single statement in BBC BASIC, especially when you include structures and PRIVATE arrays/structures):

						char *edi = VLOAD(ebp) ; // old pointer
						if (edx < (pfree + (char *) zero))
						    {
							char *eax = edi ;
							if (ecx)
								eax = VLOAD(edi + ecx) ;
							if (eax != edx)
								error (10, NULL) ; // 'Bad DIM'
						    }
						else if (edx > (pfree + (char *) zero))
						    {
							if (ecx)
								edi -= edx - pfree - (char *) zero ;
							ecx += edx - pfree - (char *) zero ;
							fixup (pfree + zero, edi - pfree - (char *) zero) ;
						    }
						if ((ecx != 0) && memcmp (pfree + zero, edi, ecx))
							error (10, NULL) ; // 'Bad DIM statement'

Here the variable names (eax, ebx, ecx, edx, esi, edi and so on) correspond to the registers used in the assembler version but don't give a clue to how the code works.

Soruk · February 2022

I've managed to get something going.
For my test library, which just consists of:

#include <stdio.h>

long long int mytest(int a, int b, double c, long long int d) {
  fprintf(stderr, "a=%d, b=%d, c=%g, d=%lld\n", a, b, c, d);
  return a;
}

I can do this:

Matrix Brandy BASIC VI version 1.22.13 (Linux/x86-64) 26 Jan 2022

Starting with 67108864 bytes free

>SYS"Brandy_Hex64",1
>SYS"Brandy_dlopen","./dltest.so" TO h%%
>P.~h%%
   155AE50
>SYS"Brandy_dlcall","mytest", 42,64,&123456789ABCDEF0,,,,,,,,,PI
a=42, b=64, c=3.14159, d=1311768467463790320
>P.&123456789ABCDEF0
1.31176847E18
>_

In this implementation, R0 is the symbol (or address when using Brandy_dlcalladdr), R1 to R11 are handled as size_t and R12-R15 are handled as double. It does have the odd effect that the order of parameters are changed!

This is on a branch so won't be included in the nightlies tonight. It will be included, I've merged the branch to master. I haven't tested on Win64 (or Win32) yet, so far the art of building a DLL for Windows is eluding me.

[Richard Russell] · February 2022

Soruk wrote: »

I haven't tested on Win64 (or Win32) yet

I don't see how it could work in Windows. And what about 32-bit Linux, that's problematical too isn't it?

so far the art of building a DLL for Windows is eluding me.

I thought it was just a case of specifying -shared in the linker command.

Soruk · February 2022

Richard_Russell wrote: »

Soruk wrote: »

I haven't tested on Win64 (or Win32) yet

I don't see how it could work in Windows. And what about 32-bit Linux, that's problematical too isn't it?

Yep, it doesn't work under Windows, it reads the parameters in the order given, unlike Linux. And I really have no idea how to fix it. I've looked at the BBCSDL source, but I am struggling to get my head around how it actually works this magic.

Under 32-bit Linux (at least on ARM - RasPi 3B+), it works fine (with the proviso that the 64-bit parameter is truncated to 32 bits).

Richard_Russell wrote: »

so far the art of building a DLL for Windows is eluding me.

I thought it was just a case of specifying -shared in the linker command.

Facepalm moment. I needed to use the MinGW gcc, not the regular one! Apart from that, and making the build script write out as .dll instead of .so, what I did for Linux works, at least as far as creating a library that could be dlopen()ed.

[Richard Russell] · February 2022

Soruk wrote: »

Under 32-bit Linux (at least on ARM - RasPi 3B+), it works fine

But the Raspberry Pi doesn't use the standard 32-bit ARM ABI, it uses the armhf (hard float) ABI, so I'm not sure whether you can draw conclusions from that. Android uses the standard ARM ABI I think, if you have a way of testing on that platform.

Soruk · February 2022

Richard_Russell wrote: »

Soruk wrote: »

Under 32-bit Linux (at least on ARM - RasPi 3B+), it works fine

But the Raspberry Pi doesn't use the standard 32-bit ARM ABI, it uses the armhf (hard float) ABI, so I'm not sure whether you can draw conclusions from that. Android uses the standard ARM ABI I think, if you have a way of testing on that platform.

I don't have a way to test on Android, and it's not a platform I've ever targeted. I've tried my test from above on my CentOS 6 x86-32 machine, and, yep, I'm also quite surprised but it didn't work either (and the 64-bit parameter wasn't picked up either. Which leaves me at a bit of a loss as to how to make this work in anything like a cross-platform manner. I'll leave it as it is for now, I've back-merged my build fixes to the sysexpand branch so if I revert the master branch to the state it was prior to merging, I haven't lost these changes.

PS. Posted from MS Edge, and so far no errors connecting via my work VPN.

[Richard Russell] · February 2022

Soruk wrote: »

I've tried my test from above on my CentOS 6 x86-32 machine, and, yep, I'm also quite surprised but it didn't work either

I think the flaw with your approach is that the SYS statement would be identical for these two function signatures:

(int, int, float, int)
(int, float, int, int)

It so happens that with the 64-bit Linux (System V) ABI, and probably with the ARM Hard Float ABI too, those two signatures do indeed result in the parameters being passed identically, so your approach works.

But that's not generally true. In the 32-bit Linux ABI, the standard 32-bit ARM ABI and both 32-bit and 64-bit Windows ABIs the parameters are passed differently in these two cases. So there has to be a way in which the SYS statement can distinguish between them.

Soruk · February 2022

I'm trying the approach you have in bbccon.c lines 514-538 from your git repo, and while it works in x86-64, I still can't get it to work in 32-bit Linux. I was able to bit-stuff my way round it (which is more than my initial implementation), but that's hardly portable between platforms.

I am truly at a loss as how to proceed with this. That you have it working means it is possible, but I cannot see what I have missed that means it is not working for me.

[Richard Russell] · February 2022

Soruk wrote: »

That you have it working means it is possible

I'm wondering what it is that you are aiming for. If you are trying to design a scheme that allows the same SYS statement to work on all platforms, that's not something I achieve or have even attempted.

Since it's common (in BBCSDL, as I assume it is in Matrix Brandy too) to need different BASIC code according to the platform, I'm not concerned that this applies to the SYS statement as well.

The flaw with your approach is not that it needs platform-specific BASIC code, but that it is impossible to make it work even if you do (for the reason I explained earlier).

(I tried posting this from Chrome rather than Edge, but the freezing is just as bad, I really think I will need to give up using this forum.)

Soruk · February 2022

I'm not trying to make my SYS statements identical to yours, and I do at least now have a way to pass floats, rather more intuitively on 64-bit Linux than 32-bit (which is a mind-bender) so it is better than before.

I am sorry you are having issues with the forum, but unfortunately try as I might I am completely unable to replicate the problem.

Out of sheer curiosity, do you have a Linux workstation setup? It might be worth testing from there... All my posts today have been from MS Edge under Windows 10, routed via the corporate VPN so there's no local-network cheating going on.

Soruk · February 2022

Further to the forum issue, I've created a new section for forum issues, and also adjusted the MTU of the server, in case it's NAT and a non-standard MTU of 1492 I have from my FTTC connection. I've set it to 1480 as that's also what my IPv6 tunnel gives me.

[Richard Russell] · March 2023

More than a year later... where do we now stand with the capabilities of SYS to pass floats? I think Matrix Brandy has moved on from what is described in this thread, but it would be helpful to know on which platforms you can successfully pass any mixture of floats and ints, and on which (if any) you can't.

A related question (and I feel I should know the answer) is how does Brandy tell the difference between an integer value of zero and a floating-point value of 0.0 - if indeed it ever has to? This is a complication in my BASICs which involves a workaround when using SYS on some platforms.

Soruk · March 2023

The parameter parser expression() handles this, and puts the correct type on the stack.

From a special debug build, doing a *FX0 via SYS:

>SYS"OS_Byte",0
Item type is STACK_STRING
Item type is STACK_INT

Matrix Brandy MOS V1.22.15 (21 Mar 2023)
>SYS"OS_Byte",0.0
Item type is STACK_STRING
Item type is STACK_FLOAT

Matrix Brandy MOS V1.22.15 (21 Mar 2023)
>SYS6,0.0        
Item type is STACK_INT
Item type is STACK_FLOAT

Matrix Brandy MOS V1.22.15 (21 Mar 2023)
>_

[Richard Russell] · March 2023

Soruk wrote: »

The parameter parser expression() handles this, and puts the correct type on the stack.

I see. It's impossible in my BASICs because the internal (variant) representation of 0 is identical to 0.0 by design, there is no way they can ever be distinguished. That's not least because floats have an 'implied 1' in the MSB position, and zero has no 1s! This has been the case right back to my Z80 BASIC.

[Richard Russell] · June 2023

Soruk wrote: »

The parameter parser expression() handles this, and puts the correct type on the stack.

This thread fizzled out somewhat. Can you confirm whether, in current builds of Brandy, SYS can successfully pass mixtures of ints and floats on all platforms, albeit possibly needing different BASIC code on different platforms? If so that's a major step towards Brandy being able to do 'anything' my BASICs can.

Soruk · June 2023

Handling mixes of ints and floats is only implemented for Brandy_dlcall and Brandy_dlcalladdr, for all other calls ints are assumed. It's also possibly the least tested feature and is only functional at all that allow dynamic library loading (so RISC OS builds, for instance, do not support this at all).

Pushing Matrix Brandy to its limits

Comments

Categories