Forums

well, unfortunate but irrelevant (if really needed, one day we'll RE that and implement it anyway probably)

So it seems that despite my best efforts :) we’ve focused on the general problem of providing metadata about the cartridge with the ROM image(s). I’ll refer again to https://web.archive.org/web/20200322151931/https://byuu.net/cartridges/boards which is about exactly this - and I actually agree that it is a good idea, but also that it’s almost impossible to solve due to the existence of old emulators, flash carts and ROM collections. That’s why I was trying to narrow the scope to homebrew, so we can start with new things.

But surely it's not really possible to separate those two things fully.

Most homebrew efforts are built standing squarely on the shoulders of the original technology, and emulators serve the dual purpose of allowing one to enjoy original conserved games, as well as development, testing and playing of new homebrew games.

I also don't really see a contradiction between the two needs, in terms of how a proposed metadata system would have to operate. Regardless of whether it's embedded in the ROM or a companion to it, the semantic contents (and IMO the syntactic contents) should be the same; and if it's suitably flexible it ought to be possible to enable homebrew developers to model the most common cases they need, and over time to model not only most original technology, but potentially some more exotic homebrew technology.

The immediate problem you pose I think is valid, even if only a minor irritation for most — can we specify some very basic variations in the standard Sega mapper using metadata, so that an emulator theoretically doesn't need to rely on heuristics and can instead 100% reliably supply the hardware that the developer wants?

Next step from there would be considering what other mapper needs might be, but I suspect that path will inevitably align with known original mapper tech (e.g. Codemasters, etc.), simply because most of it is "obvious", given the constraints on the technology at the time.
On that, I think I'd add that it's not really like the constraints on real hardware have completely opened up today, allowing an infinite range of mapper technology. Sure, it's possible to invent some exotic things, but we're still constrained by the base system hardware, as well as the practicalities of real hardware design and manufacture, so personally I say again let's put the exotic hardware to one side for now and focus on the well-known core!

Maxim wrote

So it seems that despite my best efforts :) we’ve focused on the general problem of providing metadata about the cartridge with the ROM image(s). [...] That’s why I was trying to narrow the scope to homebrew, so we can start with new things.

I doubt we can solve one problem without also providing a general solution.
But, for what is worth, emulators could add some support of some kind (a command line parameter?) to solve your specific problem, such as 'forcing' a mapper even on ROMs <= 48 KiB.

emulator.exe -forcemapper your_rom.sms

Still I think we were pretty close to the solution to the whole problem.

I'm open to any simple format for the text-file-based metadata.
Also we need a draft list for the *names* of the mappers that we already know are existing, then we will discuss the default behavior (when needed) for each of those.
I already expressed my ideas so I won't bother you more. But I can provide a draft names list - that will surely need lots of fixing.

In other areas they chose a fourCC-style naming convention but that may be more about saving space in constrained circumstances.

I think we can consider:

* Mappers:
* Sega “standard” mapper, with options for slots (some are slot 2 only), SRAM (enabled or not, SRAM size, battery backed or not). Defaults would be the “do everything” mapper that you get in emulators/flashcarts. We do not (yet) need to support the page number shift features.
* Codemasters mapper, maybe with similar options
* Maybe a generic but parameterised mapper approach to handle most of the Korean ones
* …and more for special cases like the MSXish 8/16KB slot ones and the byte flipping one
* Save functionality (in addition the the mapper)
* Serial EEPROMs in Game Gear games, parameterised as needed (size, address range?)
* Maybe a way to specify the VDP variant needed - including TMS9918a vs. SMS VDPs vs GG vs Mega Drive
* 50Hz vs 60Hz required
* FM chip supported
* Required and supported peripherals

Any of these could be optional.

I’d like for any emulator that might support this, it can be embedded in the ROM via the SDSC description as I suggested, or in a file next to the ROM (same base filename), or in a file in the same zip (same base filename or some standard name like metadata.json), or in a command line, with the ones on the right overriding the ones on the left,

Can you explain (or point towards articles) more on:

Maxim wrote

* Save functionality (in addition [to] the mapper)

and

Maxim wrote

* Serial EEPROMs in Game Gear games, parameterised as needed (size, address range?)

?

The most common save on Sega games was to battery-backed SRAM, but a few Game Gear games had a 128 byte (?) serial EEPROM. Some homebrew developers have also talked about saving to unused flash.

Maxim wrote

* Maybe a way to specify the VDP variant needed - including TMS9918a vs. SMS VDPs vs GG vs Mega Drive
* 50Hz vs 60Hz required
* FM chip supported
* Required and supported peripherals

This is indeed interesting, even if it's not something strictly bound to hardware or features present in the cartridge - but in the light of 'let's give additional info to the emulator to make this software work properly' it's something we could address too :)

Now, mappers.

I would use 'sega' as the general / feature-rich (if we don't need feature-complete after all) name for the SEGA mapper, then use `sega-<variant>` for SEGA mapper compatible mappers instead of using some other manufacturer name. Examples:

- 'sega-slot2only' (or a better name) for the slot2 only mappers regardless if they're from ichigo or raphnet (as an example, I don't even know if they ever in fact did that!)
- 'sega-slot0broken' (or a better name) for the krikzz' broken legacy SEGA mapper (if we ever need to emulate that!)

everything else I already posted regarding how an emulator should calculate the number of bits for each page register I won't repeat it again, and also everything regarding SRAM size and persistency/volatility.

Am I missing something regarding the SEGA mapper 'family'?

Then the CodeMasters' mapper.
Here we probably have only two specimen so we could just use 'codemasters' and 'codemasters-ram' (the 'Ernie Els Golf' one) for instance.
Again, should we find another very similar mapper, even coming from a different manufacturer, we would probably better create another 'codemasters-<variant>'

Then we have the MSX mappers, with the Nemesis variant. We could use 'msx' and 'msx-fixed8k' (or similar name) so that we don't reference a specific game directly - but if using a game name makes it simpler we could just agree on 'msx-nemesis'.

Finally, the korean mappers. Of course the group is not very 'homogeneous', so we should probably create a few main groups with a few variant each.
I can dive into the MEKA defined mappers list and see if I can draft that too, if we agree this is a possible way.

oh, OK I forgot one thing about the SEGA mapper. This:

The registers are thought to be undefined after a reset for games using the 831001 and 832011 ROMs and 315-5208 mapper. The 315-5235 mapper is reset to FFFC=00, FFFD=00, FFFE=01, FFFF=02 after a power-up reset.

we should decide what's the default and probably an attribute to switch to a specific variant ( 'initialized' / 'uninitialized' probably? )

Would like to take the time to reply in full later, but might I suggest that we try to approach the cataloguing of known hardware rigorously with part numbers in mind where possible.
(@sverx not sure whether this is exactly what you meant, but I read it as deciding which of the known mappers' reset behaviours we wanted to pick)

So rather than deciding what the "default behaviour" of certain things is, we try and faithfully capture the exact behaviour of known devices based on part number, and then simply allocate one particular device as the preferred "standard" Sega mapper device.

I would say that whatever known device represents the biggest superset of functionality is a good choice.

I'd also like to repeat a plea for some kind of namespacing, no matter how it's achieved, so for example:

smspower/sega/315-5235
might also alias to
smspower/sega/standard
or just
smspower/sega

I feel a little funny about calling something "sega" if it's actually a hybrid of several sega-like ideas, and a combination that maybe never existed in real hardware. Maybe there's a slightly alternative name we can use.

Games that don’t rely on it being initialised won’t care if it is, so specifying the uninitialised variant is only useful for breaking things, isn’t it?

willbritton wrote

I'd also like to repeat a plea for some kind of namespacing, no matter how it's achieved

I fail to see the reason for this.
Would you explain more why should be this important, given that SMS Power! will be the only naming authority?

Maxim wrote

Games that don’t rely on it being initialised won’t care if it is, so specifying the uninitialised variant is only useful for breaking things, isn’t it?

I'd say 'useful to *test* things' :)

sverx wrote

willbritton wrote

I'd also like to repeat a plea for some kind of namespacing, no matter how it's achieved

I fail to see the reason for this.
Would you explain more why should be this important, given that SMS Power! will be the only naming authority?

Well, yes because in short, as I think I said before I don't believe that SMSPower! ought to be the "authority", at least when it comes to the hardware identifiers!

I think the idea of a metadata specification is brilliant, but I also think the specification should be "open" - in the same sense that SDSC is open. Correct me if I'm wrong, but SMSPower! doesn't maintain a known list of authors, for example, because the author field is just a plain string.
(This stands in contrast to the Sega header, which numerically enumerates game vendors)

I think that if you don't suitably qualify the identifiers then you risk repeating essentially what is one of the flaws with iNES, which is that you have an essentially central repository of identifiers which, although (being strings rather than numbers) might seem sensible to you or I right now, might seem arbitrary and incorrect to people in the future; and working around or extending a scheme without namespacing means adding new arbitrary identifiers in an increasingly chaotic way.

To expand on my previous comment, I also think that simply calling something "sega" when it may not accurately describe any real Sega hardware could be confusing, and would certainly not imply any of the authority of Sega that some might naively assume, it seems more than sensible to me to prefix it to indicate that it's actually SMSPower's interpretation of real Sega hardware rather than something truly canonical.

Lastly, to build on a point Maxim made, if we can't rely on Github to still be in existence 20 years from now, can we rely on SMSPower! to be, at least in its current form? I shudder to think of a world without it, but designing an open standard now would protect users of it (some of whom are historical preservation enthusiasts and some homebrew creators) from problems resulting in a central authority disappearing, fragmenting, or being otherwise unable to maintain its authority.

I realise namespacing requires more bytes, and I'm sensitive to that, but since we're talking about plain (variable length) strings here, and since homebrew ROM capacity is significantly inexpensive, and since we can't retro-fit SDSC headers to original games (excepting of course Trirosmos's idea of adding binary data as a footer to ROMs) I think having the facility of plain strings provides every opportunity to keep identifiers forward compatible.

I guess in summary, I'd ask why shouldn't we build some form of namespacing into this?

Okay, here goes nothing.

Appreciate this might be a level of detail that's just not interesting right now, but it is something that is of some importance to me, and didn't want to miss the boat being part of the discussion. Anyway, feel free to tactfully ignore if needed!

I had a very quick play with seeing if I could come up with a small number of primitives that could maybe be used to build a formal mapper description. The idea would be that this probably wouldn't be embedded in or attached to ROMs, but might serve as one option for canonical backing definitions for the identifiers that do get attached to ROMs. See below for an example of that, roughly along the lines of those already proposed.

I tried to achieve a few things:

1. An extensible format
2. Something that could support arbitrarily deep hierarchies
3. Well known primitives that are generic but also tailored specifically to the problem at hand - so far I have "register", "mux" and "bus"; as well as "mem" which I think can handle both ROMs and RAMs, with some minor parameterisation.
4. Some implicit defaults where possible that might again, lend itself specifically to this problem.
5. Something declarative that might drive autogeneration of both high level software (for emulators) or HDL (for FPGAs).

In line with Trirosmos's observation, it obviously immediately tends towards an HDL, but I feel like so far it is staying declarative and relatively small in domain. Clearly I had to invent some kind of basic expression language for signal selects, which looks C-like (or Verilog-like). Again, I still think it's simple enough so far that a tool could be easily written to parse it.

The example below is just ROM bank mapping, haven't thought about RAM mapping yet.

Also I just threw it together and haven't checked my work, so it might actually be nonsense!

// a very pie in sky attempt to describe ONLY THE ROM BANK MAPPING part of a Sega mapper chip
// in terms of a small number of mapper "primitives"
{
"@id": "smspower/sega/315-5235",
"@type": "mapper",
"@sub": [
{
"@id": "reg0",
"@type": "register",
"width": 6, // could safely default to 8 here?
"wr_en": "@ce && @wr && (@addr & 0xfffd)"
},
{
"@id": "reg1",
"@type": "register",
"width": 6,
"wr_en": "@ce && @wr && (@addr & 0xfffe)"
},
{
"@id": "reg2",
"@type": "register",
"width": 6,
"wr_en": "@ce && @wr && (@addr & 0xffff)"
},
{
"@type": "mux",
"width": 6,
"input": "@reg2",
"output": "banks",
"sel": "@ce && ((@addr & 0xc000) ^ ~0x8000)"
},
{
"@type": "mux",
"width": 6,
"input": "@reg1",
"output": "banks",
"sel": "@ce && ((@addr & 0xc000) ^ ~0x4000)"
},
{
"@type": "mux",
"width": 6,
"input": "0x00",
"output": "banks",
"sel": "@ce && ((@addr & 0xfc00) ^ ~0x0000)" // protect the bottom 1,024 bytes
},
{
"@type": "mux",
"width": 6,
"input": "@reg0",
"output": "banks"
// "sel": "@ce && ((@addr & 0xc000) ^ ~0x2000)" // could be optional since previous cases not met
},
{
"@id": "mem:addr",
"@type": "bus",
"width": 20, // i.e. 256KB address space
"output": "(@banks << 14) | (0x3fff & @addr)"
}
]
}

Possible example of instantiation of a mapper

// A possible example of how that mapper could be instantiated in a header or metadata file
// Note, have played with a more compact syntax here
// also, feels like the ROM section could be implicit somehow
{
"mapper": "smspower/sega/315-5235", // instance with no parameters
"mem": { // instance with parameters
"size": 262144, // or possible use an address width instead? could also be implicit when attached to a single ROM.
// "img": "my_game.sms" // implicit in archive or embedded SDSC header
"rd": "@ce && @rd",
"wr": "0" // i.e. this is a ROM
// "addr": "@mem:addr" // this could be implicit
}
}

willbritton wrote

Well, yes because in short, as I think I said before I don't believe that SMSPower! ought to be the "authority", at least when it comes to the hardware identifiers!

Lastly, to build on a point Maxim made, if we can't rely on Github to still be in existence 20 years from now, can we rely on SMSPower! to be, at least in its current form? I shudder to think of a world without it, but designing an open standard now would protect users of it (some of whom are historical preservation enthusiasts and some homebrew creators) from problems resulting in a central authority disappearing, fragmenting, or being otherwise unable to maintain its authority.

Text files or simple static web pages on SMSPower are a lot easier to archive and access in the future than a Git repository, so I don't know if the comparison is quite apt.

You can try to decentralize where things are stored, mirror documentation and specifications, but in the end there needs to be an agreed upon standard and I don't think you can avoid having that standard be defined by comittee, especially not with just namespacing.

Either way, since it seems so far that the proposed format would end up being some form of JSON or quasi-JSON, you can always add a new tag later saying " ignore these legacy fields, use the new ones ".

willbritton wrote

I think that if you don't suitably qualify the identifiers then you risk repeating essentially what is one of the flaws with iNES, which is that you have an essentially central repository of identifiers which, although (being strings rather than numbers) might seem sensible to you or I right now, might seem arbitrary and incorrect to people in the future; and working around or extending a scheme without namespacing means adding new arbitrary identifiers in an increasingly chaotic way.

Appreciate this might be a level of detail that's just not interesting right now, but it is something that is of some importance to me, and didn't want to miss the boat being part of the discussion. Anyway, feel free to tactfully ignore if needed!

I mean... I see where you're coming from. And I do think ideally we'd be able to write HDL that describes a mapper and use it everywhere.

However, I don't see any way of making a standard like actually work in practice unless it is like iNES in that it's just arbitrary tags associated to specific hardware variations.
Requiring a parser is just a no go, imo. It makes supporting the format significantly more complex without much benefit... unless you consider people going apeshit creating a bunch of incompatible mappers that all do basically the same thing?

EDIT: to illustrate my point, I believe you're pretty much just one step away from creating a bona-fide, " Turing complete " HDL...?
That would be letting registers take the output of a mux as an input.

Trirosmos wrote

You can try to decentralize where things are stored, mirror documentation and specifications, but in the end there needs to be an agreed upon standard and I don't think you can avoid having that standard be defined by comittee, especially not with just namespacing.

There's a difference between a standard and a naming scheme.
Just as with SDSC, I have no problem at all with a committee or some authority owning the standard - that's how standards get made.

The only thing I'm saying is that the naming scheme ought to be extensible from the outset, to try to mitigate at least some (but obviously not all) of the problems that come with a single authority attempting to define things which are simply not definitive enough.

Trirosmos wrote

I mean... I see where you're coming from. And I do think ideally we'd be able to write HDL that describes a mapper and use it everywhere.

However, I don't see any way of making a standard like actually work in practice unless it is like iNES in that it's just arbitrary tags associated to specific hardware variations.

Yeah I understand that sentiment, which is why I've tried to suggest something in two parts - one which looks as much like arbitrary tags, and can be as compact as possible, and another which can back the arbitrary definitions with something formal.

To be clear, I intended HDL / software generation only as one possible option (although for me personally it's an important one ATM), but another totally viable option is that it's simply a formal language that humans can read to understand the hardware; something stronger than the informal descriptions we currently see on wikis, etc.

Trirosmos wrote

Requiring a parser is just a no go, imo. It makes supporting the format significantly more complex without much benefit... unless you consider people going apeshit creating a bunch of incompatible mappers that all do basically the same thing?

Again, I'm not suggesting a parser should be "required" in any particular sense, but I also think that parsing the expression language for code generation if one wanted could be fairly trivial (the intention is actually that it's not much more than a find-and-replace operation to generate either valid C or Verilog syntax).

So one probably wouldn't parse dynamically, only at the point that a particular type of mapper support was compiled into an emulator or built into some hardware. Even then, a totally viable alternative option would be for a human to read the expression and then translate it manually into an implementation.

Otherwise I think trying to avoid an expression syntax makes the entire exercise quite a bit harder, and I think would inevitably involve building more complicated parameters into primitives and higher level assemblies.

willbritton wrote

The only thing I'm saying is that the naming scheme ought to be extensible from the outset, to try to mitigate at least some (but obviously not all) of the problems that come with a single authority attempting to define things which are simply not definitive enough.

Yeah! I just don't think you can make JSON not extensible?
Like, with a binary format like iNES you do run into issues of just not having enough bytes in the header and such.

But with JSON, we can totally just use the " global " name space and later on someone could make a "newsetofbetterdefinedmappers/sega-slot-2-only" mapper in a new namespace.

willbritton wrote

Trirosmos wrote

I mean... I see where you're coming from. And I do think ideally we'd be able to write HDL that describes a mapper and use it everywhere.

However, I don't see any way of making a standard like actually work in practice unless it is like iNES in that it's just arbitrary tags associated to specific hardware variations.

Yeah I understand that sentiment, which is why I've tried to suggest something in two parts - one which looks as much like arbitrary tags, and can be as compact as possible, and another which can back the arbitrary definitions with something formal.

To be clear, I intended HDL / software generation only as one possible option (although for me personally it's an important one ATM), but another totally viable option is that it's simply a formal language that humans can read to understand the hardware; something stronger than the informal descriptions we currently see on wikis, etc.

Trirosmos wrote

Requiring a parser is just a no go, imo. It makes supporting the format significantly more complex without much benefit... unless you consider people going apeshit creating a bunch of incompatible mappers that all do basically the same thing?

Again, I'm not suggesting a parser should be "required" in any particular sense, but I also think that parsing the expression language for code generation if one wanted could be fairly trivial (the intention is actually that it's not much more than a find-and-replace operation to generate either valid C or Verilog syntax).

So one probably wouldn't parse dynamically, only at the point that a particular type of mapper support was compiled into an emulator or built into some hardware. Even then, a totally viable alternative option would be for a human to read the expression and then translate it manually into an implementation.

Otherwise I think trying to avoid an expression syntax makes the entire exercise quite a bit harder, and I think would inevitably involve building more complicated parameters into primitives and higher level assemblies.

Yeah, I don't know if I see much utility in a " simplified " HDL like that unless it's used to dynamically generate mapper code when running a game.

If it's something that you'd then have to either manually or automatically convert into C or Verilog at compile time... then why not just provide mapper specifications in C and Verilog, very well understood languages that everyone can look up documentation for and that can describe not only these simpler mappers but any other that might exist in the future?

fwiw, I find that diagrams and text descriptions of how mappers work are much easier to parse than even reading someone else's HDL, unless it's very well commented code, but I do realize that probably varies person to person.

But if the goal really is dynamic code generation, then I reckon that a language that can only describe mappers that do just bankswitching and not much else might not be future-proof enough to prevent the issue of emulator authors having to write ad-hoc mapper implementations.

Trirosmos wrote

Yeah! I just don't think you can make JSON not extensible?
Like, with a binary format like iNES you do run into issues of just not having enough bytes in the header and such.

But with JSON, we can totally just use the " global " name space and later on someone could make a "newsetofbetterdefinedmappers/sega-slot-2-only" mapper in a new namespace.

I totally agree, and I think my point is really around the presumptuousness of immediately claiming the global namespace and also the potential confusion of "claiming" the "sega-" prefix.
(on this point, is this family of mappers specific to only a subset of Sega products, i.e. SMS/GG and not others like MD?)
I think it would set a good convention to others to start with a namespace, but it would only be a convention because as you point out, there would not necessarily be any restriction on the string field.

Trirosmos wrote

Yeah, I don't know if I see much utility in a " simplified " HDL like that unless it's used to dynamically generate mapper code when running a game.

If it's something that you'd then have to either manually or automatically convert into C or Verilog at compile time... then why not just provide mapper specifications in C and Verilog, very well understood languages that everyone can look up documentation for and that can describe not only these simpler mappers but any other that might exist in the future?

...

But if the goal really is dynamic code generation, then I reckon that a language that can only describe mappers that do just bankswitching and not much else might not be future-proof enough to prevent the issue of emulator authors having to write ad-hoc mapper implementations.

Yeah again agree, those are all reservations I share. I guess the only reason I feel pulled in a "new language" direction at all is because if you try to use an existing formal language then which one do you use? Very few software engineers who write emulators know an HDL, and even within software engineering there is a wide diversity of languages, so why should I pick C, say, over something like Java?

On your last point to be frank I'm trying to ignore the open question of exotic future mapper designs at least in the short term. That said, I feel intuitively that any required extensibility would come from the ability to provision new primitives and add new parameters to existing primitives. But then I don't have the maths chops to prove that hunch!

willbritton wrote

I totally agree, and I think my point is really around the presumptuousness of immediately claiming the global namespace and also the potential confusion of "claiming" the "sega-" prefix.

I think it would set a good convention to others to start with a namespace, but it would only be a convention because as you point out, there would not necessarily be any restriction on the string field.

I for one think it's pretty unlikely we'll ever need another namespace, but, yeah, I don't feel super strongly about not using the global one.

willbritton wrote

(on this point, is this family of mappers specific to only a subset of Sega products, i.e. SMS/GG and not others like MD?)

Sega did create a mapper for the MD, which apparently was available to developers if they felt like their game would've required it, but it was only used for a single release and it's frankly way more complicated than it needs to be.
Krikzz then implemented something that is almost exactly like the Sega mapper, but that mapped the RAM protect bit at a different register. It also requires putting a magic string in the ROM header.
So, homebrew using both the Sega mapper and SRAM will often times have issues running on EverDrives.

Then there's at least Pier Solar and Paprium with completely custom, bespoke mappers.

willbritton wrote

Yeah again agree, those are all reservations I share. I guess the only reason I feel pulled in a "new language" direction at all is because if you try to use an existing formal language then which one do you use? Very few software engineers who write emulators know an HDL, and even within software engineering there is a wide diversity of languages, so why should I pick C, say, over something like Java?

Could even be some form of pseudocode, tbh, tho I feel like C in particular has got to be very widely understood, being the most popular systems programming language.
But HDL probably is the " right " tool for this, and there you don't have nearly as many viable options.

willbritton wrote

On your last point to be frank I'm trying to ignore the open question of exotic future mapper designs at least in the short term. That said, I feel intuitively that any required extensibility would come from the ability to provision new primitives and add new parameters to existing primitives. But then I don't have the maths chops to prove that hunch!

I guess my main two interests in defining a metadata format are " ugh, mapper detection heuristics are a pain in the ass " but also " I'd really like to make some wacky new mappers with DMA or coprocessors at some point ", so that's why I keep bringing that point back :P

To be fair, if you have registers and NANDs, then there's nothing you can't implement, but the implementation might look very dumb xD

I suspect we're going nowhere if we don't clearly define the purpose of this.

I *think* the (originally desired) purpose is not to fully describe hardware functionalities or to fully describe the hardware components present into a physical cartridge for historical/archival reasons... one could take pictures and write pages long documents for that if needed.

We need something *simple* to tell *new and updated* emulators how the attached ROM dump is to be 'handled' for the software to work properly.

So, probably because I tend to imagine simpler solutions to simpler problems before switching to the bigger picture, I think the simple solution of creating a list of unique names for existing mappers and a few attributes to describe basic variants / added functionalities is all that we need here.

If one day SMS Power! will no longer be the only naming authority for this, which I greatly doubt, they'll likely use some prefix like:

{mapper: naming-auth/mapper-name}

of they will figure out some other way themselves.

Otherwise I suspect this is just going nowhere. Also sorry if this doesn't sound respectful - I have great respects for your ideas and comments, I just think that's not the KISS approach we instead clearly need.

I feel like maybe I'm spreading a little confusion by talking about two things at once here, even though I've tried a few times to clarify myself, so let me back-pedal a bit:

I agree that there should be a really simple format which can be attached to ROMs and which describes hardware in terms of simple names.

I think we're on the same page there. The examples you gave, and the ones Maxim gave, and the shorter example I gave to that effect (at the bottom of my post a few back) I think were all much of a muchness.

The other thing I'm talking about, which is the ability to actually define what the names in the simple format means is much harder to do, and can certainly come later, but I think it will ultimately be important for sure in hardware problems (consider Everdrive's rather limited support for mappers) but I would have thought it would make emulator implementation much more straightforward too.

To repeat: it can definitely come later.

But I am thinking about it now, which is why my preference would be for a format that is at least forward looking in some of these respects.

You're right, I can probably live without namespacing on the basis that it can be retrofitted by more namespace-motivated people later, so it's a hill I guess I won't die on.

The one thing I just can't imagine the metadata being useful without is some kind of resolution mechanism so that if you give me a metadata file which specifies a "foo-mapper" then I know exactly where to go to find out how to make a foo-mapper.

For the record I'd also probably vote for tar over zip archiving to make hardware support easier, but fully acknowledge emulator authors would probably prefer the opposite.

I have no problem with keeping things simple, but there's a tradeoff between simplicity and utility, especially if you're proposing inventing a new file format and then asking emulator authors to support it. As you quite rightly pointed out, one even simpler solution here would be to pester a single emulator author to add a few command line switches.

Good! :)

So I think we should actually start, for instance by making it work for Maxim's needs first.

Is this:

{mapper:sega, sram-size:8192}

good enough or we prefer to require quoted text?

I don't think we really need these but it's a minor issue and I won't fight against that.

On whether it should be strictly JSON (which would imply quoted fields) I don't much care either, but I can definitely see emulator authors preferring a format that allows them to use a library rather than writing their own parser. Maybe it would be worth asking some?

On the specific example, and bearing in mind Maxim's later example of being able to specify other variables, like PAL/NTSC, VDP variant, etc., do we think there's some case for hierarchy (which JSON can theoretically support), or is having everything as a top level field sufficient?

I guess one way to maintain both would be to have a convention which optionally prefixes parameters with the hardware section with which they are associated, e.g.

"sram-size" is the "size" parameter of the "sram" hardware section, and "cpu-freq" is the "freq" parameter of the "cpu" seciton, etc.

In your example "mapper" has no prefix, so perhaps that indicates that there are no parameters relating to the mapper configuration - just a name is sufficient.

If we want to target simplicity of parsing on Z80 then we might prefer to avoid JSON; but in terms of good library support for emulators to drop something in, it’s the best choice and I think the possibility for extensibility and structure is worth having. We might assume that all (future) use on hardware is going to be on microcontrollers that will be able to manage with the library - is that true? How about zip support on those same microcontrollers?

I can envisage a world of flattened, newline-separated key=value - a subset of INI, I guess - if we want to make parsing as simple as possible.

Yes, we should totally invite Calindro and any other (especially developer-oriented) emulator author(s).

Maxim wrote

I can envisage a world of flattened, newline-separated key=value - a subset of INI, I guess - if we want to make parsing as simple as possible.

I'd avoid newlines, as I think it would be good to have the chance to pass the same data on a command line parameter.

As for parsing complexity, I suspect that on a computer whatever the language used to code the emulator there will be some way to get a string into tokens and then parse each token, as in this pseudocode:

split (source, "{}," params)
for each param in params
split (trim(param), ":", key_value)

(of course you need to trim and verify that both keys and values are valid, etc)

parsing on Z80 is probably better done left to right, 'ignoring' brackets and whitespace, using the `:` (or `=` if we prefer that) as a key/pair separator, and the comma as a key-pair token delimiter. It's not that hard...

Maxim wrote

We might assume that all (future) use on hardware is going to be on microcontrollers that will be able to manage with the library - is that true?

Can't speak for the rest of the hardware community, but yes I'd say that any system which is expected to process these headers and do something useful with them is more than likely capable of running a JSON parser.
Here's a minimalistic JSON parser that I'm sure would easily fit on a microcontroller.

Less luck finding a Z80 one, out of interest, but amusingly here's a 6502 JSON parser which looks surprisingly big (code-wise) at first glance.

Maxim wrote

How about zip support on those same microcontrollers?

I think this is going to be trickier and the issue is likely around peak RAM requirements. Frustratingly, I've not found much discussion out there around embedded unzipping solutions. The only github project I've seen so far which explicitly mentions the ZIP file format with minimal memory overhead is this one which is not particularly big (popular).

I do agree that ZIP is by far the most conventional way to archive and compress if one includes Windows host systems as well as Linux ones so it would be nice to support it. Just thinking aloud, but for things like flashcarts I wonder if it would be reasonable to decompress on the host system, and have the metadata files sit alongside the decompressed ROMs as an alternative to dealing with the archives direct. After all, it's practically impossible to fill a modern SD card with even decompressed ROMs.

That’s pretty much what we had planned - but it demands a certain amount of making the filename of the metadata file match that of the ROM. Hardly difficult, but potentially something to go wrong.

ZIP is basically a non-starter on anything before the Megadrive - DEFLATE requires 32KB of RAM for back-references.

You could say that "only ZIPs without compression are supported" but that would break the ease of workflow.

But do we need to decompress on a Master System?

willbritton wrote

Here's a minimalistic JSON parser that I'm sure would easily fit on a microcontroller.

Just for the record, I integrated JSMN on the RP2040 part of the Retcon AV and it works fine for JSON parsing.

It's actually quite clever - it's a header-only implementation, has no dependencies and the memory footprint for holding the parsed document is limited to 16 bytes * the number of "tokens" you want to store, tokens being keys, values and non-leaf nodes. So if you know your JSON document is likely to be small you can tune the allocation down accordingly. (note: you might also be buffering the raw JSON file in memory too, depending on your approach)
You can stream the input too, so long as you have enough memory to load each chunk of tokens, although I haven't tried this approach myself.

I haven't tried it yet, but I don't see any reason why it wouldn't work on the Z80 either, e.g. via SDCC.

Author	Message
sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Tue Feb 27, 2024 4:46 pm
	well, unfortunate but irrelevant (if really needed, one day we'll RE that and implement it anyway probably)

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Feb 29, 2024 11:07 am
	So it seems that despite my best efforts :) we’ve focused on the general problem of providing metadata about the cartridge with the ROM image(s). I’ll refer again to https://web.archive.org/web/20200322151931/https://byuu.net/cartridges/boards which is about exactly this - and I actually agree that it is a good idea, but also that it’s almost impossible to solve due to the existence of old emulators, flash carts and ROM collections. That’s why I was trying to narrow the scope to homebrew, so we can start with new things.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Thu Feb 29, 2024 1:13 pm
	But surely it's not really possible to separate those two things fully. Most homebrew efforts are built standing squarely on the shoulders of the original technology, and emulators serve the dual purpose of allowing one to enjoy original conserved games, as well as development, testing and playing of new homebrew games. I also don't really see a contradiction between the two needs, in terms of how a proposed metadata system would have to operate. Regardless of whether it's embedded in the ROM or a companion to it, the semantic contents (and IMO the syntactic contents) should be the same; and if it's suitably flexible it ought to be possible to enable homebrew developers to model the most common cases they need, and over time to model not only most original technology, but potentially some more exotic homebrew technology. The immediate problem you pose I think is valid, even if only a minor irritation for most — can we specify some very basic variations in the standard Sega mapper using metadata, so that an emulator theoretically doesn't need to rely on heuristics and can instead 100% reliably supply the hardware that the developer wants? Next step from there would be considering what other mapper needs might be, but I suspect that path will inevitably align with known original mapper tech (e.g. Codemasters, etc.), simply because most of it is "obvious", given the constraints on the technology at the time. On that, I think I'd add that it's not really like the constraints on real hardware have completely opened up today, allowing an infinite range of mapper technology. Sure, it's possible to invent some exotic things, but we're still constrained by the base system hardware, as well as the practicalities of real hardware design and manufacture, so personally I say again let's put the exotic hardware to one side for now and focus on the well-known core!

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Thu Feb 29, 2024 1:31 pm
	Maxim wrote So it seems that despite my best efforts :) we’ve focused on the general problem of providing metadata about the cartridge with the ROM image(s). [...] That’s why I was trying to narrow the scope to homebrew, so we can start with new things. I doubt we can solve one problem without also providing a general solution. But, for what is worth, emulators could add some support of some kind (a command line parameter?) to solve your specific problem, such as 'forcing' a mapper even on ROMs <= 48 KiB. emulator.exe -forcemapper your_rom.sms Still I think we were pretty close to the solution to the whole problem. I'm open to any simple format for the text-file-based metadata. Also we need a draft list for the names of the mappers that we already know are existing, then we will discuss the default behavior (when needed) for each of those. I already expressed my ideas so I won't bother you more. But I can provide a draft names list - that will surely need lots of fixing.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Feb 29, 2024 5:48 pm Last edited by Maxim on Thu Feb 29, 2024 9:17 pm; edited 1 time in total
	In other areas they chose a fourCC-style naming convention but that may be more about saving space in constrained circumstances. I think we can consider: * Mappers: * Sega “standard” mapper, with options for slots (some are slot 2 only), SRAM (enabled or not, SRAM size, battery backed or not). Defaults would be the “do everything” mapper that you get in emulators/flashcarts. We do not (yet) need to support the page number shift features. * Codemasters mapper, maybe with similar options * Maybe a generic but parameterised mapper approach to handle most of the Korean ones * …and more for special cases like the MSXish 8/16KB slot ones and the byte flipping one * Save functionality (in addition the the mapper) * Serial EEPROMs in Game Gear games, parameterised as needed (size, address range?) * Maybe a way to specify the VDP variant needed - including TMS9918a vs. SMS VDPs vs GG vs Mega Drive * 50Hz vs 60Hz required * FM chip supported * Required and supported peripherals Any of these could be optional. I’d like for any emulator that might support this, it can be embedded in the ROM via the SDSC description as I suggested, or in a file next to the ROM (same base filename), or in a file in the same zip (same base filename or some standard name like metadata.json), or in a command line, with the ones on the right overriding the ones on the left,

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Thu Feb 29, 2024 6:55 pm
	Can you explain (or point towards articles) more on: Maxim wrote * Save functionality (in addition [to] the mapper) and Maxim wrote * Serial EEPROMs in Game Gear games, parameterised as needed (size, address range?) ?

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Feb 29, 2024 9:16 pm
	The most common save on Sega games was to battery-backed SRAM, but a few Game Gear games had a 128 byte (?) serial EEPROM. Some homebrew developers have also talked about saving to unused flash.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Fri Mar 01, 2024 10:42 am
	Maxim wrote * Maybe a way to specify the VDP variant needed - including TMS9918a vs. SMS VDPs vs GG vs Mega Drive * 50Hz vs 60Hz required * FM chip supported * Required and supported peripherals This is indeed interesting, even if it's not something strictly bound to hardware or features present in the cartridge - but in the light of 'let's give additional info to the emulator to make this software work properly' it's something we could address too :) Now, mappers. I would use 'sega' as the general / feature-rich (if we don't need feature-complete after all) name for the SEGA mapper, then use `sega-<variant>` for SEGA mapper compatible mappers instead of using some other manufacturer name. Examples: - 'sega-slot2only' (or a better name) for the slot2 only mappers regardless if they're from ichigo or raphnet (as an example, I don't even know if they ever in fact did that!) - 'sega-slot0broken' (or a better name) for the krikzz' broken legacy SEGA mapper (if we ever need to emulate that!) everything else I already posted regarding how an emulator should calculate the number of bits for each page register I won't repeat it again, and also everything regarding SRAM size and persistency/volatility. Am I missing something regarding the SEGA mapper 'family'? Then the CodeMasters' mapper. Here we probably have only two specimen so we could just use 'codemasters' and 'codemasters-ram' (the 'Ernie Els Golf' one) for instance. Again, should we find another very similar mapper, even coming from a different manufacturer, we would probably better create another 'codemasters-<variant>' Then we have the MSX mappers, with the Nemesis variant. We could use 'msx' and 'msx-fixed8k' (or similar name) so that we don't reference a specific game directly - but if using a game name makes it simpler we could just agree on 'msx-nemesis'. Finally, the korean mappers. Of course the group is not very 'homogeneous', so we should probably create a few main groups with a few variant each. I can dive into the MEKA defined mappers list and see if I can draft that too, if we agree this is a possible way.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Fri Mar 01, 2024 10:59 am
	oh, OK I forgot one thing about the SEGA mapper. This: The registers are thought to be undefined after a reset for games using the 831001 and 832011 ROMs and 315-5208 mapper. The 315-5235 mapper is reset to FFFC=00, FFFD=00, FFFE=01, FFFF=02 after a power-up reset. we should decide what's the default and probably an attribute to switch to a specific variant ( 'initialized' / 'uninitialized' probably? )

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Fri Mar 01, 2024 11:22 am
	Would like to take the time to reply in full later, but might I suggest that we try to approach the cataloguing of known hardware rigorously with part numbers in mind where possible. (@sverx not sure whether this is exactly what you meant, but I read it as deciding which of the known mappers' reset behaviours we wanted to pick) So rather than deciding what the "default behaviour" of certain things is, we try and faithfully capture the exact behaviour of known devices based on part number, and then simply allocate one particular device as the preferred "standard" Sega mapper device. I would say that whatever known device represents the biggest superset of functionality is a good choice. I'd also like to repeat a plea for some kind of namespacing, no matter how it's achieved, so for example: smspower/sega/315-5235 might also alias to smspower/sega/standard or just smspower/sega I feel a little funny about calling something "sega" if it's actually a hybrid of several sega-like ideas, and a combination that maybe never existed in real hardware. Maybe there's a slightly alternative name we can use.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Fri Mar 01, 2024 11:26 am
	Games that don’t rely on it being initialised won’t care if it is, so specifying the uninitialised variant is only useful for breaking things, isn’t it?

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Fri Mar 01, 2024 1:03 pm
	willbritton wrote I'd also like to repeat a plea for some kind of namespacing, no matter how it's achieved I fail to see the reason for this. Would you explain more why should be this important, given that SMS Power! will be the only naming authority? Maxim wrote Games that don’t rely on it being initialised won’t care if it is, so specifying the uninitialised variant is only useful for breaking things, isn’t it? I'd say 'useful to test things' :)

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Fri Mar 01, 2024 2:11 pm
	sverx wrote willbritton wrote I'd also like to repeat a plea for some kind of namespacing, no matter how it's achieved I fail to see the reason for this. Would you explain more why should be this important, given that SMS Power! will be the only naming authority? Well, yes because in short, as I think I said before I don't believe that SMSPower! ought to be the "authority", at least when it comes to the hardware identifiers! I think the idea of a metadata specification is brilliant, but I also think the specification should be "open" - in the same sense that SDSC is open. Correct me if I'm wrong, but SMSPower! doesn't maintain a known list of authors, for example, because the author field is just a plain string. (This stands in contrast to the Sega header, which numerically enumerates game vendors) I think that if you don't suitably qualify the identifiers then you risk repeating essentially what is one of the flaws with iNES, which is that you have an essentially central repository of identifiers which, although (being strings rather than numbers) might seem sensible to you or I right now, might seem arbitrary and incorrect to people in the future; and working around or extending a scheme without namespacing means adding new arbitrary identifiers in an increasingly chaotic way. To expand on my previous comment, I also think that simply calling something "sega" when it may not accurately describe any real Sega hardware could be confusing, and would certainly not imply any of the authority of Sega that some might naively assume, it seems more than sensible to me to prefix it to indicate that it's actually SMSPower's interpretation of real Sega hardware rather than something truly canonical. Lastly, to build on a point Maxim made, if we can't rely on Github to still be in existence 20 years from now, can we rely on SMSPower! to be, at least in its current form? I shudder to think of a world without it, but designing an open standard now would protect users of it (some of whom are historical preservation enthusiasts and some homebrew creators) from problems resulting in a central authority disappearing, fragmenting, or being otherwise unable to maintain its authority. I realise namespacing requires more bytes, and I'm sensitive to that, but since we're talking about plain (variable length) strings here, and since homebrew ROM capacity is significantly inexpensive, and since we can't retro-fit SDSC headers to original games (excepting of course Trirosmos's idea of adding binary data as a footer to ROMs) I think having the facility of plain strings provides every opportunity to keep identifiers forward compatible. I guess in summary, I'd ask why shouldn't we build some form of namespacing into this?

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Fri Mar 01, 2024 3:35 pm
	Okay, here goes nothing. Appreciate this might be a level of detail that's just not interesting right now, but it is something that is of some importance to me, and didn't want to miss the boat being part of the discussion. Anyway, feel free to tactfully ignore if needed! I had a very quick play with seeing if I could come up with a small number of primitives that could maybe be used to build a formal mapper description. The idea would be that this probably wouldn't be embedded in or attached to ROMs, but might serve as one option for canonical backing definitions for the identifiers that do get attached to ROMs. See below for an example of that, roughly along the lines of those already proposed. I tried to achieve a few things: 1. An extensible format 2. Something that could support arbitrarily deep hierarchies 3. Well known primitives that are generic but also tailored specifically to the problem at hand - so far I have "register", "mux" and "bus"; as well as "mem" which I think can handle both ROMs and RAMs, with some minor parameterisation. 4. Some implicit defaults where possible that might again, lend itself specifically to this problem. 5. Something declarative that might drive autogeneration of both high level software (for emulators) or HDL (for FPGAs). In line with Trirosmos's observation, it obviously immediately tends towards an HDL, but I feel like so far it is staying declarative and relatively small in domain. Clearly I had to invent some kind of basic expression language for signal selects, which looks C-like (or Verilog-like). Again, I still think it's simple enough so far that a tool could be easily written to parse it. The example below is just ROM bank mapping, haven't thought about RAM mapping yet. Also I just threw it together and haven't checked my work, so it might actually be nonsense! // a very pie in sky attempt to describe ONLY THE ROM BANK MAPPING part of a Sega mapper chip // in terms of a small number of mapper "primitives" { "@id": "smspower/sega/315-5235", "@type": "mapper", "@sub": [ { "@id": "reg0", "@type": "register", "width": 6, // could safely default to 8 here? "wr_en": "@ce && @wr && (@addr & 0xfffd)" }, { "@id": "reg1", "@type": "register", "width": 6, "wr_en": "@ce && @wr && (@addr & 0xfffe)" }, { "@id": "reg2", "@type": "register", "width": 6, "wr_en": "@ce && @wr && (@addr & 0xffff)" }, { "@type": "mux", "width": 6, "input": "@reg2", "output": "banks", "sel": "@ce && ((@addr & 0xc000) ^ ~0x8000)" }, { "@type": "mux", "width": 6, "input": "@reg1", "output": "banks", "sel": "@ce && ((@addr & 0xc000) ^ ~0x4000)" }, { "@type": "mux", "width": 6, "input": "0x00", "output": "banks", "sel": "@ce && ((@addr & 0xfc00) ^ ~0x0000)" // protect the bottom 1,024 bytes }, { "@type": "mux", "width": 6, "input": "@reg0", "output": "banks" // "sel": "@ce && ((@addr & 0xc000) ^ ~0x2000)" // could be optional since previous cases not met }, { "@id": "mem:addr", "@type": "bus", "width": 20, // i.e. 256KB address space "output": "(@banks << 14) \| (0x3fff & @addr)" } ] } Possible example of instantiation of a mapper // A possible example of how that mapper could be instantiated in a header or metadata file // Note, have played with a more compact syntax here // also, feels like the ROM section could be implicit somehow { "mapper": "smspower/sega/315-5235", // instance with no parameters "mem": { // instance with parameters "size": 262144, // or possible use an address width instead? could also be implicit when attached to a single ROM. // "img": "my_game.sms" // implicit in archive or embedded SDSC header "rd": "@ce && @rd", "wr": "0" // i.e. this is a ROM // "addr": "@mem:addr" // this could be implicit } }

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Fri Mar 01, 2024 11:35 pm
	willbritton wrote Well, yes because in short, as I think I said before I don't believe that SMSPower! ought to be the "authority", at least when it comes to the hardware identifiers! Lastly, to build on a point Maxim made, if we can't rely on Github to still be in existence 20 years from now, can we rely on SMSPower! to be, at least in its current form? I shudder to think of a world without it, but designing an open standard now would protect users of it (some of whom are historical preservation enthusiasts and some homebrew creators) from problems resulting in a central authority disappearing, fragmenting, or being otherwise unable to maintain its authority. Text files or simple static web pages on SMSPower are a lot easier to archive and access in the future than a Git repository, so I don't know if the comparison is quite apt. You can try to decentralize where things are stored, mirror documentation and specifications, but in the end there needs to be an agreed upon standard and I don't think you can avoid having that standard be defined by comittee, especially not with just namespacing. Either way, since it seems so far that the proposed format would end up being some form of JSON or quasi-JSON, you can always add a new tag later saying " ignore these legacy fields, use the new ones ". willbritton wrote I think that if you don't suitably qualify the identifiers then you risk repeating essentially what is one of the flaws with iNES, which is that you have an essentially central repository of identifiers which, although (being strings rather than numbers) might seem sensible to you or I right now, might seem arbitrary and incorrect to people in the future; and working around or extending a scheme without namespacing means adding new arbitrary identifiers in an increasingly chaotic way. Appreciate this might be a level of detail that's just not interesting right now, but it is something that is of some importance to me, and didn't want to miss the boat being part of the discussion. Anyway, feel free to tactfully ignore if needed! I mean... I see where you're coming from. And I do think ideally we'd be able to write HDL that describes a mapper and use it everywhere. However, I don't see any way of making a standard like actually work in practice unless it is like iNES in that it's just arbitrary tags associated to specific hardware variations. Requiring a parser is just a no go, imo. It makes supporting the format significantly more complex without much benefit... unless you consider people going apeshit creating a bunch of incompatible mappers that all do basically the same thing? EDIT: to illustrate my point, I believe you're pretty much just one step away from creating a bona-fide, " Turing complete " HDL...? That would be letting registers take the output of a mux as an input.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Sat Mar 02, 2024 11:21 am
	Trirosmos wrote You can try to decentralize where things are stored, mirror documentation and specifications, but in the end there needs to be an agreed upon standard and I don't think you can avoid having that standard be defined by comittee, especially not with just namespacing. There's a difference between a standard and a naming scheme. Just as with SDSC, I have no problem at all with a committee or some authority owning the standard - that's how standards get made. The only thing I'm saying is that the naming scheme ought to be extensible from the outset, to try to mitigate at least some (but obviously not all) of the problems that come with a single authority attempting to define things which are simply not definitive enough. Trirosmos wrote I mean... I see where you're coming from. And I do think ideally we'd be able to write HDL that describes a mapper and use it everywhere. However, I don't see any way of making a standard like actually work in practice unless it is like iNES in that it's just arbitrary tags associated to specific hardware variations. Yeah I understand that sentiment, which is why I've tried to suggest something in two parts - one which looks as much like arbitrary tags, and can be as compact as possible, and another which can back the arbitrary definitions with something formal. To be clear, I intended HDL / software generation only as one possible option (although for me personally it's an important one ATM), but another totally viable option is that it's simply a formal language that humans can read to understand the hardware; something stronger than the informal descriptions we currently see on wikis, etc. Trirosmos wrote Requiring a parser is just a no go, imo. It makes supporting the format significantly more complex without much benefit... unless you consider people going apeshit creating a bunch of incompatible mappers that all do basically the same thing? Again, I'm not suggesting a parser should be "required" in any particular sense, but I also think that parsing the expression language for code generation if one wanted could be fairly trivial (the intention is actually that it's not much more than a find-and-replace operation to generate either valid C or Verilog syntax). So one probably wouldn't parse dynamically, only at the point that a particular type of mapper support was compiled into an emulator or built into some hardware. Even then, a totally viable alternative option would be for a human to read the expression and then translate it manually into an implementation. Otherwise I think trying to avoid an expression syntax makes the entire exercise quite a bit harder, and I think would inevitably involve building more complicated parameters into primitives and higher level assemblies.

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Sat Mar 02, 2024 1:03 pm
	willbritton wrote The only thing I'm saying is that the naming scheme ought to be extensible from the outset, to try to mitigate at least some (but obviously not all) of the problems that come with a single authority attempting to define things which are simply not definitive enough. Yeah! I just don't think you can make JSON not extensible? Like, with a binary format like iNES you do run into issues of just not having enough bytes in the header and such. But with JSON, we can totally just use the " global " name space and later on someone could make a "newsetofbetterdefinedmappers/sega-slot-2-only" mapper in a new namespace. willbritton wrote Trirosmos wrote I mean... I see where you're coming from. And I do think ideally we'd be able to write HDL that describes a mapper and use it everywhere. However, I don't see any way of making a standard like actually work in practice unless it is like iNES in that it's just arbitrary tags associated to specific hardware variations. Yeah I understand that sentiment, which is why I've tried to suggest something in two parts - one which looks as much like arbitrary tags, and can be as compact as possible, and another which can back the arbitrary definitions with something formal. To be clear, I intended HDL / software generation only as one possible option (although for me personally it's an important one ATM), but another totally viable option is that it's simply a formal language that humans can read to understand the hardware; something stronger than the informal descriptions we currently see on wikis, etc. Trirosmos wrote Requiring a parser is just a no go, imo. It makes supporting the format significantly more complex without much benefit... unless you consider people going apeshit creating a bunch of incompatible mappers that all do basically the same thing? Again, I'm not suggesting a parser should be "required" in any particular sense, but I also think that parsing the expression language for code generation if one wanted could be fairly trivial (the intention is actually that it's not much more than a find-and-replace operation to generate either valid C or Verilog syntax). So one probably wouldn't parse dynamically, only at the point that a particular type of mapper support was compiled into an emulator or built into some hardware. Even then, a totally viable alternative option would be for a human to read the expression and then translate it manually into an implementation. Otherwise I think trying to avoid an expression syntax makes the entire exercise quite a bit harder, and I think would inevitably involve building more complicated parameters into primitives and higher level assemblies. Yeah, I don't know if I see much utility in a " simplified " HDL like that unless it's used to dynamically generate mapper code when running a game. If it's something that you'd then have to either manually or automatically convert into C or Verilog at compile time... then why not just provide mapper specifications in C and Verilog, very well understood languages that everyone can look up documentation for and that can describe not only these simpler mappers but any other that might exist in the future? fwiw, I find that diagrams and text descriptions of how mappers work are much easier to parse than even reading someone else's HDL, unless it's very well commented code, but I do realize that probably varies person to person. But if the goal really is dynamic code generation, then I reckon that a language that can only describe mappers that do just bankswitching and not much else might not be future-proof enough to prevent the issue of emulator authors having to write ad-hoc mapper implementations.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Sat Mar 02, 2024 1:30 pm
	Trirosmos wrote Yeah! I just don't think you can make JSON not extensible? Like, with a binary format like iNES you do run into issues of just not having enough bytes in the header and such. But with JSON, we can totally just use the " global " name space and later on someone could make a "newsetofbetterdefinedmappers/sega-slot-2-only" mapper in a new namespace. I totally agree, and I think my point is really around the presumptuousness of immediately claiming the global namespace and also the potential confusion of "claiming" the "sega-" prefix. (on this point, is this family of mappers specific to only a subset of Sega products, i.e. SMS/GG and not others like MD?) I think it would set a good convention to others to start with a namespace, but it would only be a convention because as you point out, there would not necessarily be any restriction on the string field. Trirosmos wrote Yeah, I don't know if I see much utility in a " simplified " HDL like that unless it's used to dynamically generate mapper code when running a game. If it's something that you'd then have to either manually or automatically convert into C or Verilog at compile time... then why not just provide mapper specifications in C and Verilog, very well understood languages that everyone can look up documentation for and that can describe not only these simpler mappers but any other that might exist in the future? ... But if the goal really is dynamic code generation, then I reckon that a language that can only describe mappers that do just bankswitching and not much else might not be future-proof enough to prevent the issue of emulator authors having to write ad-hoc mapper implementations. Yeah again agree, those are all reservations I share. I guess the only reason I feel pulled in a "new language" direction at all is because if you try to use an existing formal language then which one do you use? Very few software engineers who write emulators know an HDL, and even within software engineering there is a wide diversity of languages, so why should I pick C, say, over something like Java? On your last point to be frank I'm trying to ignore the open question of exotic future mapper designs at least in the short term. That said, I feel intuitively that any required extensibility would come from the ability to provision new primitives and add new parameters to existing primitives. But then I don't have the maths chops to prove that hunch!

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Sat Mar 02, 2024 1:53 pm
	willbritton wrote I totally agree, and I think my point is really around the presumptuousness of immediately claiming the global namespace and also the potential confusion of "claiming" the "sega-" prefix. I think it would set a good convention to others to start with a namespace, but it would only be a convention because as you point out, there would not necessarily be any restriction on the string field. I for one think it's pretty unlikely we'll ever need another namespace, but, yeah, I don't feel super strongly about not using the global one. willbritton wrote (on this point, is this family of mappers specific to only a subset of Sega products, i.e. SMS/GG and not others like MD?) Sega did create a mapper for the MD, which apparently was available to developers if they felt like their game would've required it, but it was only used for a single release and it's frankly way more complicated than it needs to be. Krikzz then implemented something that is almost exactly like the Sega mapper, but that mapped the RAM protect bit at a different register. It also requires putting a magic string in the ROM header. So, homebrew using both the Sega mapper and SRAM will often times have issues running on EverDrives. Then there's at least Pier Solar and Paprium with completely custom, bespoke mappers. willbritton wrote Yeah again agree, those are all reservations I share. I guess the only reason I feel pulled in a "new language" direction at all is because if you try to use an existing formal language then which one do you use? Very few software engineers who write emulators know an HDL, and even within software engineering there is a wide diversity of languages, so why should I pick C, say, over something like Java? Could even be some form of pseudocode, tbh, tho I feel like C in particular has got to be very widely understood, being the most popular systems programming language. But HDL probably is the " right " tool for this, and there you don't have nearly as many viable options. willbritton wrote On your last point to be frank I'm trying to ignore the open question of exotic future mapper designs at least in the short term. That said, I feel intuitively that any required extensibility would come from the ability to provision new primitives and add new parameters to existing primitives. But then I don't have the maths chops to prove that hunch! I guess my main two interests in defining a metadata format are " ugh, mapper detection heuristics are a pain in the ass " but also " I'd really like to make some wacky new mappers with DMA or coprocessors at some point ", so that's why I keep bringing that point back :P To be fair, if you have registers and NANDs, then there's nothing you can't implement, but the implementation might look very dumb xD

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Sun Mar 03, 2024 3:37 pm
	I suspect we're going nowhere if we don't clearly define the purpose of this. I think the (originally desired) purpose is not to fully describe hardware functionalities or to fully describe the hardware components present into a physical cartridge for historical/archival reasons... one could take pictures and write pages long documents for that if needed. We need something simple to tell new and updated emulators how the attached ROM dump is to be 'handled' for the software to work properly. So, probably because I tend to imagine simpler solutions to simpler problems before switching to the bigger picture, I think the simple solution of creating a list of unique names for existing mappers and a few attributes to describe basic variants / added functionalities is all that we need here. If one day SMS Power! will no longer be the only naming authority for this, which I greatly doubt, they'll likely use some prefix like: {mapper: naming-auth/mapper-name} of they will figure out some other way themselves. Otherwise I suspect this is just going nowhere. Also sorry if this doesn't sound respectful - I have great respects for your ideas and comments, I just think that's not the KISS approach we instead clearly need.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Sun Mar 03, 2024 7:53 pm
	I feel like maybe I'm spreading a little confusion by talking about two things at once here, even though I've tried a few times to clarify myself, so let me back-pedal a bit: I agree that there should be a really simple format which can be attached to ROMs and which describes hardware in terms of simple names. I think we're on the same page there. The examples you gave, and the ones Maxim gave, and the shorter example I gave to that effect (at the bottom of my post a few back) I think were all much of a muchness. The other thing I'm talking about, which is the ability to actually define what the names in the simple format means is much harder to do, and can certainly come later, but I think it will ultimately be important for sure in hardware problems (consider Everdrive's rather limited support for mappers) but I would have thought it would make emulator implementation much more straightforward too. To repeat: it can definitely come later. But I am thinking about it now, which is why my preference would be for a format that is at least forward looking in some of these respects. You're right, I can probably live without namespacing on the basis that it can be retrofitted by more namespace-motivated people later, so it's a hill I guess I won't die on. The one thing I just can't imagine the metadata being useful without is some kind of resolution mechanism so that if you give me a metadata file which specifies a "foo-mapper" then I know exactly where to go to find out how to make a foo-mapper. For the record I'd also probably vote for tar over zip archiving to make hardware support easier, but fully acknowledge emulator authors would probably prefer the opposite. I have no problem with keeping things simple, but there's a tradeoff between simplicity and utility, especially if you're proposing inventing a new file format and then asking emulator authors to support it. As you quite rightly pointed out, one even simpler solution here would be to pester a single emulator author to add a few command line switches.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Mar 04, 2024 9:48 am
	Good! :) So I think we should actually start, for instance by making it work for Maxim's needs first. Is this: {mapper:sega, sram-size:8192} good enough or we prefer to require quoted text? I don't think we really need these but it's a minor issue and I won't fight against that.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Mar 04, 2024 10:06 am
	On whether it should be strictly JSON (which would imply quoted fields) I don't much care either, but I can definitely see emulator authors preferring a format that allows them to use a library rather than writing their own parser. Maybe it would be worth asking some? On the specific example, and bearing in mind Maxim's later example of being able to specify other variables, like PAL/NTSC, VDP variant, etc., do we think there's some case for hierarchy (which JSON can theoretically support), or is having everything as a top level field sufficient? I guess one way to maintain both would be to have a convention which optionally prefixes parameters with the hardware section with which they are associated, e.g. "sram-size" is the "size" parameter of the "sram" hardware section, and "cpu-freq" is the "freq" parameter of the "cpu" seciton, etc. In your example "mapper" has no prefix, so perhaps that indicates that there are no parameters relating to the mapper configuration - just a name is sufficient.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Mar 04, 2024 10:39 am
	If we want to target simplicity of parsing on Z80 then we might prefer to avoid JSON; but in terms of good library support for emulators to drop something in, it’s the best choice and I think the possibility for extensibility and structure is worth having. We might assume that all (future) use on hardware is going to be on microcontrollers that will be able to manage with the library - is that true? How about zip support on those same microcontrollers? I can envisage a world of flattened, newline-separated key=value - a subset of INI, I guess - if we want to make parsing as simple as possible.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Mar 04, 2024 10:59 am
	Yes, we should totally invite Calindro and any other (especially developer-oriented) emulator author(s). Maxim wrote I can envisage a world of flattened, newline-separated key=value - a subset of INI, I guess - if we want to make parsing as simple as possible. I'd avoid newlines, as I think it would be good to have the chance to pass the same data on a command line parameter. As for parsing complexity, I suspect that on a computer whatever the language used to code the emulator there will be some way to get a string into tokens and then parse each token, as in this pseudocode: split (source, "{}," params) for each param in params split (trim(param), ":", key_value) (of course you need to trim and verify that both keys and values are valid, etc) parsing on Z80 is probably better done left to right, 'ignoring' brackets and whitespace, using the `:` (or `=` if we prefer that) as a key/pair separator, and the comma as a key-pair token delimiter. It's not that hard...

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Mar 04, 2024 11:17 am
	Maxim wrote We might assume that all (future) use on hardware is going to be on microcontrollers that will be able to manage with the library - is that true? Can't speak for the rest of the hardware community, but yes I'd say that any system which is expected to process these headers and do something useful with them is more than likely capable of running a JSON parser. Here's a minimalistic JSON parser that I'm sure would easily fit on a microcontroller. Less luck finding a Z80 one, out of interest, but amusingly here's a 6502 JSON parser which looks surprisingly big (code-wise) at first glance. Maxim wrote How about zip support on those same microcontrollers? I think this is going to be trickier and the issue is likely around peak RAM requirements. Frustratingly, I've not found much discussion out there around embedded unzipping solutions. The only github project I've seen so far which explicitly mentions the ZIP file format with minimal memory overhead is this one which is not particularly big (popular). I do agree that ZIP is by far the most conventional way to archive and compress if one includes Windows host systems as well as Linux ones so it would be nice to support it. Just thinking aloud, but for things like flashcarts I wonder if it would be reasonable to decompress on the host system, and have the metadata files sit alongside the decompressed ROMs as an alternative to dealing with the archives direct. After all, it's practically impossible to fill a modern SD card with even decompressed ROMs.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Mar 04, 2024 4:01 pm
	That’s pretty much what we had planned - but it demands a certain amount of making the filename of the metadata file match that of the ROM. Hardly difficult, but potentially something to go wrong.

lidnariq Joined: 24 Mar 2021 Posts: 120	Posted: Tue Mar 05, 2024 7:48 am
lidnariq Joined: 24 Mar 2021 Posts: 120	ZIP is basically a non-starter on anything before the Megadrive - DEFLATE requires 32KB of RAM for back-references. You could say that "only ZIPs without compression are supported" but that would break the ease of workflow.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Mar 05, 2024 8:18 am
	But do we need to decompress on a Master System?

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Thu Mar 07, 2024 9:04 am
	willbritton wrote Here's a minimalistic JSON parser that I'm sure would easily fit on a microcontroller. Just for the record, I integrated JSMN on the RP2040 part of the Retcon AV and it works fine for JSON parsing. It's actually quite clever - it's a header-only implementation, has no dependencies and the memory footprint for holding the parsed document is limited to 16 bytes * the number of "tokens" you want to store, tokens being keys, values and non-leaf nodes. So if you know your JSON document is likely to be small you can tune the allocation down accordingly. (note: you might also be buffering the raw JSON file in memory too, depending on your approach) You can stream the input too, so long as you have enough memory to load each chunk of tokens, although I haven't tried this approach myself. I haven't tried it yet, but I don't see any reason why it wouldn't work on the Z80 either, e.g. via SDCC.

Forums

View topic - SDSC tag enhancements for homebrew