Forums

There's a perennial problem for emulation that ROM images are not enough to encapsulate the information about what is in a cartridge. However for homebrew we have options available to us. I'd therefore like to propose an extension to the venerable SDSC tag as follows:

See https://www.smspower.org/Development/SDSCHeader

Since we have filled the available space (without doing tricky things), I propose to extend the Description string. If (and only if) it starts with a '{' character then it should be parsed as JSON. The parsed document will define characteristics of the ROM, which we can discuss here. Anything left after the document may be shown as the description field currently is.

1. Mapper

We could simply enumerate mappers; so type 0 = Sega, 1 = Codemasters... or we could try to describe the mapper in its entirety, e.g. describing which addresses can be written to, which memory mapped region is affected, and how. The former seems like we're not learning from the mistakes of the iNES header, and the latter seems complicated and error-prone - and might invite people to make custom mappers for no good reason.

2. Save memory size (and type)

Most emulators will emulate save memory up to 32KB for the Sega mapper, and maybe try to trim it down to the smallest power of 2 size that is accessed. It might be nice to be explicit about it.

3. Anything else?

Example:

{ "mapper": "sega-sms-generic", "sram-size": 8192 }

Comments are welcome.

My initial motivation was to have a way for homebrew to require emulation of the standard Sega mapper, when some emulators only do this for file sizes over 64KB.

Of course, if no emulators support this, or nobody uses it, then it won't work.

1. Why JSON?

I guess the main motivation is to support emulators, rather than real hardware, although I have a bias towards the latter.

Would something like TOML or (and I hesitate to speak its name) even YAML be a better choice to cover the potential need to parse in hardware?

I appreciate that JSON kind of gives you an escape character for free, since it probably ought to start with a `{` (although other characters can be valid JSON). Does TOML have to start with `[`? If so maybe that's an option.

I also like TOML stylistically for this, because it reminds me of INI files and they feel more retro than JSON.

2. On the mappers, I've been thinking a lot about this too, and intuitively I agree that something codified would be preferable to an arbitrary menagerie a la iNES.
Codifying isn't super trivial though.

My interest is in designing a generic hardware mapper that can be reconfigured based on a header or metadata file, so it's definitely a problem I'm interested in being part of solving if it goes this direction.

It seems to me it's a very sensible idea to use the SDSC tag... even if this only solves the issue for the (newer) homebrew.
Mappers shouldn't be enumerated, having concise description is way better. "sega-generic" is a good term for instance, and there could be some alternatives such as "sega-slot2" that could be used when the generic mapper isn't adequate enough.

So your example:

{ "mapper": "sega-sms-generic", "sram-size": 8192 }

is pretty good. Of course how the SRAM is accessed depends on the mapper specifics so there's no need for other details.
(also note that "sega-sms-generic" (or "sega-generic") should work with a number of bits compatible with the ROM size, so for instance a 128 KiB ROM (8 × 16 KiB pages) will make the mapper work on 3-bit index pages, thus ignoring the upper bits if set...)

Then, back to the idea that this should also somehow be used for already existing ROM dumps, we could create JSON files to be stored in the same folder where a ROM dump is, using the same name with a .json extension (say 'Great Soccer.sms' ROM and 'Great Soccer.json' cartridge descriptor).

An emulator/flashcart should then:
- check if an SDSC header exists, and if json data is present at the Description field, use that as description of hardware.
- alternatively, check if a .json file exists with the same name as the ROM being loaded. If this exists and contains json data, use that.
- if all of the above fails, we fall back to the withstanding situation - the emulator/flashcart has to figure out by itself.

I suppose that using some markup / config structure allows sufficient flexibility to have all known and also hypothetical mappers be named and fully parameterised.

Something that might be quite clever is having the name point to a github repo path, which then contains a canonical declarative or imperative description of how the mapper works.

That way, suitably clued-up emulators could dynamically add more mapper support.

e.g.

{
"mapper": "sverx/sms-mapper",
"params": {
"bank0_register": "0xfffd",
etc.
}
}

or

[mapper]
name=sverx/sms-mapper
[params]
bank0_register=0xfffd

"Lazy" emulators could just support the well-known Sega case, but at least the github path convention would act as a basic type of namespacing and mitigate the risk of a string-based arbitrary enumeration bloat instead of a numeric one.

I suspect we should instead use the SMS Power! community as the only source of truth regarding what is valid and be the 'benevolent dictators' regarding mappers canonical names and all the details

unfortunately anyone building a ROM could theoretically write whatever they want (and even plain wrong info) in the SDSC description field... so we should probably have the external json file override info in the SDSC description field? :thinking:

sverx wrote

I suspect we should instead use the SMS Power! community as the only source of truth regarding what is valid and be the 'benevolent dictators' regarding mappers canonical names and all the details

The benevolence part is not in doubt 😄, although I would point out that some members of this community do like to turn their hand to inventing their own hardware mappers, and to me it feels wrong, however well intentioned, to act as gatekeepers to them being able to freely innovate in the field. But I do get it.
EDIT: and at the end of the day, what's to stop someone trying to overload the fields? Only a specification, convention and the support of emulator authors.
EDIT2: also, how many "valid" mappers does this community know about? Presumably it's a very small and finite list. So then what? The matter is essentially closed, unless someone wants to lobby for a new mapper one day? I think that misses a trick. Anyone who invents a new hardware mapper can circumvent the process by just ignoring it, instead of engaging with the community and offering a mapper design that others might benefit from!

sverx wrote

unfortunately anyone building a ROM could theoretically write whatever they want (and even plain wrong info) in the SDSC description field... so we should probably have the external json file override info in the SDSC description field? :thinking:

I actually think that strict emulator support here could be a great incentive for homebrewers to get their act together with respect to headers. Much like iNES, the game simply won't work on the emulator development workflow if the header is wrong.

That said, if the fallback is standard Sega header anyway, in probably 80% of cases such an incentive won't apply anyway.

Maxim's example of emulators applying some slightly tricky heuristics to supply a mapper (and there's a similar argument around save RAM) is valid. Personally I'd much prefer to take the time to write a valid header than to have to "trick" the emulator to give me the hardware I want.

willbritton wrote

I would point out that some members of this community do like to turn their hand to inventing their own hardware mappers, and to me it feels wrong, however well intentioned, to act as gatekeepers to them being able to freely innovate in the field. But I do get it.

No, sorry, I didn't mean any gatekeeping. It's the naming we'll control, not the hardware features.

JSON because XML sucks, YAML is all sorts of broken and TOML is bad at nested structure. Almost every language has a JSON order of some sort, and I feel like it’s easier to implement in Z80 if need be.

I’d like to see some proposals for the “encoded mapper”, even if we also define shortcuts for common cases. There are definitely tricky edge cases like byte flipping and bank number offsetting in the Korean mapper world. My idea is that if the whole mapper can be defined in markup, we can also define “shortcuts” for the common cases to avoid bloat. We act as BDFLs and define the shortcuts we want.

I don’t think we need to support all the commercial games yet, though. If we feel like the markup is good for homebrew then maybe we can try to apply it to the cartridge dump problem later, but I don’t want to try to do it all at once.

GitHub is not the best source of naming, it may not exist in 20 years (or be what SourceForge is today), but nor is DNS or anything else really.

sverx wrote

willbritton wrote

I would point out that some members of this community do like to turn their hand to inventing their own hardware mappers, and to me it feels wrong, however well intentioned, to act as gatekeepers to them being able to freely innovate in the field. But I do get it.

No, sorry, I didn't mean any gatekeeping. It's the naming we'll control, not the hardware features.

Ah okay, not sure I understand the distinction though — are you saying there's a list of known mappers and somehow the SMSPower community curates that list? If so, how?

I guess my suggestion of using a repo path allows people to create new names without risk of conflict, but also doesn't preclude SMSPower from controlling a list of "top level" names, either through a commons-owned repo organisation (e.g. https://github.com/smspower, which actually does exist!) or simply a mapper name which doesn't include a / symbol in it.

Maxim wrote

or we could try to describe the mapper in its entirety, e.g. describing which addresses can be written to, which memory mapped region is affected, and how. The former seems like we're not learning from the mistakes of the iNES header, and the latter seems complicated and error-prone - and might invite people to make custom mappers for no good reason.

willbritton wrote

On the mappers, I've been thinking a lot about this too, and intuitively I agree that something codified would be preferable to an arbitrary menagerie a la iNES.
Codifying isn't super trivial though.

Unless y'all want to reinvent Verilog and Verilator, that is just not feasible.
It might work for simple mappers, but how on earth would you write JSON that describes Ichigo's new mapper for instance, which has VDP DMA, can decompress graphics on the fly, flip graphics on the fly, compose sprites into background tiles on the fly, etc

I do remember Calindro mentioning something about Emulicious being able to support mapper plugins, i.e, loading mapper code as Java's equivalent of a DLL. However, besides being a possible security threat, I'm not about to include a Java interpreter on my flash cart just so I can support new mappers :P

willbritton wrote

Something that might be quite clever is having the name point to a github repo path, which then contains a canonical declarative or imperative description of how the mapper works.

Ah, yes, I do like to automatically download malware with my SMS ROMs :P

fwiw, the newest version of the EverDrive N8 Pro firmware lets a game ship alongside a FPGA bitstream specifying a custom mapper. Of course, that bitstream is highly specific to that cartridge and that FPGA and it still could be a malicious bitstream that shorts pins or something, although how much damage that could do is debatable.

sverx wrote

Then, back to the idea that this should also somehow be used for already existing ROM dumps, we could create JSON files to be stored in the same folder where a ROM dump is, using the same name with a .json extension (say 'Great Soccer.sms' ROM and 'Great Soccer.json' cartridge descriptor).

I believe that, in order to garner adoption, we do have to come up with something that can be applied to already existing ROMs as well. Having a separate file like this would definitely be the most straightforward way of achieving that.
What I dislike about this idea is that it makes ROM distribution harder. Now they have to be tarballs or, worse, ZIPs. And then flash carts would either have to ship with a ZIP library or force users to unzip all their games before running them.
Maybe that isn't that big of a deal. That's already gonna be the case for games using custom mappers on the EDN8 Pro.

What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed.
The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size.

EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics.

Maxim wrote

JSON because XML sucks, YAML is all sorts of broken and TOML is bad at nested structure. Almost every language has a JSON order of some sort, and I feel like it’s easier to implement in Z80 if need be.

Surely JSON parsing has to be harder to implement in Z80 than TOML...
Haha, at the rate things are going, I might have to write a JSON and TOML parser as my entry for the coding competition this year, which ought to guarantee no one else is disappointed to receive last place...

Agree on XML and on YAML. TOML being bad at nested structure doesn't hurt us here though does it?

Maxim wrote

GitHub is not the best source of naming, it may not exist in 20 years (or be what SourceForge is today), but nor is DNS or anything else really.

Sure, although it would only be github by convention — having something like "smspower/sega-mapper" could just be a strong hint that the definition of that mapper could be found by going to github, and for basic emulators simply doing a string comparison would be sufficient to handle that case.

If github disappears, it would probably cause issue to a number of package managers who apply similar conventions, but obviously the risk could be mitigated by allowing a fuller qualification. Then you're getting into some verbose territory.
Java has (or had?) arbitrary string namespaces with a strange reverse-URL type of convention, and plenty of languages do use fully qualified domain names which aren't the worst idea in the world.

TIL what BDFL means...

Maxim wrote

I’d like to see some proposals for the “encoded mapper”, even if we also define shortcuts for common cases. There are definitely tricky edge cases like byte flipping and bank number offsetting in the Korean mapper world. My idea is that if the whole mapper can be defined in markup, we can also define “shortcuts” for the common cases to avoid bloat. We act as BDFLs and define the shortcuts we want.

I don't think it's possible to really describe the hardware inside the json data, we can just use *previously defined* names there.

As for these names, I don't have much experience with exotic mappers to help. But the mapper list should indeed include a "none" option, a few "sega-" variants... and then there could be a whole group of "korean-xxxxx" where each one has its own details.

Still my point is that we need to be able to add this info to new and existing ROMs. Say one day fellow user Ichigo releases some games that uses a specific mapper with a specific feature, say byte flipping. We might then pick the name "ichigo-byteflip" (or some better name of course) and even if his games don't have any mapper info in the ROMs we have to be able to inform emulators and flashcarts of this, that's why I thought the separate json file could be a possible solution.

But let's discuss more, it's a very very interesting topic! :D

Trirosmos wrote

Ah, yes, I do like to automatically download malware with my SMS ROMs :P

Oh, boo!! Where's your spirit of adventure!?

But seriously, almost as soon as I wrote it I thought I should probably remove the word "imperative". I agree that executable download is a big no-no from a security perspective, but I do think that at least the idea of some basic set of declarative mapper primitives is worth exploring. Maybe not to include stuff like DMA, but who knows.

It is, IMO, the best kind of interesting problem...

Trirosmos wrote

fwiw, the newest version of the EverDrive N8 Pro firmware lets a game ship alongside a FPGA bitstream specifying a custom mapper. Of course, that bitstream is highly specific to that cartridge and that FPGA and it still could be a malicious bitstream that shorts pins or something, although how much damage that could do is debatable.

Wow, I didn't know that.
My gut doesn't like that idea personally.
So is he integrating a partial bitstream segment just for the mapper config into the whole bitstream file? If so, that's insanely devious.

Trirosmos wrote

What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed.
The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size.

EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics.

Yeah, further to my above point on this subject, I strongly agree with this.
I think companion files should only be used as a worst-case for non-standard ROMs which happen to have missing or incorrect headers.

willbritton wrote

So is he integrating a partial bitstream segment just for the mapper config into the whole bitstream file? If so, that's insanely devious.

No. The nature and quantity of NES mappers means it's much easier to just reconfigure the FPGA each time a game boots, which is what the EverDrive does.
The mapper bitstream can use all of the FPGA's resources.
Remember, the EDN8 Pro has both an ARM MCU and a second FPGA just for booting the system.

I worry this is a hairy can-of-worm problem to solve perfectly, and instead emulators supporting many mappers would be best pushed to have options to override mapper and other options during development.
Whatever description language you come with would need to cater for the actual non-trivial example (rather than cater for "generic sega mapper" example).

I'd be curious to hear about the practical problem that lead you to this idea Maxim? Maybe knowing this we can aim at best bang for bucks solution.

(Tangent: note that MEKA implement some (basic) heuristic to detect mapper when running a ROM which is not in the DB. Given how simple the heuristic is, i believe we could also try to thoroughly develop it as a standalone micro project, tested against all known roms. But whether that would matter or not depends on actual problem you are trying to solve).

Trirosmos wrote

willbritton wrote

So is he integrating a partial bitstream segment just for the mapper config into the whole bitstream file? If so, that's insanely devious.

No. The nature and quantity of NES mappers means it's much easier to just reconfigure the FPGA each time a game boots, which is what the EverDrive does.
The mapper bitstream can use all of the FPGA's resources.
Remember, the EDN8 Pro has both an ARM MCU and a second FPGA just for booting the system.

Ah, okay. I guess the mapper makes up a substantial proportion of what the Everdrive is actually doing, so that does make sense. I was thinking more of my project where the mapper is just a tiny element of the whole.

I really should find some time to look into the Everdrives more, I know next to nothing about them.

For reference:
https://github.com/ocornut/meka/blob/master/meka/meka.nam

This was never aimed at being any "official" but this is what MEKA use and this is probably the most complete list.
Note that bsittler started working on emulating GG multicarts and we are expected to add dozens of new bizarre custom mappers in coming years.

Regarding the parsing issue, I would suggest we use a very basic approach, a comma-separated list of attributes and value pairs between a { and a } so a simple

{mapper:sega-sms-generic,sram-size:8192}

would be sufficient.
After all, we don't need to care about using commas or semicolumns in the attributes or values. Whitespace between tokens could be ignored too so this

{mapper: sega-sms-generic, sram-size: 8192}

would be equivalent. The curly braces both ensure a constant first char in the Description field and a terminator char for the hardware info block.

sverx wrote

Regarding the parsing issue, I would suggest we use a very basic approach, a comma-separated list of attributes and values between a { and a }

Without meaning to sound like I'm flip-flopping, I think I do prefer some standard to none at all, so personally I'd rather see JSON than quasi-JSON.

To bang the drum just once or twice more: replace "comma-separated" with "newline-separated" and your colon character with an equals sign and I think you've basically got TOML...
(EDIT: oh except TOML needs string delimiters too :/ )

willbritton wrote

Without meaning to sound like I'm flip-flopping, I think I do prefer some standard to none at all, so personally I'd rather see JSON than quasi-JSON.

It was meant to simplify parsing (and save a few bytes ;) )
If it doesn't make things simpler, then ignore this :D

sverx wrote

It was meant to simplify parsing (and save a few bytes ;) )
If it doesn't make things simpler, then ignore this :D

No, of course you're right that it does simplify parsing, and I think the most important way it does is by neglecting string delimiters, which to my chagrin I see now that TOML doesn't, so now I'm back to INI files 🤣

It's seriously annoying there isn't a more popular super minimal standard for jobs like this.

My reasoning for preferring a standard is that for desktop software, like emulators, it's much more convenient (and maybe more secure, etc.) to use a known parser library rather than having to hand-roll one, although obviously the simpler the format gets the more trivial writing a parser becomes.

Going back to principles, whatever it is, we need:
- An escape sequence which differentiates the extended structured data from a plain text description field
- Basic dictionary semantics (key-value pairs)
- We probably don't care about nested types or array structures (?)
- We probably don't need typing in the syntax, because there would be a known schema (?)
- Strict syntactical validation feels somewhat unnecessary given the context — homebrew authors probably want header population to be as foolproof as possible.
- I do think something that's not hard to parse with a Z80 would be nice.

I do concede that JSON is a significantly forward-compatible option, and I guess really the only thing I dislike about it is the string delimiting (which I now see TOML also has) and the (probably optional) need to validate a closing brace to match the opening one; but yeah okay even though writing a Z80 parser for a subset of JSON might be slightly more annoying than I'd like, it's definitely not going to break the bank in terms of ROM space is it. 😂

In retrospect I may have been inadvertently bikeshedding...

Trirosmos wrote

What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed.
The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size.

EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics.

If it turns out that it's impossible to append the metadata do existing ROMs, either because the EverDrive expects ROMs to have a certain size and be mirrored a certain way or because it'd break compatibility with emulators using checksums to find entries in a ROM database, I think the next best option would be sverx's suggestion of a separate file.

I don't particularly care about what syntax the metadata uses. Anything that isn't a binary format is going to be annoying to implement on a flash cart anyways.

I think the reason why iNES is such a mess has nothing to do with it being a binary format and, really, moving to JSON or YAML wouldn't solve most of the issues.
The real problem is that, when iNES was created, very little was known about how the NES worked, how mappers worked and about just how dang many mappers there were.
All other issues arose from trying as hard as possible to make every subsequent version of iNES backwards compatible.

The generalised issue of providing metadata with ROMs is very well explained here: https://web.archive.org/web/20200322151931/https://byuu.net/cartridges/boards but my proposal here is to take something like the “external file” model and embed it in homebrew. Let’s not try to solve the greater issue of metadata for all ROM dumps - yet.

Embedding things in zips and then changing the extension is very common these days, and emulators have supported zips since forever. Personally I’d go for that approach for the generalised problem.

ROM databases are fine if you’re willing to spend years of your life acquiring and dumping cartridges to maintain quality. But they suck for managing metadata for homebrew and hacks.

Maxim wrote

The generalised issue of providing metadata with ROMs is very well explained here: https://web.archive.org/web/20200322151931/https://byuu.net/cartridges/boards but my proposal here is to take something like the “external file” model and embed it in homebrew. Let’s not try to solve the greater issue of metadata for all ROM dumps - yet.

But it's the same issue, tho ;)

In fact, solving it for homebrew games is a superset of solving it for existing ROM dumps, since homebrew games can do way weirder stuff.

If we define this new header that only homebrew games use, is anyone gonna adopt it, unless there's a new hot homebrew with a custom mapper that only works if emulators do support it?
Even then, aren't emulators more likely to just special case that particular homebrew instead of supporting a new header?

I think it's much better to define one way of specifying all cartridge hardware rather than sometimes embedding a header, sometimes having an external file, sometimes having both, sometimes having neither...

If I can opt my homebrew in to support for certain common features, then I’ll use it. That feature support might not be great for less well-supported things, but if someone wants to make a homebrew game that uses Meka’s nth Korean mapper’s features, wouldn’t it be great if Meka can enable it based on data in the ROM instead of heuristics? If it then only works in Meka, because Emulicious doesn’t support that mapper yet, then that’s fine.

If we first decide we have to support every necessary feature of every ROM ever released, plus special cases on top (Master Games? SG-1000 games with “40KB” layouts? Pro Action Replay?) then we’ll get stuck before we get started.

It might be worth considering that e.g. an Everdrive BIOS could possibly want to parse this data from Z80 code, but of course the parsing doesn't need to be efficient.

That is true, and that’s partly why I went with JSON - it’s easier to parse than YAML and XML, although still not very easy. Another approach would be to use something like the tagging formats used for metadata on media files.

Anyway I still believe that if we "find a solution" that works only for the (newer) homebrews, we aren't doing anything very meaningful, let me explain.

- new homebrew might not use what we would want anyway
- new homebrew might use custom (and never seen before) mappers, and we can't ask them to please let us know in advance

So I still believe we should somehow start by identifying and researching (and then give a good name) to the mappers we already know, and MEKA's mappers list is a good starting point for this.

Once we get this info, we should verify we are able to describe the existing releases correctly. Again, MEKA is quite invaluable for this.

Then we create 'official' attributes and values, and pick a data format for that.

Then we should discuss how to make it possible for this info to be 'attached' (in a loose sense) to their own ROMs. A public list? A small text file (json or whatever, that's a separate detail now)? We'll pick the option(s) that work best.

Only if we solve this, then we might see how to improve the formula for the new releases (using homebrew's SDSC tag for instance) ... but we shouldn't rely on that because it's out of our control anyway, we should just consider it as an interesting option.

sverx wrote

Anyway I still believe that if we "find a solution" that works only for the (newer) homebrews, we aren't doing anything very meaningful

Agreed, any format proposed shall support all existing ROMs if it wants to supersede what's currently done.

Trirosmos wrote

What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed.
The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size.

EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics.

As a point of reference, here's the specification for the GBX footer format that is supported by mGBA and hhugboy (and seems like support for it is also planned on SameBoy?).

In my opinion (and I stress *opinion*) any change/addition to any ROM dump binary file is a "very bad thing™"

Byuu wrote:

Quote

External board descriptions
Similar to third-party headers like iNES, the idea is to include a file with the ROM images that describes the games. You've seen examples of that from bsnes above, eg from the problem 1 section.

I personally favor this approach because an external file can be stored in plain-text, and can use an extensible markup format that allows us to add new corner cases as they are discovered, rather than have a format with limitations such as iNES permanently set in stone.

However, this approach is also not without its problems: the biggest one is simply designing the specification. How would one create a markup format that could adequately describe any arbitrary circuit board configuration, population and wiring for any game system? And how would one get the hundreds of existing emulator developers to adopt formats that will often be far more complex than the needs of their individual systems? For instance, NES emulator authors are not concerned about Flash memory save type IDs like GBA emulator authors are, and Genesis emulator authors are not as concerned about memory map layouts (where things are always linear) as SNES emulator authors are (where things are banked.)

Further, there's still the issue of what we do when a board description is released with an error in it. How do we update every copy of the game in the world with the corrected version?

And finally, there is intensely strong resistance to the idea of storing more than a single file per game image, even if inside of a ZIP archive. I do not fully understand the resistance here, but it is a reality we have to face if we want a ubiquitous solution.

Regarding the

Quote

How would one create a markup format that could adequately describe any arbitrary circuit board configuration, population and wiring for any game system?

I think we just solve this by NOT describing it in the file, but just referencing its technical details using a name.

Regarding Maxim's specific use case: if he ships a ROM plus a 'hardware descriptor' in the expected format, *updated* emulators will honor the descriptors - old ones won't, but I suspect there's no way to convince any existing emulator to enable SRAM on a <=48 KiB ROM apart from padding the ROM to 64 KiB, which defies our efforts here.

My current highly specific need is for an updated SMS VGM Player. This consists of a 16KB stub plus a variable-length amount of VGM data appended; the total may be anywhere from ~16.1KB to 4MB, not rounded to any sensible amount. Some emulators/flash carts will not work properly unless I also do something to expand the ROM to a larger size, and specifically I hoped I could coerce Emulicious into letting me debug it :)

willbritton wrote

Yeah, further to my above point on this subject, I strongly agree with this.
I think companion files should only be used as a worst-case for non-standard ROMs which happen to have missing or incorrect headers.

Sorry, I realise I misread Trirosmos's original comment here and conflated two things. What I actually meant was, I think developers (i.e. of new games) should be incentivised to get their headers right first time.

With regards to modifying existing binaries, yeah I think I'm with sverx here, it seems somewhat fraught for a few reasons.

I guess a concern with an archive is that it adds complexity to file handling in hardware, although for anything with a MCU in it, like a flashcart, feels like it's not too much to ask. Otherwise in principle, packaging files together into a logical unit seems like a very sensible thing to do. I don't know much about TAR, but is that much simpler to handle in hardware than a ZIP file, for example?

sverx wrote

I think we just solve this by NOT describing it in the file, but just referencing its technical details using a name.

Yeah think I mostly agree with this, although there does need to be a rigorous specification somewhere of what each thing actually does.
It could be a reference implementation (but in what language? multiple?) or maybe some kind of functional test specification?
At the very least, it should be a written spec that leaves emulator and / or hardware designers in no doubt as to how to implement.

There's potentially a slippery slope here which might lead one down a path of allowing composite devices to be specified in meta files (e.g. it's a standard Sega mapper + Ichigo's custom byte-flipping widget) in order to avoid having to generate completely new high level devices and associated name bloat, and as soon as you go down the slope it feels like inventing a new HDL, as per Trirosmos's first comment on this thread. That said, it is an approach I've been thinking of to solve my particular hardware problem (I don't think that having people provide a 150KB FPGA bitstream with every game is the right solution for my use case!)

willbritton wrote

I guess a concern with an archive is that it adds complexity to file handling in hardware, although for anything with a MCU in it, like a flashcart, feels like it's not too much to ask. Otherwise in principle, packaging files together into a logical unit seems like a very sensible thing to do. I don't know much about TAR, but is that much simpler to handle in hardware than a ZIP file, for example?

To be fair, I haven't actually tried to integrate a ZIP library into one of my projects.
With that said, tar is about the simplest archive format possible. Literally a 512 byte header for each file in the archive followed by the file data, verbatim. So, you can very easily read it in place.

ZIP on the other hand would require something in the order of 41 KiB of RAM?

Perhaps using only uncompressed ZIPs would simplify the code significantly, but then users might get confused as to why a completely valid ZIP file isn't being accepted by an emulator or flash cart.

Zip is a very old format but indeed the Deflate compression needs a lot of memory for a Z80 - but maybe not much for a microcontroller, something like 32KB look back plus a few KB for the trees. Uncompressed zips would be a pain to manage.

TAR is uncompressed so it’s simpler to deal with but somewhat harder to parse as it lacks a central directory.

Most emulators already support loading ROMs from zips.

Trirosmos wrote

What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed.
The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size.

Regarding possible problems with a footer, is anyone here privy to the details of what breaks on the OG SMS EverDrive?
And are those issues fixed on the X7 at all?

What happens on them if the ROM size isn't a multiple of 16 KiB? Or isn't a power of two? What happens if I append a single byte to the end of a ROM?

SMS Everdrive (not x7) will truncate ROMs to a multiple of 64KB on load, unless they are smaller than 64KB. This seems to be an oversight due to it apparently loading the data in 64KB chunks.

Some very old ROM dumps had 512B headers (from Super Magic Drive?) or 64B footers, so very old emulators were written to ignore these. However, I can’t help but feel like it’s a mistake to introduce anything that changes the checksum of the ROM file. That’s why I like the ROM+metadata in a zip approach (for non-homebrew): old emulators can ignore the metadata and load the ROM like they always did, relying on heuristics. New emulators can take advantage of the extra metadata. Checksumming tools will detect the game (even with bad metadata…) and leave the metadata alone as they list/rename/move the outer zip around.

For homebrew, it seems obvious to put the metadata in the ROM to reduce the complexity for the user, even if it means some more work for the emulator.

Not sure if this is the right direction to move the discussion in, but was just looking at the mapper IDs from Meka that Bock posted:

;-----------------------------------------------------------------------------
; Cartridge Hardware ("Mapper") Identifiers
; From MEKA sources in srcs/mappers.h
;-----------------------------------------------------------------------------
; 0 : standard sms/gg sega mapper, optional SRAM
; 1 : 32KB of RAM mapped at 0x8000-0xFFFF
; 2 : colecovisions
; 3 : code masters mapper, registers at 0x0000, 0x4000, 0x8000
; 4 : sega mapper with 93c46 eeprom
; 5 : sg-1000 (currently incorrectly emulating 4KB of RAM instead of 2KB)
; 6 : master system action replay
; 7 : terebi oekaki (graphic tablet)
; 8 : sf-7000
; 9 : sms korean mapper, mapping register at 0xA000
; 10 : sms display unit BIOS
; 11 : no mapper, ROM up to 48KB (typically 32KB)
; 12 : sms korean msx type 8KB banks, registers at 0x0000->0x0003
; 13 : sms korean for Janggun-ui Adeul
; 14 : sms 4 Pak All Action
; 15 : sg-1000 II with taiwanese memory expansion adapter, 8KB RAM from 0x2000->0x3FFF + regular 2KB RAM mapped at 0xC000-0xFFFF
; 16 : sms korean xx-in-1, register at 0xFFFF mapping 32 KB at 0x0000->0x8000 (hi-com cartridges)
; 17 : sc-3000 survivors multicart.
; 18 : sms korean xx-in-1, register at 0x0000 mapping 8 KB at 0x8000, register at 0x0100 mapping 8 KB at 0x4000, register at 0x0200 mapping 8 KB at 0xA000, register at 0x0300 mapping 8 KB at 0x6000
; 19 : sms korean xx-in-1, register at 0x2000 mapping 8x4 KB 0x4000-0xBFFF using register xored with 0x1F, 0x1E, 0x1D, 0x1C. 0x0000-0x3FFF stuck on bank 0.
; 20 : sms korean xx-in-1, register at 0xBFFC, sms+msx
; 21 : sms korean xx-in-1, register at 0xFFFE
; 22 : sms korean xx-in-1, register at 0xFFF3 and 0xFFFC (mirrored 0xBFF3, 0xBFFC)
; 23 : sms korean xx-in-1, register at 0x0000, xored with 0xFF
; 24 : sms korean xx-in-1, register at 0xFFF0, 0xFFFE, 0xFFFE (used in MD multicart with SMS games)
; 25 : sms korean xx-in-1, register at 0xFFF5, 0xFFFE, 0xFFFE (used in MD multicart with SMS games)
; 26 : sms korean xx-in-1, register at 0xFFFA, 0xFFFE, 0xFFFE (used in MD multicart with SMS games)
; 27 : sms korean xx-in-1, register at 0x2000
; 39 : sms korean xx-in-1, register at 0x2000 (11 Hap Gam-Boy)
; 40 : sms korean xx-in-1, register at 0x8000
;-----------------------------------------------------------------------------

I guess the implementation for these I can find if I browse the Meka code, but just wondering at a high level what primitive features we're actually looking at here. At a guess I'd say:

- ROM banks - at what granularity? Always 16KB or sometimes smaller?
- ROM bank control registers - with various decodings.
- Sometimes ROM bank decoding is disabled for certain blocks (e.g. lower 1KB on Sega mapper, but not on Codemasters)
- RAM banking - I guess we can assume "it varies", since the official Sega RAM banking mechanism is slightly less regular than the ROM banking mechanism.
- On-cart RAM can either be volatile or non-volatile

Exotic Ichigoesque functionality aside, comparisons with NES aside for now, just thinking about these known mappers and focusing on historic games rather than homebrew, are there any other obvious features worth calling out?

OK this is disrupting my sleep now, that's not good.
Anyway, since I was awake, I thought that the focus should be on the emulator.

What I mean is that we currently have ROM dumps and emulators have to 'guess' everything else. They do this pretty well, but this is not something we can control in any way, so I think our goal should be to make sure we could pass a minimum set of information alongside a ROM dump and have the emulator *use* this information.

This opens up a lot of new options, in my opinion. For instance, a game could have code to use different hardware, for instance to save. Also this means that these hardware info can't be written in the ROM itself - after all it also makes sense as it's the ROM to be inside the cartridge and not the other way around.

So I'm pretty much convinced that the way is to have a small text file alongside the ROM dump, or even bundled into the same archive for those emulators supporting compressed format(s).

Now, we can discuss the syntax of course, but the foundation of the idea is that we pass attributes/values pair to the emulators. I suppose we should keep the things as simple as possible by strictly defining exactly what's expected and what's not from each mapper, having additional attributes to further detail (and I admit I prefer the approach of having additional attributes when the cartridge has additional hardware than the other way around, that would be to add attributes when something is missing).

Let me post a few examples. Again, ignore the format for now.

{mapper:none}

Tell the emulator that there's no mapper in this cartridge. Z80 address space will be mapped to ROM 1:1, everything above 48 KiB will be 'out of reach'. Writes to $FFFC-$FFFF won't have effect apart from writing to mirror RAM, etc. This is good for testing how a ROM would work on a basic 'homemade' (mapperless) cartridge.

{mapper:sega}

Tell the emulator that there's a fully fledged SEGA mapper on the cartridge. The code doesn't have to *use* all the features, but the emulator should offer them. If the mapper is instead supporting just a subset of these features, we should use sega-<something> here instead.

As said before, additional hardware present in the cartridge should mean additional attributes in the hardware descriptor, so for instance there should not be any SRAM present unless:

{mapper:sega, sram-size:8192}

... and if there's a battery (or the sram is non volatile) so that SRAM contents are kept between sessions, one should have:

{mapper:sega, sram-size:8192, non-volatile}

(an attribute without value should be equivalent to attribute:true)

What I find interesting here is that one could easily test his own code in different situations, thus emulating how that would work on different cartridges. For instance, one could reduce/increment the SRAM size, or even check what happens when SRAM contents are lost (or retained!) between sessions.

Finally, I've also seen a few homebrew using Flash to save.
We might probably support that too, maybe with something like:

{mapper:sega, flash:MX29F040}

... but, well, it's all in the hands of emulators programmer to what extent they care about this (not all emulators are made for new software development...)

I'm eager to know if all this seems over-complicated or instead if it falls short of what is needed, so feedback is welcome! :D

sverx wrote

I'm eager to know if all this seems over-complicated or instead if it falls short of what is needed, so feedback is welcome! :D

TLDR; yes agreed I think that's kind of where the conversation seems to have been gravitating!

I think there are two parts here:

1. A metadata format that can be used to describe the assembly that is a cartridge. The ROM isn't the cartridge, it's only one part of the overall assembly.

2. A specification format that can be used to describe what each component in the assembly actually does, such that emulators can emulate it, or hardware can implement it.

My previous post is an attempt to try and roughly scope the domain of the second thing, as far as the majority of original games are concerned. I rather suspect we don't need a full-blown HDL here, but that there's a relatively small set of primitive cartridge components that could describe a significant majority of original SMS carts.

On the first thing, I think a fundamental question is whether to only delegate the description to a single assembly name + parameters (e.g. {mapper:sega, sram-size:8192}) or to allow a more granular and thus flexible list of components, but either way I think the approach of having named and known (or locatable) entities is a good one, for a few reasons.

sverx wrote

OK this is disrupting my sleep now, that's not good.

Are you in Sweden now? If so, I have to tell you it's probably time to wake up :D

willbritton wrote

My previous post is an attempt to try and roughly scope the domain of the second thing, as far as the majority of original games are concerned. I rather suspect we don't need a full-blown HDL here, but that there's a relatively small set of primitive cartridge components that could describe a significant majority of original SMS carts.

Yes! You posted it while I was posting mine, so I read that later.
But I do agree that MEKA is one of the two (main) sources that we should use to define the mapper 'set', the other being the mappers page, and they likely overlap anyway.

Now, of course the issue is that one of the hardest things is always naming things...

willbritton wrote

- ROM banks - at what granularity? Always 16KB or sometimes smaller?

As per de mappers page, some mappers do use 8 KiB banks.
In particular, the Janggun mapper let's you choose at will, on the fly, whether a certain portion of the address space will be broken down into a single 16 KiB bank or two 8 KiB ones.

willbritton wrote

- RAM banking - I guess we can assume "it varies", since the official Sega RAM banking mechanism is slightly less regular than the ROM banking mechanism.

Definitely, especially with that sg-1000 setup where you have 8 KiB + 2 KiB of RAM. Or the Codemasters mapper, where you can have RAM and ROM in the same 16 KiB bank.

willbritton wrote

Exotic Ichigoesque functionality aside, comparisons with NES aside for now, just thinking about these known mappers and focusing on historic games rather than homebrew, are there any other obvious features worth calling out?

Just from the mappers I know of, I'd say ROM bank shifting with the Sega mapper and obviously automatic byte flipping on the Janggun mapper.

Maxim wrote

However, I can’t help but feel like it’s a mistake to introduce anything that changes the checksum of the ROM file.

Y'all keep saying that, but I have yet to hear a reason why you feel that way. Is it because it'd break emulators using a ROM database?
Emulators that use mapper detection heuristics certainly wouldn't care.

Maxim wrote

For homebrew, it seems obvious to put the metadata in the ROM to reduce the complexity for the user, even if it means some more work for the emulator.

I don't see much reason to fragment how ROMs are distributed like that.
Having a single, consistent way of distributing any kind of ROM would be the most user-friendly option, especially if we adopted some kind of archive format like renamed ZIPs or tarballs.

sverx wrote

Tell the emulator that there's a fully fledged SEGA mapper on the cartridge. The code doesn't have to *use* all the features, but the emulator should offer them. If the mapper is instead supporting just a subset of these features, we should use sega-<something> here instead.

I'd just like to point out that, currently, the set of features that is considered a " fully fledged " Sega mapper also includes 32 KiB of SRAM.
My understanding is that is what emulators will default to when they detect the Sega mapper.
Not that it matters that much, but I guess I would've expected a game that doesn't use RAM to always have a sram-size: 0 specifier.

willbritton wrote

My previous post is an attempt to try and roughly scope the domain of the second thing, as far as the majority of original games are concerned. I rather suspect we don't need a full-blown HDL here, but that there's a relatively small set of primitive cartridge components that could describe a significant majority of original SMS carts.

On the first thing, I think a fundamental question is whether to only delegate the description to a single assembly name + parameters (e.g. {mapper:sega, sram-size:8192}) or to allow a more granular and thus flexible list of components, but either way I think the approach of having named and known (or locatable) entities is a good one, for a few reasons.

I hear you, and I do think we could come up with relatively simple syntax that could describe any " simple " mapper.
However, as a flash cart developer, I can also say it'd be nigh impossible to make that work on hardware. Even more so on flash carts with actual FPGAs.

You could argue that having mapper tags and a functional description would still be useful from an archival standpoint, but then we'd have a bunch of redundant data.

Trirosmos wrote

I'd just like to point out that, currently, the set of features that is considered a " fully fledged " Sega mapper also includes 32 KiB of SRAM.
My understanding is that is what emulators will default to when they detect the Sega mapper.
Not that it matters that much, but I guess I would've expected a game that doesn't use RAM to always have a sram-size: 0 specifier.

Honestly, I thought that too at first.
But then I realized that a "fully fledged" SEGA mapper *supports* SRAM, doesn't *include* it, so I preferred the other option, that is to add attributes when there's more hardware, instead of adding attributes to declare that something isn't present.
Same with batteries - having SRAM chips doesn't mean you also have batteries... but instead of adding an attribute that says 'we have batteries' I think it's wiser to have an attribute that says that SRAM is not volatile (which works also when using FRAM for instance). Again, one more attribute to say there's something more.

@Trirosmos, thanks for the great info on mappers, I'll definitely study the mappers page more closely.

Trirosmos wrote

My understanding is that is what emulators will default to when they detect the Sega mapper.

Emulicious applies some slightly more advanced heuristics — see here and I think I also noticed that it doesn't provision the RAM until you actually write to it (fairly moot) and as far as I can tell it assumes it's non-volatile, which is a little presumptuous.

Trirosmos wrote

However, as a flash cart developer, I can also say it'd be nigh impossible to make that work on hardware. Even more so on flash carts with actual FPGAs.

Overoptimism may be one of my defining traits, but I'm just not seeing "nigh impossible" at the moment, unless there's some factor that I'm missing.
Anyway, I'm going to give it a go, as currently I'm only implementing a Sega mapper on FPGA but would like something more flexible, so will report back when I realise you're right!

willbritton wrote

Trirosmos wrote

However, as a flash cart developer, I can also say it'd be nigh impossible to make that work on hardware. Even more so on flash carts with actual FPGAs.

Overoptimism may be one of my defining traits, but I'm just not seeing "nigh impossible" at the moment, unless there's some factor that I'm missing.
Anyway, I'm going to give it a go, as currently I'm only implementing a Sega mapper on FPGA but would like something more flexible, so will report back when I realise you're right!

I suppose I should've said " ugh, you could do it, but it's gonna be stupid ".
If a bespoke mapper is more or less " well behaved ", i.e, has a small number of mapper registers, reasonable bank granularity (no 256 byte banks for you) and uses " static " mapping (as opposed to the Janggun mapper with its switchable 16K/8K modes), then I suppose you could have a LUT of mapper addresses on the FPGA, a LUT of current bankings and some logic to automagically combine the two.

But a design like that would entirely hinge on the " well behaved " part. What if the specified mapper has 16 byte bank granularity, or dozens of registers with completely decoded addresses? How many comparators can your FPGA fit? 😅

Yeah well quite, it's certainly possible to think of mappers which are going to be hard to implement, but my point is that there ought to be a reasonable set of very common elements that can be used to implement a large proportion of the mappers we know about, if not exhaustively.

And yes, avoiding ridiculous numbers of 16 bit comparators would be nice! I did wonder whether known mappers tend to compare the same small set of bits, i.e. either the few most or least significant, plus the rest either all 1 or all 0.

In software I feel like it's much easier because fine grained comparisons don't come at any additional cost, and "register" cost is also negligible.

Getting a little OT now, but one interesting approach that might work for FPGA is the PIO model that RP2040 uses, which is effectively a nano-processor. Obviously that would come at the cost of reduced speed.
(BTW, I am now using an RP2040 as utility processor in my design which has been a great success — I have you to thank for the inspiration!)

willbritton wrote

And yes, avoiding ridiculous numbers of 16 bit comparators would be nice! I did wonder whether known mappers tend to compare the same small set of bits, i.e. either the few most or least significant, plus the rest either all 1 or all 0.

It's a mixed bag. The Codemasters mapper does incomplete address decoding, for instance, so it only cares about the two most significant bits of the address.
But the Sega mapper has to completely decode the register addresses, since they overlap RAM.

willbritton wrote

In software I feel like it's much easier because fine grained comparisons don't come at any additional cost, and "register" cost is also negligible.

It's much easier for a software emulator, but much harder for a software flash cart. I only have ~90 CPU cycles before I have to deliver data to the Z80 :P

willbritton wrote

Getting a little OT now, but one interesting approach that might work for FPGA is the PIO model that RP2040 uses, which is effectively a nano-processor. Obviously that would come at the cost of reduced speed.
(BTW, I am now using an RP2040 as utility processor in my design which has been a great success — I have you to thank for the inspiration!)

Yeah, I've thought about designing a micro-CPU especifically to handle mapper duties on a FPGA.
Of course, it'd have to be a more complex CPU than the PIOs, since those don't have arithmetic instructions and can't do comparisons other than exact X==Y.

willbritton wrote

(BTW, I am now using an RP2040 as utility processor in my design which has been a great success — I have you to thank for the inspiration!)

Heh, it's definitely my favorite MCU.

Had a walk and cleared my mind. I realized one important thing.
If you describe the cartridge simply as:

{mapper:sega}

you're implicitly saying you don't care if there's SRAM, its size, or type.
You don't care means you likely won't use it even if present so why would you worry about size and type after all? You just want the damn mapper!

So whatever the emulator wants to default to, it's fine to you.

But say your software requires it - you list it in your 'shopping list':

{mapper:sega, sram-size:8192}

again, the emulator could default to non-volatile SRAM, or not. If you need persistency, you then use:

{mapper:sega, sram-size:8192, non-volatile}

and if you want to test that your code clearly detects when SRAM is not available then you pick:

{mapper:sega, sram-size:0}

and the emulator has to abide by this, as it has been explicitly requested.

As a plus, I think it would be wonderful if one could provide such strings as a command line parameter at launch.
Say I want to test my game in different situations, I might have different command lines to invoke my emulator of choice as in:

emulator.exe -hwd:"{mapper:sega, sram-size:8192, non-volatile}" my_rom.sms

sverx wrote

again, the emulator could default to non-volatile SRAM, or not. If you need persistency, you then use:

{mapper:sega, sram-size:8192, non-volatile}

I'm not sure exactly why I object to the default to NVRAM, but I do. Perhaps it's just because Emulicious generates a .sav file alongside my ROM even if I don't need NVRAM. Otherwise I suppose it's no different from a software perspective than volatile RAM starting in a randomised state.

I suppose I should mention that I talked to ichigo and he said he has no plans of making the specs for his mapper(s) public, nor does he want them to be supported in emulators :(

Author	Message
Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	SDSC tag enhancements for homebrew Posted: Mon Feb 26, 2024 1:10 pm Last edited by Maxim on Mon Feb 26, 2024 1:14 pm; edited 1 time in total
	There's a perennial problem for emulation that ROM images are not enough to encapsulate the information about what is in a cartridge. However for homebrew we have options available to us. I'd therefore like to propose an extension to the venerable SDSC tag as follows: See https://www.smspower.org/Development/SDSCHeader Since we have filled the available space (without doing tricky things), I propose to extend the Description string. If (and only if) it starts with a '{' character then it should be parsed as JSON. The parsed document will define characteristics of the ROM, which we can discuss here. Anything left after the document may be shown as the description field currently is. 1. Mapper We could simply enumerate mappers; so type 0 = Sega, 1 = Codemasters... or we could try to describe the mapper in its entirety, e.g. describing which addresses can be written to, which memory mapped region is affected, and how. The former seems like we're not learning from the mistakes of the iNES header, and the latter seems complicated and error-prone - and might invite people to make custom mappers for no good reason. 2. Save memory size (and type) Most emulators will emulate save memory up to 32KB for the Sega mapper, and maybe try to trim it down to the smallest power of 2 size that is accessed. It might be nice to be explicit about it. 3. Anything else? Example: { "mapper": "sega-sms-generic", "sram-size": 8192 } Comments are welcome.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Feb 26, 2024 1:11 pm
	My initial motivation was to have a way for homebrew to require emulation of the standard Sega mapper, when some emulators only do this for file sizes over 64KB. Of course, if no emulators support this, or nobody uses it, then it won't work.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 1:23 pm
	1. Why JSON? I guess the main motivation is to support emulators, rather than real hardware, although I have a bias towards the latter. Would something like TOML or (and I hesitate to speak its name) even YAML be a better choice to cover the potential need to parse in hardware? I appreciate that JSON kind of gives you an escape character for free, since it probably ought to start with a `{` (although other characters can be valid JSON). Does TOML have to start with `[`? If so maybe that's an option. I also like TOML stylistically for this, because it reminds me of INI files and they feel more retro than JSON. 2. On the mappers, I've been thinking a lot about this too, and intuitively I agree that something codified would be preferable to an arbitrary menagerie a la iNES. Codifying isn't super trivial though. My interest is in designing a generic hardware mapper that can be reconfigured based on a header or metadata file, so it's definitely a problem I'm interested in being part of solving if it goes this direction.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 1:45 pm
	It seems to me it's a very sensible idea to use the SDSC tag... even if this only solves the issue for the (newer) homebrew. Mappers shouldn't be enumerated, having concise description is way better. "sega-generic" is a good term for instance, and there could be some alternatives such as "sega-slot2" that could be used when the generic mapper isn't adequate enough. So your example: { "mapper": "sega-sms-generic", "sram-size": 8192 } is pretty good. Of course how the SRAM is accessed depends on the mapper specifics so there's no need for other details. (also note that "sega-sms-generic" (or "sega-generic") should work with a number of bits compatible with the ROM size, so for instance a 128 KiB ROM (8 × 16 KiB pages) will make the mapper work on 3-bit index pages, thus ignoring the upper bits if set...) Then, back to the idea that this should also somehow be used for already existing ROM dumps, we could create JSON files to be stored in the same folder where a ROM dump is, using the same name with a .json extension (say 'Great Soccer.sms' ROM and 'Great Soccer.json' cartridge descriptor). An emulator/flashcart should then: - check if an SDSC header exists, and if json data is present at the Description field, use that as description of hardware. - alternatively, check if a .json file exists with the same name as the ROM being loaded. If this exists and contains json data, use that. - if all of the above fails, we fall back to the withstanding situation - the emulator/flashcart has to figure out by itself.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 1:59 pm
	I suppose that using some markup / config structure allows sufficient flexibility to have all known and also hypothetical mappers be named and fully parameterised. Something that might be quite clever is having the name point to a github repo path, which then contains a canonical declarative or imperative description of how the mapper works. That way, suitably clued-up emulators could dynamically add more mapper support. e.g. { "mapper": "sverx/sms-mapper", "params": { "bank0_register": "0xfffd", etc. } } or [mapper] name=sverx/sms-mapper [params] bank0_register=0xfffd "Lazy" emulators could just support the well-known Sega case, but at least the github path convention would act as a basic type of namespacing and mitigate the risk of a string-based arbitrary enumeration bloat instead of a numeric one.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 2:08 pm
	I suspect we should instead use the SMS Power! community as the only source of truth regarding what is valid and be the 'benevolent dictators' regarding mappers canonical names and all the details unfortunately anyone building a ROM could theoretically write whatever they want (and even plain wrong info) in the SDSC description field... so we should probably have the external json file override info in the SDSC description field? :thinking:

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 2:18 pm
	sverx wrote I suspect we should instead use the SMS Power! community as the only source of truth regarding what is valid and be the 'benevolent dictators' regarding mappers canonical names and all the details The benevolence part is not in doubt 😄, although I would point out that some members of this community do like to turn their hand to inventing their own hardware mappers, and to me it feels wrong, however well intentioned, to act as gatekeepers to them being able to freely innovate in the field. But I do get it. EDIT: and at the end of the day, what's to stop someone trying to overload the fields? Only a specification, convention and the support of emulator authors. EDIT2: also, how many "valid" mappers does this community know about? Presumably it's a very small and finite list. So then what? The matter is essentially closed, unless someone wants to lobby for a new mapper one day? I think that misses a trick. Anyone who invents a new hardware mapper can circumvent the process by just ignoring it, instead of engaging with the community and offering a mapper design that others might benefit from! sverx wrote unfortunately anyone building a ROM could theoretically write whatever they want (and even plain wrong info) in the SDSC description field... so we should probably have the external json file override info in the SDSC description field? :thinking: I actually think that strict emulator support here could be a great incentive for homebrewers to get their act together with respect to headers. Much like iNES, the game simply won't work on the emulator development workflow if the header is wrong. That said, if the fallback is standard Sega header anyway, in probably 80% of cases such an incentive won't apply anyway. Maxim's example of emulators applying some slightly tricky heuristics to supply a mapper (and there's a similar argument around save RAM) is valid. Personally I'd much prefer to take the time to write a valid header than to have to "trick" the emulator to give me the hardware I want.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 2:26 pm
	willbritton wrote I would point out that some members of this community do like to turn their hand to inventing their own hardware mappers, and to me it feels wrong, however well intentioned, to act as gatekeepers to them being able to freely innovate in the field. But I do get it. No, sorry, I didn't mean any gatekeeping. It's the naming we'll control, not the hardware features.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Feb 26, 2024 2:29 pm
	JSON because XML sucks, YAML is all sorts of broken and TOML is bad at nested structure. Almost every language has a JSON order of some sort, and I feel like it’s easier to implement in Z80 if need be. I’d like to see some proposals for the “encoded mapper”, even if we also define shortcuts for common cases. There are definitely tricky edge cases like byte flipping and bank number offsetting in the Korean mapper world. My idea is that if the whole mapper can be defined in markup, we can also define “shortcuts” for the common cases to avoid bloat. We act as BDFLs and define the shortcuts we want. I don’t think we need to support all the commercial games yet, though. If we feel like the markup is good for homebrew then maybe we can try to apply it to the cartridge dump problem later, but I don’t want to try to do it all at once. GitHub is not the best source of naming, it may not exist in 20 years (or be what SourceForge is today), but nor is DNS or anything else really.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 2:31 pm
	sverx wrote willbritton wrote I would point out that some members of this community do like to turn their hand to inventing their own hardware mappers, and to me it feels wrong, however well intentioned, to act as gatekeepers to them being able to freely innovate in the field. But I do get it. No, sorry, I didn't mean any gatekeeping. It's the naming we'll control, not the hardware features. Ah okay, not sure I understand the distinction though — are you saying there's a list of known mappers and somehow the SMSPower community curates that list? If so, how? I guess my suggestion of using a repo path allows people to create new names without risk of conflict, but also doesn't preclude SMSPower from controlling a list of "top level" names, either through a commons-owned repo organisation (e.g. https://github.com/smspower, which actually does exist!) or simply a mapper name which doesn't include a / symbol in it.

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Mon Feb 26, 2024 2:36 pm
	Maxim wrote or we could try to describe the mapper in its entirety, e.g. describing which addresses can be written to, which memory mapped region is affected, and how. The former seems like we're not learning from the mistakes of the iNES header, and the latter seems complicated and error-prone - and might invite people to make custom mappers for no good reason. willbritton wrote On the mappers, I've been thinking a lot about this too, and intuitively I agree that something codified would be preferable to an arbitrary menagerie a la iNES. Codifying isn't super trivial though. Unless y'all want to reinvent Verilog and Verilator, that is just not feasible. It might work for simple mappers, but how on earth would you write JSON that describes Ichigo's new mapper for instance, which has VDP DMA, can decompress graphics on the fly, flip graphics on the fly, compose sprites into background tiles on the fly, etc I do remember Calindro mentioning something about Emulicious being able to support mapper plugins, i.e, loading mapper code as Java's equivalent of a DLL. However, besides being a possible security threat, I'm not about to include a Java interpreter on my flash cart just so I can support new mappers :P willbritton wrote Something that might be quite clever is having the name point to a github repo path, which then contains a canonical declarative or imperative description of how the mapper works. Ah, yes, I do like to automatically download malware with my SMS ROMs :P fwiw, the newest version of the EverDrive N8 Pro firmware lets a game ship alongside a FPGA bitstream specifying a custom mapper. Of course, that bitstream is highly specific to that cartridge and that FPGA and it still could be a malicious bitstream that shorts pins or something, although how much damage that could do is debatable. sverx wrote Then, back to the idea that this should also somehow be used for already existing ROM dumps, we could create JSON files to be stored in the same folder where a ROM dump is, using the same name with a .json extension (say 'Great Soccer.sms' ROM and 'Great Soccer.json' cartridge descriptor). I believe that, in order to garner adoption, we do have to come up with something that can be applied to already existing ROMs as well. Having a separate file like this would definitely be the most straightforward way of achieving that. What I dislike about this idea is that it makes ROM distribution harder. Now they have to be tarballs or, worse, ZIPs. And then flash carts would either have to ship with a ZIP library or force users to unzip all their games before running them. Maybe that isn't that big of a deal. That's already gonna be the case for games using custom mappers on the EDN8 Pro. What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed. The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size. EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 2:40 pm
	Maxim wrote JSON because XML sucks, YAML is all sorts of broken and TOML is bad at nested structure. Almost every language has a JSON order of some sort, and I feel like it’s easier to implement in Z80 if need be. Surely JSON parsing has to be harder to implement in Z80 than TOML... Haha, at the rate things are going, I might have to write a JSON and TOML parser as my entry for the coding competition this year, which ought to guarantee no one else is disappointed to receive last place... Agree on XML and on YAML. TOML being bad at nested structure doesn't hurt us here though does it? Maxim wrote GitHub is not the best source of naming, it may not exist in 20 years (or be what SourceForge is today), but nor is DNS or anything else really. Sure, although it would only be github by convention — having something like "smspower/sega-mapper" could just be a strong hint that the definition of that mapper could be found by going to github, and for basic emulators simply doing a string comparison would be sufficient to handle that case. If github disappears, it would probably cause issue to a number of package managers who apply similar conventions, but obviously the risk could be mitigated by allowing a fuller qualification. Then you're getting into some verbose territory. Java has (or had?) arbitrary string namespaces with a strange reverse-URL type of convention, and plenty of languages do use fully qualified domain names which aren't the worst idea in the world. TIL what BDFL means...

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 2:47 pm Last edited by sverx on Mon Feb 26, 2024 2:48 pm; edited 1 time in total
	Maxim wrote I’d like to see some proposals for the “encoded mapper”, even if we also define shortcuts for common cases. There are definitely tricky edge cases like byte flipping and bank number offsetting in the Korean mapper world. My idea is that if the whole mapper can be defined in markup, we can also define “shortcuts” for the common cases to avoid bloat. We act as BDFLs and define the shortcuts we want. I don't think it's possible to really describe the hardware inside the json data, we can just use previously defined names there. As for these names, I don't have much experience with exotic mappers to help. But the mapper list should indeed include a "none" option, a few "sega-" variants... and then there could be a whole group of "korean-xxxxx" where each one has its own details. Still my point is that we need to be able to add this info to new and existing ROMs. Say one day fellow user Ichigo releases some games that uses a specific mapper with a specific feature, say byte flipping. We might then pick the name "ichigo-byteflip" (or some better name of course) and even if his games don't have any mapper info in the ROMs we have to be able to inform emulators and flashcarts of this, that's why I thought the separate json file could be a possible solution. But let's discuss more, it's a very very interesting topic! :D

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 2:48 pm
	Trirosmos wrote Ah, yes, I do like to automatically download malware with my SMS ROMs :P Oh, boo!! Where's your spirit of adventure!? But seriously, almost as soon as I wrote it I thought I should probably remove the word "imperative". I agree that executable download is a big no-no from a security perspective, but I do think that at least the idea of some basic set of declarative mapper primitives is worth exploring. Maybe not to include stuff like DMA, but who knows. It is, IMO, the best kind of interesting problem... Trirosmos wrote fwiw, the newest version of the EverDrive N8 Pro firmware lets a game ship alongside a FPGA bitstream specifying a custom mapper. Of course, that bitstream is highly specific to that cartridge and that FPGA and it still could be a malicious bitstream that shorts pins or something, although how much damage that could do is debatable. Wow, I didn't know that. My gut doesn't like that idea personally. So is he integrating a partial bitstream segment just for the mapper config into the whole bitstream file? If so, that's insanely devious. Trirosmos wrote What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed. The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size. EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics. Yeah, further to my above point on this subject, I strongly agree with this. I think companion files should only be used as a worst-case for non-standard ROMs which happen to have missing or incorrect headers.

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Mon Feb 26, 2024 2:53 pm
	willbritton wrote So is he integrating a partial bitstream segment just for the mapper config into the whole bitstream file? If so, that's insanely devious. No. The nature and quantity of NES mappers means it's much easier to just reconfigure the FPGA each time a game boots, which is what the EverDrive does. The mapper bitstream can use all of the FPGA's resources. Remember, the EDN8 Pro has both an ARM MCU and a second FPGA just for booting the system.

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Mon Feb 26, 2024 2:55 pm
	I worry this is a hairy can-of-worm problem to solve perfectly, and instead emulators supporting many mappers would be best pushed to have options to override mapper and other options during development. Whatever description language you come with would need to cater for the actual non-trivial example (rather than cater for "generic sega mapper" example). I'd be curious to hear about the practical problem that lead you to this idea Maxim? Maybe knowing this we can aim at best bang for bucks solution. (Tangent: note that MEKA implement some (basic) heuristic to detect mapper when running a ROM which is not in the DB. Given how simple the heuristic is, i believe we could also try to thoroughly develop it as a standalone micro project, tested against all known roms. But whether that would matter or not depends on actual problem you are trying to solve).

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 2:56 pm
	Trirosmos wrote willbritton wrote So is he integrating a partial bitstream segment just for the mapper config into the whole bitstream file? If so, that's insanely devious. No. The nature and quantity of NES mappers means it's much easier to just reconfigure the FPGA each time a game boots, which is what the EverDrive does. The mapper bitstream can use all of the FPGA's resources. Remember, the EDN8 Pro has both an ARM MCU and a second FPGA just for booting the system. Ah, okay. I guess the mapper makes up a substantial proportion of what the Everdrive is actually doing, so that does make sense. I was thinking more of my project where the mapper is just a tiny element of the whole. I really should find some time to look into the Everdrives more, I know next to nothing about them.

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Mon Feb 26, 2024 2:58 pm
	For reference: https://github.com/ocornut/meka/blob/master/meka/meka.nam This was never aimed at being any "official" but this is what MEKA use and this is probably the most complete list. Note that bsittler started working on emulating GG multicarts and we are expected to add dozens of new bizarre custom mappers in coming years.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 2:59 pm
	Regarding the parsing issue, I would suggest we use a very basic approach, a comma-separated list of attributes and value pairs between a { and a } so a simple {mapper:sega-sms-generic,sram-size:8192} would be sufficient. After all, we don't need to care about using commas or semicolumns in the attributes or values. Whitespace between tokens could be ignored too so this {mapper: sega-sms-generic, sram-size: 8192} would be equivalent. The curly braces both ensure a constant first char in the Description field and a terminator char for the hardware info block.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 3:04 pm
	sverx wrote Regarding the parsing issue, I would suggest we use a very basic approach, a comma-separated list of attributes and values between a { and a } Without meaning to sound like I'm flip-flopping, I think I do prefer some standard to none at all, so personally I'd rather see JSON than quasi-JSON. To bang the drum just once or twice more: replace "comma-separated" with "newline-separated" and your colon character with an equals sign and I think you've basically got TOML... (EDIT: oh except TOML needs string delimiters too :/ )

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 3:12 pm
	willbritton wrote Without meaning to sound like I'm flip-flopping, I think I do prefer some standard to none at all, so personally I'd rather see JSON than quasi-JSON. It was meant to simplify parsing (and save a few bytes ;) ) If it doesn't make things simpler, then ignore this :D

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 3:42 pm
	sverx wrote It was meant to simplify parsing (and save a few bytes ;) ) If it doesn't make things simpler, then ignore this :D No, of course you're right that it does simplify parsing, and I think the most important way it does is by neglecting string delimiters, which to my chagrin I see now that TOML doesn't, so now I'm back to INI files 🤣 It's seriously annoying there isn't a more popular super minimal standard for jobs like this. My reasoning for preferring a standard is that for desktop software, like emulators, it's much more convenient (and maybe more secure, etc.) to use a known parser library rather than having to hand-roll one, although obviously the simpler the format gets the more trivial writing a parser becomes. Going back to principles, whatever it is, we need: - An escape sequence which differentiates the extended structured data from a plain text description field - Basic dictionary semantics (key-value pairs) - We probably don't care about nested types or array structures (?) - We probably don't need typing in the syntax, because there would be a known schema (?) - Strict syntactical validation feels somewhat unnecessary given the context — homebrew authors probably want header population to be as foolproof as possible. - I do think something that's not hard to parse with a Z80 would be nice. I do concede that JSON is a significantly forward-compatible option, and I guess really the only thing I dislike about it is the string delimiting (which I now see TOML also has) and the (probably optional) need to validate a closing brace to match the opening one; but yeah okay even though writing a Z80 parser for a subset of JSON might be slightly more annoying than I'd like, it's definitely not going to break the bank in terms of ROM space is it. 😂 In retrospect I may have been inadvertently bikeshedding...

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Mon Feb 26, 2024 3:55 pm
	Trirosmos wrote What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed. The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size. EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics. If it turns out that it's impossible to append the metadata do existing ROMs, either because the EverDrive expects ROMs to have a certain size and be mirrored a certain way or because it'd break compatibility with emulators using checksums to find entries in a ROM database, I think the next best option would be sverx's suggestion of a separate file. I don't particularly care about what syntax the metadata uses. Anything that isn't a binary format is going to be annoying to implement on a flash cart anyways. I think the reason why iNES is such a mess has nothing to do with it being a binary format and, really, moving to JSON or YAML wouldn't solve most of the issues. The real problem is that, when iNES was created, very little was known about how the NES worked, how mappers worked and about just how dang many mappers there were. All other issues arose from trying as hard as possible to make every subsequent version of iNES backwards compatible.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Feb 26, 2024 4:17 pm Last edited by Maxim on Mon Feb 26, 2024 4:25 pm; edited 1 time in total
	The generalised issue of providing metadata with ROMs is very well explained here: https://web.archive.org/web/20200322151931/https://byuu.net/cartridges/boards but my proposal here is to take something like the “external file” model and embed it in homebrew. Let’s not try to solve the greater issue of metadata for all ROM dumps - yet. Embedding things in zips and then changing the extension is very common these days, and emulators have supported zips since forever. Personally I’d go for that approach for the generalised problem. ROM databases are fine if you’re willing to spend years of your life acquiring and dumping cartridges to maintain quality. But they suck for managing metadata for homebrew and hacks.

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Mon Feb 26, 2024 4:21 pm
	Maxim wrote The generalised issue of providing metadata with ROMs is very well explained here: https://web.archive.org/web/20200322151931/https://byuu.net/cartridges/boards but my proposal here is to take something like the “external file” model and embed it in homebrew. Let’s not try to solve the greater issue of metadata for all ROM dumps - yet. But it's the same issue, tho ;) In fact, solving it for homebrew games is a superset of solving it for existing ROM dumps, since homebrew games can do way weirder stuff. If we define this new header that only homebrew games use, is anyone gonna adopt it, unless there's a new hot homebrew with a custom mapper that only works if emulators do support it? Even then, aren't emulators more likely to just special case that particular homebrew instead of supporting a new header? I think it's much better to define one way of specifying all cartridge hardware rather than sometimes embedding a header, sometimes having an external file, sometimes having both, sometimes having neither...

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Feb 26, 2024 4:27 pm Last edited by Maxim on Mon Feb 26, 2024 4:29 pm; edited 1 time in total
	If I can opt my homebrew in to support for certain common features, then I’ll use it. That feature support might not be great for less well-supported things, but if someone wants to make a homebrew game that uses Meka’s nth Korean mapper’s features, wouldn’t it be great if Meka can enable it based on data in the ROM instead of heuristics? If it then only works in Meka, because Emulicious doesn’t support that mapper yet, then that’s fine. If we first decide we have to support every necessary feature of every ROM ever released, plus special cases on top (Master Games? SG-1000 games with “40KB” layouts? Pro Action Replay?) then we’ll get stuck before we get started.

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Mon Feb 26, 2024 4:29 pm
	It might be worth considering that e.g. an Everdrive BIOS could possibly want to parse this data from Z80 code, but of course the parsing doesn't need to be efficient.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Feb 26, 2024 4:30 pm
	That is true, and that’s partly why I went with JSON - it’s easier to parse than YAML and XML, although still not very easy. Another approach would be to use something like the tagging formats used for metadata on media files.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 5:02 pm
	Anyway I still believe that if we "find a solution" that works only for the (newer) homebrews, we aren't doing anything very meaningful, let me explain. - new homebrew might not use what we would want anyway - new homebrew might use custom (and never seen before) mappers, and we can't ask them to please let us know in advance So I still believe we should somehow start by identifying and researching (and then give a good name) to the mappers we already know, and MEKA's mappers list is a good starting point for this. Once we get this info, we should verify we are able to describe the existing releases correctly. Again, MEKA is quite invaluable for this. Then we create 'official' attributes and values, and pick a data format for that. Then we should discuss how to make it possible for this info to be 'attached' (in a loose sense) to their own ROMs. A public list? A small text file (json or whatever, that's a separate detail now)? We'll pick the option(s) that work best. Only if we solve this, then we might see how to improve the formula for the new releases (using homebrew's SDSC tag for instance) ... but we shouldn't rely on that because it's out of our control anyway, we should just consider it as an interesting option.

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Mon Feb 26, 2024 5:28 pm
	sverx wrote Anyway I still believe that if we "find a solution" that works only for the (newer) homebrews, we aren't doing anything very meaningful Agreed, any format proposed shall support all existing ROMs if it wants to supersede what's currently done. Trirosmos wrote What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed. The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size. EDIT: My main argument for attaching metadata to existing ROMs is that we could finally get rid of ROM databases and of convoluted and error-prone mapper detection heuristics. As a point of reference, here's the specification for the GBX footer format that is supported by mGBA and hhugboy (and seems like support for it is also planned on SameBoy?).

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Mon Feb 26, 2024 5:45 pm
	In my opinion (and I stress opinion) any change/addition to any ROM dump binary file is a "very bad thing™" Byuu wrote: Quote External board descriptions Similar to third-party headers like iNES, the idea is to include a file with the ROM images that describes the games. You've seen examples of that from bsnes above, eg from the problem 1 section. I personally favor this approach because an external file can be stored in plain-text, and can use an extensible markup format that allows us to add new corner cases as they are discovered, rather than have a format with limitations such as iNES permanently set in stone. However, this approach is also not without its problems: the biggest one is simply designing the specification. How would one create a markup format that could adequately describe any arbitrary circuit board configuration, population and wiring for any game system? And how would one get the hundreds of existing emulator developers to adopt formats that will often be far more complex than the needs of their individual systems? For instance, NES emulator authors are not concerned about Flash memory save type IDs like GBA emulator authors are, and Genesis emulator authors are not as concerned about memory map layouts (where things are always linear) as SNES emulator authors are (where things are banked.) Further, there's still the issue of what we do when a board description is released with an error in it. How do we update every copy of the game in the world with the corrected version? And finally, there is intensely strong resistance to the idea of storing more than a single file per game image, even if inside of a ZIP archive. I do not fully understand the resistance here, but it is a reality we have to face if we want a ubiquitous solution. Regarding the Quote How would one create a markup format that could adequately describe any arbitrary circuit board configuration, population and wiring for any game system? I think we just solve this by NOT describing it in the file, but just referencing its technical details using a name. Regarding Maxim's specific use case: if he ships a ROM plus a 'hardware descriptor' in the expected format, updated emulators will honor the descriptors - old ones won't, but I suspect there's no way to convince any existing emulator to enable SRAM on a <=48 KiB ROM apart from padding the ROM to 64 KiB, which defies our efforts here.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Feb 26, 2024 5:58 pm
	My current highly specific need is for an updated SMS VGM Player. This consists of a 16KB stub plus a variable-length amount of VGM data appended; the total may be anywhere from ~16.1KB to 4MB, not rounded to any sensible amount. Some emulators/flash carts will not work properly unless I also do something to expand the ROM to a larger size, and specifically I hoped I could coerce Emulicious into letting me debug it :)

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Mon Feb 26, 2024 6:05 pm
	willbritton wrote Yeah, further to my above point on this subject, I strongly agree with this. I think companion files should only be used as a worst-case for non-standard ROMs which happen to have missing or incorrect headers. Sorry, I realise I misread Trirosmos's original comment here and conflated two things. What I actually meant was, I think developers (i.e. of new games) should be incentivised to get their headers right first time. With regards to modifying existing binaries, yeah I think I'm with sverx here, it seems somewhat fraught for a few reasons. I guess a concern with an archive is that it adds complexity to file handling in hardware, although for anything with a MCU in it, like a flashcart, feels like it's not too much to ask. Otherwise in principle, packaging files together into a logical unit seems like a very sensible thing to do. I don't know much about TAR, but is that much simpler to handle in hardware than a ZIP file, for example? sverx wrote I think we just solve this by NOT describing it in the file, but just referencing its technical details using a name. Yeah think I mostly agree with this, although there does need to be a rigorous specification somewhere of what each thing actually does. It could be a reference implementation (but in what language? multiple?) or maybe some kind of functional test specification? At the very least, it should be a written spec that leaves emulator and / or hardware designers in no doubt as to how to implement. There's potentially a slippery slope here which might lead one down a path of allowing composite devices to be specified in meta files (e.g. it's a standard Sega mapper + Ichigo's custom byte-flipping widget) in order to avoid having to generate completely new high level devices and associated name bloat, and as soon as you go down the slope it feels like inventing a new HDL, as per Trirosmos's first comment on this thread. That said, it is an approach I've been thinking of to solve my particular hardware problem (I don't think that having people provide a 150KB FPGA bitstream with every game is the right solution for my use case!)

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Mon Feb 26, 2024 6:32 pm
	willbritton wrote I guess a concern with an archive is that it adds complexity to file handling in hardware, although for anything with a MCU in it, like a flashcart, feels like it's not too much to ask. Otherwise in principle, packaging files together into a logical unit seems like a very sensible thing to do. I don't know much about TAR, but is that much simpler to handle in hardware than a ZIP file, for example? To be fair, I haven't actually tried to integrate a ZIP library into one of my projects. With that said, tar is about the simplest archive format possible. Literally a 512 byte header for each file in the archive followed by the file data, verbatim. So, you can very easily read it in place. ZIP on the other hand would require something in the order of 41 KiB of RAM? Perhaps using only uncompressed ZIPs would simplify the code significantly, but then users might get confused as to why a completely valid ZIP file isn't being accepted by an emulator or flash cart.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Mon Feb 26, 2024 6:48 pm
	Zip is a very old format but indeed the Deflate compression needs a lot of memory for a Z80 - but maybe not much for a microcontroller, something like 32KB look back plus a few KB for the trees. Uncompressed zips would be a pain to manage. TAR is uncompressed so it’s simpler to deal with but somewhat harder to parse as it lacks a central directory. Most emulators already support loading ROMs from zips.

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Mon Feb 26, 2024 11:30 pm
	Trirosmos wrote What I'd prefer is that we add the metadata as a footer to the ROM. That way, the original ROM contents aren't modified, but no extra file is needed. The trouble with that is old emulators and flash carts seem to rely on ROM sizes being a multiple of 16 KiB, or worse, a power of two size. Regarding possible problems with a footer, is anyone here privy to the details of what breaks on the OG SMS EverDrive? And are those issues fixed on the X7 at all? What happens on them if the ROM size isn't a multiple of 16 KiB? Or isn't a power of two? What happens if I append a single byte to the end of a ROM?

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Feb 27, 2024 7:01 am
	SMS Everdrive (not x7) will truncate ROMs to a multiple of 64KB on load, unless they are smaller than 64KB. This seems to be an oversight due to it apparently loading the data in 64KB chunks. Some very old ROM dumps had 512B headers (from Super Magic Drive?) or 64B footers, so very old emulators were written to ignore these. However, I can’t help but feel like it’s a mistake to introduce anything that changes the checksum of the ROM file. That’s why I like the ROM+metadata in a zip approach (for non-homebrew): old emulators can ignore the metadata and load the ROM like they always did, relying on heuristics. New emulators can take advantage of the extra metadata. Checksumming tools will detect the game (even with bad metadata…) and leave the metadata alone as they list/rename/move the outer zip around. For homebrew, it seems obvious to put the metadata in the ROM to reduce the complexity for the user, even if it means some more work for the emulator.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Tue Feb 27, 2024 9:40 am
	Not sure if this is the right direction to move the discussion in, but was just looking at the mapper IDs from Meka that Bock posted: ;----------------------------------------------------------------------------- ; Cartridge Hardware ("Mapper") Identifiers ; From MEKA sources in srcs/mappers.h ;----------------------------------------------------------------------------- ; 0 : standard sms/gg sega mapper, optional SRAM ; 1 : 32KB of RAM mapped at 0x8000-0xFFFF ; 2 : colecovisions ; 3 : code masters mapper, registers at 0x0000, 0x4000, 0x8000 ; 4 : sega mapper with 93c46 eeprom ; 5 : sg-1000 (currently incorrectly emulating 4KB of RAM instead of 2KB) ; 6 : master system action replay ; 7 : terebi oekaki (graphic tablet) ; 8 : sf-7000 ; 9 : sms korean mapper, mapping register at 0xA000 ; 10 : sms display unit BIOS ; 11 : no mapper, ROM up to 48KB (typically 32KB) ; 12 : sms korean msx type 8KB banks, registers at 0x0000->0x0003 ; 13 : sms korean for Janggun-ui Adeul ; 14 : sms 4 Pak All Action ; 15 : sg-1000 II with taiwanese memory expansion adapter, 8KB RAM from 0x2000->0x3FFF + regular 2KB RAM mapped at 0xC000-0xFFFF ; 16 : sms korean xx-in-1, register at 0xFFFF mapping 32 KB at 0x0000->0x8000 (hi-com cartridges) ; 17 : sc-3000 survivors multicart. ; 18 : sms korean xx-in-1, register at 0x0000 mapping 8 KB at 0x8000, register at 0x0100 mapping 8 KB at 0x4000, register at 0x0200 mapping 8 KB at 0xA000, register at 0x0300 mapping 8 KB at 0x6000 ; 19 : sms korean xx-in-1, register at 0x2000 mapping 8x4 KB 0x4000-0xBFFF using register xored with 0x1F, 0x1E, 0x1D, 0x1C. 0x0000-0x3FFF stuck on bank 0. ; 20 : sms korean xx-in-1, register at 0xBFFC, sms+msx ; 21 : sms korean xx-in-1, register at 0xFFFE ; 22 : sms korean xx-in-1, register at 0xFFF3 and 0xFFFC (mirrored 0xBFF3, 0xBFFC) ; 23 : sms korean xx-in-1, register at 0x0000, xored with 0xFF ; 24 : sms korean xx-in-1, register at 0xFFF0, 0xFFFE, 0xFFFE (used in MD multicart with SMS games) ; 25 : sms korean xx-in-1, register at 0xFFF5, 0xFFFE, 0xFFFE (used in MD multicart with SMS games) ; 26 : sms korean xx-in-1, register at 0xFFFA, 0xFFFE, 0xFFFE (used in MD multicart with SMS games) ; 27 : sms korean xx-in-1, register at 0x2000 ; 39 : sms korean xx-in-1, register at 0x2000 (11 Hap Gam-Boy) ; 40 : sms korean xx-in-1, register at 0x8000 ;----------------------------------------------------------------------------- I guess the implementation for these I can find if I browse the Meka code, but just wondering at a high level what primitive features we're actually looking at here. At a guess I'd say: - ROM banks - at what granularity? Always 16KB or sometimes smaller? - ROM bank control registers - with various decodings. - Sometimes ROM bank decoding is disabled for certain blocks (e.g. lower 1KB on Sega mapper, but not on Codemasters) - RAM banking - I guess we can assume "it varies", since the official Sega RAM banking mechanism is slightly less regular than the ROM banking mechanism. - On-cart RAM can either be volatile or non-volatile Exotic Ichigoesque functionality aside, comparisons with NES aside for now, just thinking about these known mappers and focusing on historic games rather than homebrew, are there any other obvious features worth calling out?

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Tue Feb 27, 2024 9:56 am
	OK this is disrupting my sleep now, that's not good. Anyway, since I was awake, I thought that the focus should be on the emulator. What I mean is that we currently have ROM dumps and emulators have to 'guess' everything else. They do this pretty well, but this is not something we can control in any way, so I think our goal should be to make sure we could pass a minimum set of information alongside a ROM dump and have the emulator use this information. This opens up a lot of new options, in my opinion. For instance, a game could have code to use different hardware, for instance to save. Also this means that these hardware info can't be written in the ROM itself - after all it also makes sense as it's the ROM to be inside the cartridge and not the other way around. So I'm pretty much convinced that the way is to have a small text file alongside the ROM dump, or even bundled into the same archive for those emulators supporting compressed format(s). Now, we can discuss the syntax of course, but the foundation of the idea is that we pass attributes/values pair to the emulators. I suppose we should keep the things as simple as possible by strictly defining exactly what's expected and what's not from each mapper, having additional attributes to further detail (and I admit I prefer the approach of having additional attributes when the cartridge has additional hardware than the other way around, that would be to add attributes when something is missing). Let me post a few examples. Again, ignore the format for now. {mapper:none} Tell the emulator that there's no mapper in this cartridge. Z80 address space will be mapped to ROM 1:1, everything above 48 KiB will be 'out of reach'. Writes to $FFFC-$FFFF won't have effect apart from writing to mirror RAM, etc. This is good for testing how a ROM would work on a basic 'homemade' (mapperless) cartridge. {mapper:sega} Tell the emulator that there's a fully fledged SEGA mapper on the cartridge. The code doesn't have to use all the features, but the emulator should offer them. If the mapper is instead supporting just a subset of these features, we should use sega-<something> here instead. As said before, additional hardware present in the cartridge should mean additional attributes in the hardware descriptor, so for instance there should not be any SRAM present unless: {mapper:sega, sram-size:8192} ... and if there's a battery (or the sram is non volatile) so that SRAM contents are kept between sessions, one should have: {mapper:sega, sram-size:8192, non-volatile} (an attribute without value should be equivalent to attribute:true) What I find interesting here is that one could easily test his own code in different situations, thus emulating how that would work on different cartridges. For instance, one could reduce/increment the SRAM size, or even check what happens when SRAM contents are lost (or retained!) between sessions. Finally, I've also seen a few homebrew using Flash to save. We might probably support that too, maybe with something like: {mapper:sega, flash:MX29F040} ... but, well, it's all in the hands of emulators programmer to what extent they care about this (not all emulators are made for new software development...) I'm eager to know if all this seems over-complicated or instead if it falls short of what is needed, so feedback is welcome! :D

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Tue Feb 27, 2024 10:27 am
	sverx wrote I'm eager to know if all this seems over-complicated or instead if it falls short of what is needed, so feedback is welcome! :D TLDR; yes agreed I think that's kind of where the conversation seems to have been gravitating! I think there are two parts here: 1. A metadata format that can be used to describe the assembly that is a cartridge. The ROM isn't the cartridge, it's only one part of the overall assembly. 2. A specification format that can be used to describe what each component in the assembly actually does, such that emulators can emulate it, or hardware can implement it. My previous post is an attempt to try and roughly scope the domain of the second thing, as far as the majority of original games are concerned. I rather suspect we don't need a full-blown HDL here, but that there's a relatively small set of primitive cartridge components that could describe a significant majority of original SMS carts. On the first thing, I think a fundamental question is whether to only delegate the description to a single assembly name + parameters (e.g. {mapper:sega, sram-size:8192}) or to allow a more granular and thus flexible list of components, but either way I think the approach of having named and known (or locatable) entities is a good one, for a few reasons. sverx wrote OK this is disrupting my sleep now, that's not good. Are you in Sweden now? If so, I have to tell you it's probably time to wake up :D

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Tue Feb 27, 2024 10:33 am
	willbritton wrote My previous post is an attempt to try and roughly scope the domain of the second thing, as far as the majority of original games are concerned. I rather suspect we don't need a full-blown HDL here, but that there's a relatively small set of primitive cartridge components that could describe a significant majority of original SMS carts. Yes! You posted it while I was posting mine, so I read that later. But I do agree that MEKA is one of the two (main) sources that we should use to define the mapper 'set', the other being the mappers page, and they likely overlap anyway. Now, of course the issue is that one of the hardest things is always naming things...

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Tue Feb 27, 2024 11:14 am Last edited by Trirosmos on Tue Feb 27, 2024 12:03 pm; edited 1 time in total
	willbritton wrote - ROM banks - at what granularity? Always 16KB or sometimes smaller? As per de mappers page, some mappers do use 8 KiB banks. In particular, the Janggun mapper let's you choose at will, on the fly, whether a certain portion of the address space will be broken down into a single 16 KiB bank or two 8 KiB ones. willbritton wrote - RAM banking - I guess we can assume "it varies", since the official Sega RAM banking mechanism is slightly less regular than the ROM banking mechanism. Definitely, especially with that sg-1000 setup where you have 8 KiB + 2 KiB of RAM. Or the Codemasters mapper, where you can have RAM and ROM in the same 16 KiB bank. willbritton wrote Exotic Ichigoesque functionality aside, comparisons with NES aside for now, just thinking about these known mappers and focusing on historic games rather than homebrew, are there any other obvious features worth calling out? Just from the mappers I know of, I'd say ROM bank shifting with the Sega mapper and obviously automatic byte flipping on the Janggun mapper. Maxim wrote However, I can’t help but feel like it’s a mistake to introduce anything that changes the checksum of the ROM file. Y'all keep saying that, but I have yet to hear a reason why you feel that way. Is it because it'd break emulators using a ROM database? Emulators that use mapper detection heuristics certainly wouldn't care. Maxim wrote For homebrew, it seems obvious to put the metadata in the ROM to reduce the complexity for the user, even if it means some more work for the emulator. I don't see much reason to fragment how ROMs are distributed like that. Having a single, consistent way of distributing any kind of ROM would be the most user-friendly option, especially if we adopted some kind of archive format like renamed ZIPs or tarballs. sverx wrote Tell the emulator that there's a fully fledged SEGA mapper on the cartridge. The code doesn't have to use all the features, but the emulator should offer them. If the mapper is instead supporting just a subset of these features, we should use sega-<something> here instead. I'd just like to point out that, currently, the set of features that is considered a " fully fledged " Sega mapper also includes 32 KiB of SRAM. My understanding is that is what emulators will default to when they detect the Sega mapper. Not that it matters that much, but I guess I would've expected a game that doesn't use RAM to always have a sram-size: 0 specifier. willbritton wrote My previous post is an attempt to try and roughly scope the domain of the second thing, as far as the majority of original games are concerned. I rather suspect we don't need a full-blown HDL here, but that there's a relatively small set of primitive cartridge components that could describe a significant majority of original SMS carts. On the first thing, I think a fundamental question is whether to only delegate the description to a single assembly name + parameters (e.g. {mapper:sega, sram-size:8192}) or to allow a more granular and thus flexible list of components, but either way I think the approach of having named and known (or locatable) entities is a good one, for a few reasons. I hear you, and I do think we could come up with relatively simple syntax that could describe any " simple " mapper. However, as a flash cart developer, I can also say it'd be nigh impossible to make that work on hardware. Even more so on flash carts with actual FPGAs. You could argue that having mapper tags and a functional description would still be useful from an archival standpoint, but then we'd have a bunch of redundant data.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Tue Feb 27, 2024 11:27 am
	Trirosmos wrote I'd just like to point out that, currently, the set of features that is considered a " fully fledged " Sega mapper also includes 32 KiB of SRAM. My understanding is that is what emulators will default to when they detect the Sega mapper. Not that it matters that much, but I guess I would've expected a game that doesn't use RAM to always have a sram-size: 0 specifier. Honestly, I thought that too at first. But then I realized that a "fully fledged" SEGA mapper supports SRAM, doesn't include it, so I preferred the other option, that is to add attributes when there's more hardware, instead of adding attributes to declare that something isn't present. Same with batteries - having SRAM chips doesn't mean you also have batteries... but instead of adding an attribute that says 'we have batteries' I think it's wiser to have an attribute that says that SRAM is not volatile (which works also when using FRAM for instance). Again, one more attribute to say there's something more.

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Tue Feb 27, 2024 11:57 am
	@Trirosmos, thanks for the great info on mappers, I'll definitely study the mappers page more closely. Trirosmos wrote My understanding is that is what emulators will default to when they detect the Sega mapper. Emulicious applies some slightly more advanced heuristics — see here and I think I also noticed that it doesn't provision the RAM until you actually write to it (fairly moot) and as far as I can tell it assumes it's non-volatile, which is a little presumptuous. Trirosmos wrote However, as a flash cart developer, I can also say it'd be nigh impossible to make that work on hardware. Even more so on flash carts with actual FPGAs. Overoptimism may be one of my defining traits, but I'm just not seeing "nigh impossible" at the moment, unless there's some factor that I'm missing. Anyway, I'm going to give it a go, as currently I'm only implementing a Sega mapper on FPGA but would like something more flexible, so will report back when I realise you're right!

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Tue Feb 27, 2024 12:12 pm
	willbritton wrote Trirosmos wrote However, as a flash cart developer, I can also say it'd be nigh impossible to make that work on hardware. Even more so on flash carts with actual FPGAs. Overoptimism may be one of my defining traits, but I'm just not seeing "nigh impossible" at the moment, unless there's some factor that I'm missing. Anyway, I'm going to give it a go, as currently I'm only implementing a Sega mapper on FPGA but would like something more flexible, so will report back when I realise you're right! I suppose I should've said " ugh, you could do it, but it's gonna be stupid ". If a bespoke mapper is more or less " well behaved ", i.e, has a small number of mapper registers, reasonable bank granularity (no 256 byte banks for you) and uses " static " mapping (as opposed to the Janggun mapper with its switchable 16K/8K modes), then I suppose you could have a LUT of mapper addresses on the FPGA, a LUT of current bankings and some logic to automagically combine the two. But a design like that would entirely hinge on the " well behaved " part. What if the specified mapper has 16 byte bank granularity, or dozens of registers with completely decoded addresses? How many comparators can your FPGA fit? 😅

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Tue Feb 27, 2024 12:23 pm
	Yeah well quite, it's certainly possible to think of mappers which are going to be hard to implement, but my point is that there ought to be a reasonable set of very common elements that can be used to implement a large proportion of the mappers we know about, if not exhaustively. And yes, avoiding ridiculous numbers of 16 bit comparators would be nice! I did wonder whether known mappers tend to compare the same small set of bits, i.e. either the few most or least significant, plus the rest either all 1 or all 0. In software I feel like it's much easier because fine grained comparisons don't come at any additional cost, and "register" cost is also negligible. Getting a little OT now, but one interesting approach that might work for FPGA is the PIO model that RP2040 uses, which is effectively a nano-processor. Obviously that would come at the cost of reduced speed. (BTW, I am now using an RP2040 as utility processor in my design which has been a great success — I have you to thank for the inspiration!)

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Tue Feb 27, 2024 12:43 pm
	willbritton wrote And yes, avoiding ridiculous numbers of 16 bit comparators would be nice! I did wonder whether known mappers tend to compare the same small set of bits, i.e. either the few most or least significant, plus the rest either all 1 or all 0. It's a mixed bag. The Codemasters mapper does incomplete address decoding, for instance, so it only cares about the two most significant bits of the address. But the Sega mapper has to completely decode the register addresses, since they overlap RAM. willbritton wrote In software I feel like it's much easier because fine grained comparisons don't come at any additional cost, and "register" cost is also negligible. It's much easier for a software emulator, but much harder for a software flash cart. I only have ~90 CPU cycles before I have to deliver data to the Z80 :P willbritton wrote Getting a little OT now, but one interesting approach that might work for FPGA is the PIO model that RP2040 uses, which is effectively a nano-processor. Obviously that would come at the cost of reduced speed. (BTW, I am now using an RP2040 as utility processor in my design which has been a great success — I have you to thank for the inspiration!) Yeah, I've thought about designing a micro-CPU especifically to handle mapper duties on a FPGA. Of course, it'd have to be a more complex CPU than the PIOs, since those don't have arithmetic instructions and can't do comparisons other than exact X==Y. willbritton wrote (BTW, I am now using an RP2040 as utility processor in my design which has been a great success — I have you to thank for the inspiration!) Heh, it's definitely my favorite MCU.

sverx Joined: 05 Sep 2013 Posts: 3828 Location: Stockholm, Sweden	Posted: Tue Feb 27, 2024 1:33 pm
	Had a walk and cleared my mind. I realized one important thing. If you describe the cartridge simply as: {mapper:sega} you're implicitly saying you don't care if there's SRAM, its size, or type. You don't care means you likely won't use it even if present so why would you worry about size and type after all? You just want the damn mapper! So whatever the emulator wants to default to, it's fine to you. But say your software requires it - you list it in your 'shopping list': {mapper:sega, sram-size:8192} again, the emulator could default to non-volatile SRAM, or not. If you need persistency, you then use: {mapper:sega, sram-size:8192, non-volatile} and if you want to test that your code clearly detects when SRAM is not available then you pick: {mapper:sega, sram-size:0} and the emulator has to abide by this, as it has been explicitly requested. As a plus, I think it would be wonderful if one could provide such strings as a command line parameter at launch. Say I want to test my game in different situations, I might have different command lines to invoke my emulator of choice as in: emulator.exe -hwd:"{mapper:sega, sram-size:8192, non-volatile}" my_rom.sms

willbritton Joined: 06 Mar 2022 Posts: 671 Location: London, UK	Posted: Tue Feb 27, 2024 1:41 pm
	sverx wrote again, the emulator could default to non-volatile SRAM, or not. If you need persistency, you then use: {mapper:sega, sram-size:8192, non-volatile} I'm not sure exactly why I object to the default to NVRAM, but I do. Perhaps it's just because Emulicious generates a .sav file alongside my ROM even if I don't need NVRAM. Otherwise I suppose it's no different from a software perspective than volatile RAM starting in a randomised state.

Trirosmos Joined: 26 Apr 2022 Posts: 29 Location: Brinstar, Zebes	Posted: Tue Feb 27, 2024 3:40 pm
	I suppose I should mention that I talked to ichigo and he said he has no plans of making the specs for his mapper(s) public, nor does he want them to be supported in emulators :(

Forums

View topic - SDSC tag enhancements for homebrew