Forums

Sega Master System / Mark III / Game Gear
SG-1000 / SC-3000 / SF-7000 / OMV
Home - Forums - Games - Scans - Maps - Cheats - Credits
Music - Videos - Development - Hacks - Translations - Homebrew

View topic - Downloadable database of SMS games?

Reply to topic
Author Message
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Downloadable database of SMS games?
Post Posted: Wed Aug 10, 2022 11:02 am
Maybe one for the site admins, but I'm looking to get a complete database of known SMS games, including product code, region, name and ROM size.

This kind of thing gets me very close:
https://www.smspower.org/Games-Releases/SMS-EU

but crucially the ROM size is missing, I would have to click into every page and find it, although I presume this whole part of the site is backed by a database?

Is there any magic link I can use to get the entire games dataset; or failing that, anyone know the best place I can find the data?

EDIT: also it would be interesting if there were any data on which mapper (e.g. Sega / Codemasters / none / etc.) is present in the cart, but I'm sure for the most part that is easy to infer.
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14226
  • Location: London
Reply with quote
Post Posted: Wed Aug 10, 2022 11:18 am
It’s something we’ve always planned on doing (ultimately to generate meka.nam) but it’s yet to happen. The site is maxed on pmwiki which is file oriented, making such operations expensive but as they are rarely needed, it’s not a big deal. Thus it would be feasible to generate JSON or similar in PHP using pmwiki’s page APIs.
  View user's profile Send private message Visit poster's website
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 11:36 am
Maxim wrote
ultimately to generate meka.nam

Oh but that's interesting, could this help me?
https://github.com/ocornut/meka/blob/master/meka/meka.nam

What / how do I interpret the two hex fields before the title?
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14226
  • Location: London
Reply with quote
Post Posted: Wed Aug 10, 2022 11:41 am
They are checksums of the ROM, the first is the CRC32 and the second is a Meka secret checksum that is quite broken so you should ignore it.
  View user's profile Send private message Visit poster's website
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 11:45 am
Ah gotcha, well all looks like useful data, but I guess no ROM sizes again?
  View user's profile Send private message Visit poster's website
  • Joined: 01 May 2011
  • Posts: 444
Reply with quote
Post Posted: Wed Aug 10, 2022 12:26 pm
This might help for most non-Japanese releases.
  View user's profile Send private message
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 12:53 pm
BKK wrote
This might help for most non-Japanese releases.

Ah, amazing, thanks @bkk!

Related, side-note: I appear to have a copy of Lemmings that has a product code different from that found elsewhere (it's 57108 rather than 7108, i.e. I read 08 71 50 from the header, and assuming that I'm interpreting the last byte properly – I used this guide – EDIT: I definitely am because the procedure works for other carts with a non-zero third byte)

What, if anything, shall I do about documenting this somewhere?
  View user's profile Send private message Visit poster's website
  • Joined: 01 May 2011
  • Posts: 444
Reply with quote
Post Posted: Wed Aug 10, 2022 1:00 pm
It might just be an error in the header for that game, does your dump match previously dumped roms?
  View user's profile Send private message
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 1:26 pm
I'm not yet in a position to dump entire ROMs (I'm reading the header with a custom BIOS), but I'm sure I can write a routine to calculate the checksum.

However, out of interest I have just...ahem...managed to acquire another copy of a Lemmings ROM, and it too has a header of 08 71 50, just the same as my physical cart, and I've checked the crc32 of that ROM and it's f369b2d8, same as here and here, so I rather suspect that this is a transposition error documenting the original cart back in the day.
  View user's profile Send private message Visit poster's website
  • Joined: 16 May 2002
  • Posts: 1301
  • Location: italy
Reply with quote
Post Posted: Wed Aug 10, 2022 1:41 pm
willbritton wrote
I would have to click into every page and find it
This can be easily automated, I know how to do that (I often did similar things in the past on other websites, mostly to collate sport results from different sources), and I'd volunteer to do that for you, but I can't because of smspower's https, which makes it impossible for me to use my usual tools.
  View user's profile Send private message Visit poster's website
  • Joined: 01 May 2011
  • Posts: 444
Reply with quote
Post Posted: Wed Aug 10, 2022 1:42 pm
Yeah, most likely just an error. IIRC I only checked headers for games where a cover product code didn't exist, so I'm sure that's not the only one which doesn't match.
  View user's profile Send private message
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 1:54 pm
Tom wrote
willbritton wrote
I would have to click into every page and find it
This can be easily automated, I know how to do that (I often did similar things in the past on other websites, mostly to collate sport results from different sources), and I'd volunteer to do that for you, but I can't because of smspower's https, which makes it impossible for me to use my usual tools.

Thanks very much for the offer to help, @Tom, I reckon I have a few options I can try.

To misquote Jeff Goldblum, "Will....uh....finds a way 🦖"
  View user's profile Send private message Visit poster's website
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 1:55 pm
BKK wrote
Yeah, most likely just an error. IIRC I only checked headers for games where a cover product code didn't exist, so I'm sure that's not the only one which doesn't match.

Ah...it's on the outside of the box!!

I simply hadn't connected that, of course – confirmed, I can see it right there, literally in black and white now.

Okay cool, so to my original question, as a responsible citizen, is there anything in particular I should be doing to have this info corrected?
  View user's profile Send private message Visit poster's website
  • Joined: 01 May 2011
  • Posts: 444
Reply with quote
Post Posted: Wed Aug 10, 2022 2:32 pm
Well, the code on the cover is correct, so unless the code from the header is being used instead somewhere then I'm not sure that any corrections are required.
  View user's profile Send private message
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 3:27 pm
Yeah okay, wasn't sure whether the documentation was supposed to be capturing the cover code or the header code. I'm using the header codes for cartridge identification, and I guess I assumed that Meka was doing something similar, but could be wrong.

I guess the discrepancy can just be an interesting factoid swimming in my head / the ether that is this forum!
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14226
  • Location: London
Reply with quote
Post Posted: Wed Aug 10, 2022 3:33 pm
Meka uses checksums because headers are unreliable. It also uses heuristics for unknown ROMs to guess at the emulation settings needed.
  View user's profile Send private message Visit poster's website
  • Joined: 25 Feb 2006
  • Posts: 761
  • Location: Belo Horizonte, MG, Brazil
Reply with quote
Post Posted: Wed Aug 10, 2022 3:36 pm
If everything else fails, you can always try to use some web-scraping tool to extract the information from the webpages.

And, in order to find out the size of the ROMs, you could download some ROM-packs from a torrent, then write some script to collect that information from the ROMs themselves; maybe a bit of overkill, yes, but if you can't find the information anywhere, it could be a fallback.
  View user's profile Send private message Visit poster's website
  • Joined: 24 Mar 2021
  • Posts: 82
Reply with quote
Post Posted: Wed Aug 10, 2022 5:39 pm
No-intro also keeps track of (and validates, when possible) a list of name - filename - size - crc32 - md5 - sha1 for all known games for all systems, SMS included.
  View user's profile Send private message
  • Joined: 25 Feb 2006
  • Posts: 761
  • Location: Belo Horizonte, MG, Brazil
Reply with quote
Post Posted: Wed Aug 10, 2022 5:44 pm
Think of this as an introduction to data science. ;)
  View user's profile Send private message Visit poster's website
  • Joined: 06 Mar 2022
  • Posts: 217
  • Location: London, UK
Reply with quote
Post Posted: Wed Aug 10, 2022 9:35 pm
lidnariq wrote
No-intro also keeps track of (and validates, when possible) a list of name - filename - size - crc32 - md5 - sha1 for all known games for all systems, SMS included.

@lidnariq thanks so much for this, that is a totally awesome project!
  View user's profile Send private message Visit poster's website
  • Joined: 25 Feb 2013
  • Posts: 353
  • Location: Osaka
Reply with quote
Post Posted: Thu Aug 11, 2022 7:22 am
If you are interested in meka's crc, I wrote a script that calculates it given a file (usage: python3 mekacrc.py filename).
I attach it. I used it to add the homebrews to meka.nam
(see this pull request, and as by pull request discussion I will probably separate them).

I am also trying to fuse the info on smspower and the ones in meka.
The idea is to match the crc in meka with the page's name in smspower's games section, and from that work out the tags that belong to that game.
I just started working on compiling gamesCrc.txt, but the other files should be complete (byGames.txt lists all the tags for a given game name, byTag.txt lists all the games that have a specific tag, description.txt gives a description of a tag). The idea is to use a script to produce a "beefed up meka.nam" with all the information in meka.nam plus all the relevant info for the game.

On a side note, meka.nam roms appear to be all available in the usual places, except for these few exceptions:

missing SMS ba5d2776 1EBAD8829E4FF6EB Super Game 200/COUNTRY=KR/EMU_MAPPER=20
missing SMS 1df0fc90 198269AF6F442C6E Circuit, The/COUNTRY=TW/PRODUCT_NO=G-1304
missing SMS 551e6f51 FA1DF196B3DCEDE6 Super Futebol II [Game Box Série Esportes]/COUNTRY=BR/PRODUCT_NO=028560/COMMENT=Part of "Game Box Série Esportes" compilation.
missing SMS 55f929ce 5CC5DB32E6CF130A Choplifter [HTH] [SMS-GG]/FLAGS=SMSGG_MODE/COUNTRY=TW/PRODUCT_NO=HG 101-15/COMMENT=Taiwanese HTH release for Game Gear.
missing SMS 0cde0938 2DFBC7EBB5C63576 Super Game World 30 Hap/COUNTRY=KR/EMU_MAPPER=18
missing SMS c29bb8cd F5394893312A6339 Super Game World 75 Hap/COUNTRY=KR/EMU_MAPPER=18
missing SMS eb7790de 7A908E975CEC7415 Super Multi Game - Super 125 in 1/COUNTRY=KR/EMU_MAPPER=18
missing SMS ba5ec0e3 0707EB856DC38BC7 128 Hap (KR)/COUNTRY=KR/EMU_MAPPER=19
missing SMS 96e16fe4 4239547D58D3D0B9 E-SWAT [v1] [HTH] [SMS-GG]/FLAGS=SMSGG_MODE/COUNTRY=TW/PRODUCT_NO=HG 201-12/COMMENT=Taiwanese HTH release for Game Gear.
missing SMS b14b6195 C1F11922544DA4CE Wimbledon [v0] [Game Box Série Esportes]/COUNTRY=BR/PRODUCT_NO=028560/COMMENT=Part of "Game Box Série Esportes" compilation.
missing SMS 3da7f1ce 37FE6B2A21FE0710 Astro Warrior & Pit Pot [Proto]/COUNTRY=DE/COMMENT=Early Red artwork German version. Only 2 header bytes differs from final release.
missing SMS 380d7400 DD6520742A9CBCA8 Game Mo-eumjip 188 Hap/COUNTRY=KR/EMU_MAPPER=19
missing SMS 40414556 3608A3EF977E8992 The Goonies/COUNTRY=TW/PRODUCT_NO=Y-105/EMU_MAPPER=15
missing SMS 660bf6ec D093565FC7E8C376 Super Multi Game - Super 75 in 1/COUNTRY=KR/EMU_MAPPER=18
mekacrc.zip (475 B)
tags.zip (39.67 KB)

  View user's profile Send private message
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8508
  • Location: Paris, France
Reply with quote
Post Posted: Thu Aug 11, 2022 7:56 am
Roms that have been sorted but not released yet are marked with a * in Meka’s compat.txt. This is what I use to know which I should release. Some of those were already released recently.
  View user's profile Send private message Visit poster's website
  • Joined: 25 Feb 2013
  • Posts: 353
  • Location: Osaka
Reply with quote
Post Posted: Fri Aug 12, 2022 4:35 am
Thanks!
I checked with compat.txt and by searching a bit more everything matches. The only difference between compat.txt's "*" and what is out in the wild is

SMS ba5ec0e3 0707EB856DC38BC7 128 Hap (KR)/COUNTRY=KR/EMU_MAPPER=19

which is present in meka.nam but not in compat.txt
  View user's profile Send private message
Reply to topic



Back to the top of this page

Back to SMS Power!