Forums

Sega Master System / Mark III / Game Gear
SG-1000 / SC-3000 / SF-7000 / OMV
Home - Forums - Games - Scans - Maps - Cheats - Credits
Music - Videos - Development - Hacks - Translations - Homebrew

View topic - To compress or not to compress? Does 49153 equal 4194304 in 2024?

Reply to topic
Author Message
  • Joined: 05 Dec 2019
  • Posts: 56
  • Location: USA
Reply with quote
To compress or not to compress? Does 49153 equal 4194304 in 2024?
Post Posted: Sun Mar 03, 2024 3:10 am
Maxim wrote in the Discord server:
"As people have often proposed, compression in homebrew is kind of unnecessary. You have no more CPU or RAM capacity than original games, but your ROM size limit is 4MB, not 1Mbit"

As someone who enjoys making things as big as they need to be and no bigger, I'm curious why the SEGA 8-bit scene seems to have embraced storing data without compression more than some other 8-bit platforms' homebrew scenes. It's almost as if people consider 49153 bytes (one byte larger than the largest possible ROM without a mapper) no lighter than 4194304 bytes (the largest possible ROM on a reimplemented standard mapper). So in the interest of comparing the principles underlying the culture here to those of cultures I'm more familiar with, I asked around.

Another 8-bit scene has customary breakpoints at 64 KiB and 512 KiB. The 64 KiB limit of one recurring game jam has two ostensible goals: to encourage a practical project scope for a 6-person-month project and to control memory cost of a multicart of jam entries. I attribute the 512 KiB limit to a bunch of things: price of 5 V parallel flash memory pre-2020, the ROM size limit of a couple models of flash cart made from 2008 to 2020, and the number of output address lines on common mapper ICs taken from donor boards. The IC shortage of the early 2020s shook up the programmable logic market as legacy 5 V CPLDs and FPGAs got discontinued in favor of 3.3 V instant-on FPGAs. This made it easier to swallow putting the entire cart behind a level shifter. In turn, this caused older flash cart models to end production and newer single-game cartridges to carry 3.3 V mappers with more I/Os. Habits in some platforms' scenes have been slow to adapt to the new memory size tradeoffs.

One staff member wondered why even make software for an 8-bit machine in the first place, as opposed to something with an 8-bit aesthetic for a modern platform, if one is just going to throw more ROM at the problem. On the other hand, if increasing ROM size doesn't increase cost of goods, he sees nothing wrong with using ROM space to reduce development complexity.

Are there noticeable breakpoints for SMS/GG ROM size other than just mapper or no mapper?
  View user's profile Send private message Visit poster's website
  • Joined: 01 Feb 2014
  • Posts: 878
Reply with quote
Post Posted: Sun Mar 03, 2024 7:08 am
I wasn't aware of any trend to omitting compression in Sega 8-bit development.

For me personally, resource management is part of the fun of developing for an older platform. Modern development tools notwithstanding, I always approach my games like they could have been made during the console's lifetime, which includes using as little ROM as possible.

We have plenty of compression algorithms available that suit every possible need, be it high compression or fast decoding. I see no good reason not to use them except for tiles that need to be streamed from ROM for animation.
  View user's profile Send private message
  • Joined: 06 Mar 2022
  • Posts: 671
  • Location: London, UK
Reply with quote
Post Posted: Sun Mar 03, 2024 7:37 am
Interesting subject!

PinoBatch wrote

One staff member wondered why even make software for an 8-bit machine in the first place, as opposed to something with an 8-bit aesthetic for a modern platform, if one is just going to throw more ROM at the problem.


I think that's pretty specious reasoning (and apparently contradicted in that person's next statement).
ROM size is just one of many constraints and clearly removing the ROM size constraint doesn't make programming for an 8 bit system just like programming for a modern system. Even infinite ROM size wouldn't make the processor on an SMS able to perform more than a few hundred thousand 8-bit integer arithmetic operations a second - a very real constraint. Similarly the capabilities of the VDP hugely limit what's possible in terms of display.

There have been threads on this forum before positing a gap between the games we've seen and what the system is "truly" capable of, yet I think it's fair to say that we've never really seen new homebrew games / indie, as wonderful as they are, smashing some kind of conceptual barrier and delivering features that would be unthinkable on the system as it was in the 80s.

PinoBatch wrote

Are there noticeable breakpoints for SMS/GG ROM size other than just mapper or no mapper?


Personally I think the major breakpoint for ROM size, as with so many things, would be around emulator support. As far as I know, Out of the box Emulicious supports the up to 6 bit mappers that are known in original Sega cartridges but I don't think (?) that it supports up to 8 bits (which would be trivial to extend the mapper to). I'm sure that it doesn't support any system which tries to map with more than mapped 8 bits.

I think many homebrew developers default to standard Sega mappers simply because Emulator and Everdrive support give them the widest audience. Many enthusiasts don't even have a console.

I also wonder, although this is speculative, whether the VDP constraints suggest an effective ceiling on ROM space specific to game genre. Even when uncompressed, there's a limited colour depth, a limited number of tiles that can be stored in VRAM, a limited bandwidth to transfer them from ROM to VDP, and a limited rate at which the CPU can perform calculations necessary to decide whether or not each tile / sprite should be displayed.
The reason I mention game genre is that it's easy to imagine something like an RPG with ROM contents that are proportional to the amount of content in the game - the longer the story, the more ROM you will need.

ROM size aside, I've been working on a very RAM-heavy game recently, and it has struck me that SMS games with standard mappers are significantly more RAM-constrained than ROM-constrained.
I do wonder if the ecosystem (again, determined now by emulators and flashcarts) was more geared up for RAM expansion how the homebrew game scene would be different.
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14745
  • Location: London
Reply with quote
Post Posted: Sun Mar 03, 2024 8:29 am
I thought the Sega 8-bit scene was more compression-focused than other similar universes (ignoring those with tape and floppy loading to optimise for), not least because I’ve spent so much time on messing with compression algorithms. People have argued that it’s a waste of my time. In the NES world of CHR ROM it can be hugely wasteful if you start paging it for animation, and you suffer performance costs if you instead have the host CPU feeding data to character RAM. In the ROM hacking world it seems common to do “ROM expansion” - doubling the ROM size - to find space rather than trying to squeeze things into the original space.

In practical terms, I think there is a 48KB ceiling, but PinoBatch’s question was carefully aimed one byte above that. On the C programming side there was similarly a code size limit of 32KB until recently, and I don’t know how much of a speed bump it still is. But otherwise, I don’t think there’s a compelling reason to target (say) 256KB instead of 512KB or 1MB. Maybe if aiming for some off-the-shelf programmable cartridge for distribution, you’d have to consider that - but I suspect it’s not a big constraint because as mentioned above, it’s hard to make 512KB of content. Taking Phantasy Star as an example, famously squeezed into 512KB and sold at a premium price, the “story” is pretty much held in about 32KB, the space is mostly taken up by art.

For multi-carts, ichigo has gigabit limits…

The only other constraint I’d mention is Everdrive where the original and cloned models have a 1MB limit.
  View user's profile Send private message Visit poster's website
  • Joined: 25 Jul 2007
  • Posts: 732
  • Location: Melbourne, Australia
Reply with quote
Post Posted: Sun Mar 03, 2024 12:22 pm
When I look at other homebrewers projects I tend to notice a trend of using compression by default (PS Gaiden seems to be a popular one).

I on the other hand see it as a form of premature optimisation since there is no inherent benefit unless you are targeting a set size constraint.

I will tend to take the approach of using uncompressed data, then later if the need arises I can start compressing things if I need that sort of optimization.

I have also noticed that those coming from the NES scene tend to have a culture of wanting to squeeze things as small as possible, one of the factors why they always cry in agony at the thought of using a streamed data format such as PSG.
  View user's profile Send private message
  • Joined: 05 Sep 2013
  • Posts: 3828
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Sun Mar 03, 2024 3:50 pm
I don't think we're ignoring compression.

I think it's just way less important if your ROM doesn't get onto any physical cartridge. It's not that much of a big deal if it's 256 KiB or 128 KiB, and you download a .zip file anyway.

On the other hand, it's very important if you have a physical target. No-mapper cartridges are 32 KiB (typical) to 48 KiB at most, and even 'modern' Flash based cartridges could be 512 KiB 'only' so I would never suggest to skip compression altogether.
  View user's profile Send private message Visit poster's website
  • Joined: 05 Sep 2013
  • Posts: 3828
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Sun Mar 03, 2024 3:57 pm
willbritton wrote
As far as I know, Out of the box Emulicious supports the up to 6 bit mappers that are known in original Sega cartridges but I don't think (?) that it supports up to 8 bits (which would be trivial to extend the mapper to).


It already supports 8 bit registers SEGA mapper, check the two - 4-MiB each - Bad Apple entries from a few years ago.
  View user's profile Send private message Visit poster's website
  • Joined: 28 Jan 2017
  • Posts: 556
  • Location: Málaga, Spain
Reply with quote
Post Posted: Sun Mar 03, 2024 4:10 pm
Ummm...

Man, Everything is compressed. The music you play, the images, the videos you consume, the .iso of your windows / linux / osx, the .docx or xlsx files, even the network packages you send along the net....

Everything!!!

so.... why not to compress???? I found (in my case) that the zx7 compression I use massively is very good and very fast to decompress, and does not impose limits of any kind (except for ram on target), so... why not to compress???

I would think the oposite way, dude.
  View user's profile Send private message
  • Joined: 16 May 2002
  • Posts: 1356
  • Location: italy
Reply with quote
Post Posted: Sun Mar 03, 2024 4:58 pm
As much as we'd like to optimise everything, there's been lots of examples of questionable (or just outright lazy) choices throughout the years. It's no mystery that I like module files and the XM format in particular, and yet, this is one of my "favourites":
                        Patterns:
                        ---------

      ?      4  (dword) Pattern header length
     +4      1   (byte) Packing type (always 0)
     +5      2   (word) Number of rows in pattern (1..256)
     +7      2   (word) Packed patterndata size
Sure, let's use 32 bits to store a value which is always going to be 9. Even if they wanted to be forward-compatible and account for a bigger pattern header in case of future extensions to the format, why in the world would anyone want to have a 4-GB header? For each pattern in the song, no less. The most jarring part is that the size of the pattern itself is limited to a 16-bit value, so you can effectively have a 4-GB header for 64 kilobytes of data.

And keep in mind that this was in 1994, when most people didn't even know what a gigabyte was, let alone 4. Sure FAT32 would arrive a couple of years later, but...

Samples and instruments share this design choice, by the way. It's like they assumed that people would use multiple 4-GB samples within a bank of 4-GB instruments and compose songs with multiple 4-GB patterns. In 1994.
  View user's profile Send private message Visit poster's website
  • Joined: 05 Dec 2019
  • Posts: 56
  • Location: USA
Reply with quote
Post Posted: Sun Mar 10, 2024 2:24 am
Kagesan wrote
I wasn't aware of any trend to omitting compression in Sega 8-bit development.

That depends on what someone does and doesn't consider compression.

  • Expanding from 2bpp or 3bpp to 4bpp while filling VRAM: is that compression?
  • Storing only right-facing sprite cels and then flipping the tile data in software using a bit-reverse table while streaming it to VRAM: is that compression?
  • Building the world out of 16×16-pixel or 32×32-pixel metatiles: is that compression?
  • Using a sequenced sound driver instead of a VGM-style timed log of DCSG writes: is that compression?


willbritton wrote
I'm sure that [Emulicious] doesn't support any system which tries to map with more than mapped 8 bits.

There's that one Japan-only train simulator Densha de Go! 2 for another company's 8-bit handheld system. It's the only licensed game for that platform that uses 9 bits (The 64 Mega Cartridge), and I don't know whether Emulicious attempts to emulate it.

willbritton wrote
The reason I mention game genre is that it's easy to imagine something like an RPG with ROM contents that are proportional to the amount of content in the game - the longer the story, the more ROM you will need.

Rhythm games are another example of a genre with ROM size proportional to content, especially if a heavy format like PSGlib is used.

Tom wrote
Samples and instruments share this design choice, by the way. It's like they assumed that people would use multiple 4-GB samples within a bank of 4-GB instruments and compose songs with multiple 4-GB patterns. In 1994.

That's a different problem. Few mainstream instruction sets provide 24-bit integers. They provide 16-bit and 32-bit integers, meaning 65536 == 4294967295. At the time, few format designers thought to use a base-128 variable-length quantity for data sizes, as seen in standard MIDI files, WAP, ASN.1 BER, DWARF, Git, and various serialization formats used by Google and Microsoft.
  View user's profile Send private message Visit poster's website
  • Joined: 16 May 2002
  • Posts: 1356
  • Location: italy
Reply with quote
Post Posted: Sun Mar 10, 2024 6:29 am
I'm aware of that, but still, that's no reason to encode the header length (a value which is always going to be 9) to a 32-bit value, while the length of the actual pattern is limited to a 16-bit value, it's like a book having an index which is millions of times bigger than the actual content, that was my point. The header length could have been a byte/char if they really didn't want it to be a constant for future flexibility (and they did use 8-bit values (and even bitfields) in some other places, so they were aware of that possiblity). But this is unrelated to the topic at hand, so I'll just stop here.
  View user's profile Send private message Visit poster's website
  • Joined: 01 Feb 2014
  • Posts: 878
Reply with quote
Post Posted: Sun Mar 10, 2024 9:10 am
PinoBatch wrote
That depends on what someone does and doesn't consider compression.

Fair questions. Personally, I'd define compression as something that needs some sort of decoder to get the compressed data back into its original shape. Leaving out unused bitplanes is no compression at all, just an efficient way of storing data, the bit-reverse table is an edge case (I'd still say no), and using metatiles is clearly a way of compressing tilemaps. (I know little about music formats, so I can't comment on that.)

I don't really see the point in defining compression so broadly that any way of saving ROM space is included. Where are you going with this? I'm afraid I don't really undersatnd.
  View user's profile Send private message
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14745
  • Location: London
Reply with quote
Post Posted: Sun Mar 10, 2024 6:20 pm
On the side topic of using 32-bit numbers where fewer would do, it’s clearly overkill except when it isn’t (e.g. AVI 2GB limits), and the cost of relatively few bytes leaves a lot of things open for the future, even if many options are ridiculous or impossible.

Anyway, I’d say that compression is interesting and fun and I feel like part of the appeal of making games for a retro system is the limits - but ROM space is one where you can push a little further on the space vs CPU side. You can store all sorts of data uncompressed to make your game smoother, and that’s your choice; but I also like to make things that are not bigger than they need to be. A few frames to decompress artwork is not a big deal.
  View user's profile Send private message Visit poster's website
  • Joined: 14 Oct 2008
  • Posts: 513
Reply with quote
Post Posted: Mon Mar 11, 2024 4:55 pm
Maxim wrote
A few frames to decompress artwork is not a big deal.

That's true, that's only a big deal if your game is something like Dragon's Lair NES (the USA version specifically) is basically an unplayable game because it is undoubtedly trying to constantly decompress and load animated sprites into VRAM (contrast to the Japan and Europe versions which used ROM to make a much more functional game, if you need a technical comparison).
  View user's profile Send private message
Reply to topic



Back to the top of this page

Back to SMS Power!