Forums

Hi everyone,

blast'em, a known cycle accurate Mega Drive, SMS, GG and Colecovision emulator has now added the option to log VGMs for the SMS and GG platforms.

blast'em has been able to log Mega Drive VGMs for quite some time and only recently extended this function to SMS and GG.

The reason for this is that I've received many complaints about emulators being inaccurate or having issues with some games and another very glaring issue: most emulators such as KEGA and MEKA work at 60Hz instead of the correct 59.922743Hz which lead to a slight speed change and, as a consequence, pitch shift.
Unfortunately, being a very new feature added, only PSG is supported at the moment, with YM2413 support still being worked on.

While the difference between a "properly logged" VGM and a 60Hz one might not be detectable, a higher degree of accuracy is always desirable and the accuracy of blast'em logger has already been tested through MDFourier, an audio analysis tool aimed at measuring differences among console revisions (and emulators!).

I'd like to know if there is interest in adopting blast'em (which also has a very powerful debugger which, in our case, allows to write into RAM to trigger the desired track) and, generally speaking, in solutions to improve VGM logging accuracy as this might spur the developer in speeding up the development of the YM2413 support.

EDIT: unfortunately I can't post links, so you'll have to Google your way around to find information on the emulator and MDFourier.

If you want to give it a try, make sure you use the latest nightly of blast'em!

Blastem: https://www.retrodev.com/blastem/

MDFourier: https://junkerhq.net/MDFourier/

I haven’t played with it yet but I can tell you I use a debugger heavily when ripping VGMs because you often need to poke values into registers and sometimes even write custom code to explore the soundtrack data. (RAM triggers happen to be very common on SMS because so many games used the Sega “pre-SMPS” music engine.) One thing that I found very useful in Emulicious was the in-place assembler in the debugger, so I could write ad-hoc “halt and loop” code to hang the “main thread” while music plays, but as it has no VGM logging, I’d have to then port that as a binary patch in Meka.

I just checked and confirmed that the VGM spec still doesn’t allow for timing at any granularity other than 44100Hz. I think this is a necessary change to give perfect timing for VGMs - maybe you should pursue this with the relevant stakeholders before going too much further?

It is not, honestly.
44100 is just an arbitrary number, a sampling rate which (IMHO) has been a truly unfortunate choice, but it is here to stay.
And that's alright, because even considering the tiny quantization error due to not using perfect 60Hz, those are well within the margin of error of real hardware itself.

I've written a very lengthy post on 16bap which I can't link which is called "The importance of defining tolerances" which is basically how we drew the line at how much accuracy we really need.
You can find it by Googling "16bap tolerances", it should be the first result.

It gets a bit techy and with lots of numbers floating around, but the gist is that, at least in Mega Drive world, the console itself has a drift of ~0.5ms every minute which grows to ~2.67ms every minute if we factor different consoles (components age and tolerances, different ambient temperature, etc.)

Logging VGMs at ~59.92Hz will bring a tiny quantization error but it will result in a much smaller drift than the console itself naturally has, so we're within spec and ultimately not an issue, especially considering that we'll have the correct speed and pitch.

BTW, on the blast'em debugger, for example, playing a certain song is as easy as writing

set [$DE03] $8D

for Fantasy Zone Gear.
I haven't used it extensively so I can't comment on how flexible or powerful it truly is.

Most debugging emulators also have a hex editor view of memory where you can type in values. My point is that while that suffices for many games, you also need to find a point in the game where (1) the music engine is running and polling that trigger, (2) it’s not going to time out and change the music (eg a title screen timeout) and (3) it’s not going to do some work that delays the music engine, causing jitter in that nearly 60Hz interval - sometime delaying it by several frames. For SMS you can often use the “pause hack” to inject a game hang via NMI which will effectively hang the “main thread” and leave the “interrupt thread” running. On Mega Drive you may similarly find that you can hang the 68000 and leave the Z80 running music - but there will be many games where that doesn’t work, the 68000 needs to run. Finally, lots of games trigger music via function calls and you need to find the function and control how it’s called, typically (on SMS) by breaking in the debugger at a certain point and using the debugger to change a register before continuing. Being able to save and load a savestate at an exact point in the debugger is also useful here.

Maxim wrote

(3) it’s not going to do some work that delays the music engine, causing jitter in that nearly 60Hz interval - sometime delaying it by several frames.

As previously said, we're not going for "absolute perfection" (or whatever you want to call it), but we want to go as close as hardware itself is, giving it its best shot.

For example, if the least "congested" section of a game is its title screen which has some animations which introduce little delays, so be it.
That's as good as the game itself will ever play the music.

As for the debugger in blast'em, I'm afraid that I don't have much experience with it but the developer is incredibly open minded and welcomes all suggestions on making the emulator (and the debugger) better, as long as there's a good reason behind them.

If there is interest, we can make some noise and we'll surely be heard. The more the merrier, of course.

Otherwise if people are ok with MEKA and are not interested in trying to adopt a new, more accurate emulator and contribute with suggestion (and criticisms, of course), then that's that, it can't be helped.

More choice is always a good thing. VGM ripping has options for “sample accurate” and “frame accurate”, and on some sound chips it makes a very important difference; I thought on YM2612 it makes even more difference, and emulators have moved to “cycle accurate” sound chips running at their internal rate and resampling for playback exactly to capture these effects. For the SMS PSG, I can’t think of any cases where sample accuracy is needed apart from samples and those high-rate VGMs people make in trackers, so frame accurate is OK and you could even adjust those from 60Hz to 59.94 or whatever to capture the frame timing. The tone accuracy is unaffected.

donluca, what do you hypothesize is the source of the variable latency?

SavagePencil wrote

donluca, what do you hypothesize is the source of the variable latency?

It's not an hypothesis and I believe Maxim mentioned this in a previous post, but the game just displaying animations and doing other game-related calculations have a chance to "delay" the sound in ways that are unfortunately very hard to predict.

This is just from strictly the code side of things, but with MDFourier, on Mega Drives at least, we've measured fluctuation in timing even due to different ambient temperature.

That's to say there are a ton of variables in play, some of which are unfortunately completely out of our control, hence the necessity for defining tolerances and saying when something is "accurate enough".

(unrelated, but your name rings a bell... were you part of 16bap's forums or did I see you on sega-16 or some other places?)

I guess my question is why they vary from hardware revision to hardware revision. I can’t imagine CPU cycle availability being so variable. It’s not like there’s an OS “stealing” cycles…although that may not be true for flash carts, which may be snooping looking for special inputs.

I exist everywhere.

SavagePencil wrote

I guess my question is why they vary from hardware revision to hardware revision. I can’t imagine CPU cycle availability being so variable. It’s not like there’s an OS “stealing” cycles…although that may not be true for flash carts, which may be snooping looking for special inputs.

I exist everywhere.

Oh ok.

Mind that this is only for the Mega Drive AFAIK, I haven't done any testing on other consoles.

Mega Drives underwent several revisions where SEGA did very minor adjustments to the circuits so they all behave somewhat differently, although this is impossible to catch without proper analysis software such as MDFourier.

In addition to that, those revisions have all been produced at a different time and the consoles themselves have probably been used a different amount of time each, so the wear on the components is different, not to mention that the recordings Artemio did, IIRC, weren't done in a temperature controlled environment.
So if one console has been tested during summer, it could very well have slightly different timing if it had been tested during winter.

I also have to add that this about the "official" Mega Drives, because there are a billion clones and all-in-one chips that have been produced afterwards such as the Mega Drive 3 which behave differently in a more substantial way, going as far as being incompatible with certain games.

donluca wrote

44100 is just an arbitrary number, a sampling rate which (IMHO) has been a truly unfortunate choice, but it is here to stay.

a future revision of the VGM specs could introduce different sampling rates, it all boils down to store it in the header...

sverx wrote

a future revision of the VGM specs could introduce different sampling rates, it all boils down to store it in the header...

There has been endless talk in the last 10 years about this (and many other options, both as a "VGM 2.0" spec and a complete rewrite) and nothing has ever been done and I think it's not going to change anytime soon. So we might as well do the best we can with what we have.

Author	Message
donluca Joined: 13 Nov 2024 Posts: 6	Blast'em, a new accurate VGM Logger available! Posted: Fri Nov 15, 2024 5:43 pm
donluca Joined: 13 Nov 2024 Posts: 6	Hi everyone, blast'em, a known cycle accurate Mega Drive, SMS, GG and Colecovision emulator has now added the option to log VGMs for the SMS and GG platforms. blast'em has been able to log Mega Drive VGMs for quite some time and only recently extended this function to SMS and GG. The reason for this is that I've received many complaints about emulators being inaccurate or having issues with some games and another very glaring issue: most emulators such as KEGA and MEKA work at 60Hz instead of the correct 59.922743Hz which lead to a slight speed change and, as a consequence, pitch shift. Unfortunately, being a very new feature added, only PSG is supported at the moment, with YM2413 support still being worked on. While the difference between a "properly logged" VGM and a 60Hz one might not be detectable, a higher degree of accuracy is always desirable and the accuracy of blast'em logger has already been tested through MDFourier, an audio analysis tool aimed at measuring differences among console revisions (and emulators!). I'd like to know if there is interest in adopting blast'em (which also has a very powerful debugger which, in our case, allows to write into RAM to trigger the desired track) and, generally speaking, in solutions to improve VGM logging accuracy as this might spur the developer in speeding up the development of the YM2413 support. EDIT: unfortunately I can't post links, so you'll have to Google your way around to find information on the emulator and MDFourier. If you want to give it a try, make sure you use the latest nightly of blast'em!

Maxim Site Admin Joined: 19 Oct 1999 Posts: 15009 Location: London	Posted: Fri Nov 15, 2024 10:33 pm
	Blastem: https://www.retrodev.com/blastem/ MDFourier: https://junkerhq.net/MDFourier/ I haven’t played with it yet but I can tell you I use a debugger heavily when ripping VGMs because you often need to poke values into registers and sometimes even write custom code to explore the soundtrack data. (RAM triggers happen to be very common on SMS because so many games used the Sega “pre-SMPS” music engine.) One thing that I found very useful in Emulicious was the in-place assembler in the debugger, so I could write ad-hoc “halt and loop” code to hang the “main thread” while music plays, but as it has no VGM logging, I’d have to then port that as a binary patch in Meka. I just checked and confirmed that the VGM spec still doesn’t allow for timing at any granularity other than 44100Hz. I think this is a necessary change to give perfect timing for VGMs - maybe you should pursue this with the relevant stakeholders before going too much further?

donluca Joined: 13 Nov 2024 Posts: 6	Posted: Fri Nov 15, 2024 10:43 pm
donluca Joined: 13 Nov 2024 Posts: 6	It is not, honestly. 44100 is just an arbitrary number, a sampling rate which (IMHO) has been a truly unfortunate choice, but it is here to stay. And that's alright, because even considering the tiny quantization error due to not using perfect 60Hz, those are well within the margin of error of real hardware itself. I've written a very lengthy post on 16bap which I can't link which is called "The importance of defining tolerances" which is basically how we drew the line at how much accuracy we really need. You can find it by Googling "16bap tolerances", it should be the first result. It gets a bit techy and with lots of numbers floating around, but the gist is that, at least in Mega Drive world, the console itself has a drift of ~0.5ms every minute which grows to ~2.67ms every minute if we factor different consoles (components age and tolerances, different ambient temperature, etc.) Logging VGMs at ~59.92Hz will bring a tiny quantization error but it will result in a much smaller drift than the console itself naturally has, so we're within spec and ultimately not an issue, especially considering that we'll have the correct speed and pitch. BTW, on the blast'em debugger, for example, playing a certain song is as easy as writing set [$DE03] $8D for Fantasy Zone Gear. I haven't used it extensively so I can't comment on how flexible or powerful it truly is.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 15009 Location: London	Posted: Sat Nov 16, 2024 9:13 am
	Most debugging emulators also have a hex editor view of memory where you can type in values. My point is that while that suffices for many games, you also need to find a point in the game where (1) the music engine is running and polling that trigger, (2) it’s not going to time out and change the music (eg a title screen timeout) and (3) it’s not going to do some work that delays the music engine, causing jitter in that nearly 60Hz interval - sometime delaying it by several frames. For SMS you can often use the “pause hack” to inject a game hang via NMI which will effectively hang the “main thread” and leave the “interrupt thread” running. On Mega Drive you may similarly find that you can hang the 68000 and leave the Z80 running music - but there will be many games where that doesn’t work, the 68000 needs to run. Finally, lots of games trigger music via function calls and you need to find the function and control how it’s called, typically (on SMS) by breaking in the debugger at a certain point and using the debugger to change a register before continuing. Being able to save and load a savestate at an exact point in the debugger is also useful here.

donluca Joined: 13 Nov 2024 Posts: 6	Posted: Sat Nov 16, 2024 11:51 am
donluca Joined: 13 Nov 2024 Posts: 6	Maxim wrote (3) it’s not going to do some work that delays the music engine, causing jitter in that nearly 60Hz interval - sometime delaying it by several frames. As previously said, we're not going for "absolute perfection" (or whatever you want to call it), but we want to go as close as hardware itself is, giving it its best shot. For example, if the least "congested" section of a game is its title screen which has some animations which introduce little delays, so be it. That's as good as the game itself will ever play the music. As for the debugger in blast'em, I'm afraid that I don't have much experience with it but the developer is incredibly open minded and welcomes all suggestions on making the emulator (and the debugger) better, as long as there's a good reason behind them. If there is interest, we can make some noise and we'll surely be heard. The more the merrier, of course. Otherwise if people are ok with MEKA and are not interested in trying to adopt a new, more accurate emulator and contribute with suggestion (and criticisms, of course), then that's that, it can't be helped.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 15009 Location: London	Posted: Sat Nov 16, 2024 1:02 pm
	More choice is always a good thing. VGM ripping has options for “sample accurate” and “frame accurate”, and on some sound chips it makes a very important difference; I thought on YM2612 it makes even more difference, and emulators have moved to “cycle accurate” sound chips running at their internal rate and resampling for playback exactly to capture these effects. For the SMS PSG, I can’t think of any cases where sample accuracy is needed apart from samples and those high-rate VGMs people make in trackers, so frame accurate is OK and you could even adjust those from 60Hz to 59.94 or whatever to capture the frame timing. The tone accuracy is unaffected.

SavagePencil Joined: 23 Aug 2009 Posts: 228 Location: Seattle, WA	Posted: Sat Nov 16, 2024 10:39 pm
	donluca, what do you hypothesize is the source of the variable latency?

donluca Joined: 13 Nov 2024 Posts: 6	Posted: Sat Nov 16, 2024 11:46 pm
donluca Joined: 13 Nov 2024 Posts: 6	SavagePencil wrote donluca, what do you hypothesize is the source of the variable latency? It's not an hypothesis and I believe Maxim mentioned this in a previous post, but the game just displaying animations and doing other game-related calculations have a chance to "delay" the sound in ways that are unfortunately very hard to predict. This is just from strictly the code side of things, but with MDFourier, on Mega Drives at least, we've measured fluctuation in timing even due to different ambient temperature. That's to say there are a ton of variables in play, some of which are unfortunately completely out of our control, hence the necessity for defining tolerances and saying when something is "accurate enough". (unrelated, but your name rings a bell... were you part of 16bap's forums or did I see you on sega-16 or some other places?)

SavagePencil Joined: 23 Aug 2009 Posts: 228 Location: Seattle, WA	Posted: Sun Nov 17, 2024 4:45 am
	I guess my question is why they vary from hardware revision to hardware revision. I can’t imagine CPU cycle availability being so variable. It’s not like there’s an OS “stealing” cycles…although that may not be true for flash carts, which may be snooping looking for special inputs. I exist everywhere.

donluca Joined: 13 Nov 2024 Posts: 6	Posted: Sun Nov 17, 2024 11:40 am
donluca Joined: 13 Nov 2024 Posts: 6	SavagePencil wrote I guess my question is why they vary from hardware revision to hardware revision. I can’t imagine CPU cycle availability being so variable. It’s not like there’s an OS “stealing” cycles…although that may not be true for flash carts, which may be snooping looking for special inputs. I exist everywhere. Oh ok. Mind that this is only for the Mega Drive AFAIK, I haven't done any testing on other consoles. Mega Drives underwent several revisions where SEGA did very minor adjustments to the circuits so they all behave somewhat differently, although this is impossible to catch without proper analysis software such as MDFourier. In addition to that, those revisions have all been produced at a different time and the consoles themselves have probably been used a different amount of time each, so the wear on the components is different, not to mention that the recordings Artemio did, IIRC, weren't done in a temperature controlled environment. So if one console has been tested during summer, it could very well have slightly different timing if it had been tested during winter. I also have to add that this about the "official" Mega Drives, because there are a billion clones and all-in-one chips that have been produced afterwards such as the Mega Drive 3 which behave differently in a more substantial way, going as far as being incompatible with certain games.

sverx Joined: 05 Sep 2013 Posts: 4002 Location: Stockholm, Sweden	Posted: Mon Nov 25, 2024 10:04 am
	donluca wrote 44100 is just an arbitrary number, a sampling rate which (IMHO) has been a truly unfortunate choice, but it is here to stay. a future revision of the VGM specs could introduce different sampling rates, it all boils down to store it in the header...

donluca Joined: 13 Nov 2024 Posts: 6	Posted: Mon Nov 25, 2024 11:50 am
donluca Joined: 13 Nov 2024 Posts: 6	sverx wrote a future revision of the VGM specs could introduce different sampling rates, it all boils down to store it in the header... There has been endless talk in the last 10 years about this (and many other options, both as a "VGM 2.0" spec and a complete rewrite) and nothing has ever been done and I think it's not going to change anytime soon. So we might as well do the best we can with what we have.

Forums

View topic - Blast'em, a new accurate VGM Logger available!