Sega Master System / Mark III / Game Gear|
SG-1000 / SC-3000 / SF-7000 / OMV
Home - Forums - Games - Scans - Maps - Cheats - Credits
Music - Videos - Development - Hacks - Translations - Homebrew
|Goto page 1, 2 Next|
SC-3000 Tape Software Restoration ProjectPosted: Thu Apr 16, 2015 4:35 am
As Tycho has been urging me to convert the Softgold games to run on the SC-3000 multicart, I finally kicked off something I've been meaning to do for the past couple of years. The SC-3000 Tape Software Restoration Project.
The idea is to make clean digital copies of the tape audio available as 16-bit 44.1KHz mono WAV files. They work on original SC-3000 hardware and emulators like MESS, and they are highly compressible - in the order of 50:1 to 100:1. eg. the Vortex Blaster audio compresses down from approximately 14.5MB to around 172KB if you use RAR.
So this solves both the problem of degraded source audio, and the space problem as 60 seconds of normal audio is around 5MB in size and is generally not compressible by any significant margin.
I've always had a long term goal to clean up and start releasing the tape software. That's actually how the SC-3000 multicart development started - as a sideways excursion whilst looking at how to convert tape software to disk images. And after building the multicart I know most of the custom tape load routines pretty well, so I can use a few different techniques to remaster them.
I also wanted a place to record some of the neat things about the loader routines and copy protection used by the old tape software for future reference.
Anyway, here are the first two releases:
The Secret of Bastow Manor
(text / graphic adventure by Softgold)
(vertical scrolling shooter with digitized speech by Trident Technological Systems)
Note - I'm going to try to limit myself to a couple of releases per month as some of the tapes require a bit of thought (eg. the Mike Boyd ones), and all of them take time to write up.
||Posted: Thu Apr 16, 2015 4:49 am|
Really thank you!
I want to ask you some (probably trivial) things.
Are all tapes sega basic code? Or are there games that are a binary which is just loaded through basic and then executed?
||Posted: Thu Apr 16, 2015 6:03 am|
Some tapes like Bastow Manor are almost totally or totally written in basic. A text / graphic adventure does not need the performance of machine code and the Sega basic language was actually very advanced and contained a lot of useful things like drawing routines and a character set that is biased towards drawing charts, shapes, and patterns. Bastow Manor only uses embedded machine code for a bit of copy protection and to trigger the ROM load routines.
Other games like Gold Miner have a significant basic component for the convenience of drawing the screens and displaying instructions and setting up sprites but uses machine code for all the animation.
Then you have 100% machine code games like Vortex Blaster and Michael Boyd's games like Burglar Bill and Sir Roderick's Quest in which the only basic command is a CALL statement to jump to the machine code. All of the screen drawing, animation etc is handled in machine code in those games. (I haven't looked at Vortex blaster recently but I'm pretty sure the basic content is minimal)
Vortex Blaster is a single load tape ie all the machine code is hidden after the initial basic program. Michael Boyd's games use multi part loaders where the screen code and vram data is loaded direct from tape as a binary block.
||Posted: Thu Apr 16, 2015 7:12 am|
|If you understand the tape encoding then can you determine if TZX and similar formats can archive the data losslessly?|
||Posted: Thu Apr 16, 2015 7:47 am|
I am very, very happy you are doing it.
- A few bits: would you be able to provide higher-resolution scans? The 1024 wide scans are a bit tight in this day and age. The minimum I would say would be twice that, and larger would be better for preservation.
- Along with each "processed" release can you make sure to preserve a raw unprocessed audio dump of the original tape? We never know and we they come in handy in the far future.
- Would you be able to record the original audio length of each cassettes/sides if applicable? For reference.
(PS: I have increased your quota to 500 mb here and shall it ever be a problem we'll increase it again.)
||Posted: Thu Apr 16, 2015 10:07 am|
:) I can see more of my life slipping away on a hobby project.
I have had a quick look through the TZX specifications. I think the answer is probably yes, although I'm a bit busy to do a proof of concept, sorry.
However, I wonder whether the TZX style formats are necessary with this approach. Currently, no SC-3000 emulators support the TZX format, so far as I know. But there is good emulator support for actual WAV files. And using this technique allows us to shrink the WAV files down to the point where they are 'small enough'.
Having said that, if you did want to use a tape format like TZX, then the SC-3000 tapes all have a relatively simple representation. All of the ones I stepped through when building the multicart (about 70 or so) use the Basic IIIB ROM load / save routines, or at least the atoms within the routines.
For instance, even the Michael Boyd games use the SyncBits routines (that outputs 3600 ones) at the start of data blocks, then the WriteByteToCassetteOut routines from Basic IIIB ROM. So the waveforms themselves are always predictable. ie. you have sync bit blocks of 3600 one bits (2400 Hz cycle) followed by a zero (1200 Hz cycle) followed by 8 data bits and two stop bits for each data byte.
ie. a given tape may or may not follow the Basic program structure of Leader Field, Key Code, FileName, Program Length, Parity, Dummy Data, 1 second silence, Leader Field, Key Code, Program Data, Parity, Dummy Data. But the important atoms of the Leader Field (ie. the sync bits) and the format of each individual data byte are observed by all the tapes I've looked at.
As a side note, Michael Hadrup did publish LSV (Load / Save / Verify) extensions in the Sega computer magazines which provided more reliable routines (I think he said they were based on the Spectrum routines from memory). But no commercial software used those.
||Posted: Thu Apr 16, 2015 10:21 am|
Thanks Bock. I'm happy to help. I'll just try to put a limit on how much time I spend on this each month. The original multicart build took over my life for a few months :)
I've been working off Aaron's excellent collection of material. I'll just check if he minds if I use his scans.
My thinking with those zip files was to make them small and self contained. So they have high quality, but highly compressible audio, a text file with information about the title, and a smallish jpg of the cassette cover. I figured that gives them the best chance of being widely distributed and 'living' forever :) But point taken on the larger scans. I'll see what I can do.
I certainly won't get rid of the original audio recordings. But I'm not sure if there is much point making those publicly available given they are much larger (and can't be compressed) and often require processing to work with emulators / real hardware.
I'll try to remember to jot down the audio length in future.
||Posted: Thu Apr 16, 2015 10:32 am|
I would like us to extend our database and host the high quality scans here, if you or Aaron or who has been scanning them is happy with that. So I understand it is preferable to keep a smaller .JPG in the package. If you want to attach or e-mail us the bigger ones we'll post them on the site.
I realize it's not a very exciting page as is but we can build from there similarly or how we built on for the cartridge games. One key is to get people to use the emulator (MESS right now, and really another emulator should include support) to motivate the creation of screenshots, maps, etc.
300-600 KB is an acceptable size. It's important that the package are versioned (as you did so) also to convey the fact that they are manual work and may not be perfectly representative of the real data, not in a bad way mind you. I also wish we could have a catch-all binary format but that's a lot of work and your solution is reasonable. Do you mind if we start including your packages in our pages? (ZIP contents will be unaltered).
||Posted: Thu Apr 16, 2015 12:47 pm|
Original noisy audio should compress better with dedicated algorithms (e.g. FLAC), although still not as well as the generated data.
Can the clean audio be reduced in bit depth and sampling rate without affecting the results? That will save a lot of space. It ought to work at 4.8kHz with as low a bit depth as you can get.
||Posted: Thu Apr 16, 2015 7:32 pm|
The short answer is I think releasing 44.1KHz, 16 bit audio that compresses well is still the preferable solution. The lower bit depths and rates compress a bit more, but from memory not significantly more than when the source has a perfect waveform. ie. you're talking about the difference between compressing to say 200Kb vs 125Kb or something like that if I recall correctly. Also MESS and other waveform sampling / conversion tools seem to work better on higher bit rate inputs. So if the compression on the 44KHz / 16 bit is acceptable I'd rather stick with that.
Now the longer answer :)
Interesting - I just ran some FLAC tests. That gives slightly better compression than RAR on the original audio, but still only in the order of 10-15% max. And interestingly the remastered audio only compresses around 10-15% with FLAC compared to 97% + with RAR or Zip.
The second question is more interesting.
Yes, technically you can use both a lower bit rate and lower bit depth. Theoretically you can use 8-bit and 4800Hz. When I first started looking at recording files about 15 years ago I remembered that bit of encoding theory too - something like you only need to sample at twice the maximum frequency you want to reproduce (and the tape encoding works on 1200Hz / 2400Hz tones). In fact, doing that has the benefit of effectively running a low pass filter over the sampled audio by removing higher frequencies. I think in my first attempts 15 years back I settled on 8 bit and 8KHz or something like that. Unfortunately I lost all those recordings in a hard drive crash back in about 2003-4 :)
However... after playing around with this stuff for a couple of years, I prefer working with the CD Audio bit rate, or maybe 22050Hz as a minimum. Waveform sampling / conversion tools (like the MESS tape emulation and other third party tools) tend to work better when they have more data points to work with. I found MESS was a lot less predictable on audio loading when the bit rate dropped below 22050Hz and the signal was degraded somewhat. And as soon as you edit the waveforms in a sound editor (like when splicing multiple blocks together to make a full tape image), the waveforms all get shunted sideways a little bit. At higher bitrates you get more sample points in the peaks of the waveforms. If you think about it, at 44.1KHz you get umm.... each peak at 2400Hz is 1 / 4800th of a second wide. So 44100 / 4800 = 9.2 samples per half waveform. If you drop that to 22050Hz, that is about 4.6 samples per half waveform. At 4800Hz sample rate, that is only one sample per half waveform. So although it may work, you get more vulnerable to slight shifts in the waveform during editing as you drop the sample rate.
Note - the 8 bit vs 16 bit probably won't make much difference to the 'accuracy' of the recording as the exact amplitude is less important and 256 levels is probably fine. But MESS standardizes on 16 bit when writing so again I've stuck with that, especially if I'm using MESS to generate the output.
I have a bit more flexibility when using SegaWavWriter as that will output different bit rates. I just need KerrJnr to make a couple more mods to that for me :) I'll write those different techniques up when I use them for reference on the restoration project page.
||Posted: Thu Apr 16, 2015 9:06 pm|
|I experimented and found about the same things. Ultimately there is very little unique data in there and while you could reduce it to a stream of 1s and 0s for each encoded bit, it's somehow more "real" to have all the square waves.|
||Posted: Thu Apr 16, 2015 9:18 pm|
Just checking if Aaron is happy for me to pass on his scans. I don't have all the original covers, and I'm not sure I can face dragging my stuff out to rescan :)
Great. Pretty much everything should end up in that size range and I can add a comment about being a manual work. I may add an MD5 Hash on the WAV files or similar.
Yes, fine to include the zip packages in your pages. I want them widely distributed as that gives them the best chance of staying around long term.
Thanks for setting up the tape pages. I think you gave me edit access a while back, but unfortunately I didn't have the time to figure out how to create a new section / topic and edit it :)
Ok - I'd better get some work done and try to stick to my "couple of releases a month" target :)
||Posted: Fri Apr 17, 2015 2:51 am|
do you know the voltage output by the original SR-1000?
I would like to connect an mp3 player to the sc-3000, but I am worried in damaging it, so I would like to set the volume such as the voltage is the same
||Posted: Fri Apr 17, 2015 3:00 am|
I don't know what voltage it outputs, but I wouldn't worry about it. If your MP3 player output is suitable for headphones then it will be fine.
You often have to try different volume levels until you find one that works nicely (and depending on the recorded volume level of each tape, although that should not be an issue for the remastered tapes - ie. once you find a level that works for one then it should work for all). Start at about 35% volume. If that doesn't work, try 50% etc. Move up and down a bit until it works.
I usually use a mono audio cable directly from the headphone jack of my laptop to play audio into my SC-3000 cassette-in port.
Note - I don't know if you need a mono cable or not. It is possible a stereo cable will work as the WAV files are mono and the SC-3000 will only have the mono connector points, so will *probably* just pick up one channel off a stereo cable anyway.
Second note - if your MP3 player has some sort of built in real time graphic equalizer or sound modification (eg. adding 'concert hall' reverberation, bass boost etc.) that *might* affect the SC-3000's ability to recognize the tones. So turn off any sound processing if you can.
||Posted: Fri Apr 17, 2015 7:58 am|
Idea: applying some thresholds to the original recordings so they become the same clean values as the artificial ones, would preserve any inconsistent timings in the recording. That is arguably a more representative version, and should work for any encoding.
I think I have a bunch of scans from Aaron in my backlog. I also think they are not very high res.
||Posted: Fri Apr 17, 2015 10:58 am|
It'll be great if we can archive as many tapes as possible. They degrade worse than cartridges unfortunately, plus weren't as mass produced as (most) cartridges
||Posted: Fri Apr 17, 2015 3:03 pm|
Great minds think alike (or stealing my thunder, one of the two).
[ was going to confirm these recordings worked on real hardware (like a verified dump [!]) before releasing them, but I haven't got around to it yet.
Also collating 600dpi scans, some of which are on segaretro already, but also intended to be hosted here too
||Posted: Fri Apr 17, 2015 6:30 pm|
Very nice job :) I'd noticed you had been filling out more of the Sega Retro games pages over the past couple of years. They look good.
Are those all original recordings, or have you done some cleanup work on them? The two main advantages of remastering the recordings are 1. compressibility down to a small size and 2. reliability.
But that doesn't preclude making the original recordings available if you can find somewhere to host them (and as Bock said it is very useful to have them available for future reference). So don't let my project stop you :)
Have you thought about how to distribute them? I'm guessing that depending on your record settings then when you put them all together you could well have a couple GB of non-compressible data. So you may need to distribute them as a torrent or via dropbox or a cloud drive service or similar as that may be a bit large to put up on a permanent website.
If you want to try remastering yourself, then many of the tapes are trivial to remaster using the MESS save technique I cover in
By pasting that basic code into MESS, you save back an exact copy of the program you just loaded, including any special characters hidden in the filename as a copy-protection measure (eg. Help by Michael Howard) or as a convenience to help launch the program (like the 'Run' loaders for the Michael Boyd games).
ie. steps 5 & 6
5. Paste the following into MESS. This copies the file name we just loaded from $82A3 across to the SAVE file name location at $83A3
A=0:FOR X=0 TO 15:A=PEEK(&H83A3+X):POKE &H82A3+X,A:NEXT X
(then press CR)
6. Paste the following into MESS
* Saving start will appear, and press record on your tape image (see MESSUI Devices menu)
That works for
1. ALL single-load tapes
2. Any multi-load tapes that use a full basic header / data block for subsequent loads. The easiest way to recognize these is if you hear the beep for Loading start followed by beep beep for Loading End. You can then manually LOAD / SAVE the blocks.
You need more advanced techniques for remastering multi-part games that just load arbitrary blocks of data (like the Michael Boyd games). I will cover some of those next month. But the above technique will let you do a lot of your tapes if you want to.
||Posted: Fri Apr 17, 2015 7:07 pm|
As a side note, I *really* should get around to asking Francesco about the bitstream format he developed years ago for the tape images hosted on the SC-3000 Survivors website :)
That was flexible enough to quickly encode something as complex as Moonbase Alpha in a compact file.
I'll take a more detailed look at the old Survivors utilities and the example bitstream file I have for Moonbase Alpha somewhere if I can find it.
I hadn't pursued that option previously because I assumed there was no emulator support for it. Francesco wrote a flash utility to play back audio from the bitstream files. Take a look - it really is very cool. A little picture of a SR-1000 data recorder pops up and you can play back the audio.
I'm *guessing* the encoding is based on something like this:
In any case, I'll take a closer look at that. It could be a good complement to the audio remastering project.
||Posted: Fri Apr 17, 2015 7:28 pm|
Oh... wow. The latest version of MESS seems to support the bitstream format. My bitstream copy of Michael Howard's 'Help' just loaded perfectly in MESS v0160b... The .bit files are 230Kb in size and zip down to about 30Kb...
Let me spend a couple of days looking at the implications of this and see if I can reproduce the encoding from the perl scripts.
||Posted: Sat Apr 18, 2015 11:34 am|
Ok... well, the short answer is the bitstream format is very simple and MESS does seem to support it now.
The bitstream format is a text file with a .bit extension with one byte per bit.
The "0" (ascii 0x30) character is a square wave of 1200Hz, "1" (ascii 0x31) is 2400Hz and a "space" ( ascii 0x20 ) is a 833.3 us (the length of the 1200 or 2400 Hz wave ) of silence.
So it isn't particularly space efficient or flexible. ie. it is not a generic format like TZX that can support different types of encodings.
But it should work perfectly for all SC-3000 tapes that use the Basic IIIB ROM routines which is well... just about all of them :)
So I may look at adding bitstream files to the tape restoration process along with the WAV files since MESS supports them now.
It looks like Francesco started to ask MESS contributors to add support for the bitstream format to MESS around 2009, and somewhere between v0142 and v0160 someone did add support for the bitstream format to MESS. You can see some of his initial requests here
The standard Sega Basic ROM load / save routines and encoding is (more or less) based on the Kansas City Standard. That is 300 baud and the SC-3000 is 1200 baud. But otherwise it is pretty much the same.
Several years ago, the original SC-3000 Survivors (probably mostly Francesco, I'm guessing), figured that out and asked Martin Ward for assistance in modding his FFT audio analysis script to also handle the SC-3000 signal.
Martin's tape-read perl script does indeed work for FFT analysis and outputting a simple bitstream which loads in MESS, at least it does for sufficiently clean files. But in any case I can generate a bitstream file myself if I need to.
If anyone wants to play around with the tape-read script, then this is approximately what you want on the command line for an SC-3000 tape.
perl tape-read.pl baud=1200 lo=1200 hi=2400 bit=Y wav_filename.wav
That will output a .bit file and a couple of text files with tape data blocks in them. Just be aware you kind of need to know what you are doing to decide if you have a good .bit file or not, and this version of the script does not include the silence markers automatically.
Anyway, I've figured out enough of the problem that I can forget about it for a few days now, hopefully. Time for bed :)
||Posted: Sat Apr 18, 2015 12:39 pm|
|Yeah, they're just the raw .wav files, nothing fancy. I'll have to look into converting to the different formats at some point. Need to verify the working ones. Some I know are probably duds, as I couldn't get them to load on real hardware, but I recorded the wav file anyway. I was planning on trying to collate them into folders with their scans as well and eventually load them somewhere/torrent them for people to download and work on, just haven't got around to it yet. Got distracted with another ongoing preservation project for Mega LD/Laseractive Laserdiscs.|
||Posted: Sat Apr 18, 2015 8:13 pm|
A lot of the old recordings start out at duds. The trick is to use an audio editor to do stuff like applying a high pass 1Khz filter to get rid of low frequencies, low pass filters to try to get rid of high frequencies, and a technique I can't remember the name of (compression?) to equalize the volume across the recording. Sometimes you have to splice bits of part A and B together to get a good recording. And sometimes you will get an audio recording that will load on MESS but not real hardware and sometimes the other way around and save bits back from MESS or the Sega :) You just have to mix and match 'em up until you get a good result. I think we had to do all of that to repair Michael Howard's 'Help' a few years back. From memory there is a good spectrum page on cleaning up audio around somewhere if you search around.
You just need to be careful the data you get back is correct. Anything that uses the full Sega load / save routines has a checksum built in so it won't load if the checksum doesn't match. But some multi-part loader games like Michael Boyd's ones don't have any checksums.
Most of the time a 1KHz high pass filter is sufficient to fix the recordings.
The Laserdisc project sounds cool :)
Ok - here is an example bitstream file for a Hello World program
10 REM Hello World Program
20 PRINT"Hello World"
That has 1 second of silence at the start and the end. Remember, a space char is 1/1200th of a second of silence, a '0' char is a single 1200Hz tone, and a '1' is two 2400Hz tones.
You can try it with MESS v0160, and just look at it in a Hex Editor or text editor to see the content.
||Posted: Sun Apr 19, 2015 1:57 am|
I think you're talking about normalizing, but I'm not 100% on this. If you do this, try to keep it to -6dB to -10dB I believe. At least when you make songs this is an acceptable range. Any higher it might peak and get some digital clipping.
||Posted: Sun Apr 19, 2015 10:32 am|
That's the difference between normalisation and compression, in a nutshell.
Anyway, good audio editors are designed in a way to prevent clipping while normalising, by scanning the audio beforehand to find out the peak intensity and scale the volume changes accordingly, I know Goldwave does, and I think other editors do too. This doesn't prevent you from overriding that value of course but doing so would guarantee you to get clipping.
With compression you can indipendently change the volume of signals with a different loudness (e.g. make weak signals stronger and make strong signals weaker); this doesn't magically dodge clipping per se, and it's inherently harder to use compared to normalisation, but it makes wonders if the source is recorded with different peak levels throughout the place, where normalisation is almost guaranteed to ruin everything.
So yes, I think compression would be more appropriate for a task like this, given that you know what you're doing. Not that clipping a pure square wave would harm that much, but still...
||Posted: Mon Apr 20, 2015 8:13 am|
Thanks for the clarification, Tom :) I didn't have to use compression or normalization much, and it was 4 or 5 years ago when I did, so excuse my initial vague description.
The problem is the age of the tapes. They are 30 years old and on some of the tapes the volume level varies a lot just because the tape media has deteriorated. So originally the volume would have been fairly consistent across the recording. But 30 years later and you can end up with something like this:
This is the waveform from an original tape recording of Aerobat. You can see the variation in volume levels near the middle of the recording. I seem to recall I used either compression or normalization to repair a couple of tapes, but they generally weren't necessary - low / high pass filters and splicing parts from side A / B together usually did the trick.
||Posted: Fri Apr 24, 2015 12:00 pm|
I may be getting bashed for pointing this out but... for the most faithful preservation and emulation of these (or any) cassettes, wouldn't the recordings have to be in Stereo, which the emulator should then convert them to Mono when processing them?
In any case, there's no harm in saving them in Stereo as they can always be converted later to Monaural by whoever prefers them (but not the other way around).
||Posted: Fri Apr 24, 2015 2:12 pm|
|They're most likely mono tapes.|
||Posted: Sat Apr 25, 2015 3:44 pm|
|That would be a relevant distinction to make when preserving this kind of stuff... How can it be verified?|
||Posted: Sat Apr 25, 2015 7:17 pm|
I don't think there was any realistic reason to save a bitstream on a stereo cassette, since it is by definition a succession of tones on a single channel. If they saved two bitstreams in parallel on two separate channels to double the bandwidth, that would be an entirely different matter.
By reading a mono stream with a stereo head you're actually reading an altered version of the data, in case the left and right heads are calibrated slightly different, or if the magnetic tape isn't uniformly magnetised.
I know that personally I'll never get the appeal of useless stereo for mono files (as I said in the vgm topic), but I think this is a prime case of data which is natively monophonic.
||Posted: Sat Apr 25, 2015 8:42 pm|
Let's put it this way... Imagine that one stereo cassette has the game recorded only on the left channel. A mono audio file would not be an accurate representation of that tape.
That would be the most extreme example, but you see what I mean.
||Posted: Sat Apr 25, 2015 10:07 pm|
|Computer tape recorders were often strictly mono. That means the head has a single wide reader, rather than two smaller ones with a gap in the middle. This results in more ability to read the data from the tape without errors. You would need specialist equipment (or some cunning with the alignment adjustment) to detect what was actually on a tape.|
||Posted: Sun Apr 26, 2015 1:41 am|
In that case I'd advice towards always recording them in Stereo, as it can be converted to Mono anyway and... just won't do any harm, in any case.
||Posted: Sun Apr 26, 2015 9:05 am|
|Recording using a mono device should produce the best results - two partial parts of the tape is worse then the whole thing. Maybe a new(ish) stereo device is better than a 30 year old mono one, though..?|
||Posted: Mon Apr 27, 2015 10:55 am|
I don't get why you're saying this. Combining both parts should give the whole thing, while you can't separate both parts from the whole... Recording in Stereo is just better for preservation.
||Posted: Mon Apr 27, 2015 4:08 pm|
|A stereo recording has two 0.02" tracks separated by a 0.016" guard area. A mono recording could magnetise the guard area, and thus reading it back with a head spanning the guard area could give you a better signal to noise ratio than if you effectively ignored it by using a stereo head. If there's no data there (as seems to be the case in the spec) or if a mono head ignores the guard area then a stereo recording (with good calibration of the tracking) would be no worse than mono, maybe better as you would have more data.|
||Posted: Wed Apr 29, 2015 12:09 am|
Thanks for the discussion on the use of stereo vs mono recording techniques. I think this is a case where provided you can get a verifiable and repeatable recording that loads (possible with anything that has a checksum, requires multiple record passes for comparison if not), then that is good enough. Certainly the original recordings would have all been with mono datasets (and the SC-3000 only has a mono output). But I found the discussion interesting and I'll keep it in mind in future.
Disassembled Sega SC-3000 Tape Load Routines from Basic IIIB ROMPosted: Wed Apr 29, 2015 12:12 am
Last edited by honestbob on Wed Apr 29, 2015 12:33 am; edited 3 times in total
Here's something else I've been meaning to publish for future reference, but I hadn't got around to it.
This is very useful for anyone who wants to write tape emulation for the SC-3000 (or just step through a custom tape load routine) as it explains how the Sega SC-3000 actually interfaces with the data from the cassette-in port, and it shows how the main routines that all the tape software use actually work.
See attached zip file for the spreadsheet and the full text version of the file with commented disassembly of the tape load routines. Yes, god forgive me, I did start this off in a spreadsheet rather than a text file. I was young and foolish :)
Here's an excerpt explaining the important bits.
Sega Basic Level IIIB ROM Load / Save Routines Commented Disassembly
This is a disassembly and partial commenting of the Sega Basic Level IIIB ROM. In particular, the Tape Load routines are well commented. I did this to learn how to repair and remaster Sega SC-3000
tape recordings and this work also led to the great Sega SC-3000 Survivors Mk II Multicart that can play SC-3000 tape games.
From memory the Tape Save routines may only be partially documented. You are more likely to find mistakes in my comments there too. The Load routines should be well commented and mostly correct.
IMPORTANT: THERE ARE SOME ERRORS IN THIS DISASSEMBLY.
The original disassembly tool I used had some bugs in it, so not all of the instructions translated correctly. I have manually repaired those in the commented sections, but no guarrantees for the rest of the file.
Most of this was done back around 2010-2012
Overview of Sega SC-3000 tape input architecture
The Sega SC-3000 has a custom HIC-1 chip which is connected the 8255 PPI inside the Sega SC-3000. The PPI is mapped to various I/O ports.
The HIC-1 chip essentially converts the voltage level observed on the tape cassette input into either a high or low signal which sets the value of port $DD, bit 7. If the cassette input voltage level is 'high', then bit 7 of port $DD will be set (ie. one). If the cassette input voltage is low, then bit 7 of port $DD will be reset (ie. zero).
VERY IMPORTANT: THE BITS READ FROM PORT $DD ARE NOT DATA BITS. THEY REPRESENT THE CURRENT HIGH
OR LOW STATE OF THE CASSETTE INPUT.
The Basic IIIB ROM Load routines work by sampling the value of port $DD bit 7 thousands of times per second, timing how long
the input stays high or low, and then converting that to a data bit (zero or one) when the input has stayed high / low for long enough.
As a side note, that means you can digitize speech and music through the cassette-in port with a little bit of assembler magic - anyway, back on topic :)
A 'ONE' data bit is represented by two 2400Hz tones. ie. port $DD bit 7 is high for 1/4800th sec, low for 1/4800th sec, high for 1/4800th sec, low for 1/4800th sec.
A 'ZERO' data bit is represented by one 1200Hz tone. ie. port $DD bit 7 is high for 1/2400th sec, low for 1/2400th sec.
The Basic IIIB routines have some tolerance for how wide those tones can be to allow for tape slippage etc. But noise spikes or
inconsistent volume levels across the tape tend to result in tape read errors.
Cassette Input Example (ie. sampling the high / low voltage level from the cassette input) is mapped through port $DD, bit 7.
in a,$DD ; Read from PPI. Bit 7 is the high / low voltage state of the Cassette Input from the HIC-1 chip
and $80 ; Mask out the one bit we are interested in - ie. bit 7
; so and $80 results in either $80 if bit 7 was set, or $00 if it was not
Cassette Output Example (ie. setting the cassette output to a high / low voltage level) can be done by writing to the control register at port $DF
eg. you can set the cassette output high with
ld a, 9
out ($DF), a
you can set the cassette output low with
ld a, 8
Overview of main Sega SC-3000 Basic IIIB ROM tape load / save routines
All of the known commercial Sega SC-3000 tape software uses at least the Detect/Write Sync_Bits routines and
the Load/Save_Byte_To_A routines from the Basic IIIB ROM.
The only known exception to that would be Mike Hadrup's Load / Save / Verify routines. But I haven't studied those, and it was not widely used.
The main Basic IIIB ROM routines are:
Documented entry point is $3A00 which jumps to $3ACB
This is one of the two main load atoms
All known cassette loaders use sync bytes to mark the start of a data block (apart from Mike Hadrup's LSV extensions)
The Sync Bits are 3600 'one' bits in a row (ie. 3600 * 2 * 2400Hz square wave cycles). This matches save routines and published documentation.
Repeatedly sample the cassette input until we have found 255 valid 'ones' in a row
We can ignore the rest of them because Load_Byte_To_A looks for the first zero
Documented entry point is $3A06 which jumps to $3A8E
This is one of the two main load atoms
Read a single byte from the cassette input and place in register A
Look for the first Zero bit which marks the start of a data byte
Bit "1" is two cycles of 2400 Hz in a 833.3 micro Hz window
Bit "0" is one cycle of 1200 Hz in a 833.3 micro Hz window
Each byte is encoded with 11 bits
Start bit = 0, then bit 0-7 actual data with LSB first (8 bits), followed by two stop bits = 1
Because data bytes are always bounded by a 0 bit at the start and 1 bits at the end
we just search for the first 0 bit
We determine that the given bit was a zero by checking that # of iterations was > &H1D
This is the main entry point for load
If you specified a filename in LOAD "filename" then the basic parser will already have
initialized &H82A2 with string length and &H82A3 with the 16 character string
If you didn't, then &H82A2 / &H82A3 will be nulled out
This takes care of the full load including *Loading Start, the Header block with file name, file length, parity check, and the dreaded Tape Read Error message etc.
Documented entry point is $3A12 which jumps to $3A15
This is one of the two main save atoms
Write the byte in register A to the cassette output
Bit "1" is two cycles of 2400 Hz in a 833.3 micro Hz window
Bit "0" is one cycle of 1200 Hz in a 833.3 micro Hz window
Each byte is encoded with 11 bits
Start bit = 0, then bit 0-7 actual data with LSB first (8 bits), followed by two stop bits = 1
Documented entry point is $3A0F which jumps to $3A4D
This is one of the two main save atoms
All known cassette loaders use sync bits to mark the start of a data block
The Sync Bits are 3600 'one' bits in a row (ie. 3600 * 2 * 2400Hz square wave cycles)
This is the main entry point for save
The filename you specified in SAVE "filename" will already have been stored by the basic parser
at &H82A2 (string length) and &H82A3 (16 character string)
This takes care of the full save including *Saving start, the Header block with file name, file length, parity check etc.
(see the attached zip file for the disassembled routines)
Idea for simplified tape loading emulationPosted: Wed Apr 29, 2015 12:26 am
And the above post leads on to an idea for simplified tape loading emulation if anyone wants to add it to Meka :)
The above article details how the hardware works. And MESS has a very cool mechanism for using FFT analysis on WAV files to translate the audio to voltage levels and update the value of port $DD, bit 7.
But from an emulation perspective, all you need to do to emulate tape loading that works for all of the known SC-3000 commercial tape software is:
1. The Bitstream format. As discussed earlier, you can represent all of the known SC-3000 commercial software as one of three states:
- silence for 1/1200th of a second
- 1200Hz tone for a 'zero' bit
- 2 * 2400Hz tones for a 'one' bit
The Bitstream format is not space efficient (it uses one byte to represent each bit of data). But it compresses well, is simple to use, and we have other tools to work with it already.
2. Tape Emulation that runs in realtime / emulated clock time or cpu cycles / however you manage timing inside Meka that translates
- silence to 1/1200th of a second of 0 at bit 7 of port $DD
- a 'zero' bit to 1/2400th of a second of 1 at bit 7 of port $DD followed by 1/2400th of a second of 0 at bit 7 of port $DD
- a 'one' bit to 1/4800th sec of 1 at bit 7 of port $DD, then 0 for 1/4800th of a second, then 1 for 1/4800th sec, and 0 for 1/4800th sec
ie. you skip all the complex WAV file analysis
3. Tape Player controls in Meka (or keyboard shortcuts) that allow you to mount a tape, press Start and Stop, and a time counter on screen somewhere
That's pretty much it. This simplified version can not be used to save data to a Bitstream file (Bitstream is a playback format only).
Anyone keen to take a look at that idea for Meka?
Sorry Bock :)
||Posted: Wed Apr 29, 2015 7:49 am|
Based on your description, a few things come to mind...
1. There's no reason why you couldn't have arbitrary encoding formats ("turbo loaders") which would make the "bitstream" format fail.
2. The bitstream format could totally be written to by an emulator.
3. Truncating recordings to min/max loses no information that is perceptible by the system.
4. Emulators can easily support waveform data by simply picking the current sample whenever the port is read, no FFT analysis is needed. But it will be mostly real-time (subject to emulator speedup controls).
5. But an emulator can also take a high level approach and when it sees the API calls, parse the data natively, push it to emulated RAM and advance the virtual tape, in an instant.
I might have a go myself when I get a computer...
||Posted: Wed Apr 29, 2015 9:49 am|
Correct. The bitstream format only works for 1200Hz and 2400Hz tones like the IIIB ROM routines use. However that covers pretty much *all* of the available tape software. No commercial software I have seen uses any other encoding format.
So it isn't a case of Bitstream being a 'good' format. It is a 'mostly good enough' format :) And it is already supported by one other emulator (MESS) and I have some tools that let me create and edit that format. Aside from that, I'm not especially attached to it.
I haven't yet seen an example of something saved by Michael Hadrup's Load / Save / Verify routines but I'm pretty sure the encoding is totally different and wouldn't work.
I'm happy to be proven wrong :)
The Save routines work the opposite way to the Load routines. ie. you set the Cassette Out port to voltage hi / lo for arbitrary periods of time. The actual period of time is controlled by loops in the Basic IIIB code. So it is kind of the 'turbo loader' argument in reverse. There is no guarantee that what you write out to the emulated HIC-1 would conform to 1200Hz / 2400Hz pulses expected by the Bitstream format.
But it would be very easy to write a WAV file from underneath the HIC-1 emulation.
I should probably clarify writing a control byte to port $DF.
The cassette output feed is from the 8255 PPI 'Port C' bit 4 (from memory the 'Port C' is a description you will find in the 8255 PPI datasheet / documentation - it just refers to a group of pins on the 8255 chip). This particular pin is I/O mapped to the SC-3000 port $DE bit 4.
Note - I *think* you can write directly to $DE bit 4 if you want to, although the Basic IIIB ROM uses the port $DF bit set / reset feature. So stick with that for now.
Port $DF is the (write only) control register for the 8255 PPI.
If bit 7 (msb) of the control byte is set, then it sets up the PPI mode (done on startup). That is not what we want here.
If bit 7 of the control byte is NOT set, then we send a bit set / reset command to apply to ONE of the 8255 PPI Port C bits. Then bits 1 to 3 of the control byte select which of the PPI Port C outputs we should write to, and bit 0 selects the value that should be set. (Bits 4 to 6 are not used in this Bit Set / Reset format).
So for instance:
ld a, 8 ; %00001000
out ($DF), a ; ie. set bit 4 of PPI Port C to zero
; bit 7 is zero, so this is a bit set / reset command
; here bits 1 to 3 are %100 - ie. = 4,
; which is the cassette output pin / bit 4 in PPI Port C
ld a, 9 ; %00001001
out ($DF), a ; ie. set bit 4 of PPI Port C to one
I think I agree with this, but you may need to clarify your point :)
Certainly as far as the SC-3000's HIC-1 chip output is concerned, you only have a voltage high state and voltage low state based on the cassette audio input. You would need an oscilloscope and a signal generator to infer more exactly how the HIC-1 chip output changes in response to a given audio input.
But I'm pretty sure you could accurately represent any SC-3000 recording with a bit depth of 1 bit per sample - ie. high voltage / low voltage. Certainly for an emulator and a perfect waveform I think that is true.
If you want to play it back to a real SC-3000 then this is true up to a point. I'm pretty sure you will get read errors if your 'max' level is too loud or too quiet when you play it back into the cassette-in port even with a 'perfect' waveform. But you would just compensate for that by adjusting the volume control on your headphone jack.
Yes, I think you are correct. FFT analysis is necessary to look at an audio signal and try to convert it to bits / bytes (like that tape-read.pl script referred to earlier in the thread). But you are right. That is not how the HIC-1 chip and Basic IIIB code works - they sample in real time as the audio is playing.
So yes you can probably do something like just pick a threshold and treat a value over that threshold as high voltage, and something below that threshold as low voltage. I should admit I haven't actually read the MESS tape emulation code so maybe it does work that way. That would be worth trying anyway :) Good thinking.
A more flexible implementation might do some preliminary analysis on a WAV file to determine what the high / low voltage threshold levels should be for that recording to allow for differences in recording volume. That would be especially important for interpreting real audio recordings from original tapes..
But for a first pass I would start with the remastered waveforms for 'Bastow Manor' that I uploaded earlier and use that for testing with.
(Edit - I may have misinterpreted your point 5, so the following may be off topic - not sure. I thought you meant you could look for the main LOAD entry point in the Basic IIIB ROM, and then try to do an instant fast load of all the data into emulated RAM then resume program execution after the Basic IIIB LOAD call)
That would be very cool. I'm not sure if you can write a generic technique that will work in all cases though.
You could probably make it work for the full LOAD / SAVE commands as the data would be written to / read from predictable addresses. But then you would have to skip a whole pile of Z80 code and adjust register values etc. to allow for the code you have just skipped.
I'm not sure that would work for anything that calls the Basic IIIB routines in unexpected ways, and unfortunately the different tapes tend to jump into the IIIB ROM at different places.
The most generic approach is of course to hide everything in HIC-1 emulation under the port writes. MESS allows you to speed up the emulation, so if you have a 5 minute tape to load you can run MESS at say 10 times normal speed and load the tape much faster. But everything runs 10 times faster then - CPU, VDP, everything.
Awesome - good luck. It would be nice to bring Meka full circle on this. I think I probably first started to annoy Bock about this over 15 years ago :) I just didn't have enough information to be useful back then.
||Posted: Wed Apr 29, 2015 12:44 pm|
The problem with "good enough" is that, whenever a higher standard is set up in the future, there's going to be a need for redumping every single game. It's the difference between "just emulating" and "preserving".
This has happened a while ago with ISO+MP3s for CDs, and is already happening with cartridges for certain systems (dumping chip by chip against having the whole thing in one file). These redumping processes could have been avoided, had a higher stardard been set up in the beginning.
I'll bring up again the example of a cassette that only plays data on the right channel. A mono audio file would not be a good representation of that tape.
Unless the tapes themselves specify somewhere that they're indeed monaural cassettes, please record these in stereo after adjusting the azimuth so that the sound plays as crystal clear as possible.
||Posted: Wed Apr 29, 2015 6:10 pm|
Here's an excerpt from the final Sega Computer magazine from August 1988 for reference. It covers the 8255 PPI operation and ports. In the posts above I just summarized the bits you need for tape emulation. But the following might be useful too.
Note - from memory it helps to read the actual 8255 PPI datasheet as well as the Port A / B / C groups and different modes make more sense then. Basically, the 8255 PPI was a general purpose device for connecting all sorts of different devices into your system and then giving you an easier way to I/O map them and control them. The SF-7000 uses one as well.
And yes, I know I should really upload those some time :) It looks like the last four issues are still missing from the Scans section.
||Posted: Wed Apr 29, 2015 6:56 pm|
The original compact cassette specification actually states that there are two stereo channels on the tape, and for mono you just make them identical, so for archival purposes it makes sense to record in stereo. Any non standard use of the guard area to improve reliability would be lost though.
It makes no sense at all to distribute these, any more than you would want to distribute 1200dpi PNG scans - it's nice to archive, but not necessary for users to get what they want and costly given current storage and bandwidth prices, especially given the degradation issues (which seem very easy to "remaster", by the way, without all the hassle you refer to, with a little bit of code) which mean the raw recording may not even work on a real system as-is.
||Posted: Wed Apr 29, 2015 9:03 pm|
Actually doing the remastering is fairly straight-forward in most cases. It is all the validation checks, testing in emulator / real SC-3000, writing up a text file to go with it containing instructions / other information, finding a cassette cover scan etc. and packaging it all up that takes all the time. That and engaging in friendly banter on the forums :)
But I assume you are thinking something along the lines of a routine that reads the original audio WAV and converts it to a 'cleaned' WAV by just outputting hi / low states depending on whether the original audio hit a volume threshold or not.
That might work quite well in a large number of cases and would be a useful cleanup tool (I suspect you will find your code will work with some recordings and not with others). It is essentially what the real HIC-1 chip does and I assume what the tape emulation in MESS does except you get to immediately save the 'cleaned' signal back out to a WAV file. It is something I would add to the restoration toolset to use in conjunction with other remastering techniques (and as a check on other remastering techniques).
Just for general interest, here are two wave form pictures to illustrate the type of data you potentially have to analyze.
The first is taken from an original tape audio recording that was around 25 years old. You can see the waveform is not particularly 'square', and even within this tiny section of tape audio (approximately 11 / 1200th of a second / 11 bits), the 'loudest' waveforms are approximately twice as loud as the quietest waveforms.
The second is from a recording I took directly from a real SC-3000 cassette out port (with microphone boost on my laptop as the original signal level was *very* low). You can see that the SC-3000 is capable of outputting something very close to a square wave.
||Posted: Thu Apr 30, 2015 6:02 am|
If you only want to load the games, you can just make a tiny TZX file from the sound source, which can also be converted to a clean WAV file any time.
But we're also talking about preserving the contents of the cassettes "as is", imperfections and all.
||Posted: Thu Apr 30, 2015 6:39 pm|
I think that is the core of the argument (and perhaps my choice of the words 'remastering' and 'restoration' have clouded the issue.
Ultimately, the original recordings are an old and damaged representation of digital data.
My point is to preserve the software in a compact and easy to distribute form which allows use in both emulators and on original hardware. ie. an exact audio representation of the original 'ideal' audio waveforms either in a WAV (highly compressible) or in a compact file that can be used to regenerate that waveform (eg. bitstream, TZX etc.)
The exact original recordings are of some historical interest, and are necessary when checking for any errors in conversion / remastering. But as Maxim points out, none of the glorious quirks / static / stretching inherent in the original recording represent actual data. They represent chances to bang your head into a wall with a tape read error as I did many, many times as a child :)
Because the SC-3000 software is relatively rare, and so few people have it, it might as well not exist. So I want to make it accessible in the hopes it will live for a good long time yet.
I actually have a plan B for that based on the multicart. You might see that appear over the coming weeks. Shhh :)
That gets us back to the earlier question of whether or not TZX is an appropriate format for use with the SC-3000 and indirectly to my earlier point about trying not to spend too much time on this. It seems I can't help myself. Never mind :)
The TZX spec seems very flexible and allows you to represent a wide range of pulse width encoding techniques.
It is built on a couple of concepts.
1. Multiple different types of blocks. Each block type allows you to store a slightly different style of data encoding.
2. Pulse width encoding. We still assume that the audio maps to hi / low voltage states. So the question is how long each hi / low state should be (and how many of them) represent a single bit of data.
3. T States. The pulse widths are represented in T States which are specific to the processor speed of the system. eg. ZX Spectrum T state is 1 / 3500000th of a second. Amstrad CPC has 4MHz clock so T state is 1 / 4000000th of a second. Sega SC-3000... can't remember... but I do seem to recall the CPU clock is slightly different for NTSC and PAL variants, so that might be an interesting timing related detail.
4. Specialized data blocks specifically for known Spectrum ROM standard encoding and turbo encoding modes (block 10 and 11). That allows a much simpler representation because you only have to store the binary data for the audio block, and the emulator / WAV generator fills in the extra pilot pulses and start / stop bits etc. automatically. ie. you need toolset support for that.
5. ID 15 Direct Recording block. This one could be good. This is a representation of what states hi / low to play after a given number of T States, so it would slot under HIC-1 emulation well and should be reasonably compact. And you might be able to use existing third party encoding tools to generate a TZX in this format if you can pass a definition of the SC-3000 pulse widths to the encoding tools.
6. ID 19 Generalized Data Block. This is allows you to define multiple pulse widths, sync tones etc. then define the data block.
I think the most interesting / relevant block types for the SC-3000 are ID 15 and ID 19.
I'll see if I can find some time to play around with the available encoding tools for ID 15 and / or generate an example ID 19 block on a simple Hello World.
Anyway - off to work.
||Posted: Thu Apr 30, 2015 7:05 pm|
I guess the hope is that the existing tools (most of which seem to be from >10 years ago) might take a WAV and emit something compact, usable and compatible with existing emulation code. My feeling is that the tools are quite specific to the systems in question...
The clock difference between PAL and NTSC is small enough that it's well within the tape routine tolerance for detecting the frequency.
Somewhat related, I also came across Spectrum tools that reencode to a very fast turbo loader, assuming the audio is played from a modern device with low noise, no degradation over time and no tape stretching. It might be interesting to develop something similar...
||Posted: Thu Apr 30, 2015 9:01 pm|
Well, bitstream is supported by MESS and the generated audio output works with original hardware. It is a very simple format to work with. It is not flexible in that you can't represent different pulse width encodings, but it represents 99% of available SC-3000 tape software so is a good start point. And as discussed above it should only require a (reasonably) simple emulator routine to read it and generate the high / low states for port $DD bit 7 for adding to Meka.
It looks like TZX files based around ID 15 or ID 19 blocks should represent a SC-3000 signal correctly, so I will see what existing conversion tools can do with that. Some of them may allow you to set the pulse widths to search for in Direct Recording mode. And for something like MESS which I assume has TZX support in the code base you should be able to use the existing TZX code with only minor modifications - *probably* just choosing the correct CPU clock to give the T state is sufficient if it supports those ID 15 / ID 19 blocks. But I still need a proof of concept to check.
And of course MESS has good WAV support, and as discussed above that *may* be reasonably straight-forward to add to MESS (although I suspect that if you want it to work with 'original' audio recordings you might need to make it a bit smarter - eg. prescan the WAV to figure out average volume levels and be prepared to adjust your edge thresholds on the fly for different parts of the WAV). Those remastered square wave WAVs are highly compressible so those are distributable and they also work with real SC3000 hardware.
So we have three viable avenues to pursue, two of which currently work in MESS (WAV and Bitstream)
||Posted: Fri May 01, 2015 7:02 am|
|Emulators ought to try to work with raw audio (including live playback from a tape), but the bitstream format seems like a mistake to me. TZX is complicated, for good reasons, but it is flexible in ways the bitstream format isn't.|
|Goto page 1, 2 Next|