Forums

Sega Master System / Mark III / Game Gear
SG-1000 / SC-3000 / SF-7000 / OMV
Home - Forums - Games - Scans - Maps - Cheats - Credits
Music - Videos - Development - Hacks - Translations - Homebrew

View topic - Low-level Dumping SF-7000 Floppy Disks (Sega Super Control Station)

Reply to topic
Author Message
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Low-level Dumping SF-7000 Floppy Disks (Sega Super Control Station)
Post Posted: Thu Jul 14, 2022 10:04 am
Been purchasing everything I could over the past few decades, and I've finally started dumping SF-7000 Floppy Disks "the right way", using a custom Pauline controller. I've got 100 floppies so far (about 3 GB of flux data) and with about 150 to go. That include meticulously inspecting and cleaning disk surface etc.

Will post more information about the process and result.

We will need developers to:
- Pick a more suited file formats for compact release of dumps + work on adding them to emulators. Adding support for raw stream dumps would be great too.
- Reverse engineer common file systems so we can extract files along with dumps, and create tools to facilitate identifying data (checksum per file, notes about data correctness).
IMG_7716.JPEG (1011.68 KB)
IMG_7716.JPEG
IMG_7719.JPEG (968.47 KB)
IMG_7719.JPEG
IMG_7758.JPEG (486.47 KB)
IMG_7758.JPEG
IMG_7855.JPEG (539.6 KB)
IMG_7855.JPEG
IMG_7920.JPEG (282.3 KB)
IMG_7920.JPEG
IMG_7782.JPEG (341.86 KB)
IMG_7782.JPEG
IMG_7778.JPEG (322.29 KB)
IMG_7778.JPEG
IMG_7976.JPEG (842.54 KB)
IMG_7976.JPEG

  View user's profile Send private message Visit poster's website
  • Joined: 05 Jun 2010
  • Posts: 757
  • Location: Pennsylvania, USA
Reply with quote
Post Posted: Thu Jul 14, 2022 12:36 pm
I have nothing useful to add, but just wanted to say awesome work as always.
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14687
  • Location: London
Reply with quote
Post Posted: Thu Jul 14, 2022 3:57 pm
I disassembled an SF7 pirate disk some years ago, it had no file system because it was loading game ROMs into RAM directly. I think the IPL similarly doesn’t have any kind of file system; I presume Disk Basic will have the beginnings of one.

Is there any evidence yet of disks not formatted in the standard way (according to the IPL setup of the FDC)? Or do I need to write some code for the flux parser?
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Post Posted: Thu Jul 14, 2022 5:36 pm
Maxim wrote
I disassembled an SF7 pirate disk some years ago, it had no file system because it was loading game ROMs into RAM directly.

It probably has some sort of mechanism for giving a title and finding where the data is located, that's a primitive file system what we ought to write an extraction tool for (easy and probably tweaked on a case by case basis).

Quote
I presume Disk Basic will have the beginnings of one.


Correct, and many things in SF-7000 are based or forked from Disk Basic.

Quote
Is there any evidence yet of disks not formatted in the standard way (according to the IPL setup of the FDC)? Or do I need to write some code for the flux parser?


Yes.
First photo is a typical disk.
Second photo is Burglar Bill by Michael Boyds.

I just finished indexing and typing all labels, I have 283 disks (many are dev stuff) + a dozen of empty ones.

We need to figure out what's the best way to release those data (either way we will release the streams but it would be good to find the right format that suits the stuff we have).
IMG_7280.JPEG (839.12 KB)
IMG_7280.JPEG
IMG_7281.JPEG (823.63 KB)
IMG_7281.JPEG

  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Post Posted: Thu Jul 14, 2022 6:56 pm
Ought to mention, disks include other booting simili operating systems, I've got disks labelled "CP/M", "Segados", "G BASIC", "D BASIC" not sure what they all are yet.
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14687
  • Location: London
Reply with quote
Post Posted: Thu Jul 14, 2022 7:02 pm
It seems popular to lean on archive.org for this sort of thing, but certainly better to also have usable disk images where possible. I don’t know very well what is done in other parts of emulation for exotic disk images - apparently for many, pirated versions cracked to work as normal are the general case and original disks often aren’t even dumped.
  View user's profile Send private message Visit poster's website
  • Joined: 05 Sep 2013
  • Posts: 3762
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Thu Jul 14, 2022 7:39 pm
Bock wrote
Ought to mention, disks include other booting simili operating systems, I've got disks labelled "CP/M", "Segados", "G BASIC", "D BASIC" not sure what they all are yet.


CP/M is probably this. It's basically the father of DOS.
  View user's profile Send private message Visit poster's website
  • Joined: 23 Jan 2010
  • Posts: 415
Reply with quote
Post Posted: Fri Jul 15, 2022 12:16 am
sverx wrote
Bock wrote
Ought to mention, disks include other booting simili operating systems, I've got disks labelled "CP/M", "Segados", "G BASIC", "D BASIC" not sure what they all are yet.


CP/M is probably this. It's basically the father of DOS.

Not would be a primitive data base for Z80 cpu´s? I would say that it´s the father of MS Access.
edit: In fact is a Control Program father of DOS. The app was called "dBase"
  View user's profile Send private message
  • Joined: 05 Sep 2013
  • Posts: 3762
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Fri Jul 15, 2022 6:46 am
No, it's an 8-bit era operating system, very common at the time.
  View user's profile Send private message Visit poster's website
  • Joined: 29 Mar 2012
  • Posts: 879
  • Location: Spain
Reply with quote
Post Posted: Fri Jul 15, 2022 7:05 am
Yes, I had CP/M on my Amstrad CPC.

I think @ICEknight can help us with the disks. In Retrolel they're discussing how to preserve properly this kind of raw images.
  View user's profile Send private message
  • Joined: 23 Jan 2010
  • Posts: 415
Reply with quote
Post Posted: Fri Jul 15, 2022 9:32 am
sverx wrote
No, it's an 8-bit era operating system, very common at the time.

I mean that you is correct. My edit is for confirm my mistake. I had a apple II e i had a board in with a z80 for CP/M apps like as Visicalc. So you is correct is a kind of father of DOS that had some apps wrote for it. dBase for example.
  View user's profile Send private message
  • Joined: 05 Sep 2013
  • Posts: 3762
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Fri Jul 15, 2022 11:39 am
no worries. Though I never used CP/M myself I remember in the early DOS days that there was much info about how that was built on CP/M legacy.
  View user's profile Send private message Visit poster's website
  • Joined: 05 Sep 2013
  • Posts: 3762
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Fri Jul 15, 2022 11:46 am
Bock wrote
We will need developers to:
- Pick a more suited file formats for compact release of dumps + work on adding them to emulators. Adding support for raw stream dumps would be great too.


How much a disk dump takes, in the raw stream format you get out of your hardware and program?

We surely have to preserve this but I think your point is to create/use some disk image format where all the data is decoded/corrected already, right?
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Post Posted: Fri Jul 15, 2022 11:52 am
sverx wrote
How much a disk dump takes, in the raw stream format you get out of your hardware and program?

We surely have to preserve this but I think your point is to create/use some disk image format where all the data is decoded/corrected already, right?


It depends how long we read a track. Right now I'm reading 820 ms per track (about 4 rotations) at 50 Mhz, the average dump for one side takes 11 MB. For unformatted side I reduced the dumping length to 200 ms per track just to keep a data reference of the fact they were unformatted.

Some disks are redumped multiple times after cleaning. Right now I have 4.32 GB for 158 disks (316 sides) so average 14 MB/side taking account of redumps increasing the average and unformatted lowering the average.

The most common 640-sectors format, omitting errors and assuming valid data, and read via IPL functions can be binarized as 163840 bytes (this is what our .SF7 files where). I believe 80%+ could be safely released under that format.

I don't understand the technicalities well enough yet to tell if a slightly better format (e.g. DSK encoding sector location) can be enough for 100% or if we need an even lower-level format.
  View user's profile Send private message Visit poster's website
  • Joined: 05 Sep 2013
  • Posts: 3762
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Fri Jul 15, 2022 12:03 pm
I would go for the lowest hanging fruits first. If 80%+ of those disks can fit an existing format and be released in that way, let's do that first.

Then we probably need to investigate on the remaining ones, how many different schemes we need to accomodate them all - if that's at all possible of course. Hard to tell without knowing much at the moment but it's something that will come when we'll have more info.
  View user's profile Send private message Visit poster's website
  • Joined: 14 Oct 2008
  • Posts: 508
Reply with quote
Post Posted: Sat Jul 16, 2022 1:53 am
Maxim wrote
It seems popular to lean on archive.org for this sort of thing, but certainly better to also have usable disk images where possible. I don’t know very well what is done in other parts of emulation for exotic disk images - apparently for many, pirated versions cracked to work as normal are the general case and original disks often aren’t even dumped.

I've only heard recently of an effort to get original disk dumps for Apple II.
That's certainly a noble effort, hopefully 40 years later isn't too late.

I believe for some such systems as Commodore 64, the pirated "cracktos" are even celebrated as a part of history.
(I can some of the appeal in cracks for emulation, as it surely is possible to load faster, I would say as someone who has not used the original hardware. As I understand, the official 1541 disk drive was like it's own computer with own firmware and emulating copy protection would probably entail low-level emulation of the disk drive firmware. Which I imagine would also mean accurate load time emulation.)
  View user's profile Send private message
  • Joined: 25 Feb 2013
  • Posts: 384
  • Location: Osaka
Reply with quote
Post Posted: Sat Jul 16, 2022 11:00 am
I am super interested in contributing!
I already played around with the disk basic's fat

https://github.com/fabiodl/sctape/blob/main/scfloppy.py

and would love to start working on the flux data
  View user's profile Send private message
  • Joined: 14 Aug 2000
  • Posts: 740
  • Location: Adelaide, Australia
Reply with quote
Low-level Dumping SF-7000 Floppy Disks (Sega Super Control Station)
Post Posted: Sun Jul 17, 2022 3:43 pm
Interesting stuff.

Preserving disks down to the magnetic flux level is uber cool, but I feel it’s overkill, as this level of information doesn’t make it past the FDC to the computer. It does have some practical use, such as being able to identify corrupt sectors from the sector info and therefore bad dumps, no doubt.

I think we can do better than just a raw stream of digitized FM-encoding data. An XML schema that defined tags for the markers, gaps, sector info, sector data, etc… would allow the entire disk to be preserved, while allowing simpler access to the sector data with just parsing.

About the 20% of oddball disks, wouldn’t they just be system disks with their own IPL in Track 0 Sector 1 that is copied to RAM and then executed to load the rest of the disk data into RAM? That would mean the IPL on the disk would be in the standard format, and could be disassembled to see how the FDC is configured and how the rest of the disk is formatted in theory.
  View user's profile Send private message
  • Site Admin
  • Joined: 19 Oct 1999
  • Posts: 14687
  • Location: London
Reply with quote
Post Posted: Sun Jul 17, 2022 4:05 pm
That seems plausible, yes, so you could have metadata to describe the rest of the disk format, perhaps just in terms of the FDC parameters used to read it.

I imagine some copy protections may want to have corrupt sectors that trigger certain FDC behaviour or even some borderline data that triggers some non-deterministic behaviour in the FDC.
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Post Posted: Sun Jul 17, 2022 7:31 pm
Quote
this level of information doesn’t make it past the FDC to the computer. It does have some practical use, such as being able to identify corrupt sectors from the sector info and therefore bad dumps, no doubt.


That's enough of a justification. Disks are not flawless to read, some hard, some downright corrupted. Also consider that some copy protection may be using intentionally corrupted MFM sectors and I would assume the software would intentionally verify that the controller can't read them. If we don't record that information in theory the dump is incorrect and software needs to be cracked.

Also consider the typical user disks with files that happens to have corrupted sectors. If we release a "packed" version of this I think it would be appropriate and correct that we release it with information that some parts couldn't be read (with maybe a chance they eventually it'll be read).

Quote
I think we can do better than just a raw stream of digitized FM-encoding data.


Of course and we should. There are loads of formats (HxCFloppyEmulator only has nearly 30 formats) so we should be investigating what's best.

Quote
About the 20% of oddball disks, wouldn’t they just be system disks with their own IPL in Track 0 Sector 1 that is copied to RAM and then executed to load the rest of the disk data into RAM? That would mean the IPL on the disk would be in the standard format, and could be disassembled to see how the FDC is configured and how the rest of the disk is formatted in theory.


Pretty sure all the fancy disks are booting indeed, you are right Track 0 Sector 1 is "regular". https://www.smspower.org/forums/files/img_7281_577.jpeg is one of those disk.
Not sure what you are suggesting here, interested.

When I'm done with my first pass of dumping I'll open a work group to solve those problems.
  View user's profile Send private message Visit poster's website
  • Joined: 14 Aug 2000
  • Posts: 740
  • Location: Adelaide, Australia
Reply with quote
Low-level Dumping SF-7000 Floppy Disks (Sega Super Control Station)
Post Posted: Mon Jul 18, 2022 1:51 pm
I think I convinced myself it wasn't overkill by the time I finished the paragraph and typed "no doubt". :)

You are totally right. See you in the work group.
  View user's profile Send private message
  • Joined: 25 Jul 2022
  • Posts: 25
Reply with quote
Post Posted: Fri Aug 12, 2022 6:29 am
Very interesting work!
i had just got and repaired an SF-7000 (making the missing I/O cassette cartridge from scratch) and used a gotek with flashfloppy as a drive, since the replacement belt for the 3" is not yet arrived).
The only disk image i can get working was the Basic "dsk" found on the net and dumped with CPCDiskXP.
I also get booting a compilation "SF7" disk adding the Basic sector 0, but it doesn't work after the initial menu.
So, a TOSEC compilation of Sega disks will be very appreciated, and i vote for the HFE or DSK format, so the gotek's user could play with them and write real floppies if they want.
  View user's profile Send private message
  • Joined: 05 Sep 2013
  • Posts: 3762
  • Location: Stockholm, Sweden
Reply with quote
Post Posted: Fri Aug 12, 2022 7:44 am
SF7 files can be converted to DSK format using the program found here
  View user's profile Send private message Visit poster's website
  • Joined: 25 Feb 2013
  • Posts: 384
  • Location: Osaka
Reply with quote
Post Posted: Wed Aug 31, 2022 10:18 am
Useful links to compare the available formats.

https://stardot.org.uk/forums/viewtopic.php?t=19676
https://www.msx.org/wiki/Emulation_related_file_formats

I will update the post if I find more. I have yet to find the "this is it" one. However, since the source is Pauline, maybe hfe could be a natural choice.
  View user's profile Send private message
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Post Posted: Wed Aug 31, 2022 10:25 am
We need a format which can carry the fact that some sectors are damaged and unreadable by the controller. I don’t know if DSK can do that (maybe it does, i haven’t checked).

We are going to have to release dumps with known damaged sectors, so the damage is properly encoded in the dump, rather than wipe out damaged sectors and lose track of it.
  View user's profile Send private message Visit poster's website
  • Joined: 25 Feb 2013
  • Posts: 384
  • Location: Osaka
Reply with quote
Post Posted: Wed Aug 31, 2022 11:31 pm
Bock wrote
We need a format which can carry the fact that some sectors are damaged and unreadable by the controller. I don’t know if DSK can do that (maybe it does, i haven’t checked).

We are going to have to release dumps with known damaged sectors, so the damage is properly encoded in the dump, rather than wipe out damaged sectors and lose track of it.


I totally agree. I believe DSK can represent damaged sectors. It is also based on the NEC765, so it is a good match (essentially, for each sector it reports the status bits the NEC765 would report).
The weak bits can be represented with an extension, as reported at the end of this page
https://www.cpcwiki.eu/index.php/Format:DSK_disk_image_file_format

I believe the "unwritable sector" protection scheme is the only one not covered by the format.
  View user's profile Send private message
  • Joined: 25 Jul 2022
  • Posts: 25
Reply with quote
Post Posted: Fri Sep 02, 2022 1:21 pm
kamillebidan wrote

I believe the "unwritable sector" protection scheme is the only one not covered by the format.


But are there disks with protection schemes? I only have a dozen made with "c" command of the sf7000 utility, sourcing from sf7 files found on the web, but none of them seems protected (I was thinking to write them via rs232 and DSKO$ basic command too! 😄) so curious in which disks are protected
  View user's profile Send private message
  • Joined: 17 Jan 2020
  • Posts: 118
  • Location: Brisbane, AU
Reply with quote
Post Posted: Sun Sep 04, 2022 2:54 am
I had an SF-7000 in the 80's, so I'm really interested in playing with the dumped disk images. I assume they're in a format that Meka can handle?
  View user's profile Send private message Visit poster's website
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Post Posted: Sun Sep 04, 2022 7:50 pm
under4mhz wrote
I assume they're in a format that Meka can handle?


Not yet. This is what will be working on.

The 163840 bytes .SF7 format we used on our first SF-7000 releases is not a good format for all types of releases.
  View user's profile Send private message Visit poster's website
  • Joined: 25 Jul 2022
  • Posts: 25
Reply with quote
Post Posted: Sun Sep 04, 2022 8:27 pm
Bock wrote

Not yet. This is what will be working on.

The 163840 bytes .SF7 format we used on our first SF-7000 releases is not a good format for all types of releases.


Some of the disks you mentioned seem really interesting (CP/M, SegaDOS, G-Basic...)!

In the meanwhile, couldn't you post some of theese in HFE format, to start studing them?
  View user's profile Send private message
  • Site Admin
  • Joined: 08 Jul 2001
  • Posts: 8644
  • Location: Paris, France
Reply with quote
Post Posted: Wed Nov 02, 2022 3:17 pm
My friend Nanard did some work on this and settled on adding support for the Extended DSK format as a new format for SF-7000 release.

See Burglar Bill which had an odd layout incompatible with earlier .SF7 format.

  View user's profile Send private message Visit poster's website
  • Joined: 25 Jul 2022
  • Posts: 25
Reply with quote
Post Posted: Wed Nov 02, 2022 3:58 pm
Good news! i hope Extended DSK format is supported by Flashfloppy, i'd like to test some disks from your huge collection with my SF-7000 and SF-7017!
  View user's profile Send private message
  • Joined: 25 Feb 2013
  • Posts: 384
  • Location: Osaka
Reply with quote
Post Posted: Thu Nov 10, 2022 12:33 am
Bock wrote
My friend Nanard did some work on this and settled on adding support for the Extended DSK format as a new format for SF-7000 release.

This is a very sensible choice!
Did you already open a work group?
  View user's profile Send private message
Reply to topic



Back to the top of this page

Back to SMS Power!