Forums

Seeing that the support is very low so far for this format, I guees there won't be many roms released in this format. Which means that someone has to update the converter database or the end user will have to select what ever he thinks is the correct settings for every new rom released.

FluBBa wrote

Are we talking about ROM releases of old commercial Sega games or what? You only need one Sega mapper for all commercial games, though that might go against the philosophy of the emulators goal.
What about new homebrew that makes use of the mapper in the ToToTek cart (as an example) or some new unknown Korean games, you would still have to update your converter and you emulator to support those games.

I'm talking about homebrew authors, existing software is already known and I have done a lot of games and their settings already. Future software authors on the other hand have no way of specifying to emulators that they want to use a joypad, paddle, sports pad, 3d glasses, light phaser, genesis 6button controller, mappers, nationality,etc.

If new mappers come out support will be added I guess, depending upon certain things. Ideally no more "crap" mappers should really be done by people, with what you can get these days having 128MB+ all "ram like" carts should be what someone strives to standardize on the hardware end. Ideally something like an interface between a USB flash stick and the SMS, with 8 bytes of mapper number registers, 2 bytes for each page with the cartridge only detecting the last byte being written as an update. You could then disable the RAM in port 3F and have 16KB of RAM paged in/out as desired at C000-FFFF and other regions.

I've already written a software implementation of this mapper (and have full port 3f emulation), but I need to talk to hardware people first to iron out all the details. It needs to be something cheap and easy to build whilst being flexible.

FluBBa wrote

The other part is what hardware to run the game on, but that is up to the user right? I can put my cartridge in a SMS1 or a SMS2, that is what can happen in real life but not in your emulator, or?

You have the choice of what "hardware" to run the games on, my interface simply selects the ones that will work for you by "default" based on the ROM header. At the moment you have NTSC/VDP1 or 2, PAL VDP1 or 2 since that's the only major hardware changes.

FluBBa wrote

And why force a new format when you could have made it compatible with every other SMS emulator by using PK zip format and just include an extra header file in the zip? Then I might actually consider supporting it on my SMS emulator for the DS.

PKZip is an old compression library, 7Z achieves better results. I also wanted nice checksums with at least 256bits (SHA256) by default and the game settings to be uncompressed so that the interface isn't slowed down or bloated dealing with weird scenarios. As it is now it's a simple check for size, read the checksum and settings, all within the first 256bytes of the file on disk for speed.

I'm not sure why this new format needs to be backwards compatible with _some_ emulators anyhow. The old roms are still there, existing, and my tool could even be used to get them back out of the .game files if people want them in the old format for old emulators. Anyone who is working on an emulator in this day and age could add support for the format within 10 minutes easily. Emulators for the DS and other underpowered systems aren't really the future of emulation imo, you have to implement a lot of hacks to get things to work on such systems (which is fine, because it's all about getting 100% speed).

The future of emulation as I see it is perfect emulators. Until you have a system that is perfectly emulated then the future is always ahead. People say you cannot get a perfect emulation but they are wrong. By perfect I mean that no software using every trick imaginable can ever detect whether it's an emulation or the real system across numerous real consoles, which is all that matters. I don't mean to diminish your DS emulator because I think projects like that are fantastic and they interest me greatly, optimization is always a fun game. Hopefully in the future a handheld will be fast enough for complete emulation.

As to why force it, well for what I have planned with RetroCopy there needed to be a change to achieve certain features and ease of use. If the format is only ever used for RetroCopy then that is fine because I need something like it for this project. Until people have used my emulator and can see the actual direction it is going in it is hard to understand why I did some of the things I did. Either way there was a severe gap in regards to emulators dealing with ROMs and I have bridged that gap in my emulator. I'm merely releasing everything in regards to it so if anyone else in a position like mine doesn't want to blow a week or two doing it they won't have to.

I do not really care if no one uses the format outside my emulator because it's completely irrelevant to me whether they do or don't as I have done everything I need myself already. Releasing it all is merely a gesture of good will rather than needing support. I have no problems forcing people to convert their ROMs, even if it means less users. I've already put high system requirements on it, so getting "maximum market penetration" isn't my goal. :)

FluBBa wrote

Seeing that the support is very low so far for this format, I guees there won't be many roms released in this format. Which means that someone has to update the converter database or the end user will have to select what ever he thinks is the correct settings for every new rom released.

Well my plan was originally to only convert roms from my emulator (as I require hard rom dirs), basically you put your .SMS files in there and it converted them all for you and you were off and running with minimal fuss. But then I thought maybe others might want to use it so I designed an open sourced tool and a way for people to create databases and check files.

Since no one has yet shown any interest I might just go back to my old model for now and simply ask anyone who is interested to email me for the details/tools if they want them. No point in supporting a tool nobody wants. :)

Eke wrote

I second the general opinion about a new ROM format, seems to me as a "control freak" feature (no offense) and a waste of time as it is more easily handled by an internal database (especially when this new ROM format is probably going to be only supported by this emulator) or by a customizable "cartridge emulator" (rom size, mapper type, etc... supported inputs being also configurable, there is no reason you could not try to plug a light phaser while playing alex kidd, even if it was not designed for ;-) )

It most certainly is a control freak thing. :)

If I can ensure games are in the best way they can be then I will have no future problems in regards to them. Others may like supporting 10 compression formats, 512byte headers, checksumming, caches, databases and things of this nature but I don't. I created a nice fresh slate to work from instead of dealing with the past.

Eke wrote

I think it will also brings lots of confusion among current ROM sets, adding ROM file versions when it's not required. I don't know, maybe you could add these informations as a trailer so the ROM header remains intact and the file compatible with other emulators ?

Not sure what you mean? .GAME files checksums may change (as a whole file), but the flat rom DATA in them should never change. You're probably thinking in the old terms of "no formats", .GAME files can be taken forward with minimal fuss if there are errors, the worst case if there is an error is wrong settings being used by default not "it doesn't work at all". Whenever you verify your romset it would simply correct any mistakes in the header and that would be it. I don't think the days of storing ROMs on non writable media are going to be the standard in the future, even the modern consoles realize this. ;)

That said there shouldn't be any errors in the ROMs, the format already covers everything needed for every single current SMS game, and likely every single future one. It covers weird things like having 4 ports to run ROMs (BIOS, EXPANSION, CARD, CART) and even piggybacking games (no SMS games use this but Genesis games do) that future software may employ. New mappers just mean a new enumeration. I will also be testing every game when I finalize my database and to certify the 100% claim for the release version, so it should have no flaws hopefully.

Good luck testing every code path in every game. If you can figure out how to do that in finite time then you'll be a rich man :)

PoorAussie wrote

I'm talking about homebrew authors, existing software is already known and I have done a lot of games and their settings already. Future software authors on the other hand have no way of specifying to emulators that they want to use a joypad, paddle, sports pad, 3d glasses, light phaser, genesis 6button controller, mappers, nationality,etc.

I hope you do know about the SDSC header in homebrew right? I guess that could be expanded to include everything that might be needed for controler input and such.

SDSC is more for getting credits/titles/dates/version numbers into homebrew that would often be otherwise hard to distinguish and/or attribute. It doesn't have any room for expansion really, unless you overloaded something (perhaps human- and machine-readable data in the comments section). But that would be hard to "enforce", since there's no requirement for anyone to fill in the info - it's hard enough to get people to change the SDSC tag in my BMP2Tile demo *cough*.

SDSC is also a half solution. You can't go over existing system roms and apply it (at least how it is at the moment). I also don't see much point in editing actual ROM data, seems messy to me.

Either way the goal of what I did was to make things simpler, which is why I chose not to support the old way of doing things, all my loading code is very simple and portable and has every feature anyone will want in regards to PLAYING games. Certainly there is room for more features in regards to metainformation that some may need but it really isn't needed for playing the files, when it comes to pure game related information for playing it I would challenge anyone to come up with a better way... certainly I would switch to it.

Also, there is no such thing as backwards compatibility because .SMS isn't a format it's a lack of one.

Quote

The future of emulation as I see it is perfect emulators. Until you have a system that is perfectly emulated then the future is always ahead. People say you cannot get a perfect emulation but they are wrong. By perfect I mean that no software using every trick imaginable can ever detect whether it's an emulation or the real system across numerous real consoles, which is all that matters. I don't mean to diminish your DS emulator because I think projects like that are fantastic and they interest me greatly, optimization is always a fun game. Hopefully in the future a handheld will be fast enough for complete emulation.

That's a respectful goal even if I doubt perfect emulation will ever happen, even with cycle accurate chip emulation (tricks relying on very accurate hardware latency timing for example). Emulating all commercial games flawlessly (glitches included) would already be a good step forward (playing these old games being the main reason people use an emulator after all).

Anyway, in my opinion, the future of emulation should be for highly accurate but also open-source AND portable emulators because:

1/ things does not last forever: people eventually move to other things and platforms are constantly evolving . I think it's more important to leave an accurate documentation on how these old console were internally working so that anybody could reproduce their behavior even when every single Master System or NES would be reduced into dust ;-)

2/ people should always be able to improve the code and compile it for any upcoming gaming platforms. What you think is accurate might be proven wrong or completed in the future. Some people can choose to optimize the code for a specific platform. The important thing is that they have the choice and a baseground to do it.

Quote

Either way there was a severe gap in regards to emulators dealing with ROMs and I have bridged that gap in my emulator.

huh ?
I never felt there was any gap regarding the way emulators are handling ROM files. Accurate ROM databases are very similar (but less intrusive) concept and I hardly see why a new ROM format is better ?

Quote

. It covers weird things like having 4 ports to run ROMs (BIOS, EXPANSION, CARD, CART) and even piggybacking games (no SMS games use this but Genesis games do) that future software may employ.

Sorry but again I don't understand the benefits of using a new ROM format for such thing ?
About the port $3E thing (this is what this is about, right ?), you only need to emulate it correctly and could simply have various options to load ROM (load from cart slot, load from card) or to connect specific hardware (having its own emulated model off course) to the expansion slot, why should it be tied to a specific game software ?

If the purpose of your emulator is to accurately replicate the console behavior, it should not have any idea about the game species but on the contrary let the user configure it as waned, like he would have done in front of a real console, this is way more flexible in my opinion.

Same things goes for cartridges that have the ability to connect themselves to other cartridges (like game genie or sonic&knuckles), it's more flexible to have an option to virtually "connect" two cartridges together. In this case, I agree that the way the connection works is specific to each game but as you can't have a generic emulation model anwyay and need to implement each known banking scheme separately, having the description in the ROM format or letting the user choose a connection model is strictly the same.

Sorry for the long post and my approximate frenglish, but this is indeed an interesting discussion

This seems more like an emulation project to lock-in a user base. A closed source emulator with a proprietary format that the author boasts about its accuracy and features. It seems like the boasting is to lure in users and you don't seem to plan to share your information on the hardware. We know people love accuracy, or else you wouldn't be boasting about it (I assume in hopes that people will flock to your emulator). Now, your defense mechanism is to say that you're targeting a niche group... but really? That doesn't make up for closed source either. You argued quite extensively with byuu about his SNES emulator but he does everything he can to give back to that community and then some. This project seems like the exact opposite from the start and that's kind of weird for the SMS/GG community since everyone shares everything openly here.

Just a personal observation. The whole premise of this emulator seems very strange. Boasting accuracy and 3D boxes to force a closed source emulator with a proprietary format into this community (and the NES community which is just as open)? Icky.

Firstly maybe some here do not understand certain things about what I need.

1) When a user clicks on a file I need to display game settings which work with it best.
2) When a user clicks on a file I need to load other data like covers, save states, etc

You can achieve the first one relatively easy with a database. The second requires you to keep another database linking SMS/ZIP/RAR files to a checksum. This is messy.

Finally I wanted EVERY game in my hard linked rom folder to be compressed, and I did not want to have to add support for at least 3 different compression schemes that are popular today, ZIP, RAR and 7ZIP.

The easiest way to do all this is to have a new format, which has all the features you need with the best compression. There is no debate on this for me, because I have done it both ways and I know it for certain which is the best. If anyone thinks they can support .SMS/ZIP/RAR/7ZIP combined with databases and caching files in a few lines of code, then please show me.

Eke wrote

About the port $3E thing (this is what this is about, right ?), you only need to emulate it correctly and could simply have various options to load ROM (load from cart slot, load from card) or to connect specific hardware (having its own emulated model off course) to the expansion slot, why should it be tied to a specific game software ?

Knowledge about which port the game goes is obvious when you are manhandling the game. If you are holding a card game you know it goes in the card slot. There may be things in the future which take advantage of the card/exp/bios slots and expecting the user to look at a .SMS file and know is a bit silly.

Of course having the option to load things in other slots is left up to emulators to suppor, it has nothing to do with the format telling you which slot it SHOULD go in. :)

Eke wrote

If the purpose of your emulator is to accurately replicate the console behavior, it should not have any idea about the game species but on the contrary let the user configure it as waned, like he would have done in front of a real console, this is way more flexible at my opinion.

There is a difference between a raw file on a harddrive (like the .SMS file) and a game you buy. These differences include information that is important to the successful playing of the game and really should be there.

olaf wrote

Just a personal observation. The whole premise of this emulator seems very strange. Boasting accuracy and 3D boxes to force a closed source emulator with a proprietary format into this community (and the NES community which is just as open)? Icky.

Well this is just ignorance. I have said multiple times I plan to release information about the hardware that I have learned once I have finalized my emulator. I'm not about to start releasing details when there are glitches in several titles that should not be there because I haven't quite figured out the exact timing of the CRAM write delays or because I haven't tracked down every last bug yet.

Either way everything I've learned is through trial&error which makes it even harder to accurately tell people the timing of things, I can just give my best ideas and what I have done. Maybe you could tell us all why you haven't written a cycle accurate open sourced VDP and posted all the details for others yet demand it from me? Your post is offensive to me.

PoorAussie wrote

Either way everything I've learned is through trial&error which makes it even harder to accurately tell people the timing of things, I can just give my best ideas and what I have done. Maybe you could tell us all why you haven't written a cycle accurate open sourced VDP and posted all the details for others yet demand it from me? Your post is offensive to me.

Because I haven't bragged about mine or written one and I didn't demand it. If anything I danced around the idea that you should make yours open source. But okay.

P.S. I guess I missed the part in this thread where you talked about releasing information of any sort other than vaguely stating that at some point you may make it open source like MEKA.

This thread's getting a little warm :)

Most SMS emulators support only .SMS and zipped .SMS (not even multiple files per zip). Of course people will ask for RAR and 7z, and the next format, but the status quo is such that you will make almost everyone happy if your emulator works on their existing files, and they are .SMS files, maybe in a zip wrapper.

There are two classes of metadata you need: emulation info (for example, game X needs a VDP1 with NTSC timing; can run with either Japanese or Export region settings; uses a light phaser; has battery-backed RAM) and user info (cover scans, manuals, cheats, screenshots, magazine reviews...). The former is somewhat static, and is generally implemented in the form of a "database" (albeit a simplistic one). The latter is less static (for a given game it will be updated a lot more often), and is generally not implemented; the only case I can think of (in the emulation realms I deal with) is FreezeSMS, which constructs a filename from the data in its database and then looks for a metadata archive with all the info in it. This allows one to update the metadata independently of the ROM images, and to easily (legally) distribute the metadata. Some ROM auditing tools also provide metadata like region, publisher, covers and screenshots, again based on a central DB which refers to filesystem objects (for images) which are retrieved over the internet.

(Aside: one of the aims of SMS Power, when we are all 90 years old and free to work on it, is to have a large database of all information youc an imagine on every game, and to expose this in some way to emulation, perhaps through generating DB files, a web service... or even generating .game files.)

olaf wrote

Because I haven't bragged about mine or written one and I didn't demand it. If anything I danced around the idea that you should make yours open source. But okay.

P.S. I guess I missed the part in this thread where you talked about releasing information of any sort other than vaguely stating that at some point you may make it open source like MEKA.

You said and implied several incorrect things about my emulator and myself, so let's not sugarcoat it. If you call alerting people to a free emulator they will be able to use in a few weeks bragging then I think you need a reality check. Exactly what do you think I get from spending over a year of my life doing this for free? The ability to brag on forums about how great it is? If that's all I get then you should really be quiet shouldn't you because you are interrupting it with your ignorance.

I can do what I want with my work and information. If I release it to the public domain it is up to me. The fact that I am very careful to make sure I credit and thank everyone who has helped me is a sign I understand the sacrifice others have made for people like myself, and hopefully other people, unlike you, will realize and appreciate the sacrifices I have made to do what I've done.

Maxim wrote

(Aside: one of the aims of SMS Power, when we are all 90 years old and free to work on it, is to have a large database of all information youc an imagine on every game, and to expose this in some way to emulation, perhaps through generating DB files, a web service... or even generating .game files.)

I would expect by this time you could play cycle accurate games through your web browser on a holographic screen. Plus the copyrights will have expired by then, I believe. ;)

It would be cool having a museum with everything in it though, it's a good goal for the site.

PoorAussie wrote

You said and implied several incorrect things about my emulator and myself, so let's not sugarcoat it. If you call alerting people to a free emulator they will be able to use in a few weeks bragging then I think you need a reality check. Exactly what do you think I get from spending over a year of my life doing this for free? The ability to brag on forums about how great it is? If that's all I get then you should really be quiet shouldn't you because you are interrupting it with your ignorance.

I like how you get so angry in your posts they lose direction for a paragraph.

PoorAussie wrote

I can do what I want with my work and information. If I release it to the public domain it is up to me.

I wholeheartedly agree, but as I stated in the previous posts you've only vaguely stated that you may make it open source. If it isn't, that's kind of silly to do for a community that gave you so much information to work with. But again, as you said, you can do what you want with your work and information.

PoorAussie wrote

The fact that I am very careful to make sure I credit and thank everyone who has helped me is a sign I understand the sacrifice others have made for people like myself, and hopefully other people, unlike you, will realize and appreciate the sacrifices I have made to do what I've done.

I'm sorry, I didn't know you were Jesus Christ himself. I'm pretty sure last time I checked there was a page dedicated to the people who just even wrote documentation I read while working on my/brom's emulator (as well as the contributors list displayed on every page).

People here need to chill out a bit. :)

To be honest, I also don't think this "new format" is a good idea as well but its PoorAussie's emulator and he can do whatever he wants. And we should really wait and see what he does. What if he proves us all wrong? What if maybe its a better format? I think its a bit childish to attack PoorAussie even before giving his work a fair try.

I think PoorAussie deserves a full chance to do what he wants to do just like all the rest of us. We chose database/whatever, he is choose to do his own format, whats wrong with that?

Also, its not like the end. If you don't think its a good idea, go code your own emulator which does what you want (heck, I have, and pretty much every one discussing here is a SMS emu author :P). So I don't really see why people are so emotional on this.

Though, I would request that PoorAussie shares his findings. So some of us can also make/update our stuff with more accuracy. I also chose closed-source path, but I do share my findings with everyone. So after my request of current ROM format got rejected, I hope this one doesn't. ;)

stay safe,

AamirM

Out of interest, where are savestates and savegames stored? In the .game file?

First, I applaud PoorAussie's calmed and reasoned responses to the posts in this thread, some of which has been pretty rude.

Secondly, I look forward to try out your emulator, PoorAussie. Cycle accuracy sounds very interesting, and the news of this emulator should, in my opinion, have spurred responses of anticipation and joy - not a boring discussion about file formats and metadata. Furthermore, your emulator looks visually stunning and very professional.

Thirdly, in relation to the relatively uninteresting metadata discussion, I must admit, that I can not see the big problem with PoorAussie's new format. Isn't it a big advantage that homebrewers can directly present relevant information to the emulator instead of updating a centralized database file? PoorAussie's solution is decentralized. Decentralized solutions are nice.

Even if PoorAussie choose not to publish documentation or source code, the very existence of a cycle accurate emulator is a great help to the scene. Not only for playing games, but also for homebrewers wanting to test their code on something closer to the real system.

Maxim wrote

Out of interest, where are savestates and savegames stored? In the .game file?

Nah, why would you think that? :)

My basic structure at the moment is something like this :-

\Retrocopy.exe
\SMS\ (.game files)
\SMS\covers
\SMS\states

I'm not sure if I will end up supporting "savegames" like traditional emulators do. With savestates it is sort of easier forcing people to do it in one manner for every game. Having a default "last game" state that is saved on exit might be the way to go.

MarcK wrote

First, I applaud PoorAussie's calmed and reasoned responses to the posts in this thread, some of which has been pretty rude.

Secondly, I look forward to try out your emulator, PoorAussie. Cycle accuracy sounds very interesting, and the news of this emulator should, at my opinion, have spurred responses of anticipation and joy - not a boring discussion about file formats and metadata. Furthermore, your emulator looks visually stunning and very professional..

Thanks for your kind words and support. The psychology of humans means a lot of people will look upon change as something to fear, after time though something isn't so new and fearful anymore though so it's easier to accept. There is plenty of software that requires a "preprocessing" of files before it will work, and my emulator is exactly the same. It seems my downfall may have been in making people believe I was trying to take over the scene with the new format or something by releasing open source tools for conversion. I think after some time the whole issue of the ROM format will calm down, just right now it is a talking point for the above reasons.

olaf wrote

I wholeheartedly agree, but as I stated in the previous posts you've only vaguely stated that you may make it open source. If it isn't, that's kind of silly to do for a community that gave you so much information to work with. But again, as you said, you can do what you want with your work and information.

I'm not sure what open source has to do with information being released. If you had bothered to actually read my posts before jumping to conclusions (not just in this thread) you would have seen I have posted I will be releasing my findings once I have finalized them. Hardware documents make better reads than source files do, because even then people like yourself that may not understand C++ code will be able to understand english, even if it may be a mangled mix of aussie.

For you to come in and then state the direct opposite is willful ignorance.

olaf wrote

I'm sorry, I didn't know you were Jesus Christ himself. I'm pretty sure last time I checked there was a page dedicated to the people who just even wrote documentation I read while working on my/brom's emulator (as well as the contributors list displayed on every page).

http://www.smspower.org/forums/viewtopic.php?t=9240

I notice when you bragged (this is what you call notifying people right? :) ) about "your" emulator on this forum some people criticized and insulted you no good reason, and instead of learning from it you repeat similar behaviour in another emulator thread? Either way I find it highly hypocritical that people like yourself get on a high horse in regards to information being released when you have released none yourself (I do not consider rewriting hardware documents into various other languages information). Even though you were incorrect in regards to me releasing my findings if I wasn't releasing anything you don't really have any leg to stand on to say the things you did. Leave that to people who do, they don't need a champion for their cause.

I think that most of the points have been mentionned (in varied manned) I suggest release unnecessary pressure on PoorAussie and let him concentrate on working on this good software for now. None of the issue can't be solved later.

PoorAussie wrote

Maxim wrote

Out of interest, where are savestates and savegames stored? In the .game file?

Nah, why would you think that? :)

It is a progression of the ".game file = cartridge" abstraction - after all, I could bring my cartridge to a friend's house and play on his console with my savegame. I had the impression you were storing the covers in there too (making it ".game file = complete game") - if not, how do you know which file to load?

The idea of only having one savestate is a lot like the Wii emulators' interface, which you may not be familiar with. The only saving you can do is either any in-game save, or just shutting down and resuming at the same point later. The abstraction of including the entire emulator with each emulated game is perhaps a step too far, though...

PoorAussie wrote

There is a difference between a raw file on a harddrive (like the .SMS file) and a game you buy. These differences include information that is important to the successful playing of the game and really should be there.

Ok, I think I finally got your main idea now, what you want to achieve is some kind of game box "simulation", right ?
A card box that includes all the common informations stuff that usually fits on it (supported peripherals, number of players, genre, title, screenshots, manual, etc) as well as a cartridge "model" (ROM data, backup RAM data, mapper type,...), like a real card box would do.

I was misleaded by this whole "let's make a new ROM format" mention but it seems it's more a way for you to put all these informations into a single archive, and seems logical to me (and fine, as long as original ROM data file is not altered)

anyway, this is an interesting (and indeed very perfectionnist) concept, I guess we should all wait for concrete examples before critizising ;-)

MarcK wrote

Even if PoorAussie choose not to publish documentation or source code, the very existence of a cycle accurate emulator is a great help to the scene. Not only for playing games, but also for homebrewers wanting to test their code on something closer to the real system.

I disagree with this...

What the point of having a super accurate emulator when you might not be able to use it anymore in a few decade ?
Technologies change and devs move away but documentation & shared informations remain. The whole idea for an emulator is about preservation and the best way to achieve it is to publically share and release informations.

Closed-source is a different thing and I strongly respect this choice, developpers should be able to do whatever they want with their own code and to protect it if they think it's necessary.
But also remember that many closed-source emulators (or applications in general) would not have been made without other people sharing their own code and findings, I know very few people that can honestly say 100% of their programs is solely their work :-)

Bock wrote

I suggest release unnecessary pressure on PoorAussie and let him concentrate on working on this good software for now. None of the issue can't be solved later.

I guess now the username makes sense, if ya'll know what I mean... (In case you do not know what I mean, I intended it being a joke)

Maxim wrote

It is a progression of the ".game file = cartridge" abstraction - after all, I could bring my cartridge to a friend's house and play on his console with my savegame. I had the impression you were storing the covers in there too (making it ".game file = complete game") - if not, how do you know which file to load?

I still haven't quite decided the best way to do COVER linkage yet. At the moment I simply have filenames which are the hash of the ROM data, however I want to be able to include multiple covers (different regions, etc) which leads to an issue with this method. Plus hash filenames are also ugly looking in the folder.

My current thought is to link them with a single XML file in the COVERS directory since it's almost as simple. But this makes it slightly harder to share them. unless you ship off your XML file with them. Another way might be to make a file format which holds multiple JPEG files, or a XML/text file with the name as the hash which then links to jpegs. Do you have any thoughts? It kinda boils down to how the covers will be shared also I guess, will someone just make complete packs and host them for others? I'm not sure yet.

Maxim wrote

The idea of only having one savestate is a lot like the Wii emulators' interface, which you may not be familiar with. The only saving you can do is either any in-game save, or just shutting down and resuming at the same point later. The abstraction of including the entire emulator with each emulated game is perhaps a step too far, though...

Is it better like that? I haven't had the luxury of using the Wii , but I've heard positive things about the way it handles emulation. Does the Wii support multiple save slots or just one? I kind of like the idea of persistence with save states anyhow, and you could still do it with slots. ie If you have 4 slots and select slot 3, it remembers slot 3 everytime you load the game until you change slots again. Another way might be one persistent slot, and one save anywhere slot that is only temporary. So basically you keep with the idea of persistence but allow them to "quick save" at the beginning of the level and "quick load" if they die.

Though being able to rewind gameplay makes it almost impossible to die if you want to use it.... allows you to beat games like Back to the Future 3 which rely on randomness and twitch reflexes for the first level extremely easily. Takes the challenge out of games, but it's very helpful for testing and seeing areas you might not be able to reach otherwise.

PoorAussie wrote

At the moment I simply have filenames which are the hash of the ROM data

For FreezeSMS I had a map of hash to game name, so the filenames are readable. But then you have a central DB and no fallback...

PoorAussie wrote

I want to be able to include multiple covers (different regions, etc) which leads to an issue with this method.

I'm a big fan of convention over configuration. So how about a zip file (or 7z, if you must - with JPGs there's not much compression to gain) that contains files with a standard naming convention, for example <side>_<region>_<index>.<ext>, e.g.

front_eu_1.jpg
back_eu_1.jpg
back_eu_2.jpg
front_us_1.jpg
back_us_1.jpg
spine_us_1.jpg

or something to that effect. (Or does it slice entire scans on the fly?) You may need to handle the cases where there are cover variations within a region, and also where there are only incomplete scans of some variations (perhaps a fallback scheme). How is the interface for this going to work?

PoorAussie wrote

XML

...and now you have two problems.

(I'm not a huge fan of XML.)

PoorAussie wrote

will someone just make complete packs and host them for others? I'm not sure yet.

Given the size of the data (multi-GB for all variations at reasonable quality) it may end up in torrents.

PoorAussie wrote

Maxim wrote

Wii emulators' interface

Is it better like that? I haven't had the luxury of using the Wii , but I've heard positive things about the way it handles emulation. Does the Wii support multiple save slots or just one?

It's generally restrictive but it forces you to play the game more like it was intended. There's no choice to load or save, it effectively saves state on shutdown and loads state on startup. So you can effectively take very long pauses, which I remember doing back in the day (leaving the console powered on and paused for long periods as I couldn't play continuously). If the game has a save system then you can use that, of course.

If you have n save slots then someone will find a use for n+1, until n is extremely large. For example, they may choose to have a "preserved" save for each level, and/or each boss. Some games have dozens of levels.

Save state ideas:

1. Include screenshots in savestates for easier identification (with some effort you can recreate the screen from the VDP state, but a screenshot is easier)
2. Allow people to choose a textual title (to help identification)
3. Store the game playtime in the state (ApprenticeMinusDS does this, I find it interesting and it helps to tell how far through the game it is)
4. Allow marking saves as "read-only", for those saves you want to preserve (I've lost saves I wanted to keep because of this)
5. If you have a great rewind UI (not just a single-speed rewind button) then quickload/save is not so necessary

The TAS kids will probably also have some feature requests in this area.

How long is the rewind buffer? It must have some bound. Does it survive across state loads?

Maxim wrote

or something to that effect. (Or does it slice entire scans on the fly?) You may need to handle the cases where there are cover variations within a region, and also where there are only incomplete scans of some variations (perhaps a fallback scheme). How is the interface for this going to work?

It slices entire scans. If there are incomplete ones people can just photoshop something in if they want. All they have to do is maintain the correct ratios and side width, exactly like a real world wrapping.

The interface is simply going to show multiple 3d covers all at once (similar to what you see in the video, except if there are 4 covers it will be zoomed out more to show 4 rotating boxes that you can zoom in on, etc. Likely horizontally placed. So naming of the jpeg files is irrelevent.

I'm actually thinking I will just do a text file that is the SHA256 HASH, and in that text file just put in jpeg files upto maybe 8, separated by newline. This way any novice can simply append things which is easy in a text editor.

sonic1_usa.jpeg
sonic1_usabonus.jpeg
sonic1_eur.jpeg
sonic1_eur02.jpeg

Maxim wrote

...and now you have two problems.
(I'm not a huge fan of XML.)

Neither was I for a while! But with something like tinyXML it is fairly easy to add support for it and relatively easily to work with it. My GUI themeing is done in XML, though my emulator likely won't allow external themes.

Maxim wrote

Given the size of the data (multi-GB for all variations at reasonable quality) it may end up in torrents.

If the jpegs are 250KB, then you're looking at only ~100MB for SMS, if you say there are 4 variations (high assumption) that's only 400MB. Depends really on the size of each file. Going over 250KB isn't going to get too much added quality since monitors do not have enough resolution for it to matter... yet.

Maxim wrote

How long is the rewind buffer? It must have some bound. Does it survive across state loads?

It could be any length, atm due to not wanting to blow the memory budget I've got it set for like 10 seconds. Each SMS state is about ~50KB or so, assuming PAL it's 50x10x50KB, or roughly 25MB of memory. 'Course you could optimize it somewhat, but since I'm assuming a 2GHz single core processor as min spec using even 500MB of memory for such a feature is no big deal.

Loading states or doing anything which interferes with the current state will reset the buffer so to speak, there is no way to avoid that.

One thing a lot of websites do is to use numeric IDs for finding data and then to put extra human-readable stuff in the URLs, for SEO and usability. You could do the same thing, by having the program look for filenames that contain the hash; so, for example, you could have

Sonic The Hedgehog (0123456789ABCDEF00000000000000000123456789ABCDEF0000000000000000).ext

...although a 32-byte hash is rather long. (Base64 encoding will get it down to 44 chars.) Alternatively you could have the hash stored in the file and scan the directory on startup to build a map of hash to filename.

I still think "appending" JPEGs into a zip is just as easy, and tidier, than a text file.

Almost all scans in our scans section are a maximum of 250KB at screen resolution. There's just that many variants. A quick calculation suggests we have about 285MB of unique SMS game-box-only scans, and many of those are at low resolutions.

With the rewind, you could record at a much lower rate - maybe four per second, or maybe with increasing intervals at more distant times. I don't see the need for 50fps rewinding. I have in mind a "film strip" interface for rewinding, where you see the buffer as a series of frames which you can scroll through until you see where you want to be. Think of it as DVD chapter selection as opposed to winding a VHS tape back and forth looking for something.

I think you would need rewinds much more frequently. It could just be from watching Mario hard hacks (example: ZSNES with 10-frame intervals, I think) and has proven troublesome, as it takes several rewinds to get to a point where you have sufficient reaction time (falling into a hole, running into a trap before you can tell where you are. That problem could still hold true with most action games in general.).

PoorAussie wrote

It could be any length, atm due to not wanting to blow the memory budget I've got it set for like 10 seconds. Each SMS state is about ~50KB or so, assuming PAL it's 50x10x50KB, or roughly 25MB of memory. 'Course you could optimize it somewhat, but since I'm assuming a 2GHz single core processor as min spec using even 500MB of memory for such a feature is no big deal.

What about storing full-save state as "key frames", for example 1 or 2 a second, and between those key frames you just store inputs.? Can emulation be fastened and input fed to recreate the intermediary states easily ? (probably a performance problem in the case of your emulator)
You could use hybrid approach, store full states for x seconds and start dropping intermediate states in a non-linear manner for older events (and keep the inputs). A little tricky to implement but that would be the ideal solution.

Bock wrote

What about storing full-save state as "key frames", for example 1 or 2 a second, and between those key frames you just store inputs.? Can emulation be fastened and input fed to recreate the intermediary states easily ? (probably a performance problem in the case of your emulator)
You could use hybrid approach, store full states for x seconds and start dropping intermediate states in a non-linear manner for older events (and keep the inputs). A little tricky to implement but that would be the ideal solution.

If memory was tight it would be likely I would have to resort to such things. However the CPU is being burdened much more than the memory is in my emulator. Even with a full 10-11 second rewind buffer, 3d gui, opengl, etc, its only using about 65MB of memory all up. ~20MB of that is opengl and another 20MB is my gui textures/windows.

A PC that has a 2Ghz processor would have at least 1GB of memory ,so worrying about a fullscreen app using even 512MB isn't a worry, atm I'm only at a tenth of that on average. If the memory is there you might as well use it. :) My emulator will try and allocate 256MB of physical memory on startup just to ensure a relatively high end machine is running it, it's a way I plan to cut back on support.

And in regards to rewind fps, you really need at least 25/30fps on rewind for it to be effective. Sometimes it is hard coordinating it even at 50/60fps let alone less. Since memory is the only concern and I'm nowhere near a high usage it isn't really a problem imo. One of the advantages of targeting the top end of computers....I guess ;)

I really disagree with your approach.
Introducing new useless formats?
Targeting high end systems for no reason?
Breaking compatibility?

Hell, you could as well work for Microsoft at this point. Are you sure you weren't in the team that developed Vista?

Chuck Norris wrote

Introducing new useless formats?

Well they aren't useless to my design, maybe only to others.

Chuck Norris wrote

Targeting high end systems for no reason?

To cycle emulate the Master System you need a high end machine, so it's not for "no reason". Anyone could upgrade their machine for ~$150 and have RetroCopy run flawlessly on their computer. People who cannot afford $150 and are running old computers.... well they won't have the internet will they. Master Systems used to cost about $200, so to get an extremely accurate emulator running (though with more features) will cost roughly the same.

There is way too much whinging about system requirements on emulator forums, it isn't expensive to upgrade or buy a decent computer these days. And if you want good accuracy you need it, if you don't want good accuracy you don't need it. Simple.

Chuck Norris wrote

Breaking compatibility?

Compatibility with what? Only thing an emulator needs to be compatible with is original software, which it is.

Chuck Norris wrote

Hell, you could as well work for Microsoft at this point. Are you sure you weren't in the team that developed Vista?

Funny :), kinda good analogy too. MEKA runs near 100% of existing software for Master System and it works on a Pentium 200 like Windows XP. My emulator on the other hand claims 100% compatibility and needs ~2GHz and a video card to do it like Vista. If you were a layman it might not seem worth it, so I can understand that viewpoint. Still, if you don't see any advantage about cycle accuracy maybe you are not educated well enough on the matter.

Maxim wrote

One thing a lot of websites do is to use numeric IDs for finding data and then to put extra human-readable stuff in the URLs, for SEO and usability. You could do the same thing, by having the program look for filenames that contain the hash; so, for example, you could have

Sonic The Hedgehog (0123456789ABCDEF00000000000000000123456789ABCDEF0000000000000000).ext

Yeah we think alike it seems. I might go this route and just rely on the underlying OS to be fast for file searching. Best of both worlds that way.

Maxim wrote

...although a 32-byte hash is rather long. (Base64 encoding will get it down to 44 chars.) Alternatively you could have the hash stored in the file and scan the directory on startup to build a map of hash to filename.

You can reduce checksums somewhat, take every 2nd or 4th byte to form a smaller version that is usually just as secure (as an equally matched bit size) provided the checksum is spread evenly over the bits, which good ones are.

Maxim wrote

I still think "appending" JPEGs into a zip is just as easy, and tidier, than a text file.

One problem is sometimes compressing jpegs results in slightly larger files, but it would only be a few percent difference. I guess it's just more optimal on the CPU not having to decompress upwards of a MB every click. I'll have to dwell on this for a bit longer because it is tidier just packing them into a single file. Maybe a tar?

PoorAussie wrote

One problem is sometimes compressing jpegs results in slightly larger files, but it would only be a few percent difference. I guess it's just more optimal on the CPU not having to decompress upwards of a MB every click. I'll have to dwell on this for a bit longer because it is tidier just packing them into a single file. Maybe a tar?

Zip supports no-compression storage which makes it like a tar but 1000x easier to use for the average user. Good tools compare the compressed size to the zipped size and store uncompressed when it's smaller.

Decompressing it will likely be IO-bound rather than CPU-bound. A quick test on a bunch of JPEGs suggests I can "extract" 185MB in about 1.5s when it's stored uncompressed, and in about 2.5s when it's "maximally" compressed in a Zip.

Maxim wrote

Zip supports no-compression storage which makes it like a tar but 1000x easier to use for the average user. Good tools compare the compressed size to the zipped size and store uncompressed when it's smaller.

Could you explain 1000x easier? I thought most windows based archivers supported TAR? When I right click with 7Zip I can select ZIP/7Z/TAR/etc. And I'm pretty sure when I used Winrar it supported TAR also. In those programs there is no visual difference between TAR/other archives. TAR is a lot easier to handle manually in C/C++ without including the ZIP library.

PoorAussie wrote

Could you explain 1000x easier?

Newer Windowses comes with built in support for .zip files?

Out of the box, Windows supports zip and cab archives. WinZip (which sucks, but still seems dominant/common) can only add files to zips. Other, better software (I'm partial to WinRAR myself, although KZIP makes pleasingly insane compression attempts) does better but to the average person, installing 7-Zip and changing the necessary settings to work with TARs is very hard.

Maxim wrote

Out of the box, Windows supports zip and cab archives. WinZip (which sucks, but still seems dominant/common) can only add files to zips. Other, better software (I'm partial to WinRAR myself, although KZIP makes pleasingly insane compression attempts) does better but to the average person, installing 7-Zip and changing the necessary settings to work with TARs is very hard.

Good point, but are people who are capable of putting together these cover files the type of people who wouldn't be using something better than the default windows setup? Any self respecting nerd that can use Photoshop (and hence putting collections together) has something which can deal with the common compression formats... ;)

Average joe just has to place files in the right directory, not mess about with the files inside the archive. The other factor is if it's ZIP then it has the chance to be compressed mistakenly if the user isn't aware of all the issues, TAR is assured of being uncompressed and faster, it's a standard with tools already well developed (compared to something I invent), and it's easier to support than ZIP or RAR on the backend (easy header walk). Since I need all the data from the file there is zero penalty with it's format as I can load and read it entirely into memory and use offsets from a single allocation to display the JPEGs. ZIP on the other hand would require a lot of work on my part to support unless I use and include their horrible API.

So in summation, I can't see many benefits going with ZIP.... ;)

I know that the ZIP format isn't very friendly from the start but it's not that hard to implement either.
http://ndsretro.com/download/GreenBeretDS03Src.zip
I made a few functions for reading data from zip files, I bet you can fix it up quite a bit if you want to help out.
It's 5kB of file handling and 37kB for the INFLATE decompression (commented source that is).

I don't think he's bothered about small code size. A zip file library is not hard to come by, even if ZLib doesn't make it simple.

For those that are interested (probably not many here) there is a new video of the NES emulation in RetroCopy here :-

I have some more interesting SMS ones to show soon, RetroCopy will be released within the next few days, it won't be as perfect as I'd like it to be but it never really is when it comes to releasing something.

What startup values/pattern are you using for RAM on the NES? Or are you just clearing it? (most of the time the broken version of SMB starts in the minus world if emulated correctly).

FluBBa wrote

What startup values/pattern are you using for RAM on the NES? Or are you just clearing it? (most of the time the broken version of SMB starts in the minus world if emulated correctly).

Just clearing the RAM to zero. What is the "broken version" you speak of? I have noticed a few SMBs which have weird behaviour, but I thought they were hacks.

PoorAussie wrote

Well there will be an option so you can start in fullscreen mode. The plan is to have save states showing, and other little things in the TV mode that you can select. It's also to add to the classic feel. I replayed some games simply because they were a bit harder with it on an angle (like a real TV perhaps) and it's kind of mesmerizing playing it on the "emulated TV" for some reason. Due to specific features I will be adding in the future, the "TV mode" is there to stay though.

Sorry if you've already covered this, but it would be kinda cool if the TV was a 3D model and when you went to fullscreen mode, it rotated and zoomed in on it. Of course, with a 3D model, you could choose your own viewing angle, but then that would involve fancy graphical effects to emulate CRT viewing angle distortion.

on that note, since you're going all crazy with the "high-end" hardware, any chance of CUDA/OpenCL usage?

Also, have you thought about supporting 3D via nVidia's thingo?

unfnknblvbl wrote

Sorry if you've already covered this, but it would be kinda cool if the TV was a 3D model and when you went to fullscreen mode, it rotated and zoomed in on it. Of course, with a 3D model, you could choose your own viewing angle, but then that would involve fancy graphical effects to emulate CRT viewing angle distortion.

Some others have mentioned this, and it is a good idea. At the moment I haven't gone overboard with the 3D stuff since it's a little crazy doing so much on my own.

unfnknblvbl wrote

on that note, since you're going all crazy with the "high-end" hardware, any chance of CUDA/OpenCL usage?

OpenCL for the actual emulation or some other thing like filters? I was using Opengl shaders for filters early on the project before throwing them away. I haven't looked too hard at OpenCL/etc but it's possible they could be used. I'm not sure if they would be suited for emulation, but maybe they would. Either way until the standard really hits it isn't something I want to look at too indepth.

Standard 2-3 year old CPUs are fine for what I do currently. When it comes to SNES/GENESIS a little bit more may be needed. Until the "2-3 year old CPU" can't emulate the things we need maybe OpenCL will be used more. I expect someone will throw out a test soon enough though.

Author	Message
FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	Posted: Tue Jun 30, 2009 1:49 pm
FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	Seeing that the support is very low so far for this format, I guees there won't be many roms released in this format. Which means that someone has to update the converter database or the end user will have to select what ever he thinks is the correct settings for every new rom released.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 2:04 pm
	FluBBa wrote Are we talking about ROM releases of old commercial Sega games or what? You only need one Sega mapper for all commercial games, though that might go against the philosophy of the emulators goal. What about new homebrew that makes use of the mapper in the ToToTek cart (as an example) or some new unknown Korean games, you would still have to update your converter and you emulator to support those games. I'm talking about homebrew authors, existing software is already known and I have done a lot of games and their settings already. Future software authors on the other hand have no way of specifying to emulators that they want to use a joypad, paddle, sports pad, 3d glasses, light phaser, genesis 6button controller, mappers, nationality,etc. If new mappers come out support will be added I guess, depending upon certain things. Ideally no more "crap" mappers should really be done by people, with what you can get these days having 128MB+ all "ram like" carts should be what someone strives to standardize on the hardware end. Ideally something like an interface between a USB flash stick and the SMS, with 8 bytes of mapper number registers, 2 bytes for each page with the cartridge only detecting the last byte being written as an update. You could then disable the RAM in port 3F and have 16KB of RAM paged in/out as desired at C000-FFFF and other regions. I've already written a software implementation of this mapper (and have full port 3f emulation), but I need to talk to hardware people first to iron out all the details. It needs to be something cheap and easy to build whilst being flexible. FluBBa wrote The other part is what hardware to run the game on, but that is up to the user right? I can put my cartridge in a SMS1 or a SMS2, that is what can happen in real life but not in your emulator, or? You have the choice of what "hardware" to run the games on, my interface simply selects the ones that will work for you by "default" based on the ROM header. At the moment you have NTSC/VDP1 or 2, PAL VDP1 or 2 since that's the only major hardware changes. FluBBa wrote And why force a new format when you could have made it compatible with every other SMS emulator by using PK zip format and just include an extra header file in the zip? Then I might actually consider supporting it on my SMS emulator for the DS. PKZip is an old compression library, 7Z achieves better results. I also wanted nice checksums with at least 256bits (SHA256) by default and the game settings to be uncompressed so that the interface isn't slowed down or bloated dealing with weird scenarios. As it is now it's a simple check for size, read the checksum and settings, all within the first 256bytes of the file on disk for speed. I'm not sure why this new format needs to be backwards compatible with _some_ emulators anyhow. The old roms are still there, existing, and my tool could even be used to get them back out of the .game files if people want them in the old format for old emulators. Anyone who is working on an emulator in this day and age could add support for the format within 10 minutes easily. Emulators for the DS and other underpowered systems aren't really the future of emulation imo, you have to implement a lot of hacks to get things to work on such systems (which is fine, because it's all about getting 100% speed). The future of emulation as I see it is perfect emulators. Until you have a system that is perfectly emulated then the future is always ahead. People say you cannot get a perfect emulation but they are wrong. By perfect I mean that no software using every trick imaginable can ever detect whether it's an emulation or the real system across numerous real consoles, which is all that matters. I don't mean to diminish your DS emulator because I think projects like that are fantastic and they interest me greatly, optimization is always a fun game. Hopefully in the future a handheld will be fast enough for complete emulation. As to why force it, well for what I have planned with RetroCopy there needed to be a change to achieve certain features and ease of use. If the format is only ever used for RetroCopy then that is fine because I need something like it for this project. Until people have used my emulator and can see the actual direction it is going in it is hard to understand why I did some of the things I did. Either way there was a severe gap in regards to emulators dealing with ROMs and I have bridged that gap in my emulator. I'm merely releasing everything in regards to it so if anyone else in a position like mine doesn't want to blow a week or two doing it they won't have to. I do not really care if no one uses the format outside my emulator because it's completely irrelevant to me whether they do or don't as I have done everything I need myself already. Releasing it all is merely a gesture of good will rather than needing support. I have no problems forcing people to convert their ROMs, even if it means less users. I've already put high system requirements on it, so getting "maximum market penetration" isn't my goal. :)

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 2:23 pm
	FluBBa wrote Seeing that the support is very low so far for this format, I guees there won't be many roms released in this format. Which means that someone has to update the converter database or the end user will have to select what ever he thinks is the correct settings for every new rom released. Well my plan was originally to only convert roms from my emulator (as I require hard rom dirs), basically you put your .SMS files in there and it converted them all for you and you were off and running with minimal fuss. But then I thought maybe others might want to use it so I designed an open sourced tool and a way for people to create databases and check files. Since no one has yet shown any interest I might just go back to my old model for now and simply ask anyone who is interested to email me for the details/tools if they want them. No point in supporting a tool nobody wants. :)

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 2:44 pm
	Eke wrote I second the general opinion about a new ROM format, seems to me as a "control freak" feature (no offense) and a waste of time as it is more easily handled by an internal database (especially when this new ROM format is probably going to be only supported by this emulator) or by a customizable "cartridge emulator" (rom size, mapper type, etc... supported inputs being also configurable, there is no reason you could not try to plug a light phaser while playing alex kidd, even if it was not designed for ;-) ) It most certainly is a control freak thing. :) If I can ensure games are in the best way they can be then I will have no future problems in regards to them. Others may like supporting 10 compression formats, 512byte headers, checksumming, caches, databases and things of this nature but I don't. I created a nice fresh slate to work from instead of dealing with the past. Eke wrote I think it will also brings lots of confusion among current ROM sets, adding ROM file versions when it's not required. I don't know, maybe you could add these informations as a trailer so the ROM header remains intact and the file compatible with other emulators ? Not sure what you mean? .GAME files checksums may change (as a whole file), but the flat rom DATA in them should never change. You're probably thinking in the old terms of "no formats", .GAME files can be taken forward with minimal fuss if there are errors, the worst case if there is an error is wrong settings being used by default not "it doesn't work at all". Whenever you verify your romset it would simply correct any mistakes in the header and that would be it. I don't think the days of storing ROMs on non writable media are going to be the standard in the future, even the modern consoles realize this. ;) That said there shouldn't be any errors in the ROMs, the format already covers everything needed for every single current SMS game, and likely every single future one. It covers weird things like having 4 ports to run ROMs (BIOS, EXPANSION, CARD, CART) and even piggybacking games (no SMS games use this but Genesis games do) that future software may employ. New mappers just mean a new enumeration. I will also be testing every game when I finalize my database and to certify the 100% claim for the release version, so it should have no flaws hopefully.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jun 30, 2009 3:00 pm
	Good luck testing every code path in every game. If you can figure out how to do that in finite time then you'll be a rich man :)

FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	Posted: Tue Jun 30, 2009 3:26 pm
FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	PoorAussie wrote I'm talking about homebrew authors, existing software is already known and I have done a lot of games and their settings already. Future software authors on the other hand have no way of specifying to emulators that they want to use a joypad, paddle, sports pad, 3d glasses, light phaser, genesis 6button controller, mappers, nationality,etc. I hope you do know about the SDSC header in homebrew right? I guess that could be expanded to include everything that might be needed for controler input and such.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jun 30, 2009 3:39 pm
	SDSC is more for getting credits/titles/dates/version numbers into homebrew that would often be otherwise hard to distinguish and/or attribute. It doesn't have any room for expansion really, unless you overloaded something (perhaps human- and machine-readable data in the comments section). But that would be hard to "enforce", since there's no requirement for anyone to fill in the info - it's hard enough to get people to change the SDSC tag in my BMP2Tile demo cough.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 3:48 pm
	SDSC is also a half solution. You can't go over existing system roms and apply it (at least how it is at the moment). I also don't see much point in editing actual ROM data, seems messy to me. Either way the goal of what I did was to make things simpler, which is why I chose not to support the old way of doing things, all my loading code is very simple and portable and has every feature anyone will want in regards to PLAYING games. Certainly there is room for more features in regards to metainformation that some may need but it really isn't needed for playing the files, when it comes to pure game related information for playing it I would challenge anyone to come up with a better way... certainly I would switch to it. Also, there is no such thing as backwards compatibility because .SMS isn't a format it's a lack of one.

Eke Joined: 06 Feb 2009 Posts: 110 Location: Toulouse, France	Posted: Tue Jun 30, 2009 4:06 pm
	Quote The future of emulation as I see it is perfect emulators. Until you have a system that is perfectly emulated then the future is always ahead. People say you cannot get a perfect emulation but they are wrong. By perfect I mean that no software using every trick imaginable can ever detect whether it's an emulation or the real system across numerous real consoles, which is all that matters. I don't mean to diminish your DS emulator because I think projects like that are fantastic and they interest me greatly, optimization is always a fun game. Hopefully in the future a handheld will be fast enough for complete emulation. That's a respectful goal even if I doubt perfect emulation will ever happen, even with cycle accurate chip emulation (tricks relying on very accurate hardware latency timing for example). Emulating all commercial games flawlessly (glitches included) would already be a good step forward (playing these old games being the main reason people use an emulator after all). Anyway, in my opinion, the future of emulation should be for highly accurate but also open-source AND portable emulators because: 1/ things does not last forever: people eventually move to other things and platforms are constantly evolving . I think it's more important to leave an accurate documentation on how these old console were internally working so that anybody could reproduce their behavior even when every single Master System or NES would be reduced into dust ;-) 2/ people should always be able to improve the code and compile it for any upcoming gaming platforms. What you think is accurate might be proven wrong or completed in the future. Some people can choose to optimize the code for a specific platform. The important thing is that they have the choice and a baseground to do it. Quote Either way there was a severe gap in regards to emulators dealing with ROMs and I have bridged that gap in my emulator. huh ? I never felt there was any gap regarding the way emulators are handling ROM files. Accurate ROM databases are very similar (but less intrusive) concept and I hardly see why a new ROM format is better ? Quote . It covers weird things like having 4 ports to run ROMs (BIOS, EXPANSION, CARD, CART) and even piggybacking games (no SMS games use this but Genesis games do) that future software may employ. Sorry but again I don't understand the benefits of using a new ROM format for such thing ? About the port $3E thing (this is what this is about, right ?), you only need to emulate it correctly and could simply have various options to load ROM (load from cart slot, load from card) or to connect specific hardware (having its own emulated model off course) to the expansion slot, why should it be tied to a specific game software ? If the purpose of your emulator is to accurately replicate the console behavior, it should not have any idea about the game species but on the contrary let the user configure it as waned, like he would have done in front of a real console, this is way more flexible in my opinion. Same things goes for cartridges that have the ability to connect themselves to other cartridges (like game genie or sonic&knuckles), it's more flexible to have an option to virtually "connect" two cartridges together. In this case, I agree that the way the connection works is specific to each game but as you can't have a generic emulation model anwyay and need to implement each known banking scheme separately, having the description in the ROM format or letting the user choose a connection model is strictly the same. Sorry for the long post and my approximate frenglish, but this is indeed an interesting discussion

olaf Joined: 14 Oct 2006 Posts: 256 Location: NYC	Posted: Tue Jun 30, 2009 4:19 pm
olaf Joined: 14 Oct 2006 Posts: 256 Location: NYC	This seems more like an emulation project to lock-in a user base. A closed source emulator with a proprietary format that the author boasts about its accuracy and features. It seems like the boasting is to lure in users and you don't seem to plan to share your information on the hardware. We know people love accuracy, or else you wouldn't be boasting about it (I assume in hopes that people will flock to your emulator). Now, your defense mechanism is to say that you're targeting a niche group... but really? That doesn't make up for closed source either. You argued quite extensively with byuu about his SNES emulator but he does everything he can to give back to that community and then some. This project seems like the exact opposite from the start and that's kind of weird for the SMS/GG community since everyone shares everything openly here. Just a personal observation. The whole premise of this emulator seems very strange. Boasting accuracy and 3D boxes to force a closed source emulator with a proprietary format into this community (and the NES community which is just as open)? Icky.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 4:30 pm
	Firstly maybe some here do not understand certain things about what I need. 1) When a user clicks on a file I need to display game settings which work with it best. 2) When a user clicks on a file I need to load other data like covers, save states, etc You can achieve the first one relatively easy with a database. The second requires you to keep another database linking SMS/ZIP/RAR files to a checksum. This is messy. Finally I wanted EVERY game in my hard linked rom folder to be compressed, and I did not want to have to add support for at least 3 different compression schemes that are popular today, ZIP, RAR and 7ZIP. The easiest way to do all this is to have a new format, which has all the features you need with the best compression. There is no debate on this for me, because I have done it both ways and I know it for certain which is the best. If anyone thinks they can support .SMS/ZIP/RAR/7ZIP combined with databases and caching files in a few lines of code, then please show me. Eke wrote About the port $3E thing (this is what this is about, right ?), you only need to emulate it correctly and could simply have various options to load ROM (load from cart slot, load from card) or to connect specific hardware (having its own emulated model off course) to the expansion slot, why should it be tied to a specific game software ? Knowledge about which port the game goes is obvious when you are manhandling the game. If you are holding a card game you know it goes in the card slot. There may be things in the future which take advantage of the card/exp/bios slots and expecting the user to look at a .SMS file and know is a bit silly. Of course having the option to load things in other slots is left up to emulators to suppor, it has nothing to do with the format telling you which slot it SHOULD go in. :) Eke wrote If the purpose of your emulator is to accurately replicate the console behavior, it should not have any idea about the game species but on the contrary let the user configure it as waned, like he would have done in front of a real console, this is way more flexible at my opinion. There is a difference between a raw file on a harddrive (like the .SMS file) and a game you buy. These differences include information that is important to the successful playing of the game and really should be there.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 4:41 pm
	olaf wrote Just a personal observation. The whole premise of this emulator seems very strange. Boasting accuracy and 3D boxes to force a closed source emulator with a proprietary format into this community (and the NES community which is just as open)? Icky. Well this is just ignorance. I have said multiple times I plan to release information about the hardware that I have learned once I have finalized my emulator. I'm not about to start releasing details when there are glitches in several titles that should not be there because I haven't quite figured out the exact timing of the CRAM write delays or because I haven't tracked down every last bug yet. Either way everything I've learned is through trial&error which makes it even harder to accurately tell people the timing of things, I can just give my best ideas and what I have done. Maybe you could tell us all why you haven't written a cycle accurate open sourced VDP and posted all the details for others yet demand it from me? Your post is offensive to me.

olaf Joined: 14 Oct 2006 Posts: 256 Location: NYC	Posted: Tue Jun 30, 2009 5:08 pm
olaf Joined: 14 Oct 2006 Posts: 256 Location: NYC	PoorAussie wrote Either way everything I've learned is through trial&error which makes it even harder to accurately tell people the timing of things, I can just give my best ideas and what I have done. Maybe you could tell us all why you haven't written a cycle accurate open sourced VDP and posted all the details for others yet demand it from me? Your post is offensive to me. Because I haven't bragged about mine or written one and I didn't demand it. If anything I danced around the idea that you should make yours open source. But okay. P.S. I guess I missed the part in this thread where you talked about releasing information of any sort other than vaguely stating that at some point you may make it open source like MEKA.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jun 30, 2009 5:11 pm
	This thread's getting a little warm :) Most SMS emulators support only .SMS and zipped .SMS (not even multiple files per zip). Of course people will ask for RAR and 7z, and the next format, but the status quo is such that you will make almost everyone happy if your emulator works on their existing files, and they are .SMS files, maybe in a zip wrapper. There are two classes of metadata you need: emulation info (for example, game X needs a VDP1 with NTSC timing; can run with either Japanese or Export region settings; uses a light phaser; has battery-backed RAM) and user info (cover scans, manuals, cheats, screenshots, magazine reviews...). The former is somewhat static, and is generally implemented in the form of a "database" (albeit a simplistic one). The latter is less static (for a given game it will be updated a lot more often), and is generally not implemented; the only case I can think of (in the emulation realms I deal with) is FreezeSMS, which constructs a filename from the data in its database and then looks for a metadata archive with all the info in it. This allows one to update the metadata independently of the ROM images, and to easily (legally) distribute the metadata. Some ROM auditing tools also provide metadata like region, publisher, covers and screenshots, again based on a central DB which refers to filesystem objects (for images) which are retrieved over the internet. (Aside: one of the aims of SMS Power, when we are all 90 years old and free to work on it, is to have a large database of all information youc an imagine on every game, and to expose this in some way to emulation, perhaps through generating DB files, a web service... or even generating .game files.)

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 5:28 pm
	olaf wrote Because I haven't bragged about mine or written one and I didn't demand it. If anything I danced around the idea that you should make yours open source. But okay. P.S. I guess I missed the part in this thread where you talked about releasing information of any sort other than vaguely stating that at some point you may make it open source like MEKA. You said and implied several incorrect things about my emulator and myself, so let's not sugarcoat it. If you call alerting people to a free emulator they will be able to use in a few weeks bragging then I think you need a reality check. Exactly what do you think I get from spending over a year of my life doing this for free? The ability to brag on forums about how great it is? If that's all I get then you should really be quiet shouldn't you because you are interrupting it with your ignorance. I can do what I want with my work and information. If I release it to the public domain it is up to me. The fact that I am very careful to make sure I credit and thank everyone who has helped me is a sign I understand the sacrifice others have made for people like myself, and hopefully other people, unlike you, will realize and appreciate the sacrifices I have made to do what I've done.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Tue Jun 30, 2009 5:38 pm
	Maxim wrote (Aside: one of the aims of SMS Power, when we are all 90 years old and free to work on it, is to have a large database of all information youc an imagine on every game, and to expose this in some way to emulation, perhaps through generating DB files, a web service... or even generating .game files.) I would expect by this time you could play cycle accurate games through your web browser on a holographic screen. Plus the copyrights will have expired by then, I believe. ;) It would be cool having a museum with everything in it though, it's a good goal for the site.

olaf Joined: 14 Oct 2006 Posts: 256 Location: NYC	Posted: Tue Jun 30, 2009 5:59 pm
olaf Joined: 14 Oct 2006 Posts: 256 Location: NYC	PoorAussie wrote You said and implied several incorrect things about my emulator and myself, so let's not sugarcoat it. If you call alerting people to a free emulator they will be able to use in a few weeks bragging then I think you need a reality check. Exactly what do you think I get from spending over a year of my life doing this for free? The ability to brag on forums about how great it is? If that's all I get then you should really be quiet shouldn't you because you are interrupting it with your ignorance. I like how you get so angry in your posts they lose direction for a paragraph. PoorAussie wrote I can do what I want with my work and information. If I release it to the public domain it is up to me. I wholeheartedly agree, but as I stated in the previous posts you've only vaguely stated that you may make it open source. If it isn't, that's kind of silly to do for a community that gave you so much information to work with. But again, as you said, you can do what you want with your work and information. PoorAussie wrote The fact that I am very careful to make sure I credit and thank everyone who has helped me is a sign I understand the sacrifice others have made for people like myself, and hopefully other people, unlike you, will realize and appreciate the sacrifices I have made to do what I've done. I'm sorry, I didn't know you were Jesus Christ himself. I'm pretty sure last time I checked there was a page dedicated to the people who just even wrote documentation I read while working on my/brom's emulator (as well as the contributors list displayed on every page).

AamirM Joined: 03 Mar 2008 Posts: 25	Posted: Tue Jun 30, 2009 8:01 pm
AamirM Joined: 03 Mar 2008 Posts: 25	People here need to chill out a bit. :) To be honest, I also don't think this "new format" is a good idea as well but its PoorAussie's emulator and he can do whatever he wants. And we should really wait and see what he does. What if he proves us all wrong? What if maybe its a better format? I think its a bit childish to attack PoorAussie even before giving his work a fair try. I think PoorAussie deserves a full chance to do what he wants to do just like all the rest of us. We chose database/whatever, he is choose to do his own format, whats wrong with that? Also, its not like the end. If you don't think its a good idea, go code your own emulator which does what you want (heck, I have, and pretty much every one discussing here is a SMS emu author :P). So I don't really see why people are so emotional on this. Though, I would request that PoorAussie shares his findings. So some of us can also make/update our stuff with more accuracy. I also chose closed-source path, but I do share my findings with everyone. So after my request of current ROM format got rejected, I hope this one doesn't. ;) stay safe, AamirM

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Tue Jun 30, 2009 9:09 pm
	Out of interest, where are savestates and savegames stored? In the .game file?

MarcK Joined: 10 Apr 2002 Posts: 212 Location: Denmark	Thanks for your work PoorAussie Posted: Tue Jun 30, 2009 10:38 pm
MarcK Joined: 10 Apr 2002 Posts: 212 Location: Denmark	First, I applaud PoorAussie's calmed and reasoned responses to the posts in this thread, some of which has been pretty rude. Secondly, I look forward to try out your emulator, PoorAussie. Cycle accuracy sounds very interesting, and the news of this emulator should, in my opinion, have spurred responses of anticipation and joy - not a boring discussion about file formats and metadata. Furthermore, your emulator looks visually stunning and very professional. Thirdly, in relation to the relatively uninteresting metadata discussion, I must admit, that I can not see the big problem with PoorAussie's new format. Isn't it a big advantage that homebrewers can directly present relevant information to the emulator instead of updating a centralized database file? PoorAussie's solution is decentralized. Decentralized solutions are nice. Even if PoorAussie choose not to publish documentation or source code, the very existence of a cycle accurate emulator is a great help to the scene. Not only for playing games, but also for homebrewers wanting to test their code on something closer to the real system.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Wed Jul 01, 2009 1:36 am
	Maxim wrote Out of interest, where are savestates and savegames stored? In the .game file? Nah, why would you think that? :) My basic structure at the moment is something like this :- \Retrocopy.exe \SMS\ (.game files) \SMS\covers \SMS\states I'm not sure if I will end up supporting "savegames" like traditional emulators do. With savestates it is sort of easier forcing people to do it in one manner for every game. Having a default "last game" state that is saved on exit might be the way to go.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Re: Thanks for your work PoorAussie Posted: Wed Jul 01, 2009 1:50 am
	MarcK wrote First, I applaud PoorAussie's calmed and reasoned responses to the posts in this thread, some of which has been pretty rude. Secondly, I look forward to try out your emulator, PoorAussie. Cycle accuracy sounds very interesting, and the news of this emulator should, at my opinion, have spurred responses of anticipation and joy - not a boring discussion about file formats and metadata. Furthermore, your emulator looks visually stunning and very professional.. Thanks for your kind words and support. The psychology of humans means a lot of people will look upon change as something to fear, after time though something isn't so new and fearful anymore though so it's easier to accept. There is plenty of software that requires a "preprocessing" of files before it will work, and my emulator is exactly the same. It seems my downfall may have been in making people believe I was trying to take over the scene with the new format or something by releasing open source tools for conversion. I think after some time the whole issue of the ROM format will calm down, just right now it is a talking point for the above reasons.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Wed Jul 01, 2009 2:16 am
	olaf wrote I wholeheartedly agree, but as I stated in the previous posts you've only vaguely stated that you may make it open source. If it isn't, that's kind of silly to do for a community that gave you so much information to work with. But again, as you said, you can do what you want with your work and information. I'm not sure what open source has to do with information being released. If you had bothered to actually read my posts before jumping to conclusions (not just in this thread) you would have seen I have posted I will be releasing my findings once I have finalized them. Hardware documents make better reads than source files do, because even then people like yourself that may not understand C++ code will be able to understand english, even if it may be a mangled mix of aussie. For you to come in and then state the direct opposite is willful ignorance. olaf wrote I'm sorry, I didn't know you were Jesus Christ himself. I'm pretty sure last time I checked there was a page dedicated to the people who just even wrote documentation I read while working on my/brom's emulator (as well as the contributors list displayed on every page). http://www.smspower.org/forums/viewtopic.php?t=9240 I notice when you bragged (this is what you call notifying people right? :) ) about "your" emulator on this forum some people criticized and insulted you no good reason, and instead of learning from it you repeat similar behaviour in another emulator thread? Either way I find it highly hypocritical that people like yourself get on a high horse in regards to information being released when you have released none yourself (I do not consider rewriting hardware documents into various other languages information). Even though you were incorrect in regards to me releasing my findings if I wasn't releasing anything you don't really have any leg to stand on to say the things you did. Leave that to people who do, they don't need a champion for their cause.

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Wed Jul 01, 2009 4:00 am
	I think that most of the points have been mentionned (in varied manned) I suggest release unnecessary pressure on PoorAussie and let him concentrate on working on this good software for now. None of the issue can't be solved later.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Wed Jul 01, 2009 8:08 am
	PoorAussie wrote Maxim wrote Out of interest, where are savestates and savegames stored? In the .game file? Nah, why would you think that? :) It is a progression of the ".game file = cartridge" abstraction - after all, I could bring my cartridge to a friend's house and play on his console with my savegame. I had the impression you were storing the covers in there too (making it ".game file = complete game") - if not, how do you know which file to load? The idea of only having one savestate is a lot like the Wii emulators' interface, which you may not be familiar with. The only saving you can do is either any in-game save, or just shutting down and resuming at the same point later. The abstraction of including the entire emulator with each emulated game is perhaps a step too far, though...

Eke Joined: 06 Feb 2009 Posts: 110 Location: Toulouse, France	Posted: Wed Jul 01, 2009 8:18 am
	PoorAussie wrote There is a difference between a raw file on a harddrive (like the .SMS file) and a game you buy. These differences include information that is important to the successful playing of the game and really should be there. Ok, I think I finally got your main idea now, what you want to achieve is some kind of game box "simulation", right ? A card box that includes all the common informations stuff that usually fits on it (supported peripherals, number of players, genre, title, screenshots, manual, etc) as well as a cartridge "model" (ROM data, backup RAM data, mapper type,...), like a real card box would do. I was misleaded by this whole "let's make a new ROM format" mention but it seems it's more a way for you to put all these informations into a single archive, and seems logical to me (and fine, as long as original ROM data file is not altered) anyway, this is an interesting (and indeed very perfectionnist) concept, I guess we should all wait for concrete examples before critizising ;-) MarcK wrote Even if PoorAussie choose not to publish documentation or source code, the very existence of a cycle accurate emulator is a great help to the scene. Not only for playing games, but also for homebrewers wanting to test their code on something closer to the real system. I disagree with this... What the point of having a super accurate emulator when you might not be able to use it anymore in a few decade ? Technologies change and devs move away but documentation & shared informations remain. The whole idea for an emulator is about preservation and the best way to achieve it is to publically share and release informations. Closed-source is a different thing and I strongly respect this choice, developpers should be able to do whatever they want with their own code and to protect it if they think it's necessary. But also remember that many closed-source emulators (or applications in general) would not have been made without other people sharing their own code and findings, I know very few people that can honestly say 100% of their programs is solely their work :-)

mk1995 Joined: 11 Jan 2009 Posts: 116 Location: Germany	Posted: Wed Jul 01, 2009 8:48 am
mk1995 Joined: 11 Jan 2009 Posts: 116 Location: Germany	Bock wrote I suggest release unnecessary pressure on PoorAussie and let him concentrate on working on this good software for now. None of the issue can't be solved later. I guess now the username makes sense, if ya'll know what I mean... (In case you do not know what I mean, I intended it being a joke)

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Wed Jul 01, 2009 9:24 am
	Maxim wrote It is a progression of the ".game file = cartridge" abstraction - after all, I could bring my cartridge to a friend's house and play on his console with my savegame. I had the impression you were storing the covers in there too (making it ".game file = complete game") - if not, how do you know which file to load? I still haven't quite decided the best way to do COVER linkage yet. At the moment I simply have filenames which are the hash of the ROM data, however I want to be able to include multiple covers (different regions, etc) which leads to an issue with this method. Plus hash filenames are also ugly looking in the folder. My current thought is to link them with a single XML file in the COVERS directory since it's almost as simple. But this makes it slightly harder to share them. unless you ship off your XML file with them. Another way might be to make a file format which holds multiple JPEG files, or a XML/text file with the name as the hash which then links to jpegs. Do you have any thoughts? It kinda boils down to how the covers will be shared also I guess, will someone just make complete packs and host them for others? I'm not sure yet. Maxim wrote The idea of only having one savestate is a lot like the Wii emulators' interface, which you may not be familiar with. The only saving you can do is either any in-game save, or just shutting down and resuming at the same point later. The abstraction of including the entire emulator with each emulated game is perhaps a step too far, though... Is it better like that? I haven't had the luxury of using the Wii , but I've heard positive things about the way it handles emulation. Does the Wii support multiple save slots or just one? I kind of like the idea of persistence with save states anyhow, and you could still do it with slots. ie If you have 4 slots and select slot 3, it remembers slot 3 everytime you load the game until you change slots again. Another way might be one persistent slot, and one save anywhere slot that is only temporary. So basically you keep with the idea of persistence but allow them to "quick save" at the beginning of the level and "quick load" if they die. Though being able to rewind gameplay makes it almost impossible to die if you want to use it.... allows you to beat games like Back to the Future 3 which rely on randomness and twitch reflexes for the first level extremely easily. Takes the challenge out of games, but it's very helpful for testing and seeing areas you might not be able to reach otherwise.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Wed Jul 01, 2009 10:02 am
	PoorAussie wrote At the moment I simply have filenames which are the hash of the ROM data For FreezeSMS I had a map of hash to game name, so the filenames are readable. But then you have a central DB and no fallback... PoorAussie wrote I want to be able to include multiple covers (different regions, etc) which leads to an issue with this method. I'm a big fan of convention over configuration. So how about a zip file (or 7z, if you must - with JPGs there's not much compression to gain) that contains files with a standard naming convention, for example <side>_<region>_<index>.<ext>, e.g. front_eu_1.jpg back_eu_1.jpg back_eu_2.jpg front_us_1.jpg back_us_1.jpg spine_us_1.jpg or something to that effect. (Or does it slice entire scans on the fly?) You may need to handle the cases where there are cover variations within a region, and also where there are only incomplete scans of some variations (perhaps a fallback scheme). How is the interface for this going to work? PoorAussie wrote XML ...and now you have two problems. (I'm not a huge fan of XML.) PoorAussie wrote will someone just make complete packs and host them for others? I'm not sure yet. Given the size of the data (multi-GB for all variations at reasonable quality) it may end up in torrents. PoorAussie wrote Maxim wrote Wii emulators' interface Is it better like that? I haven't had the luxury of using the Wii , but I've heard positive things about the way it handles emulation. Does the Wii support multiple save slots or just one? It's generally restrictive but it forces you to play the game more like it was intended. There's no choice to load or save, it effectively saves state on shutdown and loads state on startup. So you can effectively take very long pauses, which I remember doing back in the day (leaving the console powered on and paused for long periods as I couldn't play continuously). If the game has a save system then you can use that, of course. If you have n save slots then someone will find a use for n+1, until n is extremely large. For example, they may choose to have a "preserved" save for each level, and/or each boss. Some games have dozens of levels. Save state ideas: 1. Include screenshots in savestates for easier identification (with some effort you can recreate the screen from the VDP state, but a screenshot is easier) 2. Allow people to choose a textual title (to help identification) 3. Store the game playtime in the state (ApprenticeMinusDS does this, I find it interesting and it helps to tell how far through the game it is) 4. Allow marking saves as "read-only", for those saves you want to preserve (I've lost saves I wanted to keep because of this) 5. If you have a great rewind UI (not just a single-speed rewind button) then quickload/save is not so necessary The TAS kids will probably also have some feature requests in this area. How long is the rewind buffer? It must have some bound. Does it survive across state loads?

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Wed Jul 01, 2009 10:46 am
	Maxim wrote or something to that effect. (Or does it slice entire scans on the fly?) You may need to handle the cases where there are cover variations within a region, and also where there are only incomplete scans of some variations (perhaps a fallback scheme). How is the interface for this going to work? It slices entire scans. If there are incomplete ones people can just photoshop something in if they want. All they have to do is maintain the correct ratios and side width, exactly like a real world wrapping. The interface is simply going to show multiple 3d covers all at once (similar to what you see in the video, except if there are 4 covers it will be zoomed out more to show 4 rotating boxes that you can zoom in on, etc. Likely horizontally placed. So naming of the jpeg files is irrelevent. I'm actually thinking I will just do a text file that is the SHA256 HASH, and in that text file just put in jpeg files upto maybe 8, separated by newline. This way any novice can simply append things which is easy in a text editor. sonic1_usa.jpeg sonic1_usabonus.jpeg sonic1_eur.jpeg sonic1_eur02.jpeg Maxim wrote ...and now you have two problems. (I'm not a huge fan of XML.) Neither was I for a while! But with something like tinyXML it is fairly easy to add support for it and relatively easily to work with it. My GUI themeing is done in XML, though my emulator likely won't allow external themes. Maxim wrote Given the size of the data (multi-GB for all variations at reasonable quality) it may end up in torrents. If the jpegs are 250KB, then you're looking at only ~100MB for SMS, if you say there are 4 variations (high assumption) that's only 400MB. Depends really on the size of each file. Going over 250KB isn't going to get too much added quality since monitors do not have enough resolution for it to matter... yet. Maxim wrote How long is the rewind buffer? It must have some bound. Does it survive across state loads? It could be any length, atm due to not wanting to blow the memory budget I've got it set for like 10 seconds. Each SMS state is about ~50KB or so, assuming PAL it's 50x10x50KB, or roughly 25MB of memory. 'Course you could optimize it somewhat, but since I'm assuming a 2GHz single core processor as min spec using even 500MB of memory for such a feature is no big deal. Loading states or doing anything which interferes with the current state will reset the buffer so to speak, there is no way to avoid that.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Wed Jul 01, 2009 12:23 pm
	One thing a lot of websites do is to use numeric IDs for finding data and then to put extra human-readable stuff in the URLs, for SEO and usability. You could do the same thing, by having the program look for filenames that contain the hash; so, for example, you could have Sonic The Hedgehog (0123456789ABCDEF00000000000000000123456789ABCDEF0000000000000000).ext ...although a 32-byte hash is rather long. (Base64 encoding will get it down to 44 chars.) Alternatively you could have the hash stored in the file and scan the directory on startup to build a map of hash to filename. I still think "appending" JPEGs into a zip is just as easy, and tidier, than a text file. Almost all scans in our scans section are a maximum of 250KB at screen resolution. There's just that many variants. A quick calculation suggests we have about 285MB of unique SMS game-box-only scans, and many of those are at low resolutions. With the rewind, you could record at a much lower rate - maybe four per second, or maybe with increasing intervals at more distant times. I don't see the need for 50fps rewinding. I have in mind a "film strip" interface for rewinding, where you see the buffer as a series of frames which you can scroll through until you see where you want to be. Think of it as DVD chapter selection as opposed to winding a VHS tape back and forth looking for something.

KingMike Joined: 14 Oct 2008 Posts: 513	Posted: Wed Jul 01, 2009 1:54 pm
KingMike Joined: 14 Oct 2008 Posts: 513	I think you would need rewinds much more frequently. It could just be from watching Mario hard hacks (example: ZSNES with 10-frame intervals, I think) and has proven troublesome, as it takes several rewinds to get to a point where you have sufficient reaction time (falling into a hole, running into a trap before you can tell where you are. That problem could still hold true with most action games in general.).

Bock Site Admin Joined: 08 Jul 2001 Posts: 8653 Location: Paris, France	Posted: Wed Jul 01, 2009 3:49 pm
	PoorAussie wrote It could be any length, atm due to not wanting to blow the memory budget I've got it set for like 10 seconds. Each SMS state is about ~50KB or so, assuming PAL it's 50x10x50KB, or roughly 25MB of memory. 'Course you could optimize it somewhat, but since I'm assuming a 2GHz single core processor as min spec using even 500MB of memory for such a feature is no big deal. What about storing full-save state as "key frames", for example 1 or 2 a second, and between those key frames you just store inputs.? Can emulation be fastened and input fed to recreate the intermediary states easily ? (probably a performance problem in the case of your emulator) You could use hybrid approach, store full states for x seconds and start dropping intermediate states in a non-linear manner for older events (and keep the inputs). A little tricky to implement but that would be the ideal solution.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Wed Jul 01, 2009 4:17 pm
	Bock wrote What about storing full-save state as "key frames", for example 1 or 2 a second, and between those key frames you just store inputs.? Can emulation be fastened and input fed to recreate the intermediary states easily ? (probably a performance problem in the case of your emulator) You could use hybrid approach, store full states for x seconds and start dropping intermediate states in a non-linear manner for older events (and keep the inputs). A little tricky to implement but that would be the ideal solution. If memory was tight it would be likely I would have to resort to such things. However the CPU is being burdened much more than the memory is in my emulator. Even with a full 10-11 second rewind buffer, 3d gui, opengl, etc, its only using about 65MB of memory all up. ~20MB of that is opengl and another 20MB is my gui textures/windows. A PC that has a 2Ghz processor would have at least 1GB of memory ,so worrying about a fullscreen app using even 512MB isn't a worry, atm I'm only at a tenth of that on average. If the memory is there you might as well use it. :) My emulator will try and allocate 256MB of physical memory on startup just to ensure a relatively high end machine is running it, it's a way I plan to cut back on support. And in regards to rewind fps, you really need at least 25/30fps on rewind for it to be effective. Sometimes it is hard coordinating it even at 50/60fps let alone less. Since memory is the only concern and I'm nowhere near a high usage it isn't really a problem imo. One of the advantages of targeting the top end of computers....I guess ;)

Chuck Norris Guest	Posted: Wed Jul 01, 2009 6:53 pm
Chuck Norris Guest	I really disagree with your approach. Introducing new useless formats? Targeting high end systems for no reason? Breaking compatibility? Hell, you could as well work for Microsoft at this point. Are you sure you weren't in the team that developed Vista?

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Wed Jul 01, 2009 11:29 pm Last edited by PoorAussie on Thu Jul 02, 2009 1:31 am; edited 1 time in total
	Chuck Norris wrote Introducing new useless formats? Well they aren't useless to my design, maybe only to others. Chuck Norris wrote Targeting high end systems for no reason? To cycle emulate the Master System you need a high end machine, so it's not for "no reason". Anyone could upgrade their machine for ~$150 and have RetroCopy run flawlessly on their computer. People who cannot afford $150 and are running old computers.... well they won't have the internet will they. Master Systems used to cost about $200, so to get an extremely accurate emulator running (though with more features) will cost roughly the same. There is way too much whinging about system requirements on emulator forums, it isn't expensive to upgrade or buy a decent computer these days. And if you want good accuracy you need it, if you don't want good accuracy you don't need it. Simple. Chuck Norris wrote Breaking compatibility? Compatibility with what? Only thing an emulator needs to be compatible with is original software, which it is. Chuck Norris wrote Hell, you could as well work for Microsoft at this point. Are you sure you weren't in the team that developed Vista? Funny :), kinda good analogy too. MEKA runs near 100% of existing software for Master System and it works on a Pentium 200 like Windows XP. My emulator on the other hand claims 100% compatibility and needs ~2GHz and a video card to do it like Vista. If you were a layman it might not seem worth it, so I can understand that viewpoint. Still, if you don't see any advantage about cycle accuracy maybe you are not educated well enough on the matter.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Thu Jul 02, 2009 1:19 am
	Maxim wrote One thing a lot of websites do is to use numeric IDs for finding data and then to put extra human-readable stuff in the URLs, for SEO and usability. You could do the same thing, by having the program look for filenames that contain the hash; so, for example, you could have Sonic The Hedgehog (0123456789ABCDEF00000000000000000123456789ABCDEF0000000000000000).ext Yeah we think alike it seems. I might go this route and just rely on the underlying OS to be fast for file searching. Best of both worlds that way. Maxim wrote ...although a 32-byte hash is rather long. (Base64 encoding will get it down to 44 chars.) Alternatively you could have the hash stored in the file and scan the directory on startup to build a map of hash to filename. You can reduce checksums somewhat, take every 2nd or 4th byte to form a smaller version that is usually just as secure (as an equally matched bit size) provided the checksum is spread evenly over the bits, which good ones are. Maxim wrote I still think "appending" JPEGs into a zip is just as easy, and tidier, than a text file. One problem is sometimes compressing jpegs results in slightly larger files, but it would only be a few percent difference. I guess it's just more optimal on the CPU not having to decompress upwards of a MB every click. I'll have to dwell on this for a bit longer because it is tidier just packing them into a single file. Maybe a tar?

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Jul 02, 2009 8:32 am
	PoorAussie wrote One problem is sometimes compressing jpegs results in slightly larger files, but it would only be a few percent difference. I guess it's just more optimal on the CPU not having to decompress upwards of a MB every click. I'll have to dwell on this for a bit longer because it is tidier just packing them into a single file. Maybe a tar? Zip supports no-compression storage which makes it like a tar but 1000x easier to use for the average user. Good tools compare the compressed size to the zipped size and store uncompressed when it's smaller. Decompressing it will likely be IO-bound rather than CPU-bound. A quick test on a bunch of JPEGs suggests I can "extract" 185MB in about 1.5s when it's stored uncompressed, and in about 2.5s when it's "maximally" compressed in a Zip.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Thu Jul 02, 2009 10:30 am
	Maxim wrote Zip supports no-compression storage which makes it like a tar but 1000x easier to use for the average user. Good tools compare the compressed size to the zipped size and store uncompressed when it's smaller. Could you explain 1000x easier? I thought most windows based archivers supported TAR? When I right click with 7Zip I can select ZIP/7Z/TAR/etc. And I'm pretty sure when I used Winrar it supported TAR also. In those programs there is no visual difference between TAR/other archives. TAR is a lot easier to handle manually in C/C++ without including the ZIP library.

FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	Posted: Thu Jul 02, 2009 10:50 am
FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	PoorAussie wrote Could you explain 1000x easier? Newer Windowses comes with built in support for .zip files?

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Thu Jul 02, 2009 11:42 am
	Out of the box, Windows supports zip and cab archives. WinZip (which sucks, but still seems dominant/common) can only add files to zips. Other, better software (I'm partial to WinRAR myself, although KZIP makes pleasingly insane compression attempts) does better but to the average person, installing 7-Zip and changing the necessary settings to work with TARs is very hard.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Thu Jul 02, 2009 1:46 pm
	Maxim wrote Out of the box, Windows supports zip and cab archives. WinZip (which sucks, but still seems dominant/common) can only add files to zips. Other, better software (I'm partial to WinRAR myself, although KZIP makes pleasingly insane compression attempts) does better but to the average person, installing 7-Zip and changing the necessary settings to work with TARs is very hard. Good point, but are people who are capable of putting together these cover files the type of people who wouldn't be using something better than the default windows setup? Any self respecting nerd that can use Photoshop (and hence putting collections together) has something which can deal with the common compression formats... ;) Average joe just has to place files in the right directory, not mess about with the files inside the archive. The other factor is if it's ZIP then it has the chance to be compressed mistakenly if the user isn't aware of all the issues, TAR is assured of being uncompressed and faster, it's a standard with tools already well developed (compared to something I invent), and it's easier to support than ZIP or RAR on the backend (easy header walk). Since I need all the data from the file there is zero penalty with it's format as I can load and read it entirely into memory and use offsets from a single allocation to display the JPEGs. ZIP on the other hand would require a lot of work on my part to support unless I use and include their horrible API. So in summation, I can't see many benefits going with ZIP.... ;)

FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	Posted: Thu Jul 02, 2009 9:59 pm
FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	I know that the ZIP format isn't very friendly from the start but it's not that hard to implement either. http://ndsretro.com/download/GreenBeretDS03Src.zip I made a few functions for reading data from zip files, I bet you can fix it up quite a bit if you want to help out. It's 5kB of file handling and 37kB for the INFLATE decompression (commented source that is).

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Fri Jul 03, 2009 8:22 am
	I don't think he's bothered about small code size. A zip file library is not hard to come by, even if ZLib doesn't make it simple.

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Thu Jul 30, 2009 6:45 am
	For those that are interested (probably not many here) there is a new video of the NES emulation in RetroCopy here :- I have some more interesting SMS ones to show soon, RetroCopy will be released within the next few days, it won't be as perfect as I'd like it to be but it never really is when it comes to releasing something.

FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	Posted: Thu Jul 30, 2009 8:56 am
FluBBa Joined: 21 Jul 2005 Posts: 412 Location: GBG	What startup values/pattern are you using for RAM on the NES? Or are you just clearing it? (most of the time the broken version of SMB starts in the minus world if emulated correctly).

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Thu Jul 30, 2009 9:57 am
	FluBBa wrote What startup values/pattern are you using for RAM on the NES? Or are you just clearing it? (most of the time the broken version of SMB starts in the minus world if emulated correctly). Just clearing the RAM to zero. What is the "broken version" you speak of? I have noticed a few SMBs which have weird behaviour, but I thought they were hacks.

unfnknblvbl Joined: 12 Jul 1999 Posts: 891	Posted: Thu Jul 30, 2009 1:55 pm
unfnknblvbl Joined: 12 Jul 1999 Posts: 891	PoorAussie wrote Well there will be an option so you can start in fullscreen mode. The plan is to have save states showing, and other little things in the TV mode that you can select. It's also to add to the classic feel. I replayed some games simply because they were a bit harder with it on an angle (like a real TV perhaps) and it's kind of mesmerizing playing it on the "emulated TV" for some reason. Due to specific features I will be adding in the future, the "TV mode" is there to stay though. Sorry if you've already covered this, but it would be kinda cool if the TV was a 3D model and when you went to fullscreen mode, it rotated and zoomed in on it. Of course, with a 3D model, you could choose your own viewing angle, but then that would involve fancy graphical effects to emulate CRT viewing angle distortion. on that note, since you're going all crazy with the "high-end" hardware, any chance of CUDA/OpenCL usage?

unfnknblvbl Joined: 12 Jul 1999 Posts: 891	Posted: Thu Jul 30, 2009 2:05 pm
unfnknblvbl Joined: 12 Jul 1999 Posts: 891	Also, have you thought about supporting 3D via nVidia's thingo?

PoorAussie Joined: 26 Aug 2008 Posts: 292 Location: Australia	Posted: Thu Jul 30, 2009 2:11 pm
	unfnknblvbl wrote Sorry if you've already covered this, but it would be kinda cool if the TV was a 3D model and when you went to fullscreen mode, it rotated and zoomed in on it. Of course, with a 3D model, you could choose your own viewing angle, but then that would involve fancy graphical effects to emulate CRT viewing angle distortion. Some others have mentioned this, and it is a good idea. At the moment I haven't gone overboard with the 3D stuff since it's a little crazy doing so much on my own. unfnknblvbl wrote on that note, since you're going all crazy with the "high-end" hardware, any chance of CUDA/OpenCL usage? OpenCL for the actual emulation or some other thing like filters? I was using Opengl shaders for filters early on the project before throwing them away. I haven't looked too hard at OpenCL/etc but it's possible they could be used. I'm not sure if they would be suited for emulation, but maybe they would. Either way until the standard really hits it isn't something I want to look at too indepth. Standard 2-3 year old CPUs are fine for what I do currently. When it comes to SNES/GENESIS a little bit more may be needed. Until the "2-3 year old CPU" can't emulate the things we need maybe OpenCL will be used more. I expect someone will throw out a test soon enough though.

Forums

View topic - New SMS emulator: RetroCopy