|
ForumsSega Master System / Mark III / Game GearSG-1000 / SC-3000 / SF-7000 / OMV |
Home - Forums - Games - Scans - Maps - Cheats - Credits Music - Videos - Development - Hacks - Translations - Homebrew |
Author | Message |
---|---|
|
Zest - Test Framework for the Master System
Posted: Sat Sep 02, 2023 3:33 pm
|
Hi all!
I've been polishing up an old proof-of-concept I started quite a few years back and it's now in a nice useable state. This is a test framework for the Master System/WLA-Z80 I'm calling Zest. I was finding when writing certain routines for projects I was spending a lot of time manually testing things, and didn't always notice when an optimisation or change broke some other scenario. As part of my day job I'm always writing tests for my code to define certain scenarios and assert the expected behaviour, so it seemed sensible to apply this to ASM too! Zest uses macros to provide a high level syntax for writing tests. The resulting ROM can then be run in an emulator or real hardware. An example test is: describe "increment"
it "should increment the value in A" ld a, 0 call increment expect.a.toBe 1 it "should not increment past 255" ld a, 255 call increment expect.a.toBe 255 'describe' and 'it' in the above are macros that accept a description string. They generate the code necessarily to store this description and run pre-test setup and post-test checks. Should one of these tests fail it will display the test description onto the screen to help you pinpoint what's going wrong. There are lots of examples in the repo, some of which also double up as tests for the test framework itself. There are a few features that were quite interesting to implement, so I'll discuss those below: Memory overwrite detection Zest detects if some code goes rogue and overwrites Zest's state in RAM, which could result in the program getting into an undefined state and displaying gibberish. It detects this by storing and checking a simple checksum of the test description pointers. An interesting feature is that it stores a backup of these pointers and the checksum in VRAM (in the gap in the sprite table, currently), so even if the RAM gets wiped it still has a chance of being able to recover the backup and displaying which test caused the issue. If the backup checksum is also invalid, it will just display a memory overwrite message. Timeout detection Zest has a timeout counter that it decrements each interrupt, and terminates the test if it reaches zero. This might be the case if a routine doesn't return or gets stuck in a loop. Note: this timeout detection relies on the code not disabling interrupts - I could perhaps implement a pause handler that kills the current test in such a scenario. As an aside, I found with the memory overwrite scenario that this meant the timeout could also be overwritten with a large value and cause the test to hang, so Zest also stores a simple checksum for the timeout counter as well and ends the test if it's found to be invalid. Mocking/Stubbing labels If a routine calls another label you can stub that label out and define custom behaviour. This might be useful if the routine performs a task that you want to fake, such as reading input values. Mocks are instances in RAM, so when the code calls a the mocked label it actually calls this mock in RAM. The mock starts with a call to a mediator function, which pops the address from the stack so it can retrieve the mock being called. It then then jumps to the mock's current defined handler without clobbing any registers, like it was never there. There are examples in the README and examples directory. By keeping the handler pointer in RAM it means each test can define its own routine to mock out a particular scenario. But anyway, just putting this out there in case it's of use or interest to anyone. It doesn't have any graphics assertions out of the box yet so it's mainly for testing Z80 code at the moment, so I think this is the area I'll be working on next. Some things won't be possible, such as asserting the colour palette (as there's no read for that - you'd have to instead mock out the code that sets the palette data) but it should be possible to assert that a given x/y/pattern has been added to the sprite table for example, or that a given pattern is in VRAM. I'll be using it in projects and improving it as I go :) |
|
|
Posted: Sat Sep 02, 2023 5:12 pm |
It’s very different but I wrote https://github.com/maxim-zhao/z80bench to measure cycle counts and also confirm correctness of small chunks of code, by confirming RAM or VRAM state after it returns. | |
|
Posted: Sat Sep 02, 2023 8:26 pm |
I was actually thinking at one point about implementing some sort of benchmarker into this, more for fun. The flow I'm going to be using instead is to define the different scenarios the code should handle, then using the Emulicious profiler to see the min/max/average cycles. I can then tighten the code and try to improve the stats and the tests would tell me if anything breaks.
Just thinking aloud, if it did have a profiler built in I suppose it could track how many scanlines a routine takes, then run it multiple times at increasing delays using nops and other delay tactics until it overflows to an additional scanline, and calculate cycles from that give or take. |
|
|
v0.4.0
Posted: Sat Jun 01, 2024 3:38 pm
|
Just an update on this, I've polished it up a bit more and have added some clobber detection assertions that I'm personally finding very helpful for writing quick sanity tests.
describe "myRoutine"
it "should not clobber registers" zest.initRegisters ; sets all registers to unique values call myRoutine expect.all.toBeUnclobbered ; asserts values are still the initial ones That's helped me find a lots of places where I was clobbing something without realising it. You might want to use `expect.all.toBeUnclobberedExcept` in a lot of cases, such as when you expect a value to be returned. That accepts one or more strings representing the register or register pair(s) to ignore (i.e. "a", "e", "hl"). it "should return a value in A"
zest.initRegisters call myRoutine expect.all.toBeUnclobberedExcept "a" expect.a.toBe $12 The assertion macros used to generate a lot of inline code to cater for their dynamic data. They now minimise this by using the method of generating a `call` to a fixed data-driven routine, followed immediately by the custom assertion data (like the expected value and pointer to the failure message). This means the `call` pushes the address of this data onto the stack as the return address, which the assertion routine can use `ex (sp), ix` to retrieve and utilise, and before returning it just has to increment it and swap it back to return after the data. There's a cost-benefit to writing tests so it's not always worth creating exhaustive tests for simple routines, but there have been some more complex routines I've had to write that have dozens of scenarios that wouldn't have been practical to test manually each time I made a change. I've added screenshots to the README to give an idea of how it works. There's a `template` directory that has an easy-start template if you want to give it a try. Is it ok if I pop a link to the repo in the Libraries section of https://www.smspower.org/Development/Index? |
|
|
Posted: Yesterday at 3:21 pm |
Yes, go ahead. | |