Forums

Hi all!

I've been polishing up an old proof-of-concept I started quite a few years back and it's now in a nice useable state. This is a test framework for the Master System/WLA-Z80 I'm calling Zest.

I was finding when writing certain routines for projects I was spending a lot of time manually testing things, and didn't always notice when an optimisation or change broke some other scenario. As part of my day job I'm always writing tests for my code to define certain scenarios and assert the expected behaviour, so it seemed sensible to apply this to ASM too!

Zest uses macros to provide a high level syntax for writing tests. The resulting ROM can then be run in an emulator or real hardware. An example test is:

describe "increment"

it "should increment the value in A"
ld a, 0
call increment
expect.a.toBe 1

it "should not increment past 255"
ld a, 255
call increment
expect.a.toBe 255

'describe' and 'it' in the above are macros that accept a description string. They generate the code necessarily to store this description and run pre-test setup and post-test checks. Should one of these tests fail it will display the test description onto the screen to help you pinpoint what's going wrong. There are lots of examples in the repo, some of which also double up as tests for the test framework itself.

There are a few features that were quite interesting to implement, so I'll discuss those below:

Memory overwrite detection
Zest detects if some code goes rogue and overwrites Zest's state in RAM, which could result in the program getting into an undefined state and displaying gibberish. It detects this by storing and checking a simple checksum of the test description pointers. An interesting feature is that it stores a backup of these pointers and the checksum in VRAM (in the gap in the sprite table, currently), so even if the RAM gets wiped it still has a chance of being able to recover the backup and displaying which test caused the issue. If the backup checksum is also invalid, it will just display a memory overwrite message.

Timeout detection
Zest has a timeout counter that it decrements each interrupt, and terminates the test if it reaches zero. This might be the case if a routine doesn't return or gets stuck in a loop. Note: this timeout detection relies on the code not disabling interrupts - I could perhaps implement a pause handler that kills the current test in such a scenario.

As an aside, I found with the memory overwrite scenario that this meant the timeout could also be overwritten with a large value and cause the test to hang, so Zest also stores a simple checksum for the timeout counter as well and ends the test if it's found to be invalid.

Mocking/Stubbing labels
If a routine calls another label you can stub that label out and define custom behaviour. This might be useful if the routine performs a task that you want to fake, such as reading input values.

Mocks are instances in RAM, so when the code calls a the mocked label it actually calls this mock in RAM. The mock starts with a call to a mediator function, which pops the address from the stack so it can retrieve the mock being called. It then then jumps to the mock's current defined handler without clobbing any registers, like it was never there. There are examples in the README and examples directory. By keeping the handler pointer in RAM it means each test can define its own routine to mock out a particular scenario.

But anyway, just putting this out there in case it's of use or interest to anyone. It doesn't have any graphics assertions out of the box yet so it's mainly for testing Z80 code at the moment, so I think this is the area I'll be working on next. Some things won't be possible, such as asserting the colour palette (as there's no read for that - you'd have to instead mock out the code that sets the palette data) but it should be possible to assert that a given x/y/pattern has been added to the sprite table for example, or that a given pattern is in VRAM. I'll be using it in projects and improving it as I go :)

It’s very different but I wrote https://github.com/maxim-zhao/z80bench to measure cycle counts and also confirm correctness of small chunks of code, by confirming RAM or VRAM state after it returns.

I was actually thinking at one point about implementing some sort of benchmarker into this, more for fun. The flow I'm going to be using instead is to define the different scenarios the code should handle, then using the Emulicious profiler to see the min/max/average cycles. I can then tighten the code and try to improve the stats and the tests would tell me if anything breaks.

Just thinking aloud, if it did have a profiler built in I suppose it could track how many scanlines a routine takes, then run it multiple times at increasing delays using nops and other delay tactics until it overflows to an additional scanline, and calculate cycles from that give or take.

Author	Message
eljay Joined: 17 Jun 2017 Posts: 19	Zest - Test Framework for the Master System Posted: Sat Sep 02, 2023 3:33 pm
eljay Joined: 17 Jun 2017 Posts: 19	Hi all! I've been polishing up an old proof-of-concept I started quite a few years back and it's now in a nice useable state. This is a test framework for the Master System/WLA-Z80 I'm calling Zest. I was finding when writing certain routines for projects I was spending a lot of time manually testing things, and didn't always notice when an optimisation or change broke some other scenario. As part of my day job I'm always writing tests for my code to define certain scenarios and assert the expected behaviour, so it seemed sensible to apply this to ASM too! Zest uses macros to provide a high level syntax for writing tests. The resulting ROM can then be run in an emulator or real hardware. An example test is: describe "increment" it "should increment the value in A" ld a, 0 call increment expect.a.toBe 1 it "should not increment past 255" ld a, 255 call increment expect.a.toBe 255 'describe' and 'it' in the above are macros that accept a description string. They generate the code necessarily to store this description and run pre-test setup and post-test checks. Should one of these tests fail it will display the test description onto the screen to help you pinpoint what's going wrong. There are lots of examples in the repo, some of which also double up as tests for the test framework itself. There are a few features that were quite interesting to implement, so I'll discuss those below: Memory overwrite detection Zest detects if some code goes rogue and overwrites Zest's state in RAM, which could result in the program getting into an undefined state and displaying gibberish. It detects this by storing and checking a simple checksum of the test description pointers. An interesting feature is that it stores a backup of these pointers and the checksum in VRAM (in the gap in the sprite table, currently), so even if the RAM gets wiped it still has a chance of being able to recover the backup and displaying which test caused the issue. If the backup checksum is also invalid, it will just display a memory overwrite message. Timeout detection Zest has a timeout counter that it decrements each interrupt, and terminates the test if it reaches zero. This might be the case if a routine doesn't return or gets stuck in a loop. Note: this timeout detection relies on the code not disabling interrupts - I could perhaps implement a pause handler that kills the current test in such a scenario. As an aside, I found with the memory overwrite scenario that this meant the timeout could also be overwritten with a large value and cause the test to hang, so Zest also stores a simple checksum for the timeout counter as well and ends the test if it's found to be invalid. Mocking/Stubbing labels If a routine calls another label you can stub that label out and define custom behaviour. This might be useful if the routine performs a task that you want to fake, such as reading input values. Mocks are instances in RAM, so when the code calls a the mocked label it actually calls this mock in RAM. The mock starts with a call to a mediator function, which pops the address from the stack so it can retrieve the mock being called. It then then jumps to the mock's current defined handler without clobbing any registers, like it was never there. There are examples in the README and examples directory. By keeping the handler pointer in RAM it means each test can define its own routine to mock out a particular scenario. But anyway, just putting this out there in case it's of use or interest to anyone. It doesn't have any graphics assertions out of the box yet so it's mainly for testing Z80 code at the moment, so I think this is the area I'll be working on next. Some things won't be possible, such as asserting the colour palette (as there's no read for that - you'd have to instead mock out the code that sets the palette data) but it should be possible to assert that a given x/y/pattern has been added to the sprite table for example, or that a given pattern is in VRAM. I'll be using it in projects and improving it as I go :)

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14745 Location: London	Posted: Sat Sep 02, 2023 5:12 pm
	It’s very different but I wrote https://github.com/maxim-zhao/z80bench to measure cycle counts and also confirm correctness of small chunks of code, by confirming RAM or VRAM state after it returns.

eljay Joined: 17 Jun 2017 Posts: 19	Posted: Sat Sep 02, 2023 8:26 pm
eljay Joined: 17 Jun 2017 Posts: 19	I was actually thinking at one point about implementing some sort of benchmarker into this, more for fun. The flow I'm going to be using instead is to define the different scenarios the code should handle, then using the Emulicious profiler to see the min/max/average cycles. I can then tighten the code and try to improve the stats and the tests would tell me if anything breaks. Just thinking aloud, if it did have a profiler built in I suppose it could track how many scanlines a routine takes, then run it multiple times at increasing delays using nops and other delay tactics until it overflows to an additional scanline, and calculate cycles from that give or take.

Forums

View topic - Zest - Test Framework for the Master System