[SM] Spotlight effect on Samus using layer3.

[BigDomino]

Hi everyone,

Some days ago, I started working on a new FX3 effect for my hack.
The idea was to make Samus surrounded by a luminous halo, as if she were lighting up with a lamp in the darkness.

For this purpose, I altered the "fog" tilemaps via the FX3 editor (-> I don't plan to use it anymore), changed the A/B value to make layer 3 subtract color from BG1/BG2/sprite (A=02 ; B=14), and rewrote the code that handle BG3 movements/scrolling to take Samus x/y coordinate into account.

[spoiler= Code]

Code Select Expand

Lorom

org $88DB48

; Layer 3 vertical coordinate following Samus y position.
LDA $7E0AFA ; Samus y pos. in pixels, from top border.
SEC
SBC $0915   ; Subtract screen y pos.
CLC
ADC #$0080  ; Shift layer3 position of 80 pixels down to match Samus hitbox center.
EOR #$FFFF  ; 
INC A     ; Reverse direction of the animation by changing the sign.
STA $7ECADE ; Send that value to RAM address that store vertical scroll offset for layer 3.

LDA $7E0AFA ; Samus y pos. in pixels, from border.
STA $1914,x


; Layer 3 horizontal coordinate following Samus x position.
LDA $7E0AF6 ; Samus x pos. in pixels, from left border.
SEC
SBC $0911   ; Subtract screen x pos.
CLC
ADC #$0080  ; Shift layer3 position of 80 pixels to the right to match Samus hitbox center.
    ; XXX if right arrow is pressed, add 20.
    ; XXX if left arrow is pressed, subtract 20.
EOR #$FFFF  ;
INC A     ; Reverse direction of the animation by changing the sign.
STA $7ECADC ; Send that value to RAM address that store horizontal scroll offset for layer 3.

LDA $7E0AF6 ; Samus x pos. in pixels, from border.
STA $1920,x

PLB

RTL[/spoiler]

The trick worked just as intended but there was still a problem...
In the video below, I roughly modeled the halo:

VIDEO

As usual, when you reach the left or right edge of the room, the camera can not go any further and stops following Samus.
On the contrary, the BG3 layer remains focused on Samus and, as it is designed to repeat itself in every directions, reveals a parasitic image of the halo a few meters behind...
The problem will be even more pronounced than in this video since I plan to shift the halo of a few pixels depending on the direction of Samus.

I need to find a way to prevent the periodic repetition of the BG3 but my efforts to find the routine that specifies the way it is displayed (and repeated) were in vain.
Here are the references that I could find on this subject and that I tried to study :

* http://www.metroidconstruction.com/SMMM/layer_fx_disassembly.txt
* http://forum.metroidconstruction.com/index.php?action=dlattach;topic=151.0;attach=607 (Editing Animated FX.txt by -DC-)
* http://forum.metroidconstruction.com/index.php?action=dlattach;topic=145.0;attach=3129 (Multilayer3.asm by DSO)
* http://forum.metroidconstruction.com/index.php?topic=1889.0  (see reply #5 by P.JBoy in spoiler)
* http://forum.metroidconstruction.com/index.php/topic,4434.msg60375.html#msg60375 (see first post by PHOSPHOTiDYL)
* https://jathys.zophar.net/supermetroid/kejardon/BackgroundFormat.txt
and
* The list of all RAM addresses that seem to be related to FX3 / BG3 / Layer3 in Kej./Scyzer RAM map.
[spoiler= RAM addresses]

Code Select Expand

7E:0055              Value for $2105 with 'A' BG Mode and Tile Size Setting abcdefff abcd = BG tile size (4321): 0 = 8x8 1 = 16x16, e = BG 3 High Priority, f = BG Mode
7E:005A              Value for $2109 with 'A' BG 3 Address and Size aaaaaabb a = Screen Base Address (Upper 6-bit), b = Screen Size
7E:005B              Value for $2109 during interrupt 0006 (VRAM Address of tiles and size for BG3)
7E:005E              Value for $210C with 'A' BG 3 & 4 Tile Data Designation aaaabbbb a = BG 4 Tile Base Address, b = BG 3 Tile Base Address
7E:005F              Value for $211A with 'A' Initial Setting for Mode 7 aa0000bc a = Screen Over b = Vertical Flip c = Horizontal Flip
7E:0061              Value for $2124 with 'A' BG 3 and 4 Window Mask Settings aaaabbbb a = BG 4 Window Settings b = BG 3 Window Settings
7E:0067              Value for $212A with 'A' BG 1, 2, 3 and 4 Window Logic Settings aabbccdd a = BG 4 b = BG 3 c = BG 2 d = BG 1
7E:0069              Value for $6A and $212C with 'A' Background and Object Enable (Main Screen) 000abcde a = Object b = BG 4 c = BG 3 d = BG 2 e = BG 1
7E:006A              Value for $212C (when?) Background and Object Enable (Main Screen) 000abcde a = Object b = BG 4 c = BG 3 d = BG 2 e = BG 1
7E:006B              Value for $212D with 'A' Background and Object Enable (Sub Screen) 000abcde a = Object b = BG 4 c = BG 3 d = BG 2 e = BG 1
7E:006C              Value for $212E with 'A' Window Mask Designation for Main Screen 000abcde a = Object b = BG 4 c = BG 3 d = BG 2 e = BG 1
7E:006D              Value for $212F with 'A' Window Mask Designation for Sub Screen 000abcde a = Object b = BG 4 c = BG 3 d = BG 2 e = BG 1
7E:0085              HDMA channels to turn on? Not sure for what, looks like for Layer 3 blending for something.
7E:00B9 - 7E:00BA    Value for $2111 (X scroll of BG 3)
7E:00BB - 7E:00BC    Value for $2112 (Y scroll of BG 3)
7E:05BC - 7E:05BD    High bit is a flag to write to VRAM (for BG3?). Cleared during 80:9632, when you enter a door, set in 82:E4A9
7E:195E - 7E:195F    Actual FX3 height
7E:1962 - 7E:1963    Something to do with FX3 height. Referred to if 195E is negative. Used by lava?
7E:1964 - 7E:1965    Pointer to something for VRAM in bank 8A. I'm guessing FX3 tilemap.
7E:196C - 7E:196D    FX3 something. Need to look into. (Causes rumbling)
7E:196E - 7E:196F    FX3 type (none, lava, acid, water, spores, rain, fog...) JSR index for 8067,X, in bank 90.
7E:1972 - 7E:1973    FX3 height variance?
7E:1978 - 7E:1979    FX3 height (average?) (measured from top of room down)
7E:197A - 7E:197B    FX3 height to go to
7E:197C - 7E:197D    FX3 height adjustment speed (signed)
7E:197E - 7E:197F    FX3 'C'. Bitflags: 0 = FX3 flows left, 1 = bg heat effect, 2 = 'line shift'? (Water does not affect Samus), 3 = unknown, 6 = Big FX3 tide, 7 = Small FX3 tide (priority over 6)
7E:1980 - 7E:1981    FX3, frames to wait till beginning movement. (0 means 10000)
7E:1982 - 7E:1983    FX3 'A'.
7E:1984 - 7E:1985    FX3 'B'.
7E:3800 - 7E:3EFF    Tilemap for part of layer 3? Cleared during message boxes.
7E:6000 - 7E:67FF    Page 1 of BG3 tilemap for main screen gameplay is stored here temporarily some time?
7E:6800 - 7E:6FFF    Page 2 of BG3 tilemap for main screen gameplay is stored here temporarily some time?
7E:9C00              Something for Layer 3 Horizontal Scroll HDMA (scroll value for air?)
7E:9C04 - 7E:9C22    Layer 3 horizontal scroll HDMA data (at least for water)
7E:CAD8 - 7E:CADB    Scroll offsets for BG3 during tilebar. (2 bytes horizontal, 2 bytes vertical). Copied via HDMA during tilebar
7E:CADC - 7E:CADF    Scroll offsets for BG3 during main display. (2 bytes horizontal, 2 bytes vertical). Copied via HDMA once at end of tilebar.
$8000 - $8FFF     Layer 3 graphics. (2 bit. Lotta available space scattered around.)
$B000 - $B03F     Unknown (Probably works as top tiles in Layer 3 Status Bar tilemap, but never seen written to)
$B040 - $B0FF     Layer 3 Status Bar tilemap
$B7C0 - $BFFF     Layer 3 FX tilemap?[/spoiler]

What I wanted to do could be specified as follow :
"Displays on the screen, once and only once, the complete tilemap visible in the FX3 editor (ie. the image made of 32 x 32 tiles of 8 x 8 pixels) centered on Samus (with the indicated coordinates) and fill all the remaining space around it with the same single tile, for example, one picked at the bottom right of the tilemap."

I also tried to think of a workaround :
Would it be possible to apply an enlargement ratio to the entire layer (as for some sequences in mode7) so that the half width of the tilemap equals the width of the screen ? (-> of course, the size of the halo should be decreased accordingly).

I sincerely thank anyone who can give me information or advice to solve this problem or put me on the right track.

[P.JBoy]

Only way to deal with this I can think of is to use window 1 to limit the width of BG3, the following should approximate what you're looking for

Code Select Expand


lorom

!windowBg3Mask              = $61
!window1Left                = $63
!window1Right               = $64
!windowAreaSubscreenDisable = $6D
!layer1X                    = $0911
!layer1Y                    = $0915
!samusX                     = $0AF6
!samusY                     = $0AFA
!bg3X                       = $7ECADC
!bg3Y                       = $7ECADE

!spotlightRadius = $40


; Repoint layer blending configuration 30h
org $88806E
dw FogLayerBlending


; Fog BG3 X scroll pre-instruction, after the time-is-frozen check
org $88DB48
; On entry:
;     A is 16-bit, X/Y is 8-bit
;     X is the HDMA object index
;     DB was pushed onto the stack

JSR FogBg3XScroll
PLB : RTL

; Warn if writing past end of original routine
warnpc $88DB8A


; Free space
org $88EE32
FogBg3XScroll:
PHP

; Write BG3 position, store on-screen X position of Samus in $12
LDA !samusY : SEC : SBC !layer1Y : CLC : ADC #$0080 : EOR #$FFFF : INC A : STA !bg3Y
LDA !samusX : SEC : SBC !layer1X : STA $12
CLC : ADC #$0080 : EOR #$FFFF : INC A : STA !bg3X

SEP #$20

; Window 1 left position = max(0, Samus' X position on screen - spotlight radius)
LDA $12 : SEC : SBC #!spotlightRadius : BCS +
LDA #$00
+
STA !window1Left

; Window 1 right position = min(255, Samus' X position on screen + spotlight radius)
LDA $12 : CLC : ADC #!spotlightRadius : BCC +
LDA #$FF
+
STA !window1Right

PLP : RTS


FogLayerBlending:
; On entry:
;     A and X/Y are all 8-bit

; Disable BG3 outside of window 1
LDA #$03 : STA !windowBg3Mask
LDA #$04 : STA !windowAreaSubscreenDisable
RTS


; Warn if writing past end of bank
warnpc $898000


; From RAM map:
; $61: Window BG3/BG4 mask settings ($2124). Updated during NMI
; {
;     v = ddccbbaa
;     a: BG3 window 1 mask
;     b: BG3 window 2 mask
;     c: BG4 window 1 mask
;     d: BG4 window 2 mask
;     a/b/c/d:
;         0/1: Disable mask
;         2: Enable inclusive mask
;         3: Enable exclusive mask
; }
;
; $6D: Window area subscreen disable ($212F). Updated during NMI
; {
;     v = 000edcba
;     a: Disable BG1 in window area
;     b: Disable BG2 in window area
;     c: Disable BG3 in window area
;     d: Disable BG4 in window area
;     e: Disable sprites in window area
; }


This is for a room using 30h for FX B, you'll have to account for the B = 14 code yourself

PS. here's my updated RAM map http://patrickjohnston.org/ASM/Lists/Super%20Metroid/RAM%20map.asm

[BigDomino]

Thank you P.JBoy.

Whenever I find myself in difficulties, you are there, taking your time to find solutions and offer help.
It's really a chance and a privilege to have people like you on this forum.

Your idea of ​​using a mask to hide unwanted parts of Layer 3 was really smart.
As you suggested, I changed the hijack point at the 14th entry of the pointer table ($ 88: 8052) and, after your settings for window 1, I wrote the instructions for to the color math control register B:

Code Select Expand

LDA #%10110011 ; Enable color math on BG1/BG2/Sprite/Backdrop - Subtract subscreen from main screen.
STA $71

However, I encountered a problem because the halo effect was achieved by subtracting from BG1 / BG2 & sprite, the color filling the entire peripheral area of ​​the BG3.
As BG3 was amputated right and left, the areas outside the window 1 appeared as lit as the halo itself.
(nota: This problem would not have occurred if the halo had been made by adding color).

[spoiler=Video]
https://www.youtube.com/watch?v=eDVkNXk4M4Q&feature=youtu.be[/spoiler]

By looking more closely at your code, I realized that your settings for Window 1 were limited to turning off BG3 outside the window area. The effect area for color math operations continues to cover the entire screen.

With the invaluable help of your RAM map ( :whoa: ), I looked for a way to apply a uniform color to a background layer, which could be subtracted from the graphics in the left and right areas.

I found the opportunity to do it with the following three addresses:

$ 74: Value 1 for color math subscreen backdrop color ($ 2132).
$ 75: Value 2 for color math subscreen backdrop color ($ 2132).
$ 76: Value 3 for color math subscreen backdrop color ($ 2132).

... in which I sent the following values to match the effect of the current BG3 blending:

Code Select Expand

LDA #87; LDA 10000111 blue; intensity: 7
STA $74

LDA # $47; LDA 01000111 green; intensity: 7
STA $75

LDA # $27; LDA 00100111 red; intensity: 7
STA $76

I was worried that this color could also affect the central area but, it seems that color math operations only take into account one subscreen at a given location.
In the central area, BG3 exists and has priority over the backdrop layer. Outside the central area, BG3 is disabled and the Backdrop layer is the only candidate.

[spoiler=Video]https://www.youtube.com/watch?v=v9gxPKb6VWo&feature=youtu.be[/spoiler]

Do you think this is a valid workaround?
Could this cause problems I did not think of?

[spoiler=Your code with the modifications above]

Code Select Expand

lorom

!windowBg3Mask              = $61
!window1Left                = $63
!window1Right               = $64
!windowAreaSubscreenDisable = $6D
!layer1X                    = $0911
!layer1Y                    = $0915
!samusX                     = $0AF6
!samusY                     = $0AFA
!bg3X                       = $7ECADC
!bg3Y                       = $7ECADE

!spotlightRadius = $40


; Repoint layer blending configuration 14h
org $888052
dw FogLayerBlending


; Fog BG3 X scroll pre-instruction, after the time-is-frozen check
org $88DB48
; On entry:
;     A is 16-bit, X/Y is 8-bit
;     X is the HDMA object index
;     DB was pushed onto the stack

JSR FogBg3XScroll
PLB : RTL

; Warn if writing past end of original routine
warnpc $88DB8A


; Free space
org $88EE32
FogBg3XScroll:
PHP

; Write BG3 position, store on-screen X position of Samus in $12
LDA !samusY : SEC : SBC !layer1Y : CLC : ADC #$0080 : EOR #$FFFF : INC A : STA !bg3Y
LDA !samusX : SEC : SBC !layer1X : STA $12
CLC : ADC #$0080 : EOR #$FFFF : INC A : STA !bg3X

SEP #$20

; Window 1 left position = max(0, Samus' X position on screen - spotlight radius)
LDA $12 : SEC : SBC #!spotlightRadius : BCS +
LDA #$00
+
STA !window1Left

; Window 1 right position = min(255, Samus' X position on screen + spotlight radius)
LDA $12 : CLC : ADC #!spotlightRadius : BCC +
LDA #$FF
+
STA !window1Right

PLP : RTS


FogLayerBlending:
; On entry:
;     A and X/Y are all 8-bit

; Disable BG3 outside of window 1
LDA #$03 : STA !windowBg3Mask ;61
LDA #$04 : STA !windowAreaSubscreenDisable ;6D

LDA #$B3 ; LDA 10110011 ie: enable color math on BG1, BG2, sprites, backdrop - subtract subscreen to main screen [BigDomino]
STA $71  ; Send to color math control registre B [BigDomino]

LDA #$87 ; LDA 10000111  blue ; intensity: 7h
STA $74
LDA #$47 ; LDA 01000111  green ; intensity: 7h
STA $75
LDA #$27 ; LDA 00100111  red ; intensity: 7h
STA $76

RTS


; Warn if writing past end of bank
warnpc $898000


; From RAM map:
; $61: Window BG3/BG4 mask settings ($2124). Updated during NMI
; {
;     v = ddccbbaa
;     a: BG3 window 1 mask
;     b: BG3 window 2 mask
;     c: BG4 window 1 mask
;     d: BG4 window 2 mask
;     a/b/c/d:
;         0/1: Disable mask
;         2: Enable inclusive mask
;         3: Enable exclusive mask
; }
;
; $6D: Window area subscreen disable ($212F). Updated during NMI
; {
;     v = 000edcba
;     a: Disable BG1 in window area
;     b: Disable BG2 in window area
;     c: Disable BG3 in window area
;     d: Disable BG4 in window area
;     e: Disable sprites in window area
; }[/spoiler]

Thank you again for your help and your incredible documentation.

PS: I will post a last video of the final result when finished.

[P.JBoy]

That's a lot of praise, I'm grateful :D

I think that's the ideal solution you've made, and your understanding is correct, so congrats ^_^

[BigDomino]

I have finally finished writing this patch.

It took me a long time to finalize it because I encountered a lot of unforeseen problems. :mad:

A demo video has been posted in the collaboration thread to show the final result :
https://forum.metroidconstruction.com/index.php/topic,145.msg67966.html#msg67966

Thanks again for your help, P.JBoy

[nodever2]

Great job on this, BigDomino! This definitely looks like one of my favorite patches that has been released here in quite a while. I can't wait to see hacks using this!

[BigDomino]

@ nodever2

Thank you for the positive feedback :)