3/4 topdown ysort issue?

Godot Version

Godot 4.2.2

Question

Im trying to code a 3/4 topdown game with the illusion of 3D using ysort. If the player goes up the stairs his y sort origin is being updated, ensuring correct y sort behaviour. When my character plays his attack down animation the sprite goes below the y sort origin, but the y sort origin stays the same because otherwise the character suddenly appeared in front of a tree for example. The problem is that the tile from the tilemap gets drawn in front of the sword because its y sort is lower.
This is how it looks because of ysort:
wrong_animation
The problem to fix:
Entities on top of a layer should NEVER be drawn below it, meanwhile if they are below the layer they should be drawn by y sort
Are there ways to fix this? I havent come up with any and my only solution right now is to give up the illusion of 3D which i dont really want to do