Hey sorry for the late reply. I got stuck on teraria update x)
I tried using the code in the docs you linked, but ran into a problem. When my character was standing above both of my tilemaps, that is, between two tiles, it sometimes behaved as if I'm not standing on a floor and instead I'm already on a ramp. However, the collision box kept me standing on the floor and as such, I couldn't move the character.
I'm pretty sure it could be fixed somehow, maybe by using a circle collider instead of a box collider.
In the end, I made three raycast2d nodes on my character (I posted the same question on reddit and got this idea from a user there), one originating from bottom left, second from bottom right and last one from bottom center of the players collision box.
I'm checking the tilemap using get_collider().name and if either of those raycasts collide with my Tilemap for Structures I know I'm on the ground.
Thank you anyways.