For my code, I found the solution by putting the floor as layer collision 2 and the raycast collision mask to 2:
var cast = cam.get_world().direct_space_state.intersect_ray(from,to, [rigidBod], 2, true, false)
thus bypassing hickups on other rigid bodies.
It is now working very smoothly on my Android phone.