Unfortunately, rotating the ray based off the event.relative does not quite work all around the screen, because the camera/screen pixels are not evenly spaced angularly like on a protractor. Instead, they are evenly spaced in a flat plane in front of the camera.
But you don't really need to know the math behind it, because Godot has a function which does it for you:
The camera node comes with the function project_local_ray_normal(screen_position)
to get the vector which the mouse is pointing in.
To orient the RayCast to the correct direction, you could use look_at(position,up_direction)
BUT this function only points the -Z side of the object towards the position, so make sure your ray's cast_to
property is in the direction Vector3(0,0,-1)
So for example:
extends Camera
onready var ray = $RayCast
func _input(e):
if e is InputEventMouseMotion:
ray.look_at(project_local_ray_normal(e.position), Vector3.UP)