It does seem odd to do it in physics. Are you expecting to update the direction of looking when something happens or every now and then?
If doing it with something happens, you set the frame when it happens and then just ignore it. That is, "Noise was made, everyone look towards it".
If doing it now and then, you might want your own timer. It can be slow, like 0.25 seconds per tick.
In any case, you should have a heuristic to skip the "looking left" part when nothing relevant has changed. Usually, any distance calculation is slooooow.
Tells us more?
Keep hacking; keep notes.