The program was double tapping the input because I had mouse emulation on as per default settings. These resulted in 2 presses and then 2 releases. And because the code was a little slow, before the program could even advance by one frame. Due to that, all sorts of unintended things happened.
Solution was to prevent code from running twice, and I also added a removechild before queuefree on relevant code to prevent it from being selected multiple times. Making the code faster also eliminates the problem since then you physically can't press and release on the same frame.