It depends on what you want to pool, but if you remove it from the tree I don't think it matters. If you leave it in (for performance maybe?) then you need to check process/sound/physics on them.
Also this is not a necessarily a bad design to prefetch scene properties, I had this problem initially to load levels, which I was afraid to pool due to their size and number (although I didn't try yet). If I have this problem again I will probably either look into a solution deeper, follow a system of rules or generate some kind of meta-files.