Wait till one of these stray non threat asteroids collides with another non threat and then we have a bigger problem. These rocks have been out there for millions of years and have had lots of time to clear out their current orbits long before now. Any change in their orbits will cause a chain reaction that may end all life on this particular rock. We are not gods and should not think that we are, the universe has been hear a long time and any recent changes should not be because of man.