Joey, If it isn't sticking out anymore than the panel then I have to agree with you due to the fact that it was existing. You can actually stand in front of the panel without hitting an obstruction. It's interesting to see that it does not matter where you go you will find inspectors unable to think beyond the black and white.
Ron