Without being there and seeing all the variables it is tough to give an answer.
In general in my opinion it is easer to leave the existing panel in place and re feed it.
Remember this panel will now be a sub panel and the neutrals and grounds will need to be separated. Also the grounding electrode system will need to be relocated to the new service disconnect location.
If you want to really clean things up why not go underground from the house to the pole. Use a Meter Main combo unit made for an underground feed.
