With the usual "I Am Not A Lawyer" caveat, it seems to me that, if your work ends up causing property damage or an injury, you're going to be legally liable whether or not there were permits and inspections.
If you did have permits and inspections, then that probably adds defendants--the inspector and the city/county. And arguably, if you have the "deep pockets" of the city/county sitting out the begging to be picked, that makes it more attractive to sue. So you might be in worse shape in that case.
