"In all fairness my boss is pretty good to me, in his defence"

Then thats all that matters,oppinions don't.
You have a good working relationship with your boss and an unwriten agreement. Great. Ive had the same with some of mine. (perhaps I missed that earlier)

If it works for you and him, what else matters?