With your usage, you did ok by ditching the diesels.
Your dilemma, whether the M is worth the extra cost, can only get answered by you, it's pure subjective. Will the extra money you will burn for it, deliver sufficient added experience to make you smile about it?
Also relevant, what could you do with the money that you can keep in your pocket with the 330i instead? When the answer is... don't know...you best go for the M not?
Wrt special conditions? Doubt it. M340i is selling strong apparently. The last two visits at my dealership they told me that they can make an interesting offer on the M340i, but I never asked for it actually. For me, it will certainly not be strong enough.
