I followed a similar path. Work my way up the Canon food chain and lug around a bulky heavy beast.
I went Olympus and found some great kit. Now I shoot Leica and love it, but if I’d tried Fuji first, I think I would have stuck with it. They make cameras for photographers, not wankers.