I've been in a similar position to yours: Started on the Eb tuba at the age of about 40, played it (Yamaha 321) for several years, switched to BBb (Cerveny 781), played that for more years, then decided that what I REALLY liked was an Eb and got a compensating Eb from Wessex. But I also have an old 1924 Buescher 3-valve Eb that I've finally beaten/threatened/begged/cajoled/modified into a fun little horn that actually plays in tune at A=440. In the interim, I picked up euphonium (which I play only slightly better that acceptably, and not frequently) and then went whole hog on bass trombone for the past 5 years or so, with infrequent work on the tuba(s).
In terms of switching back and forth among the Bb and Eb instruments/fingerings, I've never really had any problem -- though I confess this came as something of a surprise. But I think my response to your original question is that it will probably get better -- especially if you don't think about it much and just play. There's always a few minutes of thinking about it when I switch from one to the other, but that passes pretty rapidly. However, I've always felt that there was a different "feel" to each instrument that worked as a kind of mental switch for me. But I'm not sure how important that is.
Changing from the tubas to trombone was a real jump and took some time -- in part because of the significant difference in timbre, tone quality, and feel. And the bass trombone exacerbates this because it's fundamentally four different instruments you're playing at one time. The switch to slide rather than valves was challenging enough, but the very different "feedback" from the instrument (in terms of both tactile response and sound) took a lot of time (probably a couple of years) to grapple and feel some comfort and confidence with.
Gary Merrill
Wessex EEb Bass tuba (DW 3XL or 2XL)
Mack Brass Compensating Euph (DE N106, Euph J, J9 euph)
Amati Oval Euph (DE 104, Euph J, J6 euph)
1924 Buescher 3-valve Eb tuba (with std US receiver), Kelly 25
Schiller American Heritage 7B clone bass trombone (DE LB K/K10/112/14 Lexan, Brass Ark MV50R)
1947 Olds "Standard" trombone (Olds #3)