I think it’s unfair to lump Baker and McCoy’s era together like that. There’s a big uptick in quality when Cartmel takes over (if you gloss over Time and the Rani as a remnant of the previous administration). The production design still has issues, as seen in the Happiness Patrol, but the scripts are at least all trying to have more depth and texture than under Saward. They’re usually trying to be about something rather than just a gory romp. Paradise Towers isn’t entirely successful - it’s uneven in tone, Richard Briers is miscast, it gets a bit too caught up in citing rules - but it’s ambitious. The Kang dialogue and society is a tremendous bit of world-building. The story feels like a dip into a fully formed other world, just like Robots of Death, rather than a setting that exists only to serve its story.
It’s unfortunate that this resurgence happens when the show’s restricted in episode count and having to do so many three parters. But splitting the six parter filming budget into two three parters, allows for some all location stories, just like Spearhead, which helps temper the design problems. And by the third season, they were actively focusing on settings that they knew the designers could pull off well - hence the final season full of historical and contemporary stories.
And let’s not pretend that bad design was exclusive to the 80s. There were plenty of stories in the 60s and 70s that looked crap. Even some of the best bits of design work were marred by budgetary and design limitations - the beautiful alien forest in Planet of Evil is juxtaposed with abysmal costumes and a pretty tatty spaceship.