I think sometimes it’s just the more recent manic schedules. Much was made of Branagh not doing Thor 2 but his reasoning was pretty straighforward, He had to commit straight away and wanted to do other things first. No great falling out and he remained as a producer.
The same happened with Gary Ross on the Hunger Games, he said he wanted a break and they asked for a sequel within 18 months. It’s a modern phenomenon to an extent that directors are asked to just do one thing. David Yates seems happy to do that with Harry Potter and spinoffs but not many more.
It may be the best solution is to alternate.