Koh Yao Yai in early July?

We have 5 days/4nights to spend in the islands. Is it crazy to focus on Koh Yao Yai?

I really like the idea of getting "out of the crazy" and relaxing somewhere more remote. But given the weather (end of rainy season) and the fact that we don't have much time, should I stop fixating on Koh Yao Yai and find some place on the beaten path?

And if so, any recommendations?