The standards don’t actually mention the set model for fractions, so exactly when to introduce it is partly up to the judgement of the curriuclum writer. The reason for not introducing it in Grade 3 is that it can cause confusion as to what the whole is. If eat 6 out of 12 bananas, that’s 1/2 of the bananas. But notice that in order to interpret this I need to regard the 12 bananas as a whole. That seems a little alien, as opposed to area of length models, where I can clearly see the rectangle as a whole, or the length from 0 to 1 on the number line.
The set model is really more related to multiplication of a whole number by a fraction: $\frac12 \times 12 = 6$. That doesn’t happen until Grade 5, so that’s why you will see some people say it should go there.
But I think you could also start working with in Grade 4, as a preparation for multiplication of whole numbers by fractions. It depends on exactly how you introduce it.