Yes, I have experienced this first hand on both ends.  One person may say a horse looks thin, then another says it is just right.   Then another thinks their horse (a gelding) who looks like it is pregnant already, needs more feed!

Where is the line drawn and who is right?

I say, go by this website's "Henneke Body Scoring Condition Chart" http://www.ultimatehorsesite.com/info/weight.html to guide you.   

The rules basically apply the same way as for us.  If your weight is too high or low, your body will suffer in ways of physical stress and in the long term, may deteriorate muscle, bones, joints, etc. even faster.

Remember, horses are hardy animals, most weighing in excess of 1000 lbs, so gaining or loosing 50-100 lbs. is not unusual.  The key is to understand when a horse is on it's way to gaining or loosing weight.  Then it is time to adjust it's feed accordingly before it becomes a problem.