You say "I'm just the slightest bit overweight, you can barely tell" and then you say: "I feel so fat and I feel like others think I'm fat.".
I think I see a problem there, and it's not your weight.
Anyway, if you want to speed things up, just start a good exercise routine, and you'll get where you want to be in no time.
Took me one year. Now I'm doing triathlons.
You should see results right away if you are doing it right. Weight balance will of course depend on many factors, we really need much more info. from you to help:
'Sometimes no fat' - how? Do you sometimes eat nothing? Even lettuce has some fat (0.2g per 100g of lettuce, according to Nutrition Facts).