Oftentimes this can stem from trying to fit into what we think others expect of us:
I should be able to swim like an Olympian, always look good, always be smiling, find it easy to train hard, be thin, have the skin on a 19 year old, be strong, reliable and happy, and never have an off-day.
Of course none of us can ever measure up to that ideal so we end up feeling bad about ourselves. And since this expectation of perfection is constantly re-enforced by western society wherever we look, it can be a difficult pattern of thinking to break away from.
But here's a way to try. Seriously ask yourself this question:
your swimming (or sport, or life in general)?
Obviously this is very personal to you but your answer might be something along the lines of:
That right there is a positive identity that you can build from and face off those doubts. Something you can be proud of and live up to.
Someone once said that most of the time fear is just Forgetting Everything's All Right and perhaps it is. You're doing this because you enjoy it and because it's a positive in your life. So let's keep it real - all things considered you're actually doing just fine thanks very much!