You are not alone amigo. My guess is, you are a wee bit younger then me, but at 58 I decided to get into the sport a year ago. Better on a board then in a box! Since I've got some rather harsh lines in my face but still in shape, I appear to to be one of those old dudes that can perform. No one even says anything when I pearl or get owned by a wave.
Your issues are a part of youth, sit back and watch, you'll see your chance if you are patient. Expect to be out longer, tire THEM out by being patient. It's all about going Zen and getting in tune with the energy of the wave, not the macho behavior that masks the fact that they are scared little boys down deep without careers, money or respect. This is all they got, so see them as little kids in your mind's eye and you'll get over their childish behavior, remember, you're not a victim.
The sea gives what it gives. Pay your dues, paddle hard then harder, use the sesson as a work out to be able to zip past them later. They will take note and you'll be productive even if you don't get into the line up right away. You might not be able to PLAY right now, but you can work, so use that negative energy to gain strength and speed, keep your board straight as you paddle, commit yourself to a better diet, workout ethic and be in the moment, not the sad past or the anxiety ridden future.
Punks don't exist, only the waves. Trust yourself, you don't need to fit in with them, or be accepted by them, just be accepted by the waves.