You remember what you care about.
Much of math ability is genetic, like music and art and such, but you, I'm sure, have a mental block. Anyone who can remember which way to turn in the maze at level 7 in whatever meaningless video game you want to pick can remember ANYTHING. Look at the people who are walking encyclopedias of sports trivia.
Bible says "perfect love casts out fear," but at just your age I already knew fear makes perfect love impossible. That was a conundrum (or zen koan, if you prefer) that stayed with me for years.
I know a guy who was an Army chaplain, afraid of heights. He believed the chaplain should be able to do everything the troops had to do, so he jumped out of airplanes over 60 times.
You can overcome fear in small steps. I've been drilling my granddaughter on the times table this very week. I give her 15 min in front of the computer (flashcards Excel style), 30 min off for video games, over and over. Brain needs 15 min to transfer info from temp memory to permanent. Do it in small chunks, the 3's, the 4's, the 6's, and so on. Spend a day on each this way. Use the information in other contexts -- area of rectangles, area of triangles, solving simple equations. The more ways the information CONNECTS with other things you know, the better it will stick, but information that just hangs there in space quickly evaporates. You find yourself sitting somewhere, look at floor tiles, think about floor area = length (in tiles) x width, as if it were YOUR floor, as it might be in just a few years. Make math a part of your thinking about EVERYTHING and it will become easy.
Curious thing about our belief in free will. You can choose anything you want, but you can't choose what you want. If you could, you could choose to love math. The thing to watch out for is how much of what you want (without even thinking about it) OTHER PEOPLE choose for you. TV commercials are created by the best behavior psychologists on earth.
Good luck. I hope you know you're not the only one. 16 years ago I swore I'd teach my wife (#3) calculus. Then I scaled back to algebra. I've given up completely, but you wouldn't believe how much interior decorating I've learned.