Actually, any number ending in 1, 2 or 3 except for 11, 12, and 13 do not follow the rule either. For example, we say "fifty-first", "fifty-second" and "fifty-third".
I think it's because English has so many different influences from other languages, and the names of "first", "second" and "third" just had different origins. "Second" comes from the Latin "secundus", whereas "one" and "two" have the same origins as the German words for one and two, "eins" and "zwei".
If you think about it, we do have other words that describe ordinal position or number without using the names "one, two, three, four..." or "first, second, third, fourth...". We can talk about a first place politician as "PRIMARY candidate". We can talk about two objects by saying "BOTH of them". After we get past 2 objects though, we start grouping them together less distinctly. I'm guessing that this has a relation to why words like "first" and "second" are used instead of "oneth" and "twoth". At least "third" has some more letters in common with the word "three"!
A lot of the other number names in English match with German in odd places, which is also why we get eleven ("elf") and twelve ("zwölf") instead of "oneteen" and "twoteen".