The problem with this function is that you can have the function value of the same rational number map to two different numbers under this "function"! So, for example:
h(0.5) = h(1/2) = 1^2 / 2 = 1/2
h(0.5) = h(2/4) = 2^2 / 4 = 1
This means that h(0.5) alone is not well-defined. Depending on how you represent the number 0.5, we get different answers. It's not the number that's different, it's just the way we write it down. *That's* what makes it not well defined.
I claim that it's not well defined for every rational number other than 0. See if you can convince yourself of this fact.
A better way to define such a function, for example, would be to define:
h(r) = m^2 / n
where r = m / n and m and n are relatively prime (i.e. share no factors greater than 1). This is well-defined, because the representation of any rational number in lowest terms is unique.
So, for example, 1/2 will still map to 1/2, but 2/4 won't map to 1. Why? Because 2/4 is not in lowest terms. We must first simplify back to 1/2, and map it to 1/2.
Thus, the mapping is well-defined.
============================
Actually, as a point of interest, my Real Analysis lecturer, in one lesson, defined a function:
f(x) = {1 / n, if x is rational, and x = m / n in lowest terms
......... {0, if x is irrational
It was an example of a function that was continuous on the irrational points, but discontinuous on the rational points. It's a well defined function, and it has that interesting property.
EDIT: Expanded the discussion of the well-defined h.