well whats the first thing you notice about that expression? well it contains two variables, so it is a 'multivariate' expression - in fact it is an example of a 'multivariate polynomial', which just means a polynomial having more than one variable. polynomials having just the one variable are called 'univariate'.
so how do we factorize a multivariate polynomial?? well for simple examples like this we can pretty much follow the same method as for univariate polynomials.
what if instead of what you have there, we had the univariate polynomial 12x^2 + 17x - 5 , then how would we factorize that?
well you would look at the factors of the '12' and the factors of the '-5' and try different combinations in the two factors that could result in us finding a factorization.
well two possible factors of 12 are '4' and '3', and the only possible factors of -5 are '-5' and '1', or '-1' and '5'. so let us try some combinations to see if they work out:-
try:-
(4x + 1)(3x - 5) = 12x^2 - 20x + 3x - 5 = 12x^2 - 17x - 5
no that doesnt work out because it gives us a '-17' and not a '17', so lets try a different combination:-
(4x - 1)(3x + 5) = 12x^2 + 20x - 3x - 5 = 12x^2 + 17x - 5
so this works, so the factorization of 12x^2 + 17x - 5 is:-
(4x - 1)(3x + 5)
now what about multivariates ie. if there are more than one variable, like in your example? well we can still apply the same method as for the univariate.
we need to factorize 12x^2 + 17xy - 5y^2, well again we look at individual factors, but this time we look at the factors of '12' and of '5y^2'.
well as before possible factors of '12' are '4' and '3' (there are others but for the sake of the example we know these will work), and the only factors of -5y^2 are '-5y' and 'y' , and '5y' and '-y'.
so lets firstly try :-
(4x - 5y)(3x + y) = 12x^2 + 4xy -15xy - 5y^2
= 12x^2 - 10xy - 5y^2
we can see this doesnt work, so trying a different combination then:-
(4x - y)(3x + 5y) = 12x^2 + 20xy - 3xy - 5y^2
= 12x^2 + 17xy - 5y^2
this does work,
so, the factorization of 12x^2 + 17xy - 5y^2 is:-
(4x - y)(3x + 5y)
this is just a case of playing around with various factors until you hit on the right ones, just like in the univariate case. except if you're dealing with two variable examples like the one here then you will have factors that themselves have a variable in them.
ps. there is a formal method for doing multivariate factorizations and its called 'kroneckers method for multivariates'. if you ever do mathematics at a higher level you could well cover it.