This was a hard one for me and I had to Google information for the area of an equiangular hexagon. The text I reference below shows some nice pictures of equiangular hexagons (their Fig 1), that nicely answer your question about equiangular vs. equilateral. The equiangular hexagon given in the problem is what Ball calls an (ab)^3 hexagon, with a=1 and b=r. After you draw out this (ab)^3 hexagon, you will note that you can just fit it inside of an equilateral triangle, whose sides are each (1+2r) long. After you draw this out, you can see that the area of the hexagon is just the area of the equilateral triangle minus the areas of 3 little equilateral triangles, each with a side of r length. Since the area of an equilateral triangle is given as ((sqrt 3)/4)*(length of side)^2, you can calculate the area of the hexagon as [((sqrt3)/4) * (1+2r)^2] - 3 * [((sqrt3)/4) * (r)^2] = ((sqrt3/4)*(r^2 + 4*r + 1).
Save that formula as Eqn 1.
Now inscribe the triangle ACE as described in the problem. Use the Law of Cosines to find the length of a side of the inscribed triangle: c^2 = a^2 + b^2 -2ab Cos C, where the length of two sides is 1 and r, and the included angle C is 120 degrees. Since the Cos of 120 = -1/2, the length of the side of the triangle becomes c = sqrt(r^2 + r + 1). You will notice that each side of the inscribed triangle is opposite a 120 degree angle, so the triangle must be an equilateral triangle. As described above, the formula for the area of an equilateral triangle is: A = [((sqrt3)/4) * side^2], or
A insc triangle = [((sqrt3)/4) * (r^2 + r + 1). Save that equation as Equation 2.
Now the problem says that the ratio of the area of the inscribed triangle to the area of the hexagon is 0.7, so you have: (Equation 2)/(Equation 1) = 0.7. Substituting in the numbers and rearranging this in the form useful for the quadratic equation, you end up with 0.3*r^2 - 1.8*r + 0.3 = 0. You can plug the coefficients into the quadratic formula: [-b +/- SQRT(b^2-4ac)]/2a. As long as (b^2-4ac) is positive, you are dealing with real number roots and you can see that this is the case, with b=-1.8, a=0.3 and c=0.3. Then notice that the problem asks for the sum of the possible values of r. From the quadratic formula there will be two values for r, one that is + the square root term and the other that is - the square root term. When you add these, the square root terms will cancel out and leave you with the sum of the values of (1.8 + 1.8)/(2*0.3) = 6.
I have no idea how people could be expected to do this problem along with 24 other problems in a 1 hour time period! Maybe there is an easier way, but the way I give above does eventually get the right answer.