In a sphere of radius R, inscribe a cylinder of radius r, height h.
Draw a diagram of this. Actually, if a 3D diagram is too tricky, you can just draw the 2D vertical cross section that contains the axis of the cylinder. The sphere then shows up as a circle, and the cylinder as a rectangle inscribed in the circle.
Draw a line from the common center, C, of sphere and cylinder to the center, B, of one base of the cylinder, then from there to A, the endpoint of a radius of the cylinder base, which is also on the sphere (circle), then back to C.
ABC is a right triangle with its right angle at B; legs AB=r and BC=h/2; and hypotenuse AC=R. So it follows that:
r^2 = R^2 - (h/2)^2
The volume of the cylinder is
V = π r^2 h
= (π/4)(4R^2 - h^2)h
To maximize V wrt h,
0 = dV/dh = π(R^2 - (3/4)h^2)
h^2 = (4/3)R^2
h = 2R/√3
r^2 = R^2 - (h/2)^2 = R^2 - (1/3)R^2 = (2/3)R^2
V = π r^2 h = (4π / 3√3)R^3
= V[sphere]/√3
To check that it is a max, not a min,
d^2 V/dh^2 = - (3π/2)h^2 < 0 ==> MAX