First, by Chain Rule, we have to determine the following derivatives
d/dx (x² sinx) = x² cosx + 2x sinx
d/dx (x² cosx) = -x² sinx + 2x cosx
d/dx (x cosx) = -x sinx + cosx
Simplify the expression
∫ x² e^(3x) sin(3x) dx, let y = 3x
= (1/27) ∫ y² siny e^y dy
The trick for each integration by parts is to isolate the exponential term as the integrating factor
∫ y² siny e^y dy, Integrate by Parts, u = y² siny, dv = e^y dy
= ∫ u dv
= uv - ∫ v du
= y² siny e^y - ∫ y² cosy e^y dy - 2 ∫ y siny e^y dy ............... [1]
∫ y² cosy e^y dy, Integrate by Parts, u' = y² cosy, dv' = e^y dy
= u' v' - ∫ v' du'
= y² cosy e^y + ∫ y² siny dy - 2 ∫ y cosy e^y dy ................. [2]
Substitute [2] into [1]:
∫ y² siny e^y dy = y² siny e^y - [ y² cosy e^y + ∫ y² siny dy - 2 ∫ y cosy e^y dy ] - 2 ∫ y siny e^y dy
.
∫ y² siny e^y dy = y² siny e^y - y² cosy e^y - ∫ y² siny dy + 2 ∫ y cosy e^y dy - 2 ∫ y siny e^y dy
.
2 ∫ y² siny e^y dy = y² siny e^y - y² cosy e^y + 2 ∫ y cosy e^y dy - 2 ∫ y siny e^y dy
.
∫ y² siny e^y dy = (1/2) y² siny e^y - (1/2) y² cosy e^y + ∫ y cosy e^y dy - ∫ y siny e^y dy ........... [3]
∫ y cosy e^y dy, Integrate by Parts, let U = y cosy, dV = e^y dy
= ∫ U dV
= UV - ∫ V dU
= y cosy e^y + ∫ y siny e^y dy - ∫ cosy e^y dy .................. [4]
Substitute [4] into [3]:
∫ y² siny e^y dy
= (1/2) y² siny e^y - (1/2) y² cosy e^y + y cosy e^y + ∫ y siny e^y dy - ∫ cosy e^y dy - ∫ y siny e^y dy
= (1/2) y² siny e^y - (1/2) y² cosy e^y + y cosy e^y - ∫ cosy e^y dy ............... [5]
∫ cosy e^y dy, Integrate by Parts, U' = cosy, dV' = e^y dy
= ∫ U' dV'
= U' V' - ∫ V' dU'
= cosy e^y + ∫ siny e^y dy, Integrate by Parts, U'' = siny, dV'' = e^y dy
= cosy e^y + ∫ U'' dV''
= cosy e^y + U'' V'' - ∫ V'' dU''
= cosy e^y + siny e^y - ∫ cosy e^y dy
==> 2 ∫ cosy e^y dy = cosy e^y + siny e^y
==> ∫ cosy e^y dy = (1/2) cosy e^y + (1/2) siny e^y ................... [6]
Substitute [6] into [5]:
∫ y² siny e^y dy
= (1/2) y² siny e^y - (1/2) y² cosy e^y + y cosy e^y - (1/2) cosy e^y - (1/2) siny e^y
= (1/2) y² siny e^y - (1/2) y² cosy e^y + (1/2) cosy e^y - (1/2) siny e^y
Substitute by y = 3x and add an arbitrary constant C and you're done
∫ x² e^(3x) sin(3x) dx
= (1/27) ∫ y² siny e^y dy
= (1/54) [ y² siny e^y - y² cosy e^y + cosy e^y - siny e^y ] + C
= (1/54) [ 9x² sin(3x) e^(3x) - 9x² cos(3x) e^(3x) + cos(3x) e^(3x) - sin(3x) e^(3x) ] + C
= (1/54) e^(3x) [ 9x² sin(3x) - 9x² cos(3x) + cos(3x) - sin(3x) ] + C
= (1/54) e^(3x) [ (9x² - 1) sin(3x) - (9x² - 1) cos(3x) ] + C
= (√2 /54) (9x² - 1) sin(3x - π/4) e^(3x) + C