For individual characters, Insert Symbol works well - most of what you need will be in the Symbol font.
If you need to format equations, though, you'll need to use Equation Editor. You should be able to find this in the Office installation hierarchy under Tools or something along those lines. To use it, if you don't have the toolbar button displayed, go to Insert > Object and select Microsoft Equation from the list. It has several different groups for different kinds of symbols and placeholders. The placeholder ones make sure the formatting is correct, and can be nested - so if you want to write an expression for the square root of the determinant of a matrix of polynomial fractions, you can.
Equation Editor has some problems, though - at least it did when I last used it substantially, some years ago now. Be prepared for file sizes to balloon dramatically if you use lots of equations, and don't expect to be able to read them on a system running any other version of Office (though most of the time it will probably be OK). NEVER edit your document with any other version of Office, because it's likely that some of your equations will be scrambled if you do.
For the mathematical professional, though, there's no substitute for LaTeX (or, to avert flames, similar systems such as TeX, AMSTeX, etc). This allows you to use text codes for every imaginable mathematical operation (and probably a whole bunch more you've never heard of). This keeps file sizes small, allows you to edit your document on any software on any system, and automatically handles layout, cross-referencing, bibliography and more. Also, it's free. The learning curve is steep, though, so it's not worth learning for casual use. I recommend using an editor like WinEdt which has many features designed expressly for TeX/LaTeX editing. (Not free, but not too expensive either.)
To get many of the benefits of LaTeX without having to learn all the codes, you can try software like Scientific Word. This is essentially a visual front-end around a LaTeX/word processing combination package, so you can enter formulae directly in an Equation Editor-like interface together with text. However, you are obviously then tied to the software, and it's not very cheap ($180 US for a student licence, $525 US for an academic licence).