Concrete, stone, concrete block, and other masonry foundation materials have very little insulating value - a foundation wall can account for approximately 20% of heat loss. Adding insulation will keep your basement warmer, make the floors above warmer and more comfortable, plus it will lower heating bills. Insulating is often the first step toward creating additional living space for a growing family, home office, or similar use. The Model Energy Code recommends R-11 insulating values for basement walls in most of the country and R-15 for the northernmost states.
For an unfinished basement where you don't want or need finished walls and want to leave insulation exposed, a good choice might be a blanket insulation that is covered with a white flame-resistant polypropylene facing, such as CertainTeed's Basement Insulation.
For finished basements it's best to install traditional faced or unfaced batt insulation between studs in a framed wall. Although you can attach foam insulation panels to the walls between wood furring strips, Fiberglas is much cheaper. Building and fire codes require that foam or Fiberglas batt insulation be covered with drywall or other finish wall.