Well, the reason for chemo is because there may be individual cancer cells in your body that they didn't catch. It provides an added level of prevention of a recurrence.
Probably you should discuss your concerns with your doctor. Ask how he intends to control the side effects. Also, don't be shy about getting a second, or third, opinion.