If your surgeon thinks that the follow-up chemo would be of little value, you might want to hold off, to save the chemo for if you need it in the future. Chemo is rough, and side effects can be many. I personally, would hold off unless the chemo was strongly recommended. However, you must keep on top of your scans and bloodwork. With or without chemo, your cancer may come back.
You must be sure about your decision, and it is yours. If/when the cancer comes back, you can't blame yourself for not having the chemo (if that is your choice.) I was told for my cancer that they'd never know if the chemo had any effect. I stopped after 7 rounds.