My mother went through this, but she's 88. That is the difference.
What you risk is an increased chance that the cancer will come back. I don't think a diet will provide the same level of protection. Yes, quality of life is very important. You don't want to get more years of bad health instead some years of good health. But, additional treatment might give you many years of good health. You're still relatively young. 20 years from now the deciding factors might be different--like it is for my mother.
Talk to your doctors. Tell them your concerns. How do they plan to treat you to reduce those concerns? What side effects do they expect you to have? How long will you have them? What do they expect to accomplish?