They thought you didn't have control over anything but you choices.
And if you can't control something you shouldn't worry about it.
Here's that Epictetus guy again:
Some things are in our control and others not. Things in our control are opinion, pursuit, desire, aversion, and in a word, whatever are our own actions. Things not in our control are body, property, reputation, command, and in one word, whatever are not our own actions.