There could be some benefit in taking Xanax few times a week for a short time period even though you've managed without it. Your mind might know that you can have a break from anxiety for few hours if needed but your body might not comprehend it unless you do it a few times. It's a way of establishing control and that itself can lessen anxiety.
In no way can you treat Xanax like caffeine pills but you shouldn't be too afraid to use it either. Toxicity wise on it's own it's very safe. With dependency/withdrawal wise it still takes months of daily use at much higher doses. In recreational use people seek euphoria and a sense of carelessness. When looking for those tolerance grows very fast and even with no tolerance the dosage is bigger than what would be appropriate for you.
In your case all you need is disconnecting/suppressing the "fight or flight" reflex. In therapeutic use, besides taking far bigger doses, one way that can cause problems is "taking it when needed" when that's several times a day every day. Even without exceeding the recommended daily dosage, that will create a physical dependency usually with a very strong psychological component at that point.
Being aware of benzo addiction is very good, but the usual dangers apply less to you because of your personality and non existent history of drug abuse.