Where in the United States can you buy gold bars or coins?

I would like to know what is the best way to buy gold coins or bullion in the U.S. I am not a coin collector; I don't care about that stuff. I want legit, recognized, gold coins that I can purchase from a reputable place with their certificate and KEEP with me - not have a piece of paper saying I own them and they are locked in some vault somewhere. Just as a form of security and piece of mind for me, I guess. I also thought about giving them as gifts to my small nieces and nephews rather than toys etc. once in a while.

Does anyone know what is the best way to go about doing this? I don't really care where the coins are from as long as they are authentic, certified, etc.

It is legal too, right? To buy coins from here or anywhere else? I'm obviously very ignorant about the subject. I had thought about doing this about 4 years ago when the price was like 0 per ounce, now I check and it is like 00 and I want to kick myself for not buying then.

Thanks!
By the way, I realize that the price of gold is ridiculously high right now, so I probably won't be buying very soon, but I would like to buy when it is reasonable again.