As others said, under armor is great. It fits snug to the body and allows you to layer on top of it without adding a bunch of bulk. Getting an afternoon tee time will help secure part of the warmest part of the day, too, so it won't be quite as chilly.
On those extra chilly days, I use ear muffs to keep my head warm.
I still have a golf hat on, but the ear muffs go over the hat nicely, and I still have easy access to my ball marker.
I'm in Colorado Springs, and for the past 8+ years, have played golf at least once every month of the year. We get snow, but with the low humidity, it generally doesn't stay long.
I know a lot of places frown on it, but during the winter (only) I wear jeans to stay warm. I use a long sleeve 'performance' shirt under a polo, and usually an insulated pull-over. That keeps me plenty warm as long as the sun is shining and the wind is down.
The coldest I ever played in was 26 degrees, yes, it was cold-ish, but no wind and sunny made it very tolerable. By the 4th hole, it was in low 40's and very enjoyable.
Under Armour base layer, mid weight fleece, topped of by a wind shirt or wind vest. Towards the end of November I'll start using gloves on both hands. Mizuno and FootJoy make reasonably priced winter gloves. Both Under Armour or Jockey make great base layer bottoms and I'll usually skip the normal golf pants and go with wind pants. And the flask is a necessity up here...