Consider getting one of those specialized "beanies" that basically just covered the top of your head and your ears. Getting my ears clipped by branches when it's cold is just the worst and I used to wear the balaclava for primarily that reason. That got too hot though so i got the beanie thing. Don't really like the balaclava til it's 20 or less along with some wind.
dont know about you but at 25 with a balaclava on while riding i would sweat my nuts off.
Sounds like you might have it pulled down too far.
The Smartwool 'clava is nice--it's quite thin and comfortable right up into the 40's but with additional layers (a neck gaiter over it for example to cover more of your lower face) it's good as cold as anyone would ride. Basically, using it as a "base layer." Wore mine with the smartwool neck tube for this morning's commute--19 degrees and I was fine. Maybe a little too warm even. I wear the 'clava nearly every day in the season and it's held up fine going on 5 years. It is possible to pull it below my chin, WC, but I think they may have changed the design a bit so can't speak for the newer ones.