Most men's jackets actually come with hoods and are very warm on their own. Even in this picture,you can see that the fellow has a hood, or some sort of neck thing attached to his jacket. That's because most fashion designers are fully aware that man scarves are unattractive.
The only guys who can pull off this look are well-dressed gay guys. And they do a great job. But if that group does not include you, please ditch the scarf...and any ridiculous striped shirts you may have.