Differences facial skin of men and women are not many. It's just that men's skin is thicker and generally not as sensitive as a woman's skin. So it requires different treatment with women. Although you do not wear makeup, it does not mean that your facial treatment can be of origin.
As with women, facial care needs special attention. You may wash your face too often or even rarely. It may even be wrong to use a face soap or wash your face at the wrong time. It will cause acne bumps and dull or dry skin.
Generally experts recommend men wash his face twice a day, ie in the morning and evening. If too often or more than twice, your skin can be a risk of dryness. The men can wash their faces in the morning while taking a shower, before leaving for work. At night, men can wash their faces with face soap that has scrubs, to remove dust and oil that stick after a day of activity.
Good luck!