Ossan means “middle aged man” but its connotation is quite derogatory. It stems from a word “Ojisan” which means uncle or middle aged men in general.
Uncles related by blood and middle aged men in general are both called “ojisan” in Japanese. You may find this surprising but there are similar instances in the world. In some language they do not even distinguish a father from an uncle. On the other hand there are some culture where grandparents on the mother sides are called differently from the grandparents on the father’s sides. Similarly in the Japanese language there is no equivalent for the English word “sister”. The Japanese word “ane” means elder sister and the word “Imoto” means younger sister. There is no Japanese word for sister in general. Whether the sister is older or younger always matters.
Having said that Ossan is more impolite and less softer than ojisan, both may not sound too nice anyway as they both have connotations that they are old men.
The female equivalent of ojisan is obasan, however there is no female equivalent word for ossan.
Some argue that men may be called ossan or ojisan from the age of around 30 or 45 etc, However, in some situations grade school children may called a 17 year old boy “ojisan.”
This would be because historically Japan used to be a country of village culture and people in the same community used help one another as if they had been relatives and middle aged men in general helped raising the children in the community as if they had been their uncles. This would be more true for females since middle aged or elderly women actively took part in raising children in the community as if they had been aunts or grand mothers.
Although it would not make any difference in having a connotation of “middle aged man” or an “old man”, the politest form among the variants referring to middle aged male would be “ojisama” which is a word with respectable connotations.