where do i draw the line?

i work in fremont and there is this homeless man who hangs out there. i am not sure if he is mentally challenged or if he has done too many drugs, but he walks around and has random out bursts of yelling, body convulsions, dancing, and singing. however at the same time he can be perfectly normal. him and i have held many normal conversations. which i think had lead to my problem. latley he has been hanging around my work more. knocking on the window trying to get my attention, coming into the office, pacing out side on the sidewalk, and most recent he walked me to my car at night.

this is where i start to get a worried. he has never said or done anything inappropriate to me. but i don't feel safe alone with him, especially at night. i don't want to be the rude person who is mean to him, but i want to protect myself. where do i draw the line? and what do i say to him?

it's not like i can say "hey, i am not comfortable around you. please leave" i feel that would just make him mad. or can i say that? dr. mike says i am naive and he has decided to walk me to my car from now on when he is here. but there are times i am at work alone.

so am i being over dramatic? like i said above he has never done or said anything inappropriate. however i wouldn't let a sane man walk me to my car at night.