The Creepy Neighbor.

I know that everyone has a strange neighbor of some sort, I mean my poor sister has one that likes to blast porn at strange hours of the day (awkward!) and I'm sure there are some who like to spy on you or throw trash in your yard or whatever. Mine is seriously starting to cross the line. 

We live next to an older couple who have older kids living with them and they seemed great when we first moved in- they came to introduce themselves immediately, say "hi" every time we see them, and they love L. L loves them too and gets really excited every time he sees them. They will take our trash down to the street if we forget, water my dead plants since our house is currently out of commission (thanks, Artemis), and are generally very good people. The wife has a firery personality and will get into anyone's face if they do something that bugs her (which is quite frightening, and she's only 4'10.") However, it's the husband that is making my skin crawl.

He comes out every so often to chat with Nick and me (if we're outside,) but most of the time its when I'm the only one home. I didn't think anything of it since Nick is gone until late due to school and work, but I've noticed that the neighbor takes off right after Nick does get home. Once he said something (and I can't remember his exact words) about him being my boyfriend and he's always saying stuff like "we better not get caught or people will talk." I thought it was strange, but I figured that maybe he was being silly or something. We had never talked for longer than ten minutes and I know that I have never said anything to give him the idea that I was interested in him in any way. Then he kissed my cheek and I avoided him for a month.Things seemed to be back to "normal" friendly neighborliness, until last night:

CN: One day we'll have fun

Me: what?

CN: you know, without the wife or husband barging in

me: what? (I had been studying for twelve hours at this point and I know my brain wasn't functioning correctly. He couldn't possibly mean what I think he does.)

CN: I'm down. You're down?

Me: uh that depends on what you mean by "fun."

CN: whatever you want it to mean. 

Then he walked away before I could say anything else or possibly burn myself with a lighter to make sure I hadn't fallen asleep while studying and was having a nightmare. Even if I didn't love my husband beyond reason, CN is the same age as my dad and not my type in any way. He's pretty yucky. Seriously, I have no idea what to do. I don't want to go over to his house to tell him that his behavior is wildly inappropriate for fear of his wife, and I definitely don't want to invite him to mine. Clearly this guy has invented some sort of attraction between the two of us and I seriously don't know how to handle it. I'm afraid to go outside and have been running to the car with L in hopes of avoiding him all together. Nick knows all of this and has from the beginning. He says I just have to tell him straight out that his advances are not ok, and I agree with Nick. I'm just a chicken. And sleep deprived at the moment. 

What would you do in this situation, besides move, because I really do love my house? Is it time to "man"-up and face him or just keep avoiding him until he gets the message?