I Don't Know What to Think

Maybe you wise and wonderful mommies out there can help me with this. We have some new neighbors and there are two little boys that Little Man has started to socialize with. One of the boys is 9 and the other 6. They play video games together, at our apartment. Not that I would mind if Little Man went over there, but there hasn't been an invitation in that direction. Which is fine too.
Anyways, here is my little dilemma. The older boy, Rudy, let's call him, is um, well, rude. Rude and loud and he calls both Little Man and his brother some marginal names. He says things like "what the!" and "you suck". Which as I mentioned are marginal in my book. He is manipulative and I guess I really don't like him at all. But Little Man LOVES him. Wants to always play with him. If it were left up to me, I would never, ever have him over, I think he is brat. But my boy, my precious boy, my boy who has so little by the way of joy in his life, delights in his company. And I am stuck. I try to do some guidance by way of saying things like, "no shouting", "please don't call the other boys names" and "I think you could...instead", but still the kid really is a monster. So what would you do? What do you think I should do? Talk to his mom? I could, we have chatted quite a bit, try to keep guiding the kid into treating my boy better? What?
Edited to add: I am less worried Little Man will pick up the way Rudy acts and more concerned about Little Man being manipulated and used for his awesome toys. Rudy will threaten to leave unless Little Man concedes to what this child wants, including things like allowing Rudy to play a one player game that leaves the other two out of it. The younger brother seems quite acclimated to his brother extortion and threats to tattle or leave if he doesn't get his way and so the little brother just lets him have his way. I think Rudy is pretty marginal in his treatment of my boy, but once more, I have been guiding and putting up with it because Little Man loves the time they spend at our house. I am trying to allow Little Man to work things out for himself and not spending all my time hovering over the kids. But geez, it is hard, because he is so vulnerable.
And Chag? I thought about you when I said "mommies", and almost said parents instead, but I reminded myself you hadn't commented in a long time and I thought perhaps you weren't reading, so I went ahead. I will be more careful in the future! My apologies and thanks for the advice.
P.S. I called Dr. Tall and Thin's office, but the nurse said she hasn't been able to pin him and get him to answer the question for Little Man's PICC line insertion. So we continue to wait. And worry. Next week.