What happens if I claim land that contains someone else's hearth fire?
Their hearthfire will still be there. Removing other people's hearthfires on your claim is not considered a criminal act, but you still have to be strong enough to be able to do it, and hearthfires are a bit tough.
What happens if someone logs in and he finds himself on my land?
Assuming you haven't given anyone but yourself any rights on your claim:
He will be able to teleport back to his hearthfire even though it's on your claim. He may or may not be able to walk around on your claim after teleporting without performing the criminal act of trespassing to do so. I don't remember whether it's considered trespassing in that case.
If he was standing on some part of the land that now belongs to you, he won't be able to log back on at his last location, but instead will have the option to log back on at his hearthfire. If you have also destroyed his hearthfire, he will be moved to a random location in the world instead.