Largely, it doesn't fit the notion of "physical activity" or "physical skill" - poker is a mental activity. Otherwise, any skilled activity where there is competition between people - two people seeing who can complete a crossword first, Jeopardy!, pub quizzes, etc. - would also have to be called sports.
There is a lot of physical skill in poker. Much more than those other games you mention. The mental aspect is only part of it. Bluffing and otherwise not giving away your hand takes physical skill.
Most of the world considers chess to be a sport. I think the definition of "sport" isn't so hard and fast, there are many qualities that they share, but not all sports share every quality. Judging by recent Olympic standards, the criteria of competition, media and spectator involvement, and monetary dividends seem altogether as important as physical exertion. And poker is actually physically exerting -- this has been measured on players while playing, and try playing for 12-14 hours a day, for several days in a row, see if physical stamina doesn't start to enter the picture.
Well, that's a silly argument for exclusion. The whole issue which people disagree about is how the term "sport" is defined, or rather, whether the current or most widely accepted definition is the best one or the one that should be applied. Most people arguing poker is a sport are doing this by citing evidence that people have other criteria besides physical exertion to determine what is a "sport". In other words, they are contesting the meaning of the word. Simply replying "sports requires exertion" is kind of like arguing that something should be illegal because it is already against the law. (People actually use the logic with the issue of legality of drugs...)
I think that when a non-strenuous game like chess becomes very popular, the players seek to elevate the status of the game by calling it a "sport". This is probably only done because they seek to distinguish their game from others. This usage is not in line with many trusted sources for the word "sport" (see above, here's another). Chess is a sophisticated, classic, and popular game, just like poker.