Usually not, unless they are (1) fully healthy, and (2) sitting somewhere important where you don't have time to bring replacements. Upgrades are too expensive, compared to building new units. This is especially true for one-level upgrades. If you skip 2-3 levels ahead, say upgrade level 1 to level 4, then it makes more sense as the starting price is so low compared to the upgraded price.

Maybe upgrade them, but have a care; a major advantage of infantry is how cheap it is to produce and maintain. By upgrading you prevent your being able to spam them in future if this is needed, which is why I never upgrade regular infantry, or militia, past lvl.2. It is far better to research armoured cars.
Motorised infantry can be thought of as a complement to regular infantry; they possess similar properties, including armour class, with the exception that one is an offensive unit, the other defensive. Use them as such; regular for defence, and motorised for offence.
Mechanised infantry is entirely different. It is both slower and more powerful than motorised infantry, and is neither offensive nor defensive. I myself prefer to use it stacked with medium tanks and SP artillery for a punch through the enemy line, with motorised infantry and light tanks being used to exploit the breakthrough. Defensively it is useful, but usually most goes to the front line.
With the exception of the armoured car, no tank is as effective against infantry as infantry itself. Infantry have at least the utility of tanks, and, once commandoes and motorised infantry are used, are perhaps even the more valuable of the two. Quite apart from the fact that diversity is needed, a) in order to counter the enemy, who may have various units himself, effectively; and b) in order to use less of one resource, such as oil, so as not to have a shortage thereof while simultaneously experiencing a glut of food and possessing a useless grain production city.

With Pan Asian, it's usually not a bad play to upgrade your infantry especially by skipping a level. Their terrain advantages make it worth upgrading considering they are the best in the game. Otherwise, I'd say be careful and think it through. It's rarely worth upgrading only one level at a time. You need to consider the opportunity cost. I find myself often needing resources for other things, food is needed for both naval units, ordnance, and recruiting stations. Goods, for those same, but swap out naval units for air. Obviously you need manpower for every unit.

You should not consider it if you only play offensive just a waste of resources but may be if you play defensive

The best way to play offense is to take undefended land and then use terrain to your advantage to repel counter-attacks. Infantry and other defensive units are key to a good offense. Attacking units can kill things, but they are not good at holding onto your gains. You always need a good defense. Especially if you're playing offense.

You need to play a mixture of both. I've won three rounds by waiting for players to move all their troops to attack another player, then sending my own troops to come in from behind and take his undefended core provinces. Too many people make the mistake of not protecting existing territories; it doesn't matter how much you annex if you lose what you have.

The best thing is making non-existent undiplomatic nations in your neighbor, taking initiative first always helps in keeping your country safe instead of defending. But in the worst scenario, it might even end up in having to go for war with all neighbors. To play safe, may be it is the best to make friendly diplomatic with neighbor, I will choose the former option though.

The problem is, invading unfriendly, but not warlike, countries almost always makes AI's not like you. You also have the expansion bonus to take into account. That said, the objective of the game is the accumulation and retention of victory points.