However, as suspected, the 1.5 Mbps video bitrate limitation (along with the encoder settings) can be problematic with some content.
To illustrate this, I have taken this Cornell Laboratory of Ornithology HD video and re-encoded it to 320x180 (0.8 Mbps, 11.5 MB) using QuickTime 7.1.2 and 640x360 (1.6 Mbps, 23.1 MB) using QuickTime 7.1.3, with their respective default "Export Movie to iPod" settings.
Although the re-encoded 640x360 video looks fairly good in most areas, unfortunately in some spots the results are not pretty.
Here is a screengrab of the original high definition video, scaled down to 640x360 in QuickTime Player:

Here is the HD video re-encoded to 640x360 using QuickTime 7.1.3's "Export Movie to iPod" function:

Here is the HD video re-encoded to 320x180 using QuickTime 7.1.2's "Export Movie to iPod" function, and scaled up to 640x360 in QuickTime Player:

While there is (not surprisingly) a noticeable improvement going from 320x180 to 640x360 when viewing video at larger screen sizes, a higher bitrate (or different settings) for the latter would have been preferred. Unfortunately, it's likely that hardware limitations of the iPod prevent use of higher bitrate H.264, since it may be too difficult for the iPod's CPU to decode cleanly. (This is in contrast to last year's 320x240 H.264 limitation, which seems now to have been an intentional software limitation.) Overall though, the video looks pretty reasonable, and will satisfy most users, even with TV playback from the iPod.
[Update 2006-09-14]
It appears that several third party video encoders do not support the Baseline Low Complexity Profile required by the iPod for 640x480 H.264 video. For example, video encoded at that resolution directly from DVD using the popular Handbrake application does not work on the iPod. MPEG4 video continues to work fine, however.