When I would listen to music and surf the net over 3G data at the same time, the music would sometimes skip. Completely unacceptable. Other applications would often have a few second lag too before I could interact with them. This even was true for answering the phone. If I was for example reading email and a call came in, the phone would go to the call screen, but wouldn't allow me to answer it right away. The swipe-to-answer finger gesture wouldn't work as the screen would be unresponsive for a few seconds. If the caller didn't stay on the line long enough I'd miss the call. Very irritating. This was never a problem on the 3G with previous versions of iPhone OS.
So, I downgraded back to OS 3, following these good instructions.
While reports of lag with the 3G are common, people have not been reporting it with the iPhone 3GS. While the iPhone 3GS has a faster CPU, I suspect it has more to do with memory. The 3G has 128 MB RAM, while the 3GS has twice that, at 256 MB. That's a huge difference, presumably saving on page-outs to flash memory. (The iPhone 4 has 512 MB RAM, which is actually twice as much as the iPad.)
I don't know if Apple will be able to address this performance issue on the 3G with future iOS 4 point updates. Hopefully they will, but no guarantees. At this point the iOS 4 features are not a big loss though, since multitasking of third party apps isn't offered on the 3G anyway, and folders as currently implemented in iOS 4 aren't something I will miss much. However, in the future, iOS 4 will be more of a necessity, as applications will eventually start requiring iOS 4 or higher to function.