Many times the project I had worked on eventually we have to tune in JVM when it was production.
Here is some recipes from expert:
1. Introduce: J2EE Performance Tuning

2. Improving Java Application Performance and Scalability by Reducing Garbage Collection Times and Sizing Memory

3. Trouble-Shooting and Diagnostic Guide contains many tips how to find problems that are often related to performance

Reference from Sun:
HotSpot FAQ
JVM Tuning whitepapers
JSDK 5 JVM tuning

Update: 02-12-2008

Inside the Java Virtual Machine