There's probably a bottleneck in your config. Either at the hypervisor level or inside the VM.
On my instance, I have 150 tickets/day, up to ~40 agents and up to ~140 customers simultaneously connected. I'm running Apache 32-bit/MySQL 32-bit/Strawberry Perl 32-bit/Windows 2008 R2 64-bit in a VM with 4 CPUs on Hyper-V (Windows 2008 R2), on 4 year old hardware. I think my access times are slightly better than yours.
Ensure your VM performs optimally. It may be worth to install OTRS in a different VM, on a different hypervisor and see if you get the same performance.
Make sure you follow chapter 6 from the admin manual for performance improvements but really for 2-3 customers and 2-3 agents you shouldn't need to do make any special performance performance tweaks for OTRS to runs decently fast.