After playing with the SRV for a few days, I decided to file a Bug Report for the way the SRV targeting controls work.
As you can see here, they say its by design. So I did what they suggested, and posted in the Dangerous Discussions forum to get more people posting about it. Could use some help if you agree that targeting in the SRV should work just like the ships Targeting.
Thanks