For anyone who has experienced frame-rate stutter issues in VR, NVIDIA has issued a hot-fix driver. This driver is not yet available via driver update, but you can download it from NVIDIA Support.
These issues have been there since around May 2020. Recently with the 3000 series cards it has been more apparent.
If you are experiencing such issues in VR and your NVIDIA driver version is lower than the one in that page, feel free to give the new driver a go. It has certainly fixed my issues. You can also choose to wait for the driver update to be rolled out via the official NVIDIA channels.
While this hotfix has solved my graphics issues in VR, it is worth pointing out that advanced users and overclockers have recommended (in the comments below) to skip this hotfix until the full fix is done, if you use tools such as afterburner, fpsvr and/or other monitoring software.
About a week or two after I posted this topic, NVIDIA released the official driver. This means the link in this post no longer works and displays an error. Please download the latest NVIDIA driver to have your issue solved. At the time of posting this update, I am using version 461.40 from the official update: