I have tried several options - i.e. using the SuperSU to Unroot but somehow its not that "clean" and have some errors during Reboot to Recovery (even having the Android logo with the red exclamation mark).
Thus, with several retries, I found out the best and safest method would be using Skipsoft Android Toolkit. The latest Nexus 7 (2013) Toolkit at V 1.0.6
(Note: Please ensure you enable Settings --> Storage --> USB Connection --> Camera (PTP) to allow the PC recognise the device).
After the device has been unrooted, we can still use the Toolkit to Root the device back.
[NOTE: Its advisable to purchase/donate 4 Pounds to get the latest Nexus 4 Toolkit. It will solve many of the "pains" of Rooting/Unrooting Android 4.3 for Nexus 4].