The caves are located in the heart of Vijayawada City at "Kasthuribaipet and Mogalrajpuram". The caves are said to be excavated in 5th Century AD. The idols of Lord Nataraja, Vinayaka and Arthanareeswara can be seen here.
The Mogalarajapuram temple has an 'Ardhanarisvara' statue which is thought to be the earliest in South India and one of its kind. One of the caves has an overhanging cornice with artificial windows. The idols of Lord Nataraja and Vinayaka in one cave are still in good condition compared to the other worn out images of deities. It is also a place of archaeological importance.