Although Heian Jingu Shrine was only established during the Meiji era, its architecture recreates the Chodo-in, the imperial government complex of Heian-kyo. As a result, its sense of scale and space feels very different from Kyoto's typical temples and machiya townhouses. In particular, the vermilion buildings around Otenmon Gate and Daigokuden Hall, paired with the expansive courtyard, offer a relatively direct sense of the grandeur of Heian-kyo's imperial architecture. It remains quite distinctive among Kyoto's many temples and shrines.
There aren't many complete, full-scale structures where you can experience what a large Heian-period palace looked like today. Much of it can only be imagined through ruins, reconstructed models, or later reconstructions, so even though Heian Jingu itself is a modern building, I still think it's well worth seeing.
Unfortunately, I didn't have time to visit the shrine garden in the back on this trip. The garden was designed by Meiji-era landscape architect Ogawa Jihei and is especially famous for its weeping cherry blossoms in spring. I'd like to go inside and see it next time I have the chance.
Also, anyone who collects goshuin shouldn't miss it. There is an amulet office on the grounds of Heian Jingu where you can receive goshuin and omamori charms, making it convenient to stop by after worshiping.