I know there is a comment on this somewhere within the literature, but I cannot recall where it is at the moment. Suffice it to say that Annuminas does not appear to have been a fortress, but rather a center for culture and art.

My guess would be that Annuminas was abandoned over a period of years, finally becoming deserted sometime after TA 1409. The simple reason being that the Kings of Arthedain did not possess the military strength to defend the ancient capitol of Arnor (much as the Dunedain of Gondor eventually lost Minas Ithil and Osgiliath to the enemy).

With the defeats of Arthedain by the Witch-king, the remaining Dunedain were forced to seek refuge where they could - Fornost Erain (Norbury or 'north burg'), Amon Sul and the forts along the North Downs being the places which could most easily be defended.
It is suggested, however, that Aragorn planned to occupy Annuminas in the FO and make it once again the capitol of a restored Arnor.