There is no clear explanation from Tolkien, if that is what you are asking. And I don't believe there is any general consensus. I sort of favor its being a town, but I know I've discussed the matter with a few people who feel it was probably just Radagast's home.

"Rhosgobel" is Elvish (Sindarin, or close it) and means "Brown Hay". The "hay" is an English term for a hedge, a defensive hedge surrounding a town or homestead. We Americans would more likely translate it as "Brown Hedge" since we don't use "Hay" to refer to anything but a dried grass used to feed animals (and in the South it's a semi-formal greeting when spelled "Hey", but that's another matter altogether).

Tolkien put hedges around both homesteads (like Beorn's in THE HOBBIT) and towns (such as Ephel Brandir on Amon Obel in the forest of Brethil).

Although most people associate Radagast almost strictly with animals, there is some reason to believe he also associated with Men and Elves. For instance, Beorn says he used to see Radagast on occasion. Of course, some people suggest that Beorn might be a special case given that he is a skin-changer. But Radagast's home was known to Elrond's messengers, and it appears he used the Westron when speaking with Gandalf (although we cannot be sure of that, but Gandalf's account of his meeting with Radagast seems to imply they were speaking as men rather than exchanging thoughts).

And Radagast's location in western Mirkwood seems to put him in with the Woodmen and Beornings. So, as Gandalf was most closely associated with Arnor, Saruman with Gondor (after returning from his journeys into the east), it would seem reasonable to suggest that Radagast was most closely associated with the men of the Vales of Anduin.

Therefore, a permanent or semi-permanent abode among men doesn't seem unreasonable to me. The book does not say Rhosgobel was deserted when Elrond's messengers arrived -- only that he wasn't there.

------------------
Parma Endorion: Essays on Middle-earth, Revised Edition