"..and the house of Elrond was a refuge for the weary and the oppressed, and a tresury of wise councel and wise lore... and harboured the Heirs of isildur... because he knew in his wisdom that one should come from there line to whom a great part was appointed..."

Also Gandalf was in closet councel with Elrond of all the Elves or inhabitents of M.E. he was also one of the three people that knew where and when the Istari came from.

As for skill with a sword.. didn't he defend Celeborn at some stage?? two of them i a battle or something... they losing.. and him and Celeborn escaped or soemthing.. and he saved his life or something??
i don't recall what happened... i think it is in unfinished tales or Sil.. somewhere... my unfinished is on loan to someone... so i can't check.

Beren Valthôl