"I am Gandalf the White, but Black is mightier still"
All pedant arguments aside. Tolkien sets Black with evil and White with good (and no..this has nothing to do with race, skin colour, hair or clothes).

And Olorin: Gandalf says this before he strips Saruman of rank. You are also right Olorin in saying that Saruman is connoted with Black after his fall. Gandalf says he will set his silver (Shadowfax) against Black before he rides to battle against Saruman.