To understand the difference, we have to ask what the target is and what the cause of the attraction? Here is an example.
John was attracted TO teaching. He was attracted BY the promise of a long summer vacation.
Teaching is the target of the attraction. A long summer vacation is the cause of the attraction.