Ivey attended high school at Highland Park High School, in Highland Park, Illinois. There, he ranked number seventeen in the list of historic Highland Park scorers with 721 points scored. He scored a high school career high of 31 points over Rich Central High School in 1982. He graduated the same year.
Ivey attended college at Boston University. He was elected to the All Conference second team in 1985-1986 and was one of the three captains of the team that year. Ivey played 119 games in his four year college career spanning from 1982 to 1986.