Key Events in Black Higher Education : The Journal of Blacks in Higher Education
Read through the link above and think of the hurdles and legal fights that have had to take place over the past 200+ years just to legally be allowed into certain colleges and certain programs.
1948: The U.S. Supreme Court rules in Sipuel v. University of Oklahoma that Ada Sipuel be admitted to the law school at the University of Oklahoma. The ruling states that blacks have the right to a legal education of the same quality as whites.
Harvard was founded in 1636. I'd say most of the top universities in the US were founded by the mid 1800s. It wasn't until the mid-1900s that the Supreme Court even gave blacks the right to a higher education in law. That's not to say black people weren't attending colleges, but white people made sure the options were attend a black college or we're going to make it as difficult as possible for you.
White families that have been in the US have had multiple hundreds of years of educational and opportunity advantages and despite court rulings the playing field still isn't level in 2019. Even if the Supreme Court ruling in 1948 magically leveled the playing field it's still such a massive head start that isn't going to disappear in 3 or so generations.