I wonder whether someone knowledgeable has quantified it.
Were there free black workmen in the north before and after the civil war? Sure, but their numbers were incredibly tiny. The great bulk of black Americans were south of the M-D line and were slaves. There were some free blacks in the south, but their numbers were dwarfed by the slaves.
Overall the numbers of blacks involved in the actual construction of homes, industries, etc. was tiny. You can say blacks helped build industrial America (true), but they were just a tiny percent of the total work force. As the growth of America spread across the country post-war again the numbers of emancipated blacks involved in the creation of industrialized America was a tiny fraction of the work force.
Most blacks still lived down south working on farms now mostly sharecroppers. The great black migration to the north didn't mostly happen until the 20th century and after WWII.
Many parts of the country (like the upper Midwest) were almost totally devoid of blacks until the 20th century. Some people blame sundown laws driving out blacks, but most likely it was just a matter of many blacks not leaving the south even after emancipation.
Of course the phrase is being using by many black agitators to claim they are owed billions-trillions of dollars for all the unpaid or underpaid work they did "building the country."
But like many leftist claims (including the numbers of Indians in N. America pre-European settlement) it should be taken with a grain of salt.
