The Census Results

Let's imagine two countries with equal populations of 100,000: Richistan and Pooritania. They have the same income distribution, except that at each point on the distribution, individuals in Richistan have twice the income of people in Pooritania. The 1000th richest person in Richistan makes twice as much as the 1000th richest person in Pooritania and the 90,000th richest person in Richistan makes twice as much as her counterpart in Pooritania.

Let us suppose that over the next 20 years, 20,000 people, randomly distributed across the income distribution, move from Pooritania to Richistan. As a result of their move, their incomes go up by 50%.
Everyone else in Pooritania and Richistan just gets an increase of 5%.

What would the statistical result be? Median wages in Richistan would stagnate, and the incomes of the lowest quintile would go down. Much like the census results. But everyone would be materially better off.

Some in Richistan might still object that it has become more unequal (or, equivalently, more diverse). These objections are not necessarily unreasonable -- there are costs as well as benefits of greater diversity. But the objections are to the presence, rather than the existence, of poor people.