First, let's see why it can't work when the police buy illegal guns directly from their owners. If the police buy those guns for a price lower than the market rate for illegal guns, then the owners will just get rid of guns they don't need (ones that are old, defunct or redundant); in this case the supply either stays the same or increases (because the owners who sold the guns they didn't need might use some of the revenue from those sales to buy more guns). Now if the police buy illegal guns for a price higher than the market rate, then pretty much every owner of an illegal gun will sell it to the law, then go right back to the market, replace the gun he sold with a cheaper one, and pocket the difference. In this case as well the supply either does not change or increases.
Clearly, therefore, buying back from the owners doesn't work. But what if the government bought illegal guns directly from the dealers? Could that work? Let's see. Illegal gun dealers sell their product at some market clearing price that brings them profit. In order to divert the sale of illegal guns from individual customers to the government, the government would have to outbid individual buyers; i.e., it would have to offer gun dealers to buy their product for a price higher than its market value. Then, gun dealers would simply sell all their guns to the police instead of to the gangsters, and the streets would be free of illegal guns, right? Right?
Wrong, of course. Government resources are huge, but finite. Suppose (completely unrealistically) that the government devotes all its available funds to above market price gun purchases, so that every illegal gun currently produced is being sold to the police. There are now a lot of gangsters running around who need guns but can't buy them because the government outbids the highest price they are able to offer. There is untapped demand. And because the government is (by necessity) paying prices above the market value, a newly entering dealer can tap that demand by selling guns to gangsters at precisely the market-clearing rate. The customers will want to buy from that dealer, and the dealer will make a profit. Illegal guns are back on the streets, and the government is completely broke.
It's one of the amazing paradoxes of democracy: every college kid who has taken Econ 101 knows the gun buyback program cannot possibly work. And yet it continues, and will continue for at least some time in the future. Because politicians do not care whether their policies actually work; they only care whether voters think that they're doing something about the problem.
There truly are few more depressing and infuriating sights than that of some commissioner of some police department being photographed beside a heap of "illegal guns that our law enforcement has just taken off the streets."