Vendor trash is community shorthand for items not worth keeping or trading — you sell them to an NPC vendor for scraps or just leave them on the ground.
Example: most of what drops is vendor trash. Learning to walk past loot is a genuine skill, and a loot filter does the walking for you.
Related: loot filter