It matters just like balancing a check book matters.
Say we start with 50 balls. The odds of picking the desired number are 1/50. The odds of picking the second number are 1/49 and so forth.
If the ball is returned then the odds of picking a number would be 1/50 every time. The odds of winning are substantially different between the 2 methods.
The balls in lotteries I have watched are always kept in a tray and never returned. If you have ever played the lottery you know that the winning numbers are always all different.