Older games were purely harder for economic reasons. This industry is a business and will continue to be a business until its decline. In arcades, games were made to be hard to empty your pockets faster and better. Some games had incentives like Free Plays for achieving a certain goal such as getting first place in a race. Other games had contests behind them, awarding the winner with a prize, however, players often paid down lodsadosh just for a chance to get that prize. Some of the more notorious games (Carnevil, I'm looking at you) had certain settings where it would literally be impossible to pass a certain part of the game.
Now that we have console games that don't need to be fed quarters every time you die, a necessity for difficulty is non-existent. People rather want to market their games to the masses, allowing as many fans to purchase this game. Today, games want to be accessible for as many people as possible, though some games focus on a certain demographic and others like to reach a wide,yet varied audience.