IMO this is where the kinds of concepts discussed in The Stanley Parable come into play. Simply put, when you reduce them to their basic elements, every game is like this. You can say "real" games have more free will or interactivity, but it's only as much as the game designers are willing to give us. In any kind of story based game there may be multiple paths to different events or multiple outcomes, but all of these are still following some kind of programmed, linear path. When you think about it, there really isn't a tremendous amount of difference between "Hold the right arrow key and see random shit" and "Hold the right arrow key and click the mouse and press the buttons at random times and see random shit". Whether it be the plot or (even in a sandbox game, for instance) the world we're put in, every game is laid out for us in advance.
I can agree that entries like Dear Esther aren't what some would traditionally consider games. They take minimalism and linearity to extremes other games do not. That being said, just because they're unconventional doesn't mean you can oust them from the medium entirely. It's a lot like 4'33", for those of you who've heard of it. it's minimalist and abstract to the point where (IMO) the amount of actual sound content wouldn't warrant a "Song of the Year" award or anything, but it's still music in the fundamental sense.