That's basically author Frederick Reiken's argument in this provocative post, about novels that approach time in unconventional ways.
Why can we accept flashbacks so easily, even in the most conventional narratives? Because novels use "block time," conceiving of time as space. That makes jumping between 2010 and 1986 as conceptually simple as a flight from Atlanta to Denver. For a science fictional example, think of Billy Pilgrim in Slaughterhouse-Five:
Billy has become "unstuck in time," and has no control over which part of his life he will wind up in next. All the moments of his life - present, past, future - can be arrived at as if they were geographic places. What I'll suggest is that a conventional, realistic work of literary fiction makes use of the same apparatus minus the spasmodic time travel. Instead of the protagonist becoming unstuck in time, the author simply moves the reader around within the block-time map. In other words, the reader is the one who becomes unstuck, because the story can go anywhere it wants.
Reiken also says if you want to understand how all time travel stories work, just read two stories by Robert Heinlein: "All You Zombies –" and "By His Bootstraps." He says:
If you can chart and understand the paradoxes that are exploited in those two stories (the first hinges on the so-called "grandfather paradox;" the second hinges on the eponymous "bootstrap paradox"), you will understand the mechanics of all time travel genre stories ever written.
For more thoughts on time travel, plus a list of novels that play with chronology, check out the original post at The Book Lady's Blog.