People act like the worst accusation you can hurl at storytellers is "They made it up as they went along." As if having a Master Plan is the same as good storytelling. In fact, it's frequently the other way around.
Often the best storytelling is improvisational, like jazz or chess. Whether you're talking about books, television, movies or comics, the cleverest and most fascinating stories often come out of a writer's desperation in the face of a roadblock. Obviously, when it comes to television, or a series of movies, these roadblocks may come from outside — an actor can die, get pregnant or quit, or a studio executive may demand more shiny robots — but these roadblocks exist in every medium.
Even if you're a novelist working in absolute isolation, you'll eventually have to try and publish your book, and you'll be faced with feedback on your carefully honed story. But also, the roadblocks may come from inside you — maybe you've decided that one of your characters would turn out to be a worm living in a synthetic human skin, but when you come to write that startling revelation, you discover that it doesn't work. It doesn't make sense, or it doesn't jibe with what you've already established about the character. The synthetic human skin no longer fits.
So really improvisation is a necessity, not just somehow the mark of a Bad Storyteller. Everybody does it, although some more than others. In the case of Lost, for example, the actor Adewale Akinnuoye-Agbaje decided to abandon the show after a single season, forcing the producers to go back to the drawing board on a huge storyline they'd planned for his character, Mr. Eko. This sort of thing happens all the time.
As Joss Whedon told Laura Miller in a 2003 Salon.com interview about the final season of Buffy The Vampire Slayer:
The master plan does not have a master plan. Television ultimately finds itself, and after it finds itself, it finds itself changing. I'd have a year plotted out, maybe two years in advance. And I had the major points that I knew I needed to hit and they would serve as anchors and we'd get from one to the next, and that was great. But the rest you deliberately don't have a master plan for, because you don't know what's going to happen. Apparently people seem to be responding to this Boreanaz fellow [David, the actor who plays Angel]. Apparently Seth [Green, who plays Oz] is gone. Apparently this villain isn't working out and this one's popping like crazy. You need to improvise, you always need to.
TV's like whitewater rafting: Without rocks, there wouldn't be rapids, and it wouldn't be as much fun. Rolling with it gave us Tara and Willow coming out. It gave us Spike falling in love with Buffy. Rolling with it found out that Anya and Andrew were comic geniuses. You plan your ideas and themes, and then you let the rest form naturally, and then it feels real. It doesn't feel like you're imposing something on everybody. Ultimately, the staff-who are the biggest fan-geeks in the world, and I'm including myself-when they watch an episode have to feel the way the audience does, and more importantly the characters have to feel the way the audience does. If the audience doesn't buy that Buffy's brought back from the dead, then Buffy can't buy it. They've got to go, "I can't believe this has happened. It's horrible." If the audience is feeling the loss of Angel and feeling that she can't have a relationship with Riley, she's got to feel the same way. You feel that out.
You tend to hear about the ideas that didn't work out, and the ones that came out of nowhere, more on television than in other media. Because television is so high-profile, and so many people work on it, and so much material gets generated in the course of a TV season. So for example, we know that Javier Grillo-Marxuach was dead against having a romance between the Middleman and Lacey on The Middleman, but the show's other writers pushed for it, and it ended up being one of the best parts of the show.
Here's the thing — improv storytelling and midstream ideas aren't just a cruel necessity, they're a huge part of what make the best books, TV shows, movies, etc. worth our time. There are a few reasons for this, off the top of my head:
1) The people live! They breathe! They struggle and cry! The thing about a story outline is, it's usually all plot, with just a few ideas for how that'll grow into a story. Unless all of the characters are just unimaginative robots — and there's nothing wrong with that — then the characters will start adding a lot of complexity to the mix as soon as they're more than just a set of names and actions in a bare outline. Oftentimes, it seems like the better the characters, the more likely they are to wander off the path a bit.
But also, great characters rise to the challenge when the story takes a seemingly random turn. If outside circumstances, rather than your own creative frustrations, force you to swerve in a whole new direction, then your characters will suddenly show some thrilling new dimensions. Really great characters will make that detour in the story feel not just natural, but more fascinating than whatever you'd originally planned.
2) Good stories feel like anything can happen, because it can. I'm a big believer in "narrative energy," or the kind of static-electric crackle you get from a story where the inventiveness is cranked up to maximum. It's impossible to quantify, or even to identify really, but you can tell when a creator or set of creators is having fun coming up with stuff, and when they're just grimly plodding towards the finish line along a predetermined path. Given the choice between a story that gets too random, and one that carefully lays all the bricks one after the other, in a totally straight line, I'd take the former every time.
A lot of bad storytelling feels like a grade-school essay: It tells you what it's going to tell you, then it tells you what it's telling you, then it tells you what it's told you. All of the dominos are set up in slow motion, then toppled in even slower motion. Obviously, out-of-nowhere storytelling can also be hideously bad (hello, Heroes!) but nothing is worse than the plod.
3) Seat-of-the-pants storytelling makes a mythos bigger and stranger. A truly great swerve is like a really great retcon — it recasts everything that's come before in a new light, and adds an extra dimension to everything. To make all the pieces fit, you end up having to add a whole new layer of freaky to your backstory and your worldbuilding. And if you do it right, it winds up being more memorable and cooler than whatever painted backdrop you'd put up originally.
4) Lesbian Willow. That is all.
Thanks to Sheerly Avni for suggesting this topic and providing the Whedon quote!