With Margot at the Wedding, writer-director Noah Baumbach has pulled off the rather dubious accomplishment of making an incisive, involving film about a thoroughly detestable group of people. Seriously, if you thought the bickering Brooklyn family at the center of Baumbach's last film, The Squid and the Whale, was insufferable, you ain't seen nothing yet.

Let's start with the title character, a neurotic, uptight author (Nicole Kidman) who is only capable of experiencing joy when making someone else miserable. Against her better judgment, Margot agrees to attend the wedding of her estranged sister Pauline (Jennifer Jason Leigh), who currently lives in their old childhood summer home on Long Island. A blubbering mess of emotions where her sibling is more of a Type-A ice queen, Pauline is rushing into marriage with local loser Malcolm (Jack Black) to the pleasure of nobody in her family.

Of course, Margot shouldn't be so quick to pass judgment, since her track record with men isn't stellar either. As we soon learn, the real reason she's come to Long Island is to escape her eternally patient husband Jim (John Turturro, playing the only decent soul in the picture) and hook up with a fellow writer (Ciarán Hinds). Tagging along for this roller-coaster family reunion is her pampered teenage son Claude (newcomer Zane Pais), who has the same blind admiration for Margot that Walt Berkman did for his overbearing dad in The Squid and the Whale.

Baumbach has freely acknowledged that he drew heavily on his own life to craft his previous film, and while it's not being billed as such, Margot at the Wedding also feels semi-autobiographical. Shooting the film almost entirely handheld and largely eschewing exposition, he establishes an intimacy that suggests he knows and understands these people. This may sound like a backhanded compliment, but few contemporary filmmakers are as skilled at writing such unapologetically obnoxious characters—either Baumbach is just preternaturally gifted or he really needs to start hanging out with nicer folks. Joking aside, you have to admire the writer-director for sticking to his vision and not forcing redemption on people who don't ask for it. Had Margot at the Wedding been directed by someone like, say, Joel Schumacher or even a more respected veteran such as Sydney Pollack, you can bet the film would have ended with a scene where Margot and Pauline finally lay all their issues on the table, then cry, kiss and make up. But Baumbach deliberately denies both his leading ladies and the audience that catharsis. In fact, his film doesn't end so much as stop; he understands that this family will never truly be healed and that the best they can do is go their separate ways before they make things worse.

Aside from Baumbach's compellingly cynical worldview, the other reason to seek out Margot at the Wedding is to watch the way his cast subverts their typical screen personas. Take Jack Black, for example—like most of his characters, Malcolm is a loudmouthed jerk prone to fits of rage when he doesn't get his way. While Black normally plays this role for broad laughs, here he taps into the deep desperation that exists in that kind of personality. Meanwhile, the common knock against Kidman is that she's too cold and remote a screen presence to elicit any real affection. In Margot, she takes that criticism to its logical endpoint, by playing a cold, remote character you're not supposed to have any affection for. This is the meanest role Kidman has played since 1995's To Die For and one gets the sense that she's enjoying herself immensely.

Leigh is the only one of the three leads who doesn't seem to be playing a variation on her screen self, perhaps because she's always been something of a chameleon. Nevertheless, the actress is quite good in what is probably the trickiest of the three roles. Our inclination is to pity Pauline for having to put up with her sister's cruelty and her fiancé's buffoonery, but Leigh and Baumbach are careful to show us that she's not a naive innocent. Like everyone in Margot at the Wedding, her actions are motivated by a combination of anger, hurt and naked self-interest. This may not make any of them likeable, but it does make them human.