I have to agree with the recommendations to see Winter Soldier, not only because it's necessary to understand what's going on in AoU, but because it's a good spy novel sorta story. No points for guessing who the Winter Soldier really is if you've kept even moderately up with comics in the last two decades.Iron Man 3 is one of those films that I could only really stomach once, not because it's bad, but because the tone is too bleak compared to the previous two films. Downey Jr. continues to be awesome as Stark, there's still plenty of quips and humorous moments, and overall it's an enjoyable film. And I'll add one item that was missed on the list of things this film brings to AoU, and that's that Tony's suits can now be AI-piloted.
The Dark World is a love it or leave it film. It's not a bad movie, at least in my opinion, but it's just not as strong as the others in Phase 2. Perhaps there's some sort of big pay-off coming up in Ragnarok, but right now it seems to be more a film that was moving things along than a story I could really get into.
And Guardians is an awesome film, worth seeing even if it isn't relevant to AoU. I like to think of it as the palate cleanser of Phase 2, sort of the light-hearted action film that the first Iron Man was. When you see it, you'll understand why a lot of us who saw it back in theaters think that it's like James Gunn accidentally tapped into the UF-verse.
After you've made your way through those films, Age of Ultron is definitely worth seeing. Ultron is one of the most insufferable villains Marvel has ever created, but James Spader makes him awesome. And the Maximoffs end up being one of the best parts the film. Plus, the Hulkbuster scene is every bit as amazing as the trailers promised. There's really no reason I can think of not to see it.