LAST EDITED ON May-01-15 AT 11:18 AM (EDT)
There are things in IM3 and Cap 2 that would be good to know, but pretty much all of them are called back and mostly explained in the film. Someone who has only seen the "Phase 1" movies + the Avengers is pretty much fine going into AoU.You don't need to see Guardians at all if you have even a basic knowledge of Marvel Comics history before going to see Ultron, but I will say that you SHOULD watch it, because it is really entertaining and I suspect you'd enjoy quite a lot of the things in it.
Also, AoU's banter levels: Excellent.