In our personal experience, a 20 footer is fine as long as you live in a 2-3 bedroom without a lot of clutter. Containers take A LOT of boxes. It's the big furniture items you collect with a house (and kids) that push you over the edge into 40 footer space.
We moved from a 2500 sq.ft. house (+ big basement). 2 adults + 2 young kids. Our 40 footer still had about 2 empty meters left at the end. Granted, we have very little stuff according to most people.
we moved all the clutter from a 4 bed house (no kids)
so a huge number of boxes (about 200 I think) but only a small amount of the furniture - 2 sofas, 1 sideboard, a couple of tables and a large desk and a couple of bikes. That fitted in a 20 ft container with a few inches to spare.