Issues after Depo

My husband and I decided that we were ready to have another child at the end of last year. My last Dep shot was 1 Aug 2006. I was on the shot for a year and a half after the birth of my son. My first year on Depo went fairly well, my only side effect was the mood swings. I really enjoyed not having a menstrual cycle. After that, it just started doing weird things with my body. We discussed the option of coming off the shot, but decided to wait a few more months. My body went really crazy after coming off the shot. My cycles were completely random and still have not sorted out. It only took 3 months to get pregnant with my son, and we weren't even trying. We both would like a sibling for our son before he gets too much older. Both my husband and I have siblings that we are not close to because of an age gap. He is 10 years younger than his brother, and I am 13 years older than my sister. My ideal age gap is 2-4 years between my son and his sibling. I was expecting to conceive shortly after coming off the Depo. However, I talked to some of my neighbors and co-workers and they have told me that it took them 2-3 years to get pregnant after coming off the shot. Is this a normal thing, or am I just too impatient?