I think I angered the pregnancy goddess

Ok all, I am getting very frustrated and starting to think that I was not supposed to have babies (or even one). I am 35 and up until I met my husband I was not fond of being around children for extended periods of time and until my 3rd year being married, I never wanted to have a baby. Before I met my husband I was too busy with my career and social life to slow down for a kid. Funny thing is, I am actually ready to have a baby, and strangely something that I spent so much money trying to stop is so hard to start. Man, if I knew that having a baby was so hard, I would not have wasted my money on birth control.

I have been actively trying for the last 6 months, I have tracked my cycle for the last 4 years (initially trying to avoid the days I was ovulating, now trying to be there), and started chlomid 3 months ago. I have done tests, ultrasounds, and they all say that nothing is wrong with me.

I am going through so many emotions right now, frustration, sadness, confusion. What is wrong with me, have I angered the pregnancy goddess and cursed to wonder??? Should I give up...or do whatever it takes???