Those Longing Looks

My husband and I have been TTC now for over four years unsuccessfully. We have been married for five years and have actively not prevented getting pregnant for six years. My husband has been very supportive we tried Clomid for about a year and then decided to give up on that because I wanted a break from the emotional roller coaster that it was putting me on. Currently we do not have the finances for IVF and we do not have any fertility clinics in our area so we would have to travel at the minimum of four hours to even get a consultation for one with having all expenses out of pocket because our insurance does not cover anything related to infertility. So far the doctors have not found anything wrong with either my husband or myself so we should be able to get pregnant we just haven't yet. I am 32 years old and not sure when to just kind of say forget about it. I was hoping to be on my second child by now but we are still hoping for our first. What hurts the most is seeing my husband's longing look at children playing and saying how adorable they are and just seeing the look in his eyes it really just breaks my heart. Anyone know how to comfort their husbands in this situation? I try so hard to but just not sure what to say to him and we are still undecided about adoption at this point.