Husband's Longing Look

My DH and I have been TTC for four years now. We have been married for five years and have not prevented getting pregnant for six years. We were on Clomid for about a year and we decided to take a break from that because I needed a break from the hormonal roller coaster that it was putting me on. My husband has been extremely supportive. However, when I hear about someone being pregnant or having a child I want to be supportive and happy for them but at the same time it is disappointing for me because of how hard my DH and I have tried. It has been extremely hard when I see my DH looking at children playing and him saying how adorable they are and to see that look of longing in his eyes. How do you guys react to that and how are you supportive to your DH's and what do you say to your DH's when this comes about? I am supportive to my DH and when is the time to just give up. I am 32 years old and my DH is 38 years old. We have not ruled out adoption but would like to have a child of our own. I guess adoption would be an option down the road but just not sure down the road. Any ideas or suggestions or anyone else experience this with their husbands?