I'll just let you know, we purchased hull #314 1 year before our son was born. We were worried about not using the boat after he came along, but... My best sailing yet was his first and second summer. Last summer (year 3) was a bust due to a big move, but this year we're already rearing to go.
Sailing with an infant is a breeze (pun intended). We rigged up a hammock and off we went - food source came along as the deck hand.
Year 2, I solo'd all over the local hot spots with just me and him. Immobile enough that I could place him in the cockpit under motor, or down below when under sail. Sailed 4 - 5 nights a week, and had an absolute ball. Rafted up with other local club members at least every second weekend, and he got passed around from boat to boat to boat. His first steps were taken on the boat rafted up with 6 boats tied together over a long weekend. Definitely left him with a sitter for the races tho...
My headspace is that if I made excuses to not go when he was 1 or 2, I'd make more and bigger excuses as he got older, so we just built it into our lives, and now that he's really moving around, I'm glad we did, because he's already a bit familiar with the boat, and we're familiar with the routine of packing to have him on the boat, and how me and the Mrs. switch off to keep things under control when needed.
If you can afford to keep the boat, believe me - it's easy to adjust to an infant on board. Our friends have a Catalina 36. Still a comfy boat, but not nearly as easy to single hand as a J/30. I didn't think twice about pulling into any slip for the night, even in a moderate breeze, if we were out for the day. Big and comfy sometimes means more hands needed, and the J/30 gave me plenty of incredible nights with the wee one and me, plus mamma bear when she wasn't at work paying for my adventures