Buying a 2 family home with my mother? Good Idea or Bad Idea?

I'm considering buying a 2 family home with my mom. I would get the bigger unit and my mom will get the smaller unit. I am recently married and my mom love my husband to death and vice versa and me and my mother have a great relationship and she is the only person in the world that I would buy a house with besides my husband. We live in New York City and homes are very expensive especially in good neighborhoods. I was considering purchasing a 2 family home with my mom and I would take on the owner's unit and my mom would take the tenant's unit but we would co-own it. I spoke to my mom about it and she was ok with taking on the small unit because it's just her and her partner of 14 years. My husband and I are will eventually start having kids so it would be best if we get the bigger unit. This will make the mortgage cheaper and allow us more flexibility in our income. Not to mention my mom will be closer to her grandchildren which is what she wanted anyway and we would have a baby sitter :). For the most part we are young. I'm 25 and my husband is 27. My mom is only 47 so she is also still pretty young herself. Me and my mom are just a like when it comes to being responsible with bills, etc. We have great credit scores and everything. I just want to make sure we are doing the right thing. My husband said he's ok with it. I just want to make sure I'm doing the right thing. Most couples try to move away from their parents not with them. So I wonder if I'll regret it later even if I do love my mom to death and we have great relationship. My mom is not the type to be in me and my husband's business so I don't have to worry about that either. Me and my mom are both renting and I just don't want to throw money away. Technically the mortgage will be split four ways instead of 2 and i think that will make all of the difference. I know I'm rambling. But what do you guys think. What would you do? Is there anything that I'm not considering that I should be thinking about when making a decision like this?