Is he ultimately responsible, no matter what?
It may be my Navy background and of my understanding the responsibility and authority of the captain of a ship, but for me that answer is yes. He has the authority. He also gets the responsibility. They go together hand in hand. No ifs, ands, or buts.
You didn't have to marry her. You (hopefully) did your homework. You studied her character in advance. You lived with her. You have the authority during the marriage. It may not all be in your control, but it is always your responsibility. It may not be fair. That is just the way it works, at least in my mind.
If something is not right, you have to recognize that and fix it. You might as well take control of the rudder, because you will be called to accout for the course of the ship and its safe passage.