It depends....
In calm water with not a lot of moving around you might be alright. If there is any chop or swell, I'd think twice about it. How agile are people and do they know what they are doing? I'd be more inclined to go out with a large group knowing that people can take care of themselves. If you have a problem, it doesn't take much for it to turn in a disaster, particularly if people don't know what they are doing and panic.