Do not get rid of them without replacing their use with something equal. Their use being the only remotely viable way to store things and feel that there is a decent chance it will still be there when you get home from work (mostly because they can be hard to find). Its bad enough I have to remake a new boat every time I log in. Over the weekend I cant even logout for lunch and come back to a floating boat.
They are not invincible. 2 people hitting it with stock tools can take a door out in about 20 min. Don't sleep in this, it can easily take more than 20 min trying to do it in other ways. If you can see the structure from above the water, you can definitely hit it with cannon bears and/or grenades. The trick is you cant see the graphic of the cannon ball or grenade, so you have to play around until you see the green numbers. If its deeper than this, I do not know if the cannons and grenades stop working, or if the difficulty to aim just gets too much. Regardless, switch to the costly explosive barrels. There is a YouTube video showing how to do it.
Overall, if structures were better these wouldn't bee needed. However I would expand upon them rather than remove them. Undersea terrain is a largely untapped real estate. Id like to see more resources down there, more viable ways to build down there. The submarine should be much more accessible to the average player. I think it would be neat to have a 2nd race that lives normally underwater and has a breath timer above. They could have a submarine comp that flies through the air, we'll call it.....an air ship!
For real, it should take much more work to break into something than it does to build it. Especially compared to how easy it is to do so currently. There's a reason castles were built out of stone and had to be taken by siege.