The Gregorian Calendar became a political football in the 1500s with the Pope and religion wanting their influence combined with government officials wanting their own particular political stamp on it. Ergo, we wound up with the mess we have today.
You still need an extra day each year, call it New Year's or Year's End, that won't be a M-Tu-W-Th-F-Sa-Su. It will just be an extra odd day. Every four years, you will have to add another extra day in kind (Leap Day). Except there will be three times every 400 years where you won't do this.
So, add one day each year and another 97 days every 400 years. Do that, and it will work. Outside of that, you could ditch that whole cesium clock and add 0.310625% to the length of a second and you could have your 364 day calendar. Of course that would screw up the day-time portion. Eventually, midnight would occur during the day. But at least the calendar would be right.
The Gregorians may not have been good at naming months, but they got the time fix right. The problems is that the rotation of the earth is not in sync with the revolution of the earth around the sun.
