A good portion of the size difference (though certainly not all) is that ntp.org's ntpd has a lot of drivers to interface with physical devices [1], while OpenNTPD doesn't. If you don't own the physical devices, the LoC in the drivers are kind of a wash: they're not useful to you, but you also won't be running them. A stripped-down NTP-client-only build option would be nicer, though. PHK's suite of ntpd replacement tools seems to be going in that direction, offering a client-only tool for edge servers that just need to sync time over the network.
On OpenBSD the integration with physical devices is done via builtin software. I run a stratum 1 OpenNTPD server on OpenBSD, with a GPS attached via nmea [1]. The GPS data shows up via the sensors framework, which OpenNTPD interfaces with [2].
OSX really needs adjtimex or ntp_adjtime for high-precision clock synchronization for PLL loops (unless there's another way). If XNU devs wont do it, I may consider adding a new call to the sysent table via a kext (easier to do because of advances in modern open source rootkits). There's a partial port from a BSD to XNU for a starting point. XNU devs ignored that patch author entirely the first time. It's really a shame when collaboration for useful improvements are turned down.
[1] https://github.com/bsdphk/Ntimed [2] https://news.ycombinator.com/item?id=8781435