Sometimes screen sharing service on your headless Mac mini may crash. This should fix the issue:
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.screensharing.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.screensharing.plist