Hashcat identify hash format by modes. See a list of hashcat modes.
sha512crypt (mode 1800)
Take the just the hash from passwd and remove all unrelated data (user, gecos, etc)
hashcat -m 1800 -o output.txt shadow.hash rockyou.txt --forceNote that --force ignores all warnings.
salted SHA512 (mode 1710)
hashcat -m 1710 -o cracked-hash.txt hash:salt rockyou.txt --force