An anchor transaction accepts any 64 bytes (512 bits) as the hash. This allows you to choose almost any hashing algorithm. Based on your requirements choose a fast algorithm like SHA-2 or BLAKE2b or a slow algorithm like BCrypt. The algorithm isn’t stored in the transaction, it’s up to you pick and use one consistently or to store which algorithm was used.
To hash a document, read it to apply the hashing function on the binary data. Structured data should be serialized (e.g. as JSON). Beware that serialization process should be deterministic; in other words, if you provide the same data, you should always get the same serialized data.
For public records, simply apply the hashing function on the data and use that for the anchor transaction. This could be applicable for public archives, allowing them to prove the authenticity of stored documents.
This should not be used for private data as the hash could leak information. For instance, if an organization would create a hash from client-data existing of the customer name, address, etc, an unauthorized party might be able to recreate the hash and find out if a person is a client with this company.