Lecture 21: Hash functions (2024)

Hash functions

Hash tables are one of the most useful data structures ever invented.Unfortunately, they are also one of the most misused. Code built using hashtables often falls far short of achievable performance. There aretwo reasons for this:

• Clients choose poor hash functions that do not act like random numbergenerators, invalidating the simple uniform hashing assumption.
• Hash table abstractions do not adequately specify what is required of thehash function, or make it difficult to provide a good hash function.

Clearly, a bad hash function can destroy our attempts at a constantrunning time. A lot of obvious hash function choices are bad. For example,if we're mapping names to phone numbers, then hashing each name to itslength would be a very poor function, as would a hash function that used onlythe first name, or only the last name. We want our hash function to use all of the information in the key. This is a bit of an art. While hash tables are extremely effective when used well, all too often poor hash functions are usedthat sabotage performance.

Recall that hash tables work well when the hash function satisfies thesimple uniform hashing assumption -- that the hash function should look random.If it is to look random, this means that any change to a key, even a small one,should change the bucket index in an apparently random way. If we imaginewriting the bucket index as a binary number, a small change to the key shouldrandomly flip the bits in the bucket index. This is called informationdiffusion. For example, a one-bit change to the key should causeevery bit in the index to flip with 1/2 probability.

Client vs. implementer

As we've described it, the hash function is a single function that mapsfrom the key type to a bucket index. In practice, the hash functionis the composition of two functions, one provided by the client andone by the implementer. This is because the implementer doesn't understandthe element type, the client doesn't know how many buckets there are, andthe implementer probably doesn't trust the client to achieve diffusion.

The client function h_clientfirst converts the key into an integer hash code,and the implementation function h_implconverts the hash code into a bucket index. The actualhash function is the composition of these two functions,h_client∘h_impl:

To see what goes wrong, suppose our hash code function on objects is thememory address of the objects, as in Java. This is the usual choice. Andsuppose that our implementation hash function is like the one in SML/NJ; ittakes the hash code modulo the number of buckets, where the number of bucketsis always a power of two. This is also the usual implementation-side choice.But memory addresses are typically equal to zero modulo 16, so at most1/16 of the buckets will be used, and the performance of the hash table willbe 16 times slower than one might expect.

Measuring clustering

When the distribution of keys into buckets is not random, we say that the hashtable exhibits clustering. It's a good idea to test yourfunction to make sure it does not exhibit clustering with the data. With anyhash function, it is possible to generate data that cause it to behave poorly,but a good hash function will make this unlikely.

A good wayto determine whether your hash function is working well is to measureclustering. If bucket i contains x_i elements,then a good measure of clustering is (∑_i(x_i²)/n) - α. A uniform hash function produces clustering near 1.0with high probability. A clustering measure of c > 1greater than one means that the performance of the hash table is slowed down byclustering. For example, if all elements are hashed into one bucket, theclustering measure will be n²/n - α =n-α. If the clustering measure is less than 1.0, the hashfunction is spreading elements out more evenly than a random hash functionwould; not something you want to count on!

Unfortunately most hash table implementations do not give the client away to measure clustering. This means the client can't directly tell whetherthe hash function is performing well or not. Hash table designers shouldprovide some clustering estimation as part of the interface. Note that it'snot necessary to compute the sum of squares of all bucket lengths; pickinga few at random is cheaper and usually good enough.

The reason the clustering measure works is because it isbased on an estimate of the variance of thedistribution of bucket sizes. If clustering is occurring, some buckets willhave more elements than they should, and some will have fewer. So there will bea wider range of bucket sizes than one would expect from a random hashfunction.

For those who have taken some probability theory:Consider bucket i containing x_i elements. For each of the nelements, we can imagine a randomvariable e_j, whosevalue is 1 if the element lands in bucket i (with probability1/m), and 0 otherwise.The bucket size x_i is a random variable that is the sum of all these random variables:

x_i = ∑_j∈1..n e_j

Let's write 〈x〉for the expected value ofvariable x, andVar(x) for thevariance of x, which is equal to〈(x - 〈x〉)²〉 =〈x²〉 - 〈x〉².Then we have:

〈e_j〉 = 1/m
〈e_j²〉 = 1/m
Var(e_j) = 1/m - 1/m²
〈x_i〉 = n〈e_j〉 = α

The variance of the sum of independent random variables is the sum of theirvariances. If we assume that the e_j are independentrandom variables, then:

Var(x_i) = n Var(e_j) = α - α/m = 〈x_i²〉 - 〈x_i〉²
〈x_i²〉 = Var(x_i) + 〈x_i〉²
= α(1 - 1/m) + α²

Now, if we sum up all m of the variables x_i, and divide by n, as in the formula, we should effectively divide this by α:

(1/n) 〈∑ x_i²〉 = (1/α)〈x_i²〉 = 1 - 1/m + α

Subtracting α, we get 1 - 1/m, which is close to 1 if m is large, regardless of n orα.

See Also

Hashprice Explained: A Crucial Metric in The Mining Industry

Now, suppose instead we had a hash function that hit only one of everyc buckets. In this case, for the non-empty buckets, we'd have

〈e_j〉 = 〈e_j²〉 = c/m
〈x_i〉 = αc
(1/n) 〈∑ x_i²〉 - α = (1/n)(m/c)(Var(x_i) + 〈x_i〉²) = 1 - c/m + αc
= 1 - c/m + α(c-1)

If the clustering measure gives a value significantlygreater than one, it is like having a hash function that misses a substantialfraction of buckets.

Designing a hash function

For a hash table to work well, we want the hash function to have twoproperties:

• Injection: for two keys k₁ ≠ k₂,the hash function should give different results h(k₁) ≠h(k₂), with probability m-1/m.
• Diffusion(stronger than injection):if k₁ ≠ k₂, knowing h(k₁) givesno information about h(k₂). For example, if k₂ isexactly the same as k₁, except for one bit, then every bit inh(k₂) should change with 1/2 probability compared toh(k₁). Knowing the bits of h(k₁) does not give anyinformation about the bits of h(k₂).

As a hash table designer, you need to figure out which of theclient hash function and the implementation hash function is going toprovide diffusion. For example, Java hash tables provide (somewhat weak)information diffusion, allowing the client hashcode computation tojust aim for the injection property. In SML/NJ hash tables, the implementationprovide only the injection property. Regardless, the hash table specificationshould say whether the client is expected to provide a hash code withgood diffusion (unfortunately, few do).

If clients are sufficiently savvy, it makes sense topush the diffusion onto them, leaving the hashtable implementation as simple and fast as possible.The easy way to accomplish this is to breakthe computation of the bucket index into three steps.

1. Serialization: Transform the key into a stream of bytes that contains all of the information in the original key. Two equal keys must result in the same byte stream. Two byte streams should be equal only if the keys are actually equal. How to do this depends on the form of the key. If the key is a string, then the stream of bytes would simply be the characters of the string.
2. Diffusion: Map the stream of bytes into a large integer x in a way that causes every change in the stream to affect the bits of x apparently randomly. There are a number of good off-the-shelf ways to accomplish this, with a tradeoff in performance versus randomness (and security).
3. Compute the hash bucket index as x mod m. This is particularly cheap if m is a power of two, but see the caveats below.

There are several different good ways to accomplish step 2: multiplicative hashing, modular hashing, cyclic redundancy checks, and secure hash functions such as MD5 and SHA-1.

Frequently, hash tables are designed in a way that doesn't let the client fully control the hash function. Instead, the client is expected to implement steps 1 and 2 to produce an integer hash code, as in Java. The implementation then uses the hash code and the value of m (usually not exposed to the client, unfortunately) to compute the bucket index.

Some hash table implementations expect the hash code to look completely random, because they directly use the low-order bits of the hash code as a bucket index, throwing away the information in the high-order bits. Other hash table implementations take a hash code and put it through an additional step of applying an integer hash function that provides additional diffusion. With these implementations, the client doesn't have to be as careful to produce a good hash code,

Any hash table interface should specify whether the hash function is expected to look random. If the client can't tell from the interface whether this is the case, the safest thing is to compute a high-quality hash code by hashing into the space of all integers. This may duplicate work done on the implementation side, but it's better than having a lot of collisions.

Modular hashing

With modular hashing, the hash function is simply h(k) = k mod mfor some m (usually, the numberof buckets). The value k is an integer hashcode generated from the key.If m is a power oftwo (i.e., m=2^p),then h(k) is just thep lowest-order bits of k. TheSML/NJ implementation of hash tables does modular hashing with m equal to a power of two. This is very fast but thethe client needs to design the hash function carefully.

The Java Hashmap class is a little friendlier butalso slower: it uses modular hashing with mequal to a prime number. Modulo operations can be accelerated byprecomputing 1/m as a fixed-point number, e.g. (2³¹/m). A precomputed tableof various primes and their fixed-point reciprocals is thereforeuseful with this approach, because the implementation can then usemultiplication instead of division to implement the mod operation.

Multiplicative hashing

A faster but often misused alternative is multiplicative hashing,in which the hash index is computed as⌊m * frac(ka)⌋. Herek is again an integer hash code,a is a real number and frac is the function that returns the fractionalpart of a real number.Multiplicative hashing sets the hash index from the fractional part ofmultiplying kby a large real number.It's faster if this computation is done using fixed point rather than floatingpoint, which is accomplished by computing (ka/2^q)modmfor appropriately chosen integer values of a, m, and q. So qdetermines the number of bits of precision in the fractional part of a.

Here is an example of multiplicative hashing code,written assuming a word size of 32 bits:

val multiplier: Word.word = 0wx678DDE6F (* a recommendation by Knuth *) fun findBucket({arr, nelem}, e) (f:bucket array*int*bucket*elem->'a) = let val n = Word.fromInt(Array.length(arr)) val d = (0wxFFFFFFF div n)+0w1 val i = Word.toInt(Word.fromInt(Hash.hash(e)) * multiplier div d) val b = Array.sub(arr, i) in f(arr, i, b, e) end

Multiplicative hashing works well for the same reason thatlinear congruential multipliers generate apparently random numbers—it's likegenerating a pseudo-random number with the hashcode as the seed. Themultiplier a should be large and its binary representation should be a"random" mix of 1's and 0's. Multiplicative hashing ischeaper than modular hashing because multiplication is usuallyconsiderably faster than division (or mod).It also works well with a bucket array of sizem=2^p,which is convenient.

In the fixed-point version,The division by 2^q is crucial.The common mistake when doing multiplicative hashing is to forget to do it,and in fact you can find web pages highly ranked by Googlethat explain multiplicative hashingwithout this step. Without this division, there is little point to multiplyingby a, becausekamodm= (kmodm) * (amodm) mod m. This is no better than modular hashing with a modulus of m, and quite possibly worse.

Cyclic redundancy checks (CRCs)

For a longer stream of serialized key data, a cyclic redundancycheck (CRC) makes a good, reasonably fast hash function.A CRC of a data stream is the remainder after performing a longdivision of the data (treated as a large binary number), but using exclusive orinstead of subtraction at each long division step. This corresponds to computinga remainder in the field of polynomials with binary coefficients. CRCs can becomputed very quickly in specialized hardware.Fast software CRC algorithms rely on accessing precomputed tables of data.

Cryptographic hash functions

Sometimes software systems are used by adversaries who might try to pickkeys that collide in the hash function, thereby making the system have poorperformance. Cryptographic hash functions are hash functions that try tomake it computationally infeasible to invert them: if you knowh(x), there is no way to computex that is asymptotically faster thanjust trying all possible values and see which one hashes to the right result.Usually these functions also try to make it hard to find differentvalues of x that cause collisions. Examples of cryptographic hashfunctions are MD5 and SHA-1. Some attacks are known on MD5, but it isfaster than SHA-1 and still fine for use in generating hash table indices.

Precomputing hash codes

High-quality hash functions can be expensive. If the same values are beinghashed repeatedly, one trick is to precompute their hash codes and storethem with the value. Hash tables can also store the full hash codes of values,which makes scanning down one bucket fast. In fact, if the hash code is longand the hash function is high-quality (e.g., 64+ bits of a properly constructedMD5 digest), two keys with the same hash code are almost certainly thesame value. Your computer is then more likely to get a wrong answer from acosmic ray hitting it than from a hash code collision.