You are also paying for craftsmanship for diamonds. It takes a skilled and therefore well paid craftsman to properly cut them. It also takes a long time to do it. Beyond that, the amount of jewelery quality diamond is low compared the the amount of diamond that is available. Still, the price is artificially high because DeBeers essentially has a monopoly.
In the end, gold and diamonds are expensive because people will pay that much for them. Really, it's just like anything else. Things cost what people will pay and they will pay more for rare things that are pretty.
Sadly, that is also true of gasoline, with the added problem that it is now a necessity for most people. The scary part is that we would all pay whatever it costs to buy gas. We complain, but in the end we have no choice and they really could charge whatever they want. The only limitation is at a certain price people couldn't buy enough of it to be profitable and they tend to stay very close to that line to make as much profit as possible.