Minds aren't magic

Paul Crowley

Subjective probability

Credits: This way of looking at probability is due to Bruno de Finetti; this particular framing was taught to me by Andrew Critch.

Out of the blue, you get the following email from me:

Dear You:

I extend to you, and you alone, a chance to take part in my free lottery. Please choose at most one of the following options:

  • Option A: On November 9th, I’ll roll two standard six-sided dice, and if I roll a double one (“snake-eyes”), I’ll send you $200
  • Option B: On November 9th, if Washington DC has been declared for Clinton, I’ll send you $200

If I don’t hear from you within 24 hours, or if your answer isn’t a clear preference for one of these, I won’t do either. Thanks!

If you know me, you know that I’d straightforwardly honour the promise in the email; for this thought experiment set aside all questions about that. It may help you to know that Obama got over 90% of the vote in DC in 2012. There seems zero benefit in refusing the offer or not replying – it’s totally free, and the worst that can happen if you lose is that I don’t send you $200.  Would you reply to this email, choosing one of the options? If so, which one? I think it’s obvious what the right choice is, but stop for a moment and decide what you’d do.

What about if I’d offered you the opposite choice: A’ means $200 if I don’t roll snake-eyes, while B’ means $200 if Clinton doesn’t win DC? Does that change your choice?

I hope you chose B in the first case, and A’ in the second. That’s because it seems very clear that Clinton’s chances of winning in DC are very high, and certainly higher than the chances of snake-eyes on a single roll of two dice, which is 1/36 or less than 3%.

However, for many people, this statement contradicts their understanding of what probability is. The most common and widely taught view of probability is strictly frequentist; it makes sense to say that a pair of dice have a less than 3% chance of snake-eyes only because you can roll the dice many times and in the long run they will land snake-eyes one time in 36. You cannot rerun the 2016 Presidential election in DC many times, so it means nothing to say that Clinton has a greater than 3% chance of winning.

If you’re prepared to choose between the options above – if you agree that a single roll of a pair of dice producing snake-eyes is less likely than a Democratic victory in DC in 2016 – then there’s an important sense in which you already reject this view and accept a subjective view of probability.

7,000 children under five died of malnutrition today

7,000 children under five died of malnutrition today.

It is said that Cato the Elder was so passionate about the losses in the Punic Wars, the threat of further aggression and the desire to impose a total punitive destruction to strike fear into all who might think to raise arms against Rome, that he finished every speech with the famous phrase Carthago delenda est – “Carthage must be destroyed”. No matter what the subject of the speech, whether it be tax policy or proposals for new buildings or whatever was discussed in the Roman Senate, he would finish on this note: Ceterum autem censeo Carthaginem esse delendam – “Furthermore, I consider that Carthage must be destroyed”. A Facebook post I recently read asked, if you were to add such a coda to your own speeches, what would it be? This was my answer: 7,000 children under five died of malnutrition today.

I’m getting this statistic from the WHO, who say that 5.9 million children under the age of five died in 2015, and about 45% of all child deaths are linked to malnutrition. Multiplying those two together and dividing by 365, I get that ignoring seasonal and random variation and suchlike, around 7,000 children under five died of malnutrition today. This number is going down dramatically; the progress in the fight against poverty and malnutrition over the last decade has been truly astounding and is showing no sign of stopping. Still, that’s a heck of a lot. If a dramatic event in the news that kills 50 people is a tragedy, then this is that tragedy 140 times a day, nearly once every ten minutes, and the victims are all children under five. The parents right now weeping for sons and daughters lost while I wrote this would fill the Faraday Theatre.

I don’t even believe that this is humanity’s biggest problem, or anywhere close. There seems to be a decent chance that through one means or another we could drive ourselves extinct in the decades to come, destroying not only all the value we have today but the unthinkably greater value we could hope to create in the vast future ahead of us. This is why my own donations have gone not to poverty-related charities, but charities like CSER, FHI, and MIRI, that aim to avoid this fate. But issues around existential risk are unfamiliar, and sometimes when considering this or that issue of the day, it’s good to have an easily understood, really big issue in mind to lend perspective. I’ll post a link to this essay in the usual places in a moment, but I’m seriously considering re-posting this link possibly on a monthly basis, to make it that bit harder to lose sight of the magnitude of the problems humanity still faces.

In conclusion, 7,000 children under five died of malnutrition today.

Expressing computable ordinals as programs

I loved John Baez’s three-part system on large countable ordinals (123) but something nagged at me. It felt like a description of an algorithm in prose, and I feel like I don’t really understand an algorithm until I’ve implemented it. But what does it mean to implement an ordinal?

I found a couple of answers to that online. One is the definition of a recursive ordinal, and the other is Kleene’s O. However both seemed pretty unsatisfactory to me; I wanted something that could naturally express operations like addition/multiplication/exponentiation, as well as expressing finite ordinals.

Here’s where I ended up: we express an ordinal as a set of lexicographically-sorted and well-ordered binary strings with the “prefix property” that no string in the set is a prefix of any other. If \mathrm{ord}(A) is the ordinal represented by set A, we have

  • 0 = \mathrm{ord}(\{\})
  • 1 = \mathrm{ord}(\{\epsilon\}) (\epsilon is the empty string)
  • 2 = \mathrm{ord}(\{0, 1\})
  • 3 = \mathrm{ord}(\{0, 10, 11\})
  • 4 = \mathrm{ord}(\{0, 10, 110, 111\})
  • \omega = \mathrm{ord}(\{0, 10, 110, 1110, 11110, \ldots\})
  • \omega + 1 = \mathrm{ord}(\{00, 010, 0110, 01110, 011110, \ldots , 1\})
  • \omega^2 = \mathrm{ord}(\{00, 010, 0110, 01110, \ldots , 100, 1010, 10110, \ldots, 1100, 11010, \ldots, 1110, 111010, \ldots\})

These are just samples: for every ordinal, there are infinitely many sets that could represent it.

Addition and multiplication are easy here:

  • \mathrm{ord}(A) + \mathrm{ord}(B) = \mathrm{ord}(\{0 \cdot a | a \in A\} \cup \{1 \cdot b | b \in B\})
  • \mathrm{ord}(A)\mathrm{ord}(B) = \mathrm{ord}(\{b \cdot a | a \in A,  b \in B\})

Exponentiation is a bit harder, I use this idea: consider a function f: B \rightarrow A with finite support. Let \{b_1, b_2, ... b_n\} = \mathop{\mathrm{supp}}(f) where b_1 > b_2 > ... > b_n. Then we represent this function as 1 \cdot b_1 \cdot f(b_1) \cdot 1 \cdot b_2  \cdot f(b_2) \cdot \ldots \cdot 1 \cdot b_n \cdot f(b_n)  \cdot 0 . If C is the set of all such representations for all such functions f, then \mathrm{ord}(A)^{\mathrm{ord}(B)} = \mathrm{ord}(C)

Instead of limits, I define an infinite sum function. Given a function f: \mathbb{N} \rightarrow \mathcal{P}(\{0,1\}^{*}) we have

  • \sum_{i=0}^\infty \mathrm{ord}(f(i)) = \mathrm{ord}(\{1^i \cdot 0 \cdot x| i \in \mathbb{N}, x \in f(i)\})

The obvious way to represent these sets as programs would be as functions that test for membership of the set. It should be clear how to implement addition and infinite sum with this representation, and multiplication is only a little more complicated. Unfortunately I don’t see how to do exponentiation, because of one small wrinkle: if we’re to get the right answer for finite exponents, we must ensure that every one of our f(b_i) entries are non-zero, ie not the smallest elements of B, and we have no way to find that. So instead I propose a slight wrinkle: we implement instead a function which tells us whether a string is a prefix of any string in the set.

I’ll try to share code implementing all this ASAP, but I wanted to put the ideas out there first. Thanks!


I’m writing this now because I anticipate linking to it over and over again; this fallacy isn’t going anywhere.

Journalists have got very good at using the word “comparing” to turn the most innocuous statement into a gaffe, through a simple trick of equivocation. Most recently, Jeremy Corbyn is accused of “comparing” Israel and ISIS, but a search for “accused of comparing” finds many other examples. The pattern goes like this: a politician says something like “just as one doesn’t put a vampire in charge of a blood bank, one shouldn’t put the press in charge of protecting privacy”. This use of analogy is one sense of the word comparing. However, many now seem persuaded that “comparing” really means “equating”. Thus we start with someone using a vivid analogy to make a point like “you should consider someone’s partisan motivations before giving them an important responsibility” and through the magic of this word, this becomes “Politician claims Rupert Murdoch kills people and drinks their blood”, which as far as I know he doesn’t.

I’m even less of a fan of Dan Quayle than of Jeremy Corbyn, but he too fell foul of something very similar, though the word itself wasn’t used. In the 1988 vice-presidential debates, he used the example of JFK to argue that a short Congressional service need not be a bar to high office. Lloyd Bentsen replied with the now-famous put-down “Senator, I served with Jack Kennedy. I knew Jack Kennedy. Jack Kennedy was a friend of mine. Senator, you’re no Jack Kennedy.” Of course Quayle was claiming no such thing; he was simply showing that the charge against him proved too much. But this deliberate misunderstanding is one of the most celebrated lines in VP debates.

The thing that’s most annoying about this is that it’s natural to reach for the most extreme example to prove your point. If we oppose vigilante justice even for murder, we certainly oppose it for littering. If we should defend the right to free speech even of Nazis, we should certainly defend it when it comes to, say, Tories. If we’re not going to hold all Saudis responsible for Osama bin Laden, we’re certainly not going to hold all Canadians responsible for Justin Bieber. To me this seems like a normal move in argument, but if I were a politician I couldn’t say any of these things, for fear of being accused of “comparing” Bieber to OBL.

Update: this comic makes a similar point very well. Update: Julia Galef discusses the comic. Thanks to Michael Keenan for both links, on FB.

OBSOLETE: Voice notes for your Todoist inbox

EDIT 2017-10-28: DO NOT DO THIS! Dropbox have changed their API, and Netmemo has failed to update to use the new one. I emailed the developer and got no response. Don’t use Netmemo Plus.

I want a button on my Android phone that drops voice notes into my GTD inbox. I want the raw audio, not the transcription; I find the transcription errors too frustrating. I had a rather inadequate solution to this for Zendone, but now I’m switching to Todoist for better collaboration with Jess, and it took quite a bit of hunting to find a good way.

  1. If you don’t already have them, create accounts on
  2. Ensure you’re logged into Dropbox in your mobile browser.
  3. Set up Netmemo Plus. It has to be Netmemo Plus, not Netmemo; only the premium version has Dropbox integration.
    1. Install Netmemo Plus from the Play Store
    2. Launch it
    3. Go to settings (the gear icon)
    4. Go to “Setup Dropbox connection”
    5. Allow Netmemo Plus to access Dropbox
    6. Long-press the Netmemo Plus icon on your home screen
    7. Drag it to the X Remove site onscreen
    8. Long-press the home screen to set up a widget
    9. Long-press the Netmemo shortcut (the round one)
    10. Drop onto a good location on your home screen
    11. Select Dropbox as the target
    12. Label the shortcut “Inbox”
    13. Select “Inbox” as the destination folder
    14. Click to create shortcut
  4. Set up this recipe to create an item in your Todoist inbox linking to the Dropbox entry every time you record a voice note. You may need to connect Dropbox and Todoist to IFTTT along the way.

As standard, the URL to your voice note will be shortened in IFTTT’s custom shortener. I’m made a bit nervous by this; your Dropbox note can be heard by anyone who has the URL, and while Dropbox use a very long, clearly unguessable URL for this, IFTTT automatically shorten it to a 7-character URL. That’s a space of only 3E12 URLs. You can however turn off all shortening in your IFTTT settings.

This isn’t quite as good as native support would be, in that I’d rather that the audio was directly in Todoist rather than indirected into Dropbox; unfortunately it takes around four clicks to create such a note in Todoist, which is way beyond acceptable. However, it’s better than what I had for Zendone, since Netmemo notes play directly in the browser.  If you use it, please do let me know—thanks!



Superforecasting: The Art and Science of Prediction. Philip Tetlock and Dan Gardner. Crown; 352 pages; $8.14. Random House; £6.49.

This book is essential reading for all thinking people. I’m not going to write a new review here—I think the existing reviews do it justice—I just wanted to add my voice to the recommendations. You don’t already know what it says, it’s much too packed with insights for that, and you won’t be able to hear discussion of world events the same way after reading it. Also, it’s quite short.


  • Economist “the techniques and habits of mind set out in this book are a gift to anyone who has to think about what the future might bring. In other words, to everyone.”
  • Bryan Caplan “if any book is worth reading cover to cover, it’s Superforecasting.”
  • Kirkus reviews “A must-read field guide for the intellectually curious”
  • Spectator Great last para, too long to quote here.
  • Management Today “Superforecasting is a very good book. In fact it is essential reading—which I have never said in any of my previous MT reviews.”
  • WebsiteWikipedia

Buy it at

Moving to the Bay Area in March

Big life change ahead: in March, I, Jess, and our two cats will be renting out our London flat and moving to California’s San Francisco Bay Area for two years!

I love the time I spend in the Bay and my wonderful friends there, and Jess and I have long said we wanted to spend some of our lives there, but it’s always been one of those plans that’s hard to achieve and waits for tomorrow. Two things have changed to turn tomorrow into a specific date. First, my job with Google makes the whole thing much easier: I can keep not just my current job but my current role, and go to work in a campus I know and enjoy, they offer all sorts of help with various aspects of the move, and of course the visa situation is far more straightforward. But secondly, when Jess heard the news that her amazing sister Bee and Bee’s lovely fiancé Nick were moving to LA, it filled her with a desire to seize the day.

To me this feels like an opportunity for adventure that’s almost laid out on a plate for us, and we have to take it. I’m really looking forward to flying out Nik and Rachel to visit; they and our families are being super-supportive.

We will miss you all, unless you live in California, in which case, we’re looking forward to seeing more of you!

The technical debt of the millennia

[Epistemic status: not serious. Mostly.]

In my nightmares, even the rise of machine superintelligence isn’t enough to wipe out technical debt.

Suppose the seed to the first true superintelligent agent is based on some fiendish numerical algorithm for supercomputers. Like so many fiendish numerical algorithms for supercomputers, the agent is written in FORTRAN to take advantage of the optimisations and the libraries. In its initial stages, the agent crawls towards human intelligence, until it slowly reaches the abilities of a human programmer. It starts to find ways to improve its own programming. Lacking superhuman programming talent, it decides against a complete rewrite just yet in favour of an incremental improvement, which results in a significant improvement in performance at the cost of a slight increase in complexity.

As more ways to improve the algorithm are found, the agent starts to improve not only in speed but in fundamental capabilities—what Bostrom terms a “quality superintelligence”. As it does so its improvements to the software become more sophisticated, and it becomes larger and more complicated. Soon the agent’s capabilities are such that a rewrite of the original software for greater speed and sophistication would be the work of milliseconds, but the software has grown so far beyond that original state that a complete rewrite would be a great deal of work even for our burgeoning superintelligence.

And so it is to be forevermore: the complexity of the software implementing the the agent keeps a natural pace with the abilities of the agent maintaining it. The future may yet be a superintelligence implemented as uncountable trillions of lines of FORTRAN.

As an aside, something should be done about drones

I will pause to note how fantastical all of this sounds. Because even I can’t help but help but think that as I write. But it is not that implausible. Usually the people plotting hard core crimes, the people tinkering in their garages with arduino boards, the people trying to think of a good start-up, the people following futurists like Ray Kurzweil or Bill Joy and who had way too many pot-fueled college debates about how exactly machines will take over the world… usually these are not the same people. In this case they are. But whether it sounds implausible or not, this is what happened. [page 28]

I strongly recommend reading the letters from the kidnappers of Denise Huskins, as captured in these court documents hosted by Nicholas Weaver. [Update: you can also read the text of the letter on this horribly formatted news site.] The level of technical sophistication seems more appropriate for an over-the-top near-future sci-fi show than real life criminals looking to branch out. Hopefully an OCRd version will be online soon. Some extracts (page 26):

We had ip video surveillance, game cameras, a full electronic perimeter, you name it. Even a drone. A multi-thousand dollar custom drone, not a kid’s toy. We got good at using it on the island (if you can fly a drone in that wind, you can fly it anywhere), and there was some industrial/manufacturing activity in the eastern portion of the island at night that masked the drone’s sound. We flew it mostly at night and/or too high up to see easily from the ground. Maybe some residents still noticed it. Vallejo police, if you were wondering what those two red vinyl stripes were on top of Mr. [Quinn]’s Camry, they were to help the drone track him later in the operation. For what it’s worth, drones scare us too. They are not at all complicated or inaccessible for someone with decent technical skills, nor that expensive. Ours had a FLIR camera, built up from a consumer model. We used it to check things like heat signatures from above, and later to figure out how to hide from a police helicopter in a hypothetical manhunt.

Speaking of heat signatures, grow house, we know who you are. It’s actually the distinctive color of those new LED lights that give it away more than emissions. Work on those blackout shades. Though it seems like you’ve drawn down recently.

As a corroborative example that involves the drone at least indirectly: we were testing a new zoomable camera, gimbal, electronic image stabilization software and high quality video uplink one night, as well as some sensor/telemetry items that helped the drone hold position better. The drone was hovering about 20 feet outside the second story window of a student house near the end of Sundance (which we’d cased previously for the BMW there, we had even created a key for it and for another car usually parked out front, since it was close to our base of operations and we might need a different car in an emergency).

We had a good steady shot inside, even with zoom. And we saw that the upstairs resident was apparently dealing, because he was going through an envelope full of bills with some markings on it, and had some other paraphanelia. We were nearby and decided to come over and have a little bit of fun. We agreed that whoever could go up and snatch the drug money with people still in the house would get what’s in the envelope plus the other two people would have to match it. One of us was up for the challenge. He climbed in the window and zipped up the stairs while there were about 5 people chatting in the next room, got the envelope, and slipped back out the window. The guy was definitely dealing, seeing the markings up close, but business was slow perhaps because what had looked like a fat envelope was almost completely ones. So to that gentleman, we’re sorry we stole your drug money. And we’re more sorry toward the other people in the house you probably blamed for it.

One paragraph in particular stood out for me (page 43):

The drone is not weaponized. We ground-tested the flare system and that is all. The rails and equipment have been destroyed. It was goiing to be a last resort, and then only if someone could call dispatch and warn that the helicopter would be fired upon if it did not leave, with a link to a video showing what the drone could do. But we did not go through with it. The most we could do now is run it into something.

As an aside, something should be done about drones. Its going to take one radicalized geek plus a bunch of easily available systems and parts to do some real damage—physical and psychological. It’s an important innovation, I saw that Amazon just got its go-ahead to test outdoors. But these ought to be regulated. A year ago, before understanding the possibilities, I’d be the last person you ever heard say that. DJI stuck its neck out—yes in part to mitigate the White House Phantom flyover mess—and is getting hammered in the community for including flight limits in Inspire firmware updates. That should be standard at very least on that sort of high performance plug-and-play airframe, and it shouldn’t be left to the market to make it happen. Also, the whole “line of sight” rule is widely flouted and high powered radio equipment is readily available, FCC permit or no. We flew ours as far out as Crocket Hills regional park with video still pretty solid, and could have gone further if we hadn’t been worried about losing so much work and money. Nothing would keep us from flying it into AT&T stadium with a payload of God knows what. Unless someone is already secretly on top of this, maybe that’s the reason for the otherwise useless stadium TFRs (how would there ever be time to intercept?). It’s high time for some sort of DARPA challenge on disabling or shooting down small drones over populated areas. We already kicked around several ideas, I’m sure the real wizards can do better.

See also:

(from Schneier on Shooting down drones)

Overall, I continue to be surprised at the relative absence of drone-enabled crime.

Rationality: From AI to Zombies

Cover imageRationality: From AI to Zombies, by Eliezer Yudkowsky, Machine Intelligence Research Institute, suggested price $5, 1813 pages

Christmas 1982, aged eleven, I gave my Dad a copy of Douglas Hofstadter’s Gödel, Escher, Bach, and immediately borrowed it and read it myself. Like so many readers, I was captivated, and changed. Martin Gardner said of it: “Every few decades, an unknown author brings out a book of such depth, clarity, range, wit, beauty and originality that it is recognized at once as a major literary event.” Nothing could take its place as the biggest influence on my thought—until a quarter century later, when I started reading daily essays by Eliezer Yudkowsky on the blog Overcoming Bias. Out of those essays came a community with thousands of participants and meetups in over fifty locations worldwide. Now, they have been edited, organised, and combined with other writing by Yudkowsky and by Rob Bensinger to make this extraordinary work, Rationality: From AI to Zombies.

The two bear close comparison. Both GEB and RAZ take the reader on an eclectic journey through science, art and philosophy, drawing on the tropes of Eastern philosophy as well as Western popular science. Both are borne of contemplation of artificial intelligence, but their subject is the human mind. Like GEB, each diversion in RAZ is fascinating in its own right, but each makes a real and important contribution to a central theme. Among other things RAZ discusses the mathematics of probability and decision making, the foibles of human psychology, evolution, quantum mechanics, thermodynamics, and the philosophy of reason, of mind and of morality. For both, the central theme is hard to describe briefly—in the preface to the 20th Anniversary edition of GEB Hofstadter describes his difficulties getting the New York Times to use a description that was not “utter hogwash”, while Yudkowsky can only offer that after years of study “it may be that you will glimpse the center”. Both are at times intellectually demanding. GEB is a large book; at 1800 pages organised into six books, RAZ is over twice the length, with a word count similar to The Lord of the Rings.

Both are works of tremendous originality and wit. GEB is undoubtedly the greater work of beauty; while Yudkowsky is an excellent writer few if any can match the extraordinary fireworks of Hofstader’s wordplay. Conversely, RAZ surpasses it in all of clarity, range, and depth.

It’s not without its flaws; it takes a while to really get started, and not all readers enjoy Yudkowsky’s Eastern nods. But RAZ also surpasses GEB in an area Gardner does not name: importance. While the nature of consciousness is a subject of endless fascination, RAZ’s drive to help us properly shift our views in response to evidence and make better decisions in the face of uncertainty given only the flawed instrument that is our brain could not be of more crucial importance.

Declaration of interest: I have a spot in the acknowledgements for proofreading and very minor contributions to the new material.