Eliezer Yudkowsky Quotes

Many philosophers - particularly amateur philosophers, and ancient philosophers - share a dangerous instinct: If you give them a question, they try to answer it.

Reality is always lawful," said Harry, "even if we don't know the law.

One last time, Akon thought of all his fear, of the sick feeling in his stomach and the burning that was becoming a pain in his throat. He pinched himself on the arm, hard, very hard, and felt the warning signal telling him to stop.
Goodbye, Akon thought; and the tears began falling down his cheek, as though that one silent word had, for the very last time, broken his heart.
And he lived happily ever after.

Boy, you resolve not to go down the path of a Dark Lord and the universe starts messing with you the instant the Hat comes off your head. Some days it just doesn't pay to fight destiny. Maybe I'll wait until tomorrow to start on my resolution to not be a Dark Lord.

This was made for you, I think. For the person you're going to become.
A weapon to fight Death, in its form as the shadow of despair that falls on human minds and drains away their hope for the future.

That's what I'd tell you if I thought you could be responsible for anything. But normal people don't choose on the basis of consequences, they just play roles. There's a picture in your head of a stern disciplinarian and you do whatever that picture would do, whether or not it makes any sense. A stern disciplinarian would order the students back to their rooms, even if there was a troll roaming the hallways. A stern disciplinarian would order students not to leave the Hall on pain of expulsion. And the little picture of Professor McGonagall that you have in your head can't learn from experience or change herself, so there isn't any point to this conversation. People like you aren't responsible for anything, people like me are, and when we fail there's no one else to blame.

If what you believe doesn't depend om what you see, you've been blinded as effectively as by poking out your eyeballs.

Reality usually delivers results a little worse than the 'worst-case scenario'. It's called the planning fallacy, and the best way to fix it is to ask how long things took the last time you tried them. That's called using the outside view instead of the inside view. But when you're doing something new and can't do that, you just have to be really, really, really pessimistic. Like, so pessimistic that reality actually comes out better than you expected around as often and as much as it comes out worse. It's actually really hard to be so pessimistic that you stand a decent chance of undershooting real life.

If the students see that rules are for everyone ... for professors too, not just for poor helpless students who get nothing but suffering out of the system ... why, the positive effects on school discipline should be tremendous.

Someday," said the Boy-Who-Lived, "when the distant descendants of Homo sapiens are looking back over the history of the galaxy and wondering how it all went so wrong, they will conclude that the original mistake was when someone taught Hermione Granger how to read.

I should have known. I should have known from the beginning. I was raised in another world. A world where royal blood is not a license to rule, a world whose wizards do more than sneer from their high towers, a world where life is not so cheap, where justice does not come as a knife in the night, a world where we know that the texture of a race's skin shouldn't matter -
And yet for you, born in this world, to question what others took for granted; for you, without ever touching the Sword, to hear the scream that had to be stopped at all costs -
"I don't trust you either," Hirou whispered, "but I don't expect there's anyone better," and he closed his eyes until the end of the world.

Remember, if you succeed in everything you try in life, you're living below your full potential and you should take up more difficult or daring things.

You couldn't leave your home planet while it still contained a place like Azkaban.
You had to stay and fight.

Have I ever remarked on how completely ridiculous it is to ask high school students to decide what they want to do with the rest of their lives and give them nearly no support in doing so? Support like, say, spending a day apiece watching twenty different jobs and then another week at their top three choices, with salary charts and projections and probabilities of graduating that subject given their test scores? The more so considering this is a central allocation question for the entire economy?

I'm wondering if there's a spell to make lightning flash in the background whenever I make an ominous resolution.

Without thinking about it at all, Harry stepped in front of Hermione.
There was an intake of breath from behind him, and then a moment later Hermione brushed past and stepped in front of him. "Run, Harry!" she said. "Boys shouldn't have to be in danger.

John Kenneth Galbraith said: Faced with the choice of changing one's mind and proving that there is no need to do so, almost everyone gets busy on the proof.

Unfortunately, I am also just now realizing that I've only been going along with society all my life, and that I never thought the matter through for myself, until now.

During World War II, there had been a project to sabotage the Nazi nuclear weapons program. Years earlier, Leo Szilard, the first person to realize the possibility of a fission chain reaction, had convinced Fermi not to publish the discovery that purified graphite was a cheap and effective neutron moderator. Fermi had wanted to publish, for the sake of the great international project of science, which was above nationalism. But Szilard had persuaded Rabi, and Fermi had abided by the majority vote of their tiny three-person conspiracy. And so, years later, the only neutron moderator the Nazis had known about was deuterium.

The only deuterium source under Nazi control had been a captured facility in occupied Norway, which had been knocked out by bombs and sabotage, causing a total of twenty-four civilian deaths.

The Nazis had tried to ship the deuterium already refined to Germany, aboard a civilian Norwegian ferry, the SS Hydro.

Knut Haukelid and his assistants had been discovered by the night watchman of the civilian ferry while they were sneaking on board to sabotage it. Haukelid had told the watchman that they were escaping the Gestapo, and the watchman had let them go. Haukelid had considered warning the night watchman, but that would have endangered the mission, so Haukelid had only shaken his hand. And the civilian ship had sunk in the deepest part of the lake, with eight dead Germans, seven dead crew, and three dead civilian bystanders. S

There are no surprising facts, only models that are surprised by facts; and if a model is surprised by the facts, it is no credit to that model.

If you're unhappy whenever other people don't picture you exactly the same way you picture yourself, that's already dooming yourself to always be unhappy. No one ever thinks of us just the same way we think of ourselves.

That left two possibilities, really.
Possibility one: Magic was so incredibly opaque, convoluted, and impenetrable, that even though wizards and witches had tried their best to understand, they'd made little or no progress and eventually given up; and Harry would do no better.
Or ...
Harry cracked his knuckles in determination, but they only made a quiet sort of clicking sound, rather than echoing ominously off the walls of Diagon Alley.
Possibility two: He'd be taking over the world.
Eventually. Perhaps not right away.

There are no hard problems, only problems that are hard to a certain level of intelligence. Move the smallest bit upwards [in intelligence] and some problems move from "impossible" to "obvious." Move a substantial degree upwards, and all of them will become obvious.

If cryonics were a scam it would have far better marketing and be far more popular.

I will reluctantly teach you enough trivia for a passing mark on the Ministry-mandated portions of your first-year finals. Since your exact mark on these sections will make no difference to your future life, anyone who wants more than a passing mark is welcome to waste their own time studying our pathetic excuse for a textbook. The title of this subject is not Defence Against Minor Pests. You are here to learn how to defend yourselves against the Dark Arts. Which means, let us be very clear on this, defending yourselves against Dark Wizards. People with wands who want to hurt you and who will likely succeed in doing so unless you hurt them first! There is no defence without offence! There is no defence without fighting! This reality is deemed too harsh for eleven-year-olds by the fat, overpaid, Auror-guarded politicians who mandated your curriculum. To the abyss with those fools! You are here for the subject that has been taught at Hogwarts for eight hundred years! Welcome to your first year of Battle Magic!

I'm not a psychopath, I'm just very creative.

Once you tell a lie, the truth is your enemy; and every truth connected to that truth, and every ally of truth in general; all of these you must oppose, to protect the lie. Whether you're lying to others, or to yourself.

If you handed [character] a glass that was 90% full, he'd tell you that the 10% empty part proved that no one really cared about water.

If you see your activities and situation originally, you will be able to originally see your goals as well. If you can look with fresh eyes, as though for the first time, you will see yourself doing things that you would never dream of doing if they were not habits.

You can only arrive at mastery by practicing the techniques you have learned, facing challenges and apprehending them, using to the fullest the tools you have been taught, until they shatter in your hands and you are left in the midst of wreckage absolute ... I cannot create masters. I have never known how to create masters. Go, then, and fail ... You have been shaped into something that may emerge from the wreckage, determined to remake your Art. I cannot create masters, but if you had not been taught, your chances would be less. The higher road begins after the Art seems to fail you; though the reality will be that it was you who failed your Art.

If you don't sign up your kids for cryonics then you are a lousy parent.

Existential depression has always annoyed me; it is one of the world's most pointless forms of suffering.

But then human beings only understood each other in the first place by pretending. You didn't make predictions about people by modeling the hundred trillion synapses in their brain as separate objects. Ask the best social manipulator on Earth to build you an Artificial Intelligence from scratch, and they'd just give you a dumb look. You predicted people by telling your brain to act like theirs. You put yourself in their place. If you wanted to know what an angry person would do, you activated your own brain's anger circuitry, and whatever that circuitry output, that was your prediction. What did the neural circuitry for anger actually look like inside? Who knew? The best social manipulator on Earth might not know what neurons were, and neither might the best Legilimens.

To worship a sacred mystery was just to worship your own ignorance.

No rescuer hath the rescuer, Godric Gryffindor had written. No Lord hath the champion, no mother and no father, only nothingness above.

Litmus test: If you can't describe Ricardo 's Law of Comparative Advantage and explain why people find it counterintuitive, you don't know enough about economics to direct any criticism or praise at " capitalism " because you don't know what other people are referring to when they use that word .

I only want power so I can get books.

Right. Because he seemed like such a normal kid. And he is a normal kid, he is just what you'd expect a baseline male child to be like if Darth Vader were his doting father.

Asimov's "The Relativity of Wrong":3 When people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together.

Maybe you just can't protect people from certain specialized types of folly with any sane amount of regulation, and the correct response is to give up on the high social costs of inadequately protecting people from themselves under certain circumstances.

Homo sapiens didn't become the dominant species on this planet by having the sharpest claws or hardest armor - though I suppose some of that point may be lost on wizards. Still, it's beneath my dignity as a human being to be scared of anything that isn't smarter than I am.

If you want to win this argument with Dad, look in chapter two of the first book of the Feynman Lectures on Physics. There's a quote there about how philosophers say a great deal about what science absolutely requires, and it is all wrong, because the only rule in science is that the final arbiter is observation - that you just have to look at the world and report what you see. Um ... off the top of my head I can't think of where to find something about how it's an ideal of science to settle things by experiment instead of arguments -

By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.

When we look at others we see personality traits that explain their behaviour, but when we look at ourselves we see circumstances that explain our behaviour. People's stories make internal sense to them, from the inside, but we don't see people's histories trailing behind them in the air. We only see them in one situation, and we don't see what they would be like in a different situation. So the fundamental attribution error is that we explain by permanent, enduring traits what would be better explained by circumstance and context.

Trying to navigate the halls of Hogwarts was like ... probably not quite as bad as wandering around inside an Escher painting, that was the sort of thing you said for rhetorical effect rather than for its being true.
A short time later, Harry was thinking that in fact an Escher painting would have both pluses and minuses compared to Hogwarts. Minuses: No consistent gravitational orientation. Pluses: At least the stairs wouldn't move around WHILE YOU WERE STILL ON THEM.

The media thinks that only the cutting edge of science, the very latest controversies, are worth reporting on. How often do you see headlines like 'General Relativity still governing planetary orbits' or 'Phlogiston theory remains false'? By the time anything is solid science, it is no longer a breaking headline.

When there are lots of possibilities, most of the work goes into just locating the true answer - starting to pay attention to it. You don't need proof, or the sort of official evidence that scientists or courts demand, but you need some sort of hint, and that hint has to discriminate that particular possibility from the millions of others. Otherwise you can't just pluck the right answer out of thin air. You can't even pluck a possibility worth thinking about out of thin air.

Are you seeking power because that is your true desire, Brennan?
Or because you have a picture in your mind, of the role that you play as an ambitious young man, and you think it is what someone playing your role would do?

That which the truth nourishes should thrive.

What people really believe doesn't feel like a BELIEF, it feels like the way the world IS.

If I could create a world where people lived forever, or at the very least a few billion years, I would do so. I don't think humanity will always be stuck in the awkward stage we now occupy, when we are smart enough to create enormous problems for ourselves, but not quite smart enough to solve them.

By and large, the answer to the question "How do large institutions survive?" is "They don't!" The vast majority of large modern-day institutions some of them extremely vital to the functioning of our complex civilization simply fail to exist in the first place.

The point being that when there are lots of possible answers, most of the evidence you need goes into just locating the true hypothesis out of millions of possibilities - bringing it to your attention in the first place. The amount of evidence you need to judge between two or three plausible candidates is much smaller by comparison. So if you just jump ahead without evidence and promote one particular possibility to the focus of your attention, you're skipping over most of the work.

A single Lie That Must Be Protected can block someone's progress into advanced rationality. No, it's not harmless fun.

When you are older, you will learn that the first and foremost thing which any ordinary person does is nothing.

You'd think there'd be some kind of more dramatic mental event associated with updating on an observation of infinitesimal probability -" Harry stopped himself. Mum, the witch, and even his Dad were giving him that look again. "I mean, with finding out that everything I believe is false.

Also: seriously, broomsticks? He was going to fly on, basically, a line segment? Wasn't that pretty much the single most unstable shape you could possibly find, short of attempting to hold on to a point marble? Who'd selected that design for a flying device, out of all the possibilities? Harry had been hoping that it was just a figure of speech, but no, they were standing in front of what looked for all the world like ordinary wooden kitchen broomsticks. Had someone just gotten stuck on the idea of broomsticks and failed to consider anything else? It had to be. There was no way that the optimal designs for cleaning kitchens and flying would happen to coincide if you worked them out from scratch.

Like that's the only reason anyone would ever buy a first-aid kit? Don't take this the wrong way, Professor McGonagall, but what sort of crazy children are you used to dealing with?"
"Gryffindors," spat Professor McGonagall, the word carrying a freight of bitterness and despair that fell like an eternal curse on all youthful heroism and high spirits.

I've seen people severely messed up by their own knowledge of biases. They have more ammunition with which to argue against anything they don't like. And that problem - too much ready ammunition - is one of the primary ways that people with high mental agility end up stupid, in Stanovich's "dysrationalia" sense of stupidity.

Why does any kind of cynicism appeal to people? Because it seems like a mark of maturity, of sophistication, like you've seen everything and know better. Or because putting something down feels like pushing yourself up.

I don't care where I live, so long as there's a roof to keep the rain off my books, and high-speed Internet access.

As you would be kind to others, be kinder to yourself as well.

The people I know who seem to make unusual efforts at rationality, are unusually honest, or, failing that, at least have unusually bad social skills.

When you walk past a bookshop you haven't visited before, you have to go in and look around. That's the family rule.

Hermione," Harry said seriously, as he started to dig down into the red-velvet pouch again, "don't punish yourself when a bright idea doesn't work out. You've got to go through a lot of flawed ideas to find one that might work. And if you send your brain negative feedback by frowning when you think of a flawed idea, instead of realizing that idea-suggesting is good behavior by your brain to be encouraged, pretty soon you won't think of any ideas at all." Harry put down two heart-shaped chocolates beside the book. "Here, have another chocolate. Besides the one from earlier, I mean. This one is to reinforce your brain for generating a good candidate strategy.

What is deadlier than hate, and flows without limit?
Indifference.

Sometimes Harry thought the deepest split in his personality wasn't anything to do with his dark side; rather it was the divide between the altruistic and forgiving Abstract Reasoning Harry, versus the frustrated and angry Harry In The Moment.

When we are young we believe that we know everything, and so we believe that if we see no explanation for something, then no explanation exists. When we are older we realise that the whole universe works by a rhythm and a reason, even if we ourselves do not know it. It is only our own ignorance which appears to us as insanity.

Never give anyone wise advice unless you know exactly what you're both talking about.

And you threaten to abandon us to Voldemort if we do not comply with your wishes."
Harry's voice was razor-sharp. "I regret to inform you that you are not the center of the universe. I'm not threatening to walk out on magical Britain. I'm threatening to walk out on you. I am not a meek little Frodo. This is my quest and if you want in you will play by my rules."
Dumbledore's face was still cold. "I am beginning to doubt your suitability as the hero, Mr. Potter."
Harry's return gaze was equally icy. "I am beginning to doubt your suitability as my Gandalf, Mr. Dumbledore. Boromir was at least a plausible mistake. What is this Nazgul doing in my Fellowship?

When the basic problem is your ignorance, clever strategies for bypassing your ignorance lead to shooting yourself in the foot.

Are you familiar with the economic concept of 'replacement value'? Hermione's replacement value is infinite! There's nowhere else I can go to buy another one!

Our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

There is no justice in the laws of nature, no term for fairness in the equations of motion. The Universe is neither evil, nor good, it simply does not care. The stars don't care, or the Sun, or the sky.
But they don't have to! WE care! There IS light in the world, and it is US!

Note to self: Overthrow government of magical Britain at earliest convenience.

My parents were early adopters, and I've been online since a rather young age. You should regard anything from 2001 or earlier as having been written by a different person who also happens to be named 'Eliezer Yudkowsky.' I do not share his opinions.

Is there some amazing rational thing you do when your mind's running in all different directions?" she managed.

"My own approach is usually to identify the different desires, give them names, conceive of them as separate individuals, and let them argue it out inside my head. So far the main persistent ones are my Hufflepuff, Ravenclaw, Gryffindor, and Slytherin sides, my Inner Critic, and my simulated copies of you, Neville, Draco, Professor McGonagall, Professor Flitwick, Professor Quirrell, Dad, Mum, Richard Feynman, and Douglas Hofstadter."

Hermione considered trying this before her Common Sense warned that it might be a dangerous sort of thing to pretend. "There's a copy of me inside your head?"

"Of course there is!" Harry said. The boy suddenly looked a bit more vulnerable. "You mean there isn't a copy of me living in your head?"

There was, she realized; and not only that, it talked in Harry's exact voice.

"It's rather unnerving now that I think about it," said Hermione. "I do have a copy of you living in my head. It's talking to me right now using your voice, arguing how this is perfectly normal."

"Good," Harry said seriously. "I mean, I don't see how people could be friends without that.

He was personally responsible for becoming more ethical than the society he grew up in.

a belief is only really worthwhile if you could, in principle, be persuaded to believe otherwise. If your retina ended up in the same state regardless of what light entered it, you would be blind .

Don't let your life be steered by your reluctance to do a little extra thinking.

The next time, Mr. Potter, that you choose to escalate a contest rather than lose, you may lose all the stakes you place on the table.

It was sometimes praised as a noble public endeavor, but nearly all other Atlanteans found more important things to do on any given day than help. Even the Atlantean nobles ignored the prospect of somebody other than themselves obtaining unchallengeable power, which a less experienced cynic might expect to catch their attention. With relatively little support, the tiny handful of would-be makers of this device labored under working conditions that were not so much dramatically arduous, as pointlessly annoying. Eventually time ran out and Atlantis was destroyed with the device still far from complete.

Did ghosts not require a salary? Was that it? Or was it literally impossible to fire anyone in Hogwarts even if they died?
Now it seemed that Professor Snape was going about being absolutely awful to everyone who wasn't a Slytherin and it hadn't even occurred to anyone to terminate his contract.
And the Headmaster had set fire to a chicken.

Most Muggles lived in a world defined by the limits of what you could do with cars and telephones. Even though Muggle physics explicitly permitted possibilities like molecular nanotechnology or the Penrose process for extracting energy from black holes, most people filed that away in the same section of their brain that stored fairy tales and history books, well away from their personal realities: Long ago and far away, ever so long ago.

Don't trust thoughts because you think them, but because they obey specific trustworthy rules.

Complex adaptations like "being a little selfish" and "not being willing to
work without reward" are human universals. The strength might vary a bit
from person to person, but everyone's got the same machinery under the
hood, we're just painted different colors.

Lies propagate, that's what I'm saying. You've got to tell more lies to cover them up, lie about every fact that's connected to the first lie. And if you kept on lying, and you kept on trying to cover it up, sooner or later you'd even have to start lying about the general laws of thought. Like, someone is selling you some kind of alternative medicine that doesn't work, and any double-blind experimental study will confirm that it doesn't work. So if someone wants to go on defending the lie, they've got to get you to disbelieve in the experimental method. Like, the experimental method is just for merely scientific kinds of medicine, not amazing alternative medicine like theirs. Or a good and virtuous person should believe as strongly as they can, no matter what the evidence says. Or truth doesn't exist and there's no such thing as objective reality. A lot of common wisdom like that isn't just mistaken, it's anti-epistemology, it's systematically wrong. Every rule of rationality that tells you how to find the truth, there's someone out there who needs you to believe the opposite. If you once tell a lie, the truth is ever after your enemy; and there's a lot of people out there telling lies

And I know that I'd just make myself unhappy by comparing that reality to... something perfect that I built up in my imagination.

So the next time you doubt the strangeness of the future, remember how you were born in a hunter-gatherer tribe ten thousand years ago, when no one knew of Science at all. Remember how you were shocked, to the depths of your being, when Science explained the great and terrible sacred mysteries that you once revered so highly. Remember how you once believed that you could fly by eating the right mushrooms, and then you accepted with disappointment that you would never fly, and then you flew. Remember how you had always thought that slavery was right and proper, and then you changed your mind. Don't imagine how you could have predicted the change, for that is amnesia. Remember that, in fact, you did not guess. Remember how, century after century, the world changed in ways you did not guess.
Maybe then you will be less shocked by what happens next.

Hermione's eyes lit up with a terrible light of helpfulness and something in the back of Harry's brain screamed in desperate humiliation.

Textbook science is beautiful! Textbook science is comprehensible, unlike mere fascinating words that can never be truly beautiful. Elementary science textbooks describe simple theories, and simplicity is the core of scientific beauty. Fascinating words have no power, nor yet any meaning, without the math.

Trying and getting hurt can't possibly be worse for you than being ... stuck.

Human beings," said the Ship's Confessor, "cannot designate a 'current best candidate' without psychological consequences. Human rationalists learn to discuss an issue as thoroughly as possible before suggesting any solutions. For humans, solutions are sticky in a way that would require detailed cognitive science to explain. We would not be able to search freely through the solution space, but would be helplessly attracted toward the 'current best' point, once we named it. Also, any endorsement whatever of a solution that has negative moral features, will cause a human to feel shame - and 'best candidate' would feel like an endorsement. To avoid feeling that shame, humans must avoid saying which of two bad alternatives is better than the other.

I've lived the lives of all the characters in all my books, and all their mighty wisdom thunders in my head.

You know, Mr. Lupin," Harry said, "it really takes a baroque interpretation to think that somebody would be walking around, pondering how death is just something we all have to accept, and communicate their state of mind by saying, 'The last enemy that shall be destroyed is death.' Maybe someone else thought it sounded poetic and picked up the phrase and tried to interpret it differently, but whoever said it first didn't like death much." Sometimes it puzzled Harry how most people didn't seem to even notice when they were twisting something around to the 180-degree opposite of its first obvious reading. It couldn't be a raw brainpower thing, people could see the obvious reading of most other English sentences. "Also 'shall be destroyed' refers to a change of future state, so it can't be about the way things are now.

The human brain cannot release enough neurotransmitters to feel emotion a thousand times as strong as the grief of one funeral. A prospective risk going from 10,000,000 deaths to 100,000,000 deaths does not multiply by ten the strength of our determination to stop it. It adds one more zero on paper for our eyes to glaze over.

Since the rise of Homo sapiens, human beings have been the smartest minds around. But very shortly - on a historical scale, that is - we can expect technology to break the upper bound on intelligence that has held for the last few tens of thousands of years.

You've got to have an imagination to make it to the stars. The sort of species that wouldn't invent science fiction, probably wouldn't even invent the wheel -

The truly important problems are often the ones you're not even considering, because they appear to be impossible, or, um, actually difficult, or worst of all, not clear how to solve.

Doing worse with more knowledge means you are doing something very wrong.