Range, page 10
Unsurprisingly, there was a group of Calculus I professors whose instruction most strongly boosted student performance on the Calculus I exam, and who got sterling student evaluation ratings. Another group of professors consistently added less to student performance on the exam, and students judged them more harshly in evaluations. But when the economists looked at another, longer-term measure of teacher value added—how those students did on subsequent math and engineering courses that required Calculus I as a prerequisite—the results were stunning. The Calculus I teachers who were the best at promoting student overachievement in their own class were somehow not great for their students in the long run. “Professors who excel at promoting contemporaneous student achievement,” the economists wrote, “on average, harm the subsequent performance of their students in more advanced classes.” What looked like a head start evaporated.
The economists suggested that the professors who caused short-term struggle but long-term gains were facilitating “deep learning” by making connections. They “broaden the curriculum and produce students with a deeper understanding of the material.” It also made their courses more difficult and frustrating, as evidenced by both the students’ lower Calculus I exam scores and their harsher evaluations of their instructors. And vice versa. The calculus professor who ranked dead last in deep learning out of the hundred studied—that is, his students underperformed in subsequent classes—was sixth in student evaluations, and seventh in student performance during his own class. Students evaluated their instructors based on how they performed on tests right now—a poor measure of how well the teachers set them up for later development—so they gave the best marks to professors who provided them with the least long-term benefit. The economists concluded that students were actually selectively punishing the teachers who provided them the most long-term benefit. Tellingly, Calculus I students whose teachers had fewer qualifications and less experience did better in that class, while the students of more experienced and qualified teachers struggled in Calculus I but did better in subsequent courses.
A similar study was conducted at Italy’s Bocconi University, on twelve hundred first-year students who were randomized into introductory course sections in management, economics, or law, and then the courses that followed them in a prescribed sequence over four years. It showed precisely the same pattern. Teachers who guided students to overachievement in their own course were rated highly, and undermined student performance in the long run.
Psychologist Robert Bjork first used the phrase “desirable difficulties” in 1994. Twenty years later, he and a coauthor concluded a book chapter on applying the science of learning like this: “Above all, the most basic message is that teachers and students must avoid interpreting current performance as learning. Good performance on a test during the learning process can indicate mastery, but learners and teachers need to be aware that such performance will often index, instead, fast but fleeting progress.”
* * *
• • •
Here is the bright side: over the past forty years, Americans have increasingly said in national surveys that current students are getting a worse education than they themselves did, and they have been wrong. Scores from the National Assessment of Educational Progress, “the nation’s report card,” have risen steadily since the 1970s. Unquestionably, students today have mastery of basic skills that is superior to students of the past. School has not gotten worse. The goals of education have just become loftier.
Education economist Greg Duncan, one of the most influential education professors in the world, has documented this trend. Focusing on “using procedures” problems worked well forty years ago when the world was flush with jobs that paid middle-class salaries for procedural tasks, like typing, filing, and working on an assembly line. “Increasingly,” according to Duncan, “jobs that pay well require employees to be able to solve unexpected problems, often while working in groups. . . . These shifts in labor force demands have in turn put new and increasingly stringent demands on schools.”
Here is a math question from the early 1980s basic skills test of all public school sixth graders in Massachusetts:
Carol can ride her bike 10 miles per hour. If Carol rides her bike to the store, how long will it take?
To solve this problem, you would need to know:
A) How far it is to the store.
B) What kind of bike Carol has.
C) What time Carol will leave.
D) How much Carol has to spend.
And here is a question Massachusetts sixth graders got in 2011:
Paige, Rosie, and Cheryl each spent exactly $9.00 at the same snack bar.
Paige bought 3 bags of peanuts.
Rosie bought 2 bags of peanuts and 2 pretzels.
Cheryl bought 1 bag of peanuts, 1 pretzel, and 1 milk shake.
What is the cost, in dollars, of 1 bag of peanuts? Show or explain how you got your answer.
What is the cost, in dollars, of 1 pretzel? Show or explain how you got your answer.
What is the total number of pretzels that can be bought for the cost of 1 milk shake? Show or explain how you got your answer.
For every problem like the first one, the simple formula “distance = rate × time” could be memorized and applied. The second problem requires the connection of multiple concepts that are then applied to a new situation. The teaching strategies that current teachers experienced when they were students are no longer good enough. Knowledge increasingly needs not merely to be durable, but also flexible—both sticky and capable of broad application.
Toward the end of the eighth-grade math class that I watched with Lindsey Richland, the students settled into a worksheet for what psychologists call “blocked” practice. That is, practicing the same thing repeatedly, each problem employing the same procedure. It leads to excellent immediate performance, but for knowledge to be flexible, it should be learned under varied conditions, an approach called varied or mixed practice, or, to researchers, “interleaving.”
Interleaving has been shown to improve inductive reasoning. When presented with different examples mixed together, students learn to create abstract generalizations that allow them to apply what they learned to material they have never encountered before. For example, say you plan to visit a museum and want to be able to identify the artist (Cézanne, Picasso, or Renoir) of paintings there that you have never seen. Before you go, instead of studying a stack of Cézanne flash cards, and then a stack of Picasso flash cards, and then a stack of Renoir, you should put the cards together and shuffle, so they will be interleaved. You will struggle more (and probably feel less confident) during practice, but be better equipped on museum day to discern each painter’s style, even for paintings that weren’t in the flash cards.
In a study using college math problems, students who learned in blocks—all examples of a particular type of problem at once—performed a lot worse come test time than students who studied the exact same problems but all mixed up. The blocked-practice students learned procedures for each type of problem through repetition. The mixed-practice students learned how to differentiate types of problems.
The same effect has appeared among learners studying everything from butterfly species identification to psychological-disorder diagnosis. In research on naval air defense simulations, individuals who engaged in highly mixed practice performed worse than blocked practicers during training, when they had to respond to potential threat scenarios that became familiar over the course of the training. At test time, everyone faced completely new scenarios, and the mixed-practice group destroyed the blocked-practice group.
And yet interleaving tends to fool learners about their own progress. In one of Kornell and Bjork’s interleaving studies, 80 percent of students were sure they had learned better with blocked than mixed practice, whereas 80 percent performed in a manner that proved the opposite. The feeling of learning, it turns out, is based on before-your-eyes progress, while deep learning is not. “When your intuition says block,” Kornell told me, “you should probably interleave.”
Interleaving is a desirable difficulty that frequently holds for both physical and mental skills. A simple motor-skill example is an experiment in which piano students were asked to learn to execute, in one-fifth of a second, a particular left-hand jump across fifteen keys. They were allowed 190 practice attempts. Some used all of those practicing the fifteen-key jump, while others switched between eight-, twelve-, fifteen-, and twenty-two-key jumps. When the piano students were invited back for a test, those who underwent the mixed practice were faster and more accurate at the fifteen-key jump than the students who had only practiced that exact jump. The “desirable difficulty” coiner himself, Robert Bjork, once commented on Shaquille O’Neal’s perpetual free-throw woes to say that instead of continuing to practice from the free-throw line, O’Neal should practice from a foot in front of and behind it to learn the motor modulation he needed.
Whether the task is mental or physical, interleaving improves the ability to match the right strategy to a problem. That happens to be a hallmark of expert problem solving. Whether chemists, physicists, or political scientists, the most successful problem solvers spend mental energy figuring out what type of problem they are facing before matching a strategy to it, rather than jumping in with memorized procedures. In that way, they are just about the precise opposite of experts who develop in kind learning environments, like chess masters, who rely heavily on intuition. Kind learning environment experts choose a strategy and then evaluate; experts in less repetitive environments evaluate and then choose.
* * *
• • •
Desirable difficulties like testing and spacing make knowledge stick. It becomes durable. Desirable difficulties like making connections and interleaving make knowledge flexible, useful for problems that never appeared in training. All slow down learning and make performance suffer, in the short term. That can be a problem, because like the Air Force cadets, we all reflexively assess our progress by how we are doing right now. And like the Air Force cadets, we are often wrong.
In 2017, Greg Duncan, the education economist, along with psychologist Drew Bailey and colleagues, reviewed sixty-seven early childhood education programs meant to boost academic achievement. Programs like Head Start did give a head start, but academically that was about it. The researchers found a pervasive “fadeout” effect, where a temporary academic advantage quickly diminished and often completely vanished. On a graph, it looks eerily like the kind that show future elite athletes catching up to their peers who got a head start in deliberate practice.
A reason for this, the researchers concluded, is that early childhood education programs teach “closed” skills that can be acquired quickly with repetition of procedures, but that everyone will pick up at some point anyway. The fadeout was not a disappearance of skill so much as the rest of the world catching up. The motor-skill equivalent would be teaching a kid to walk a little early. Everyone is going to learn it anyway, and while it might be temporarily impressive, there is no evidence that rushing it matters.
The research team recommended that if programs want to impart lasting academic benefits they should focus instead on “open” skills that scaffold later knowledge. Teaching kids to read a little early is not a lasting advantage. Teaching them how to hunt for and connect contextual clues to understand what they read can be. As with all desirable difficulties, the trouble is that a head start comes fast, but deep learning is slow. “The slowest growth,” the researchers wrote, occurs “for the most complex skills.”
Duncan landed on the Today show discussing his team’s findings. The counteropinion was supplied by parents and an early childhood teacher who were confident that they could see a child’s progress. That is not in dispute. The question is how well they can judge the impact on future learning, and the evidence says that, like the Air Force cadets, the answer is not very well.*
Before-our-eyes progress reinforces our instinct to do more of the same, but just like the case of the typhoid doctor, the feedback teaches the wrong lesson. Learning deeply means learning slowly. The cult of the head start fails the learners it seeks to serve.
Knowledge with enduring utility must be very flexible, composed of mental schemes that can be matched to new problems. The virtual naval officers in the air defense simulation and the math students who engaged in interleaved practice were learning to recognize deep structural commonalities in types of problems. They could not rely on the same type of problem repeating, so they had to identify underlying conceptual connections in simulated battle threats, or math problems, that they had never actually seen before. They then matched a strategy to each new problem. When a knowledge structure is so flexible that it can be applied effectively even in new domains or extremely novel situations, it is called “far transfer.”
There is a particular type of thinking that facilitates far transfer—a type that Alexander Luria’s Uzbek villagers could not employ—and that can seem far-fetched precisely because of how far it transfers. And it’s a mode of broad thinking that none of us employ enough.
CHAPTER 5
Thinking Outside Experience
THE SEVENTEENTH CENTURY was approaching. The universe was one in which celestial bodies moved around the stationary Earth powered by individual spirits, ineffable planetary souls. The Polish astronomer Nicolaus Copernicus had proposed that planets moved around the sun, but the idea was still so unorthodox that Italian philosopher Giordano Bruno was censured for teaching it, and later burned at the stake as a heretic for insisting there were other suns surrounded by other planets.
Their spirits may have been driving, but the planets also needed a vehicle for motion, so they were assumed to be riding on pure crystalline spheres. The spheres were invisible from Earth and interlocked, like the gears of a clock, to produce collective motion at a constant speed for all eternity. Plato and Aristotle had laid the foundation for the accepted model, and it dominated for two thousand years. That clockwork universe was the one German astronomer Johannes Kepler inherited. He accepted it, at first.
When the constellation Cassiopeia suddenly gained a new star (it was actually a supernova, the bright explosion at the end of a star’s life), Kepler recognized that the idea of the unchanging heavens could not be correct. A few years later, a comet tracked across the European sky. Shouldn’t it have cracked the crystalline spheres as it traveled, Kepler wondered? He began to doubt two millennia worth of accepted wisdom.
By 1596, when he turned twenty-five, Kepler had accepted the Copernican model of planets orbiting the sun, and now he posed another profound question: Why do planets that are farther away from the sun move more slowly? Perhaps the more distant planets had weaker “moving souls.” But why would that be? Just coincidence? Maybe, he thought, rather than many spirits, there was just one, inside the sun, which for some reason acted more powerfully on nearby planets. Kepler was so far outside the bounds of previous thought that there was no evidence in existence for him to work from. He had to use analogies.
Smells and heat dissipate predictably farther from their source, which meant that a mysterious planet-moving power from the sun might as well. But smells and heat are also detectable everywhere along their path, whereas the sun’s moving soul, Kepler wrote, is “poured out throughout the whole world, and yet does not exist anywhere but where there is something movable.” Was there any proof that such a thing could exist?
Light “makes its nest in the sun,” Kepler wrote, and yet appears not to exist between its source and an object it lights up. If light can do it, so could some other physical entity. He began using the words “power” or “force” instead of “soul” and “spirit.” Kepler’s “moving power” was a precursor to gravity, an astounding mental leap because it came before science embraced the notion of physical forces that act throughout the universe.
Given how the moving power seemed to emanate from the sun and disperse in space, Kepler wondered if light itself or some light-like force caused planetary motion. Well, then, could the moving power be blocked like light? Planetary motion did not stop during an eclipse, Kepler reasoned, so the moving power could not be just like light, or depend on light. He needed a new analogy.
Kepler read a newly published description of magnetism, and thought maybe the planets were like magnets, with poles at either end. He realized that each planet moved more slowly when it was farther in its orbit from the sun, so perhaps the planets and the sun were attracting and repelling one another depending on which poles were nearby. That might explain why the planets moved toward and away from the sun, but why did they keep moving forward in their orbits? The sun’s power seemed somehow to also push them forward. On to the next analogy.
The sun rotates on its axis and creates a whirlpool of moving power that sweeps the planets around like boats in a current. Kepler liked that, but it raised a new problem. He had realized that orbits were not perfectly circular, so what kind of strange current was the sun creating? The whirlpool analogy was incomplete without boatmen.
Boatmen in a whirling river can steer their boats perpendicular to the current, so maybe planets could steer in the sun’s current, Kepler surmised. A circular current could explain why all the planets move in the same direction, and then each planet steered through the current to keep from getting sucked into the center, which made the orbits not quite circular. But then who was captaining each ship? That brought Kepler all the way back to spirits, and he was not happy about it. “Kepler,” he wrote to himself, “does’t thou wish then to equip each planet with two eyes?”
