View transcript
- This is the slide that should have been. It's kind of an overture or a splash page of things I've done that are reasonably relevant in the last 20 years. This is a poster all over New York subway stations or bus stations. For the Tribeca Film Festival. Great stories are timeless. And I hope you can see the Egyptian theme. So what, and you can buy books that will tell you there's only one story, or there are seven, or 150, you can pick. So, what about the old stories? This is the oldest map found. It's two inches by one, it's a stone. You can carry it with you. It shows two perspectives at the same time, an overview and the paths. And it was found in a cave. This is the previous oldest map. This is even older, using your hand to show where things are, and gestures preceded anything on stone or paper. And they're another important form of visual communication that I've been studying and can't have time to tell you about. Time, so we have space, maps, we have time, as old stories. This is the Chauvet cave. It looks like it's meant to be a hunt. We have explanations, this is making bread in an Egyptian tomb. We have stories, this is one from China of a journey of an emperor. So we have space, time, explanations, stories and number. This is 70,000 years ago, up to 100, and a kind of tally. These kinds of things are found all over the world. And those are the four stories that are, or four ways of visualizing that seem to appear all over the world. So kinds of discourse. I did a big survey looking into film and narrative and philosophy and linguistics and so forth, coming, trying to see what do people think stories are. So one way is to look at kinds of discourse. Description, explanation, story, conversation. So description is a state of affairs in time or space. So I showed you two ancient examples of that. And there are many as you walk around this wonderful museum. Explanations add causality. And they add a beginning, middle and end, an outcome. Stories seem to add, now here there's some dispute, they seem to add a narrative voice. This is what the philosophers say. In my mind, they also add agency, this is a Tversky original. And I'll explain that a little more. Emotion, suspense, the sorts of things that Scott was exemplifying beautifully. Conversation, now a story can build on explanations and descriptions, so it's not that discourse is pure. Conversations are brief interchanges, not like what we're doing now, right? And they're interactive, and we talk about visualizations that are interactive. And they, again, can contain all of the above. Stories, they have a narrative voice or a perspective. There's a point of view. There is typically emotion, suspense, desire would be one of them. Stories override data. In some ways, they're the antithesis of data. So this is known in psychology as the Volvo effect. You do all the research on your new car, Volvo's the safest, most reliable, best gas mileage and so forth. Your uncle had a lemon, and you don't touch a Volvo. So again, my point of view on stories is they add agency, a kind of human mind or human-like mind that can control events or try to control events. And that's a different kind of causality. And this leads to personification and anthropomorphism. And it's rampant. I'm not saying it's a bad thing. It's rampant both in the way we talk, we talk about chemical bonding, attraction, these are words that we use that human beings use. And anthropomorphism, you can see in all kinds of cartoons. And Larry Gonick, I'm a big fan of his, his guides, cartoon guides to science. Okay, thought is discrete and linear. By linear I don't mean logical. I mean one thought after another. The way we walk through, from place to place. Our minds go from thought to thought. They can go like this, right? It's not necessarily linear. But thought is discrete, our views of space, time, number, actions, explanations, stories are all broken into parts. This is a longer story that I won't have time to tell now, but I've got a lot of research backing. So the parts have natural breaks, like my joints. And actions have natural breaks by the, accomplishing goals and sub-goals. They form hierarchies that we call partonomies. So I can divide my body into arms and legs. And my arms and legs into parts, and so forth. Joints are important. They connect the parts. So explanations and stories are discrete and step by step. Here's one example of students who showed us step by step how to put together a TV cart. This is a cycle, we gave people descriptions of cycles, like a seed turns into a flower, creates a new seed. We asked them to put something down on paper that shows that. And we did do seed to seed. Most people make us lines, 'cause time is linear. Thinking about, it's not the same seed that it returns to. This is again a longer story. Elements of visual narratives. We can use depictions that are more or less iconic. We can use figurative depictions, like scales of justice. I have a lot to say about that. We can use schematic forms like dots and lines. We can use all kinds of symbols like words and symbols. And often in comics and in visual explanations of science, the words don't form sentences, and that's fine. Meaningful schematic marks. So these are again schematic marks, but they have meaning. And we've shown that in a bunch of experiments that you could call empirical semantics. So points represent places or ideas, depending on whether you have maps or networks of ideas. Lines are paths or relations connecting the ideas. An arrow is an asymmetric relationship. An asymmetric line indicating an asymmetric relation. Then we have containers of different forms. They can be areas, they can be sets that contain things. Again, a longer story. So sketch maps like one of our students, are points and lines connecting places and relations. Distances and direction are wildly inaccurate. But they still work. Why, because they're used in context. And most diagrams have a conceptual, or stories, have a conceptual context. It might be shared information that we share, it might be walking or driving in the ground, where what you see disambiguates. Not always, but the assumption is they're used in a context. Language too. I can't use highly mathematical terms here. Not just because I don't know them, but because you might not. Okay, so empirical semantics. We've shown in a series of experiments that boxes, bar graphs are interpreted as discrete comparisons, and lines as trends, in spite of conflicting data. So, information graphics are for inferences. And you get different inferences depending on how you display the information. And I want to encourage more research on that. These are bad diagrams, I can apologize for why. But it's the same data plotted. It's shown as a matrix, or shown as lines. When we get lines, we get lots of inferences about temporal sequences. When we use those dots in a matrix, we get many more inferences. And they're about everything. So depending on what you want to, the kinds of inferences that you want people to make, choose your infographic. Okay, arrows. We gave people these simple diagrams, only one, with or without an arrow. With an arrow, we get causal interpretations. People tell us in sequence what happens in a car brake or a bicycle pump or a pulley system. Without arrows, we get structural descriptions. They tell us where the parts are. And we get the reverse, if I give a structural description, I'm gonna get a diagram like the one on the left. If I give and tell people to diagram it, if I give people a functional behavioral description, and ask them to diagram it, I don't get labeled parts, but I get arrows. So arrows change meaning. This is a human arrow. And there were helicopters overhead looking for a getaway car. And the people on the ground knew where it was. And they formed an arrow to show the helicopter pilot where to go. It's a gem. Okay. So I've told you about the elements from iconic to more depictive, to this frames, and frames break up events in comics. Scott has shown this nicely. And how you display frames on a page matters. And the sizes and so forth. So, but we also have place in a page. So vertical, you're fighting gravity. So everything good goes up. And up is more good, powerful. The horizontal dimension is more, we have some body asymmetries. They seem to affect value. But otherwise it's pretty neutral. Reading order has a huge effect. And again, there's this much research backing that. And this is a curiosity, this is an Excel sheet in Hebrew, with English words. But it comes out of a Hebrew program. And because Hebrew is written right to left instead of left to right, they've plotted time backwards. And this is terribly confusing because the Western way of having time go left to right is what predominates science, and certainly in, all over the world. Okay, creating explanations. Again, I'm giving you tips of icebergs. We had junior high students learn a lesson in chemical bonding, this is work with Eliza Bobek. So she taught the lesson over two sessions, using all kinds of visuals and verbals. The students were tested. They were divided in half. Half made verbal explanations, the normal way of describing science. Half made visual explanations. Then they had a second test. Those groups improved from just making the explanation. They didn't read or learn anything more, just making the explanation. But the ones who made visual explanations improved way more. We think it's because visual explanations give you a check for coherence and they give you a check for completeness, and a platform for inference. Here is some of the visualizations they made. They aren't what they saw. We have sharks grabbing electrons. We have stick people giving them. We have boiling pots. So these, you can see how clever they are. And they come from the students' heads. Ambiguity came up yesterday. We did a study of architects. And they make discoveries in their own sketches. Experienced architects are better than novices. The ambiguity allows them to reconfigure. It's one of the reason they used to resist CADCAM programs. Similar work on artists, this is the work of Andrea Kantrowitz. Artists make discoveries as they draw. The drawing emerges. It's a wordless conversation between the eye and the hand and the page. And language doesn't do it. Okay, this is the year of Leonardo. And he invented new ways to draw. He thought through drawings. He didn't use math. There was not that much math known. He was an autodidact. He didn't know math. He thought through drawings. All his amazing discoveries came from drawings. And he invented new ways to draw that are diagrammatic, long before Diderot. Even later. Cutaways, enlargements, juxtapositions, changing perspectives, so here are some examples. Cutaways, enlargements, juxtapositions, multiple perspectives. Again. They're gorgeous. So visual narratives, whether they're descriptions or explanations, they break up time and space. Again, Scott's book, which I recommend, talks about this in detail. They show time, space, action, causality, emotion in the frames, between them. And they connect the segments, a kind of visual anaphora. Again, my term. And they have a secret ingredient, figures of depiction. More figures of depiction than the Greeks had words for. I tried to find them. So, I want to say one word on culture, 'cause that hasn't come up. We have studiesdcomics, kids' books and diagrams, east, west, also language effects. And we find consistent the views on the East, that they're more socially integrated, and that Western people are more individual. We found that reflected in diagrams for math and comics and in book covers. So that's again a longer story. Jumping to visual anaphora, this is a lovely cover from the New Yorker. And you can follow the red book. It's connecting each frame to the other. So what the anaphora does is it takes something from frame n, carries it over to frame n plus one, and that provides continuity. Sometimes you don't want continuity, and that's fine. Here are some things that comics can do that I don't see very often in visualizations either of information or of data. And they are devices that I think designers could use. So time is superimposed on space. You've got the frames indicating the temporal relations superimposed on a background, and you can follow both. This is from Larry Gonick. And this is quite common. You're showing temporal migrations on a map. Chris Weir. This is a comic on Feynman. And again, it's showing you the spatial, the walk he and his student are taking. And you can follow them on the walk, and you zoom in and out to see them. Okay, figures of depiction. We have metaphors, puns, personification. I don't know what you want to call it. Puppet governments. We have metaphor. This is again, this is Windsor McKay, showing the streets of New York that still look like that, even though this is 100 years old. And rolling up to become a treadmill, reflecting what life feels like sometimes. This is a New Yorker cover. I think you can't see it. But that O is a moon, it's the O of New Yorker. It's shining, it's the O for Obama. It's the issue that came out after he was elected. It's shining on the Lincoln Memorial. So again, just so much figurative language there. And for me, it brought tears. So we have Krazy Kat, this is a kind of visual onomatopoeia. Rhythm, to show that the chase is really fast. The frames get slanted. Here, we have more onomatopoeia. Sammy Sneeze is going to sneeze. and when the choo comes, the frame breaks. Okay, and he's surprised or taken aback. Here we show two perspectives, the perspective of the teacher bringing this new Chinese kid into class, and what the child sees, these unfriendly faces. So double perspective. A million examples of that. Compare and contrast. Another compare and contrast. This is Asaf Konuka. And you're seeing this child getting this wonderful robot all in color. Below it, you see the child labor that made it. This is a one-frame story, again compare and contrast. She's barging out of the house disgusted. And he doesn't even notice 'cause he's so busy looking at a date line. Figure-ground, you can tell us two stories at once. A background, this is sound and the shot. Here you're getting two things happening presumably at the same time, again by using figure-ground. And you can decipher that visually. Here you have a foreground, now, and a background story. Again, you can, there are tons of examples of this. Again, you can figure it out. You can break frames. This is the comic on Feynman. He's showing that he's, he's at MIT, entertaining a bevy of women, and passing a letter back to his fiance over the map, that's showing the distance between them. And she's at Columbia, and she's only got one guy. Okay. But breaking the frame, and again you can get it. Here is the three little pigs, breaking the frame, meant for kids. The first pig that the wolf eats comes out of the, out of the story, pokes his head back in, and says, you know, get out of the story, it's safe out here. Okay, meta, children's book, Harold is drawing his story. I have, I'm gonna end with designing the world. We now design, almost every bit of the world is designed. And our, we put our mind out into the world. So our book shows supermarket shelves reflect categories, hierarchies, orders. Buildings reflect repetition, symmetry, embedding, one to one correspondences. Our place settings are one to one correspondences, repetition, cycles, embeddings. Look how much intelligence is built into the way that we organize the world. And it allows foreigners, small children to decipher the world. The world isn't just organized the way our minds are organized, it's diagrams. This is a diagram overhead of an airport, telling where each kind of vehicle can go, and each kind of person. This is down the street from me. It tells you where you can walk, where buses can go, where cars can go, when you can walk, when you can't walk. The world is a diagram. So the world is a diagram, it's designed spaces created by actions that enable abstractions, like categories and one to one correspondences. So it's combining the actions, the gestures, the spaces, the abstractions. And there's a word, spraction, that I use to describe that cyclical way of thinking. And I end with this. Gratitude to NSF and the other sponsors. So thank you.