View transcript
- All right, test for you. Whatever you see, duck or rabbit, flip it, change it to the other one. How did you do that? What did you do in your brain? There was some switches, or dials, or knobs, or sliders, that you moved in around your head to push yourself towards the other interpretation. This is what perceptual psychologists will study, will use artificial stimuli like this to figure out how those switches and knobs work. These things don't actually happen in the real world. This thing is perfectly titrated to be half way between the two possibilities so that we can explore those switches and knobs, but they actually rarely happen in the real world. Real world objects are almost always veridically recognized right away. Finally did find one example of a real-world object where it takes me about ten seconds of flipping back and forth to figure out whether this is a purse or dachshund. It does, after about second, nine or ten I do convince myself that it is a purse, but it takes some time. So that's the only real world example I know of. Other wise, you have to do these perfectly titrated things in the lab. So in the real world this rarely happens, but in visualization this kind of hopping back and forth between possibilities is constant. You're always looking for different patterns and different interpretations and different comparisons of the data in any visualization. Visualizations are ambiguous figures. Let me give you my favorite example of this. This is one from the, more from the education and spatial's cognition literature So here's a graph of an object's motion. Notice that the y-axis' position and the x-axis' time, and I'd like you to imagine some real world situation involving real objects that could produce these data. I'll give ya a couple seconds to think about it. I'm not gonna call on anyone. But you show this to high school students and college students, and a substantial number of them will suggest things like, this is a graph of a ball rolling down a hill. And I hope you're realizing that this is not a good example for this graph. That is seeing a rabbit when you really wanna see a duck in this image. Actually, more specifically, more like a small pile of ducklings. You wanna break it up into three different pieces and you wanna look at that first piece and you wanna say, well the position is not changing but time is, so it's an object at rest. And then both are changing, so it's an object in motion, and we don't actually know which way it's moving. And then the purple bit at the end is an object at rest again. So that was a really different way of seeing this kind of ambiguous figure either as a rabbit or as a bunch of ducks. So it's critical to be able to learn how to see the right view. So how do we changes these knobs and sliders in our brain to see the right view? Let's go back to the rest of the perception literature and look for ambiguous figures. How can you see either the face or the vase in this image? Well, I like to use the metaphor of this sort of audio mixing board. I spent a little bit too much time in the AV club in high school. I don't know if you could tell, God that nerdy past. But this is actually a pretty veridical metaphor for how your brain is going to change what you see. You can turn up the gain or turn down the gain on aspects of this image. So for the case of the face vase, you can turn up the gain on the darker region of space and that's gonna cause you to be more likely to see the face instead of the vase. And then you can turn it down and we're back to our ambagious past. Here's another example. Soon as you looked at these berries, you immediately turned up the gain on those bigger, juicer blueberries. And the image in the back of your head, somewhere in the visual cortex literally changed so that you're seeing something different on the screen. And then you wanted to make sure that they actually were bigger, juicer than the red ones, and so you turned that back down and you turned up the gain on the red. You're constantly moving these gain knobs in your visual system. This process can be so powerful it really does literally change what you see, least for the representation in the back of your head, that I'm sure most of you in this room have seen this demonstration of watching a basketball game between the white shirted team and the black shirted team. And your job is to count the passes from the white shirted team, which causes you to turned up the gain on the light shirts, so you're seeing something like this in the back of your brain, and then as the video goes on, you're so, you have this gain knob turned up so high, that you don't notice what you would notice if you weren't counting the white team, that there's a gorilla walking through the middle of the basket ball game. I think that 90% of you in this room have seen this demo. If you haven't, go to that webpage on the bottom. Even if I'm ruined it for the other 10%, I promise there's other great stuff on the page, I won't tell you what, that you will find interesting. So the goal, one of the main goals of our lab, is to figure out how you turned these knobs, and switches and sliders in your visual system in order to see different views of the same visualization. Here's another example. This is one that was generated by a former post-doc in the lab, Audrey Michael, and she interviewed high school teachers to find out what kind of relatively simple visualizations still had multiple interpretations that students would struggle with. Here's a good example. Seasonable temperatures in Sydney and Chicago for two different months, January and July. And watching you all turn the knobs and sliders in your visual system, and you're all browsing through about three or four, at minimum, interesting patterns in this very tiny dataset. And we wanna know, let's say, why is it that I suddenly wanna live in Sydney? Well the average temperature's higher than Chicago. And one of the ways you do that might be to compare the two means, and so if I wanna grab the mean from the red connection, maybe I move up the slider on the red, and that gets me part of the way, but how do we actually get the mean now, right? I'm focusing on the red, but this isn't helping me know how I get the average out of those two. So now it's time to make stuff up, and come up with new possibilities, hypotheses generation, right? How do you get the average of these two red bars? Do you do something like, intuitively I feel like maybe I get this invisible box and then I bisect it. Or maybe I get this box over here and then I bisect it. Or maybe I draw a line between these two points and then I bisect it. If I'm gonna teach this young student how to get an average out of these bars, I need to have some sense of how I do it in my brain, and I don't know how to do it. So what we do is we generate these possibilities and then we try to empirically test between them to figure out how I do, or maybe how I should, do this simple operation. And what we find is that what we know so far, our model of the visual system and how these switches and knobs work, is not sufficient, right? None of that last bit of stuff that I did is present on this slider over here. So that slider model is not gonna be sufficient. It's on my bad list for the moment. We're gonna get rid of it for a second. So if you have these two collections and you get the average of one verses the average of another, I starting to have some sense of how I do that. But what if the collections are bigger? Let me flip to a more complicated example for a moment. So who has the higher salary on average at this company? The male of the female employees? So there's an averaging that you're doing across these two. How did you do that? Going back to tomorrow's previous talk, you could pull out the values from these individual bars, from those marks, by looking at the top's of the positions or their lengths, right? So you averaging together the tops of the positions or the individual lengths of the bars. Or you're doing something else, some other proxy, something ridiculous like the total area of all of the gray pixels or all of the black pixels. No way you're doing something that silly in your head. You know what, if you were doing something like that, it would really mess you up if the number of bars in the male and the female collection were different, because then the area of pixels would still be different, would always be different, regardless of the averages. And we have a condition like this, and this drops your accuracy in a big way. So there's at least a major contribution of this weird proxy of the total number of pixels to these average judgments for, and these are college students, right? This isn't for little kids. So you're, it's sometimes hard to tell what your visual system is secretly doing, but you can empirically unpack it. Let's go back to our other graph. What else do you wanna pull from here? You wanna that, another reason I would rather live in Sydney is this seasonal variation. The range of temperatures is much smaller in Sydney as compared to Chicago. So that's another visual pattern that you picked out to pick up this higher level point. Well, the main reason why the high school teacher introduced this example is because they use it to talk about hemispheric differences in seasonality, right? The fact that Sydney's got their summer in January, and then they talk about globes, and angle of the sun, and et cetera. In order to pull out that pattern, you actually have to inhibit that last pattern. The difference in the range, that's irrelevant. That quantitative metric difference is irrelevant. It's the qualitative polarity difference in who's value is bigger when, that's critical for understanding this seasonable swap. So you have to see that that's a minus and that's a plus relationship, and ignore the metric differences in those two sets of bars in order to connect the visual pattern here with the rest of the lesson, and so that's tough. So one person, at different times, can see different views of the same data. Even at the same time, two different human beings can also see two different views of the same data. So one can see the rabbit, and one can see the duck. Here's my favorite example of that. If you go to an image search and you type in, I will be charitable, complicated dense slide, you get this image, and this is one that we could spend some time critiquing and talking about a better way to set up this particular very important point, but the important point here is that the person who made this sees this fundamentally differently than I do, right? When I look at this I don't know how to turn my knobs and switches in order to see what data support bullet point number two. The author does, and when the author sees bullet point number two, their knobs and switches are gonna turn. They're gonna see the duck, and they're gonna notice that the image will actually change, and they'll see that the ratio of the purple to green in the second bar, just making stuff up now, is bigger than the ratio of the purple to green in the fifth bar, right? Here's the trick, though. I think that they really believe that because they see it that way, I do too. There's this phrase in the education literature known as the cursive expertise, or the cursive knowledge, that once you see it that way, you really feel, you don't notice that you're not explaining it or showing it to other people. So we wanted to simulate this in the lab. Here's a study from a PhD student named Cindy Xiong, and a collaborator named Lisanne van Weelden from Utrecht. They showed people a simple visualization. So Jennifer, you're gonna be my research subject here. So I'm gonna tell you, I'm gonna make you an expert. I'm gonna put some expert glasses on you in about 20 seconds. See this mythical European country in these awesomely vaguely political party names? These are political polling data. See those top two parties, the green one and the blue one? So the green ones was one in the lead, but then those two parties had a debate and the green candidate insulted the spouse of the blue candidate. Aw, look at their popularity drop. But then, a month later the green candidate, who's just neck and neck with the blue candidate, ran into a burning building and saved a puppy, saved a whole pile of puppies, put them in a bag, brought them back out, right? And so you could see how their popularity spikes, 'cause cameras caught it all. All right, I want you to forget all that. Just forget that story.

- [Audience Member] They were purses not puppies.

- They were purses not puppies, and that was. It's a good thing the camera was far away, very nice. So now we've really made this salient, right? Purses, puppies, all this, it's stuck in your head. You're cursed with expertise. I want you to forget everything you just saw on the left. I'm gonna grab some random person from the rest of the Exploratorium. We're gonna bring them in here, and we're gonna ask them to circle, highlight, the most salient, interesting pattern on that unannotated graph on the right. Can you simulate for me what they're gonna draw? And so you have to kind of put on the lens of somebody who doesn't know that story from the left graph. And you're gonna circle something, and you're gonna circle something, and you're gonna circle something, and all the data for everybody in this room are gonna look like this. You're gonna think that other people see what you see, that this top pair of lines is just objectively more salient. Maybe you're right, maybe it is, it's the ones on the top, you could think of a hundred reasons why you might be right. So we have another group of people we tell exactly the same story, but we tell it about the bottom two lines. Look, da, da, da, da, and the puppies, right? And then we say forget everything we just said, here's an unannotated copy of the graph. We're gonna grab a person from outside. They don't know the story. We remind them they don't know the story, and you draw that. So now you think that other people are gonna see what you see, but it's a completely different pattern. So this is that cursive expertise, or cursive knowledge, but simulated in the lab. So how do you get people to see what you see? One of our sources of inspiration for the way that we study how to get other people to see through your glasses is data journalists, so your Groegers, your Popovichs, some of the folks in this room. You could add Propublica or Economist to this list. And they know that you can't just show the raw data and expect people to see what you see, and so they're really good at adding highlighting and annotation, that lets you understand from the analyst or author's point of view what you should be seeing, right? You've all got it now. So here's what they're gonna do. Well, here's what they're not gonna do. If I let you stare at this, it's gonna take you about 15 seconds. This is the kind of thing we time in the lab to figure out okay, there's a point, and then I gotta go and is it, hm, there's a thing and hm, and you have to learn, figure out what pattern in the data supports these sentences on top. They're not gonna do it this way. They're gonna do it like this. Unemployment is stated, is higher than stated goals, and then there's a little clicker, like a little stepper button. They don't really use that any more, now it's more scrolly telling, right? So now you're gonna go like this with your mouse, or like this with your phone, and you're gonna see this progression of views, where they're gonna show you what they see as they simulated talk over this visualization. So I think you notice that unemployment is getting better, yes, but remember that there was this 8% goal over here, and actually there's a lot of area over that line within the curve, every dot's 10 thousand unemployed people. And especially notice, that even recently, we're still substantially over this goal. Everybody in the room followed along, everybody saw what I see when I walk through this. And I think this is a really cool example of coordinating a cognitive, linguistic, high-level argument with what sets a perceptual configurations of those knobs and switches best go along with those states. So to sum up, visualizations are ambagious figures. You could see this one this way, or this way. We study how you control these different views in a visualization. When you're explaining data, or teaching people how to read a graph, it's tough because you see the visualization in one way, duck or rabbit? And your brain assumes that other people see it like you see it. And what we learned from this research is you really need to explicitly show what you see, and one inspiration for how to do that right, how to coordinate language and vision at the same time, is to follow this annotation and painting strategy that a data journalist, or I'm sure many of the other informal science education folks in this room, I'd love to learn from you in this respect, do. So thanks.