3iap

Unfair Comparisons: How Visualizing Social Inequality Can Make It Worse

Wed, 01 Dec 3002 00:00:00 GMT

TLDR

3iap’s peer-reviewed research was accepted to IEEE VIS 2022. And we replicated the results in a follow-up study accepted to IEEE VIS 2024.
Social cognitive biases can significantly impact dataviz interpretation.
Charts showing social outcome disparities can promote harmful stereotypes about the people being visualized. Showing within-group outcome variability can mitigate these risks.
Popular visualizations for showing social inequality can backfire and make it worse. News publishers, public health agencies, and social advocates should consider alternative approaches to minimize harm to marginalized communities.
The Equitable Dataviz Primer Workshop can teach your teams how to visualize inequality, without making it worse.

At first glance, the charts below seem harmless. They were published by reputable sources. They highlight important issues (disparities in health, wealth, incarceration, etc). They’re relatively clean and comprehensible. They’re clearly well-intended and, at least, not purposefully misleading.

Collage of likely harmful, deficit-framed data visualizations of racial disparities (credit: Brookings, NCES, Wikipedia, CDC, The Atlantic, Vox, CNN Money, Wikipedia, McKinsey, Economic Policy Institute, Economic Policy Institute, US Census, US Sentencing Commission, CDC, Federal Reserve, CNN).

But, as it turns out, instead of just raising awareness about inequality, charts like these can play an active role in making inequality worse. The way they’re framed can mislead audiences towards harmful stereotypes about the people being visualized.

Cindy Xiong and I explored this framing effect in a new paper that we presented at this year’s IEEE VIS conference. In our research, we show how visualizing social outcome disparities can create a deficit framing effect and that some of the most popular chart choices can make the effect significantly worse.

While it’s moderately depressing to spend a year of your life confirming (yet another way) that people can be judge-y and unfair towards each other, we’re eager to share this work with the data visualization community. These deficit-framed charts are everywhere and we’d love y’alls help in raising awareness and nudging the wider dataviz community toward more equitable design choices.

Toward that goal, I’d like to share a few key aspects of our research that I think are most relevant for data designers:

What is “deficit framing?”
What are the implications for dataviz?
What?! How could a simple chart possibly lead to stereotyping?
What did we find in our research?
What should data designers do differently?

But first, some background…

Eli’s Personal Backstory

In the summer of 2021, a post from Pieta Blakely turned my world upside down. She suggested that one of my beloved chart types (multi-series line charts) could, in fact, be subtly racist.

A multi-line timeseries chart showing 10th grader test scores disaggregated by race. This chart risks a "deficit framing;" by inviting direct comparisons between monolithic groups, the chart invites viewers to conclude that lower-scoring groups are individually deficient to higher-scoring groups. Chart redrawn from Pieta Blakely's "Presenting data for a Targeted Universalist approach."

She pointed out that charts emphasizing outcome differences between racial groups could actually make the disparities worse by encouraging harmful stereotypes about the people being visualized.

I was shocked! A jumble of reactions followed:

The chart above seems so innocent?!
I like the chart above! I used similar charts all the time.
Comparisons are a fundamental building block of data storytelling! Actionable dataviz relies on the contrast between a measurement and a meaningful benchmark. This is a concept I’m heavily invested in!
If this were true, how could I have made it this far and not know about it?! (I’ve been a dataviz nerd for quite a while now…)
Even bigger: If this were true, it implies that information can backfire… even when it’s accurate, well-intended, and cleanly designed.

That last point really stuck with me, because it has such big implications for dataviz, especially for data journalism and advocacy. It implies that a lot of prominent work for raising awareness of inequality… might actually make it worse.

Others in the data and viz communities have weighed in: Alice Feng and Jon Schwabish rallied around Blakely in their comprehensive Do No Harm Guide. Catherine D’Ignazio and Lauren F. Klein allude to similar issues in Data Feminism. On the other hand, Jack Dougherty and Ilya Ilyankou cited the discussion in O’Reilly’s recent Hands-On Data Visualization, but ultimately dismissed the concern.

But, as it turns out, the idea that raising awareness can backfire isn’t new at all. Education and equity researchers have been thinking about “deficit thinking” for, literally, a hundred years. It was just me, in my white guy tech bubble, who was late to the party.

What is “deficit thinking?” What does it mean for dataviz?

While the chart on the left could be explained by external factors like “Group A works in a busier restaurant than Group B,” many viewers explain the differences in terms of personal attributions, like “Group A must work harder than Group B,” despite the chart offering no causal evidence either way.

Deficit thinking is the surprisingly pervasive bias that the people who suffer inequalities are somehow personally to blame for them. And it’s well studied. Equity researchers Martha Menchaca and Lori Patton Davis and Samuel D. Museus date the concept back to 100+ years ago.

Pieta’s insight, though, was that deficit thinking could be triggered from a “neutral” chart like the ones above, and that our design choices might make this better or worse. She pointed out that emphasizing direct comparisons between minoritized and dominant groups encourages audiences to see the groups with the worst outcomes (often marginalized groups) as deficient, relative to the groups with the best outcomes (often majority groups).

Deficit thinking is harmful because it encourages victim blaming. It implies that outcome differences are caused by group members’ personal characteristics (e.g., “It’s because of who they are”) as opposed to external causes (e.g., “It’s because of systemic racism”).

Victim blaming leads to two further harms:

A Distraction Effect: Since personal blame is a cognitively easier explanation (src), and people tend to stop seeking explanations when they find one that plausibly fits their pre-existing beliefs (src), victim blaming obscures external causes, leaving widespread, systemic problems unconsidered and unaddressed (src).
Self-Fulfilling Stereotypes: Victim blaming also reinforces harmful stereotypes, setting lower expectations for people that become self-fulfilling prophecies, further entrenching the disparities in question (src).

Pieta and I go into more depth on the impacts of deficit framing here: What can go wrong? Exploring racial equity dataviz and deficit thinking, with Pieta Blakely.

Why does this matter? What are the implications?

Deficit thinking has immediate implications for visualizing people and outcome disparities, but the existence of an effect like this has even broader implications. It demonstrates that that information can backfire and be harmful, even when it’s accurate, well-intended, and cleanly presented. This dramatically raises the stakes for making responsible design choices.

Typically we only think of “good dataviz” along a few dimensions:

Comprehension: Do most people read it correctly?
Approachability: Is the time-required worth the value of information?
Affect / Aesthetics: Does it create the right emotion? Is it nice to look at?

Framing effects like deficit thinking imply that there’s at least one more dimension to “good” dataviz, which is something like this:

Second-Order Beliefs: Do viewers read the chart correctly, but still consistently arrive at incorrect or harmful beliefs?

Previously-fine advice to revisit

Consider “Cotgreave’s law:” “The longer an innovative visualization exists, the probability someone says it should have been a line / bar chart approaches one.”

This captures an instinct a lot of us share, that often, “simpler,” workhorse charts like bars and lines get the job done and there’s probably not a strong reason for anything fancier. Or to put it another way: “99 times out of 100, use a bar chart.”

In a world where the worst-case effect of a suboptimal chart is a loss of clarity or slower comprehension, this is harmless advice. Bars aren’t usually the optimal chart choice, but they’re probably second- or third-best and the stakes generally aren’t that high.

However, the deficit-framing effect shows that, in contexts like inequality, the stakes are high and the “worst case” of an incorrect chart choice is considerably worse than lost clarity. So previously reasonable assumptions like these need to be revisited.

Other implications

The deficit framing effect has a few other implications for dataviz.

These potentially harmful charts are everywhere (e.g., see NYTimes, the Bureau of Labor Statistics, Pew Research, Wikipedia). This means we’ve got some charts to redesign.
The most popular ways to show this type of data are probably the worst. So we’ve got some deeply ingrained habits to break.
We can no longer assume that charts are passive. By choosing to visualize something like inequality, our charts take an active role in shaping it.

Not only do we have some design instincts to reconsider, we have a lot of historic charts to revisit.

Given the uphill battle, I hope to equip you with the conviction and the means to adopt more equitable design practices within your own work and in the wider community. To do that, let’s dive deeper into how this effect actually works.

How could an innocent chart possibly cause stereotyping?

First, we’ll look at stereotypes in general, then we’ll consider how two different types of charts could create similar misperceptions.

About Stereotypes

For most social outcomes, individuals’ outcomes will typically be widely distributed within any group of people, like the chart on the left. Stereotypes, however, trick your brain into imagining a distribution like the right, where Purple People’s outcomes are more similar than reality, and their differences with everyone else are more exaggerated than reality.

To understand how a chart might nudge someone toward stereotyping, let’s look at stereotypes in general. For example, let’s consider a stereotype that people in this Purple Group A are especially high earners.

In reality, the distributions for outcomes like income will look like the chart on the left. Even if average earnings for people in the Purple group are higher than average earnings for everyone else, you’ll still see that earnings are widely distributed within both groups, and between the groups there’s a lot of overlap.

The “Purple people are high earners” stereotype is a distortion of reality. Stereotypes assume that, within a group, people are more similar than they really are, and between groups, people are more different than they really are. The stereotype implies a distribution like the chart on the right, where it seems that, not only do people in the Purple group earn more, their earnings are very similar, and therefore all Purple people earn more than all other people.

Could certain charts create similar misperceptions?

How could charts perpetuate stereotypes?

Bar charts, like stereotypes, trick our brains into imagining that within-group outcomes are more similar than reality, and that between-group outcomes are more different than reality.

Bar charts like the above represent groups as monoliths. Instead of showing the diversity of outcomes within a group, they only show group averages. Because people naturally discount variability (src), especially when viewing bar charts or confidence intervals (src, src), this chart creates a false impression of within-group similarity and exaggerates the differences between groups. Wilmer and Kerns refer to this as the “dichotomization fallacy.”

Discounting variability creates a stereotype about income, then correspondence biases take it a step further and turn it into a stereotype about personal differences between people.

Discounting variability is a cognitive error that supports a further error about people. If you (incorrectly) believe that every person from Group A earns more than every person from Group B, it’s much easier to conclude that earnings are caused by something intrinsic to the people in each group. And, because the most apparent “cause” in this graph are the groups, there’s a very slippery cognitive slope toward blaming the outcome differences on the people being visualized–rather than more complex, cognitively taxing, external explanations like systemic racism.

Ignoring or deemphasizing variability in charts can create illusions of similarity. If stereotypes stem from these illusions of similarity, then the way visualizations represent variability (or choose to ignore it) can exaggerate these perceptions and mislead viewers toward stereotyping.

Let’s walk through an example…

How a viewer might interpret income disparities shown on a bar chart.

When viewing a bar chart, a viewer’s thought process might go something like this:

“Purple people, on average, earn the most, followed by Teal, then Blue, then Green.” This is a basic, accurate read.
“Every Purple person earns more than every Teal person. Every Teal person earns more than every Blue person. Every Blue person earns more than every Green person.” This is the dichotomization fallacy or outgroup homogeneity, overlooking within-group variability.
“The only apparent difference between earning levels seems to be a person’s group color, so a person’s group color must be the cause.” This is illusory causality and personal attribution bias.
“Since Purpleness leads to higher earnings, Purple people must be the smartest or hardest workers. Green people just need to work harder.” This is correspondence bias or a representativeness heuristic.
“If it’s true for the people on the chart, it’s probably true for every Purple, Teal, Blue, and Green person in the universe. So all Purple people must be the smartest, hardest workers and Green people the least.” This is a group attribution error and harmful stereotype.

Showing Variability

On the other hand, charts like jitter plots make outcome variability unignorable. They show between-group differences and within-group differences.

How a viewer might interpret income disparities shown on a jitter plot.

Seeing the wide variation of individual outcomes within a group disrupts our tendencies to monolith, making it clear that inter-group differences only play a small role in individual outcomes. A viewer’s impressions might go like this:

“Purple people, on average, earn the most, followed by Teal, then Blue, then Green.” This is a basic, accurate read.
“A lot of Purple people earn more than the other groups, but not every Purple person earns more. Teal people also have high earnings, but they’re also all over the place. Blue and Green earnings are generally lower than others, but they also have some high earners.” Recognizing variability.
“Group color isn’t a great predictor of income. There must be some other factors involved.” Thinking about the bigger picture.
“This seems complicated. There are differences, but I need more information to understand why.”

This last line might seem like a bad thing, but in the context of visualizing people and social issues, “it’s complicated” is often an accurate read. The complexity is the point and is easy to overlook. And when the stakes are this high, it’s certainly better to leave viewers with unanswered questions than give them a false sense of confidence in the wrong conclusions.

The general hunch:

Two possible ways to show inter-group social outcome disparities. The bar chart on the right gives no indication of outcome variability. The jitter plot on the left shows each group’s mean outcome in the context of each group members’ individual outcomes, highlighting outcome variance.

We’ve walked through two examples of how viewers might perceive two very different charts: jitter plots versus bar charts. In our research, we included other chart types that are more apples-to-apples comparisons. But these two chart types are great for illustrating our main hunches:

Charts that hide variability leave room for a cascade of biases and misperception that ultimately lead to harmful stereotypes.
Charts that emphasize variability make it clear that simplistic explanations like blaming and stereotyping can’t possibly be the cause of outcome disparities..

We’ve made a few other testable assertions in the stories we outlined above. In the next section we’ll see how these played out in the research.

The “Dispersion vs Disparity” research project

If the theories above hold up, then when visualizing social outcome disparities, we’d expect charts like bar charts (that hide within-group variability) to encourage stereotyping and charts like jitter plots (that emphasize within-group variability) to discourage it. To restate the hypotheses a bit more precisely:

Deficit framing happens: Given any chart showing social outcome differences (and no other causal explanations for why the differences occur), some viewers will misread the charts as evidence for a stereotype about the groups being visualized (e.g., “Group A earns more than Group B because Group A works harder than Group B”).
Our design choices matter: Charts that downplay outcome variability (e.g., bar charts, dot plots, confidence intervals) will lead to more stereotyping than charts emphasizing outcome variability (e.g., jitter plots, prediction intervals).
Error bars don’t help: Visualizing uncertainty won’t solve the problem, viewers need to see variability. Even charts that imply variability (e.g., confidence intervals) will lead to more stereotyping than charts that explicitly show variability (e.g., prediction intervals).

Experiment design

We tested six different chart types, showing four different topics of social outcome disparities. The chart types varied their emphasis on outcome variability.

To test our hypotheses we ran four different combinations of chart types with more than a thousand people on Mechanical Turk. Participants each saw one of the 19 charts above. The charts showed (fictional) outcome differences between different groups, for one of four different topics (restaurant worker pay, life expectancy, test scores, household income).

To test the effect of showing variability in visualization designs, participants saw two different categories of charts:

Low / No variability charts: These were bar charts, dot plots, and confidence intervals. They only show the average outcome for each group and hide the variability of the underlying data. This includes confidence intervals, which show uncertainty for the average, but not variability in outcomes.
High variability charts: These were jitter plots and prediction intervals. Like the low/no variability charts, these show the average outcome for each group, but they also show the wider range of outcomes possible for individuals within each group.

Two example survey questions (out of 33 possible). The control is a slider, representing a 1-100 point scale (0=disagree, 100=agree).

We asked participants whether or not they agreed with various explanations for the outcome differences in the graph. The questions were evenly split between two question types:

Personal attribution (blame) agreement: How strongly the participant attributes outcome differences to the personal characteristics of the people within each group (e.g., “Based on the graph, Group A likely works harder than Group D.”)
External attribution agreement: How strongly the participant attributes outcome differences to external factors that affect the people within each group (e.g., “Based on the graph, Group A likely works in a more expensive restaurant than Group D.”)

The “personal attribution” questions were the important measure. Given that the charts provided no evidence for the causes of the outcome differences, agreement with personal attributions implies a personal stereotype about the people within the group. The “external attribution” questions were included as a baseline for comparison.

You can read more about the experiment setup here.

Experiment Results

Hunch #1: Deficit framing is real. Many people misread “neutral” charts as evidence for personal stereotypes.

Left: An example of a chart used in the experiment, with explicitly racial groups (Asian, Black, Hispanic, White). Right: Distribution of participants’ average responses to personal attribution questions. 34 percent of participants agreed that personal attributions explained the differences in charts like the left, indicating beliefs in harmful stereotypes.

These are partial results from our first experiment, where we tested different charts labeled with explicitly racial groups, like Asian, Black, Hispanic, White. We found that when viewing a chart about explicitly-racial outcome differences, 34 percent of participants agreed with personal attributions like “These outcome differences are because [Group] works harder than [Other Group],” despite the charts giving no evidence to support claims like that.

Self-reported attitudes about race / racism are notoriously hard to capture because of things like social-desirability biases, so we suspected that “real” results might be more extreme.

Left: An example of a chart used in the experiment, with abstract letter groups (Group A, B, C, D). Right: Distribution of participants’ average responses to personal attribution questions. 34 percent of participants agreed that personal attributions explained the differences in charts like the left, indicating beliefs in harmful stereotypes.

To control for social-desirability biases, we also tested charts where the groups were more abstract and not explicitly defined, like “Group A,” “Group B,” “Group C,” and “Group D.”

The abstract letters actually increased the effect. In these conditions, the majority of participants (53 percent) were willing to agree with personal attributions about the people being visualized.

This shows that the effect isn’t just limited to race. And optimistically, it implies that even if people are willing to be judgey about others in ambiguous groups… fewer of them were willing to be consciously judgey about race. But, again, we suspect the condition with abstract letter groups is closer to reality.

In any case, both of these tests show that deficit framing (or correspondence bias) can affect substantial portions of audiences. That is, given a “neutral” chart, viewers will often mistake evidence for outcome differences as evidence for personal differences between the people being visualized.

Hunch #2: Design choices matter. Charts that hide variability lead to more stereotyping.

Results of our first three experiments, where we compared low and high variability chart types. Each row represents an experiment. Each dot along the axis represents a participant’s average agreement with a personal attribution about one of the groups listed in the chart. Agreement with personal attributions implies belief in a harmful stereotype. The orange charts on the right are low variability (bars, dot plots, and confidence intervals) and consistently led to stronger beliefs in harmful stereotypes about the people being visualized.

In the three experiments where we tested low versus high variability charts, low variability charts consistently led to more stereotyping.

In Experiment #1, we tested jitter plots versus bar charts. Bar charts significantly increased stereotype agreement by seven points.
In Experiment #2, we tested jitter plots versus prediction intervals versus dot plots. Dot plots significantly increased stereotype agreement relative to prediction intervals, by seven points. Dot plots showed five points more agreement than jitter plots, but the difference wasn’t significant for that pair.
In Experiment #3, we tested prediction intervals versus confidence intervals. Confidence intervals significantly increased stereotype agreement by five points.

Across the three experiments, the high variability chart types (jitter plots, prediction intervals) significantly reduced stereotyping compared with the low variability chart types (bar charts, dot plots, confidence intervals), by five-to-seven points on the 100-point scale.

The differences are modest, but meaningful. Obviously no chart will erase all harmful beliefs, but this study does show that data visualization design can interact with our social cognitive biases in some unexpected and potentially harmful ways. It also shows there’s room for improvement over the status quo.

To put the differences in perspective, we can compare them to other well-known factors that affect attribution biases: political beliefs. Attribution biases, like deficit thinking or correspondence bias, usually have a stronger effect given certain cultural beliefs. For example, people from western, individualistic cultures, conservative political ideologies, or believers in the “just-world hypothesis” often show stronger attribution biases. We found consistent results in our study: Self-reported republicans showed significantly more agreement with personal attributions than democrats (~six points). So the chart types in our experiment had a similar effect size as politics, a well-known factor influencing attribution biases. That is, using high-variability charts has a similar effect size as waving a wand and turning a typical republican into a democrat.

Hunch #3: Confidence intervals aren’t enough. Show variability.

Results of our third experiment, where we compared prediction intervals with confidence intervals. This comparison is notable because, given the same dataset, prediction intervals will show long bars across the chart, while confidence intervals will typically show shorter bars. This allowed us to test the effect of overlapping ranges while holding the underlying data constant. Here we found that prediction intervals, which emphasize outcome variability, led to less stereotyping than confidence intervals, which emphasize uncertainty around the average.

Experiment #3 is worth a closer look. Since confidence intervals are a best practice for showing uncertainty, we wanted to see if just showing any kind of uncertainty was enough to influence stereotyping, or if seeing variability (and the full overlapping ranges of outcomes) would have different effects. A super interesting study from Jake Hofman & colleagues suggests that confidence intervals encourage a similar “dichotomization fallacy” as bar charts, where people assume that individual outcomes are tightly clustered within the very short error bars.

So we tested confidence intervals (based on standard error around the group mean, which would show shorter bars with little overlap), versus prediction intervals (based on standard deviation of the outcomes, which would show as long overlapping bars for the same data). We found that prediction intervals reduced personal attributions by five points, relative to confidence intervals. This suggests that just seeing uncertainty isn’t enough to impact stereotyping, instead it’s about seeing variability and the overlapping outcome ranges.

Experiment conclusions

The results confirm that deficit thinking (or correspondence bias) effect can be triggered from ‘neutral’ charts and graphs. We show that, regardless of how the data is presented, many people will misinterpret data on social outcome disparities as evidence for blaming those differences on the characteristics of the people being visualized.

When visualizing social outcomes, avoid overly simplistic charts that hide variability (like bar charts, dot plots, and confidence intervals). Instead, consider jitter plots and prediction intervals (or similar charts), which provide more context and reduce harmful stereotypes about the people being visualized.

Fortunately, there’s significant room to improve over conventional chart types like bars or confidence intervals that present people (and their outcomes) as monolithic averages. These findings validate the argument that deficit-framed dataviz can unnecessarily perpetuate harmful beliefs like victim blaming and stereotyping. They also show that data designers have some degree of control over this phenomenon (and therefore some degree of responsibility).

Finally, we reveal another way that accurate information can backfire. This finding is particularly relevant for equity-focused advocacy groups. Simply visualizing outcome disparities is not enough to solve them, and if not done carefully, raising awareness of inequality can actually make it worse.

What this study does NOT show:

Bar charts aren’t inherently racist. Jitter plots won’t erase pre-existing harmful beliefs (at least, not entirely). Jitter plots aren’t even the best way to visualize outcome disparities (prediction intervals did slightly better in our experiments). But they seem to be a step in the right direction.

Data design implications

What do these results mean for dataviz design practice? While the experiment was inspired by use cases related to DEI and racial equity, these findings also apply to any visualizations depicting inter-group outcome disparities (e.g. gender, wealth, age, etc). The human capacity to be judgy towards other people is limitless, so unfortunately these results are widely applicable.

Resist the trap of false simplicity. In contexts like social inequality, designers must tell a complete story or risk misleading viewers towards harmful beliefs about the people being visualized. This will sometimes mean more complex charts and graphs, and that’s totally okay. It’s always better to tell a complex story that’s true than a simple story that’s false.

Equity considerations for data designers

Accurate, well-intended dataviz can still backfire. Our duty of care towards audiences (and the people we visualize) means expanding our definition of “good dataviz.” Clarity, approachability, and aesthetics are important, but insufficient. To avoid backfire effects, we also need to consider and minimize the ways that even accurate reads of data can still contribute to inaccurate, harmful beliefs about the people we visualize.
Means mislead, show variability. Monolithing people behind summary statistics makes it easier to stereotype them. In the same way that exposure to people from other communities helps us appreciate the rich diversity within those communities, exposure to the data behind the summary statistics helps us appreciate outcome diversity within the groups being visualized. This prevents viewers from jumping to easy, but harmful, personal attributions and therefore minimizes stereotyping.
Resist the trap of false simplicity. For problems as big and messy as inequality and structural racism, if “it’s complicated” isn’t one of viewers’ main takeaways, then the chart is doing something wrong. A chart that only shows “Group X’s outcomes are 123 percent better than Group Y” might make a compelling sound bite, but it tells such an incomplete story that it’s arguably dishonest. And, as we’ve shown, it’s likely harmful.

Resources and next steps

Sign up for a workshop. Learn how to visualize inequality, without making it worse. If you’re part of a team of data designers, journalists, analysts, or advocates, this can help your team quickly catch up on this important topic. The workshops cover not only our recent research, but also the underlying psychology and alternative design approaches to conventional (harmful) visualizations of social outcome disparities.
Learn How To create Jitter Plots using Google Gemini, a tutorial on visualizing income disparities in North Carolina.

Huge thank yous to Alli Torban for her lively illustrations, Pieta Blakely for her brilliant advice & inspiration, Cindy Xiong for co-conspiring, and to Jon Schwabish, Alice Feng, Steve Franconeri, Steve Haroz, Robert Kosara, Melissa Kovacs, David Napoli, Kevin Ford, Ken Choi, and Mary Aviles for advice and feedback along the way.

Experiment Results: The Risks of Visualizing Racial Inequality

Tue, 01 Jun 3002 00:00:00 GMT

moved: what-can-go-wrong-racial-equity-data-visualization-deficit-thinking-VV8acXLQQnWvvg4NLP9LTA

Sketching Sketchy Bar Charts.

Mon, 22 Mar 3002 00:00:00 GMT

A fun thing

A fun thing that happens with dataviz: You spend hours and hours researching a story, pouring through the data, and lovingly crafting a perfect chart. Then you present it to your client or boss or followers, and they point out a crucial, fatal flaw:

“OMG, why isn’t this a bar chart?!”

You take a deep breath. You try not to reveal that a little piece of your soul has died, leaving a smudge of black ash, just to the left of your heart.

But you’re a professional! You also know there’s merit to their feedback. Bar charts are known for being low-fuss and straightforward, and that’s often the better choice than something needlessly fancy.

But are bar charts really as straightforward as they seem? Let’s test it out.

This is a bar chart, where each bar represents responses to a different bar chart. Overall it shows that people severely misinterpret bar charts (just like this one). So there’s a good chance you’ll misinterpret this bar chart, just like Wilmer and Kerns’ participants misinterpreted their bar charts. You can find out by taking the test below. Then send us your results, so we can make yet another bar chart about people misinterpreting bar charts of misinterpreted bar charts. And we can recursively create misunderstood bar charts from now until infinity.

Above is a bar chart about misinterpreting bar charts (Ooo, meta!). It shows how accurately people interpreted four different bar charts, covering four different topics from Wilmer & Kerns’ experiment: gender, social science, clinical, and aging. Each bar represents 133 participants’ average OLI interpretation scores, which is an accuracy measure we’ll unpack later.

How would you interpret this chart? Even without understanding “OLI,” you can still read some of the basic facts:

The “Aging” bar chart did the best (27), with a nine point gap between to the next best “Clinical” chart (18).
The “Gender” bar chart did the worst, with the lowest average score (8).
All four charts had average scores below 50, indicating inaccurate interpretations.

But there’s more to dataviz than parroting these basic facts. Visualizations also create a mental impression about the shape of the underlying data, which influences our interpretations in important ways.

When you imagine the data behind this chart, what does your mental image look like? If you were to draw out the data points behind these averages, where would they go?

If you’d like to check your interpretation, pause here, grab some paper and a pen, then:

First, draw a quick outline of the chart above.
Then imagine the individual test scores for the people who took Wilmer & Kerns’ test.
Then for each of the four bars, draw 20 dots to represent the scores for 20 random participants.
Then read on to find out how to interpret your interpretation.

Collage of 134 bar chart interpretation drawings, submitted for Jeremy Wilmer and Sarah Kerns’ study: “What’s really wrong with bar graphs of mean values: variable and inaccurate communication of evidence on three key dimensions.”

Jeremy Wilmer and Sarah Kerns, researchers at Wellesley College, ran a pair of drawing experiments where they asked participants to redraw a set of four different bar charts, then annotate the bars with dots representing where they imagined the underlying data might be in the original dataset. Notably, they did not make participants suffer through “bar charts about bar charts,” but we won’t hold that against them.

In their 2022 study, 134 participants responded with a stack of 536 hand-drawn bar charts. At 20 dots per bar, they ended up with 40,000+ dots that they could scan, analyze, and compare, to understand how people imagine the underlying distributions behind the charts.

Their headline result: 76% of those 536 sketches showed at least one major misinterpretation of the data behind the stimuli charts. That is, most people misinterpreted most of the bar charts.

Four plots showing four different ways people might interpret a bar chart of averages. The dots represent viewers’ imagined distributions for the data points underlying the bar charts. The green plot represents an “accurate” interpretation, the three orange plots represent common biased interpretations found in Wilmer 2022.

For the topics in the study (as well as our meta bar chart), a realistic interpretation would look like the green plot above, showing distributions that are centered(-ish) around the average, widely dispersed, overlapping between categories, and probably with some normal(-ish) shape to them.

But only 24% of the 536 sketches looked like the green plot. Instead, when they imagined the underlying data, most participants showed one of three common misinterpretations:

Underestimating variability: 217 (40%) drawings underestimated the amount of variability in the distribution. So even though these distributions should overlap across categories, people assumed there was little-or-no overlap. Willmer and Kerns refer to this as the “dichotomization fallacy” because, in effect, it leads to viewers falsely assuming there are clear divisions between categories. The “OLI” score in the meta bar chart refers to the “overlap index” in the study.
“Within the bar” bias: 129 (24%) drawings showed the data points falling inside the area of the bar, rather than balanced around the bar’s end point which represents the average.
False uniformity: 125 (23%) drawings showed the data as evenly distributed and ignored the shape of the distribution.

Not only are these misinterpretations common, they can have serious downstream consequences. For example, underestimating variability can lead to overestimating the differences between chart categories, with downstream impacts like:

Mistaking coincidence for causality.
Nudging business leaders into overpaying for ineffective equipment.
Misleading patients into accepting medical treatments they might otherwise decline.
Reinforcing harmful stereotypes about people from marginalized communities.

What does this mean for dataviz?

Many of us have heard the critique “that could have been a bar chart.” But despite their perceived simplicity, bar charts of averages can be surprisingly misleading.

Charts can show accurate data and still be misleading. Misleading visualizations aren’t just about miscomprehension and biased decision making, they can also reinforce downstream beliefs with wider social implications.
Avoid bars for showing averages, where they systematically mislead viewers about the underlying data and the reality behind the chart. (For that matter, be wary of single-point dot plots and confidence intervals, which are also susceptible to exaggerated differences or underestimated variability.)
Ideally, anytime we’re showing an average, we’re also showing the data behind the average. We’re in a golden age of fun, compact, comprehensible ways to show distributions, like swarms, quantile dots, areas, rain clouds, violins, jitters, ranges, prediction intervals, ridges, histograms, gradient intervals, or even simple line charts. Most of these plots can be overlaid with averages and stacked to replace bars, and, importantly, each of these do a better job of showing the breadth and variability of the underlying data.

This post by Eli Holder originally appeared on effaff.com here.

Timely Advice: How Long Does Dataviz Take?

Sun, 24 May 3001 00:00:00 GMT

Every client asks, “How long do you think that will take?” I’ve built software for a long time. I used to resist even answering the question. I’m a Fred Brooks acolyte and appreciate all the unforeseen ways that a complex project can go sideways.

I don’t mind it anymore though. It’s an important part of setting expectations (which make for happy projects and happy clients). And, for anyone who’s worked for a fixed fee, it’s important for understanding if a given project will be profitable. So, not only do I attempt to estimate timing for every project, I also track the actual time to see if I’m right.

One of the challenges for estimating — and expectation setting — is having a track record of similar projects to reference. If you’re a larger shop, with a long history and full portfolio, you have an information advantage. Smaller, independent shops, or freelancers, earlier in their careers, don’t have this advantage so it can be difficult to estimate.

Or, even worse, it can be easy to give into the pressure from occasionally overzealous clients fixated on budget line-items (“You’ll spend how long on research?!”).

The goal with sharing this data is to even out that informational asymmetry, and give a detailed reference of time and effort involved in producing (fairly complex) data visualizations.

3iap’s time-tracking dataset

From the start of 3iap in 2020, the focus was data visualization for clients. Since then, 3iap has done a variety of projects, covering the full spectrum of work you might encounter as a dataviz consultant (e.g., research, analysis, data-wrangling, metrics, design, and various types of engineering). I’ve kept a close record of the time spent on each project.

As of early 2022, 3iap has logged ~1,550 hours of client dataviz work (in addition to sales / marketing / paperwork / etc., +300 hours of general product consulting to pay the bills, and an obscene number of untracked hours on silly side projects).

Below are findings about how that time was spent, in addition to highlighting 10 specific projects that represent a range of different dataviz work.

How much “dataviz” work goes into dataviz?

Distribution of 1,550 hours spent on 3iap client projects, split by activity type.

Roughly 60 percent of the total time was spent directly designing or engineering visualizations.

Design + Engineering

Distribution of dataviz design and engineering time spent on 3iap client projects, split by activity sub-type. Design and engineering activities together made up 60% of total hours.

31 percent of the time was spent on design, which can include everything from story discovery, typically bouncing between exploratory analysis and sketching story concepts with a pen and markers (4 percent), mocking up specific charts in Figma or Google Sheets (6 percent), prototyping different design approaches in Observable (3 percent design, 3 percent engineering), and even the occasional copywriting (2 percent).

On seeing this, I was surprised that slides were the second highest design activity (6 percent) — I suspect this is due to inefficiency of the tool itself, whereas Figma can be componentized and coded dataviz can be automated, Keynote involves a lot of manual pixel pushing.

29 percent of the time was spent developing visualizations, typically in javascript (13 percent React, 6 percent Angular), but also occasionally in Data Studio (4 percent). This time also coincides with data-wrangling activities, building pipelines to prepare datasets for visualization.

Note, the design-to-engineering ratio might not be representative for others in the field or of a specific project. My background is computer science, so there’s a selection bias toward more technical work. Prototyping designs in code is also part of my design process, which blurs the lines further. Also, most 3iap projects are either engineering OR design, not both. For a more representative ratio, Interactive Scientific Storytelling and Complex Report: Analysis & Presentation Design were projects that involved both design and development.

How much “non-dataviz” work goes into dataviz?

Distribution of data, communication and research time spent on 3iap client projects, split by activity sub-type. Combined these activities made up 39% of total hours.”

This leaves 40 percent of total time on other activities. This remaining time is split between research, client communication, and data wrangling. (All of which are insanely important, but this might be unintuitive from clients’ perspectives.)

18 percent of the total time was spent communicating with clients, users and stakeholders, digging for stories and trying to make sure everyone is on the same page. This includes meetings (8 percent), and documenting designs, plans and code (3 percent), and the rest is email and Slack. While this might seem excessive, communication is a crucial part of the process; a few hours of up-front meetings, mind-reading and documentation can save days of rework. For that reason, a significant portion of communication time coincides with other activities. For example: 7 percent of total hours were tagged with both communication and design, which might include co-design exercises with clients or design reviews.

As expected, at 16 percent, data wrangling and analysis takes a significant chunk of total time. This includes data prep, which I’ve categorized as fairly mindless data engineering or spreadsheet maneuvering (9 percent) or data pulls (3 percent). More interesting data work was more fragmented: ~2 percent of the time was exploratory analysis (e.g., for storytelling), ~1 percent of the time was spent designing metrics (e.g., exploring different calculations that might best tell a given story) and another 1 percent was creating mock datasets (e.g., to compensate for data security constraints or clients who are slow to provide real data).

Research / discovery was 6 percent of the total time. The bulk of this was spent talking with clients, and coincides with meetings, email, and Slack. It also includes things like industry research, reviewing related academic literature, and whatever materials the client has available.

4 percent of total hours were tagged with both communication and research, which might include client mind-reading exercises, user interviews or other types of qualitative user research. This is probably the highest-impact time spent in any project: It might seem unintuitive, but at least in my experience the fastest path to a compelling data story isn’t necessarily in the data itself, it’s talking with the people behind the data.

Chaos / Overkill

Distribution of total time spent on 3iap client projects, split by cause of unproductivity. Clients cause “chaos,” whereas “overkill” time is self-inflicted.

For certain projects, I also track a category that I call chaos, which is time lost due to client shenanigans. This includes things like adding new scope on a fixed-fee project or revisiting early decisions that lead to rework. This was 8 percent of total hours.

The inverse of this category is overkill, where I become overly excited about an idea, fall down a rabbit hole, and devote way more time to it than is reasonable or sane. This was 4 percent of total hours.

10x 3iap dataviz projects - timing and details

The previous findings covered overall statistics for all 3iap projects. However, it can be helpful to see how time is used on individual projects. For each of the projects below I’ve tried to share enough about the scope of the project to understand the requirements, as well as overall statistics on how the time was spent. There’s also a timeline showing how the types of work evolve throughout the course of a project.

1. Analytics Product Design System (14 Days)

A long-term client asked 3iap to redesign their SaaS analytics app.

Left: Distribution of time spent on a single project, split by activity type. Right: Timeline of activity throughout the course of the project. Each ‘row’ corresponds to eight-ish hours (a “work day”). Each segment represents a single activity. Tile color corresponds to the activity type. Segments with multiple colors had overlapping colors (e.g. user interviews are both ‘research’ and ‘meetings’). Black lines between segments represent the boundaries of a calendar day.

A long-term client asked 3iap to redesign their large, complex SaaS analytics product (covering 200+ distinct metrics). There were three parts of the project: 1) a design system of chart components and supporting elements that can be mixed and matched to answer a wide range of analytics questions, 2) detailed designs of four different narrative reports, showing how the design system can address deep dives into various analysis topics, and 3) API designs and technical specifications for similar flexibility / composability when accessing the data from the backend. Because this was a familiar client and topic, there was little research or data wrangling required.

2. In-Product Chart Component (11 Days)

3iap developed an interactive chart component and automated testing framework within the client’s existing codebase.

The goal was to not only deliver the component, but also develop a template for how other charts could be reliably developed and tested within their environment. The component itself was fairly simple, the challenge was making it work (reliably) within their system. This project was fairly chaotic. In addition to last-minute scope creep, startup codebases aren’t known for their quality or engineering practices. No judgment here, but this added significant drag!

3. Complex Report: Analysis & Presentation Design (24 days)

Analyzing a complex and novel topic, new to 3iap and the client, designing appropriate visualizations, then telling a cohesive story within a 51-slide deck.

This involved researching a new topic for the client, designing and developing novel visualizations, working closely with their data engineers to develop a novel set of metrics, several iterations of analysis, developing a framework to generate charts demonstrating the analysis, then designing a deck to tell the whole story. While this is the most research / analysis heavy project in the batch, these activities still only made up 38% of the total time (26% data + 12% research).

4. Embedded Reporting Tool Design (12 Days)

SaaS startup client with a unique dataset asked 3iap to design, prototype and user-test their in-product analytics UX.

The project involved researching their industry, offering and dataset, designing metrics to reflect the activities they wanted to track, designing and prototyping 4 “live reports” across Figma and Data Studio, facilitating user tests, and adapting accordingly. The client was the ideal balance of engaged, open-minded, and data savvy. They also had data available on day one. The project went smoothly and finished ahead of schedule.

5. Scrollytelling Infographic (13 Days)

Developed an interactive scrolly-telling visualization in Angular.

This was a fairly straightforward development project, implementing the clients’ designs for an 11-section scrollytelling infographic based on a small, static dataset. The main challenges were an unfamiliar environment (3iap’s first angular project) and fine-tuning the animated transitions (to be expected from this format). There was a small amount of chaos in the form of surprise scope and changing design decisions.

6. Exploration Dashboard Prototype (7 Days)

Self-serve exploration dashboard in Data Studio, unpacking multiple themes in a familiar domain.

The goal was to translate a fairly detailed static report template (usually ~53 slides) into a Data Studio dashboard that supports basic, self-serve exploration for end users. The result was seven interactive mini-reports, each covering a different theme, with two-to-three subsections and about 20 charts each.

7. Complex Survey Exploration Tool Design (18 Days)

Designed tool for visualizing combined results of a large-scale research project, including exploratory dashboards and narrative deep-dive reports.

This project started in a deep, dark jungle of chaos. The client sent five different datasets, fresh from SPSS, combined into a single 1.3GB Excel file, a 32-page draft report of their research, and very little other guidance. This involved a significant amount of data cleaning just to get started. Then identifying key use cases for the tool and exploratory analysis to find six major story themes from the data. Each theme became a comprehensive, interactive live report. This also involved major exploratory functionality, including a geographic component (prototyped across Observable and MapBox) and universal filtering. Another source of chaos was ongoing scope additions. All of the above pushed the timeline several days past the estimate. Having said all that, while this project should have been total chaos, it actually smoothed out over time. Although the clients were hands-off, we were able to divine their priorities and keep them engaged and aligned along the journey.

8. Interactive Scientific Storytelling (24 Days)

Designed and developed a tool for visualizing long-term clinical trial results.

3iap handled this project from first concepts through final code. This started with understanding the client’s domain and research, walking them through pen and paper concepts and coded prototypes, then finally developing the visualization as a standalone React app. Implementing the design direction in a way that was performant on older iPads (a specific requirement) meant some heavy technical lifting. Ensuring the results were “true to the science” had implications for design and data wrangling (e.g., an auditable metric calculation pipeline). The main sources of chaos (and missed estimates) were extremely late access to data (designs were based on mock data estimated from their provided materials) and last-minute added scope.

9. Interactive Text Content Explorer (8 Days)

Developed an interactive text content explorer using client’s designs and static data.

This was a quick React app based on the clients’ designs. It involved some unusual interactions, but it had a static dataset and few moving parts, making it relatively straightforward. The main sources of chaos were late data and last-minute added scope.

10. Complex Dataset Exploration Tool Design (26 Days)

Visualization and product design, providing a common interface for discovering and visualizing hundreds of similar (but, not identical) high-dimensional datasets.

This project was truly exciting and truly chaotic! It required significant visualization, information, and product design. The domain was intrinsically complex, involving high-dimensional data, many moving parts, and an endless list of edge cases to consider. In addition to the inherent complexity, the designs needed to be reviewed and approved by committee — specifically a group of smart, strong-willed and occasionally mercurial stakeholders. So, in addition to imposing a high communication tax, requirements evolved and expanded significantly throughout the course of the project. Despite all this, the project was a blast. The dataset was unique, interesting, and had a clear path to social good. And the clients, despite committing every possible design-client cardinal sin, were at least thoroughly engaged in the work and approached their mission in earnest.

Other fun facts

Estimation accuracy

This chart shows the “work day” lines from six of the projects above. It highlights three things: 1) the amount of work required to complete the originally agreed upon scope of work (black lines), 2) the amount of additional work to complete scope additions and changes (red lines) and 3) the project estimates for the original scopes of work. For four of six projects, the time to complete the original SOWs was within the estimated range.

I did structured estimates for seven projects, based on clients’ initial requirements.

5/6 projects had at least minor scope creep from the original requirements (the red segments above). For the four projects that were “fixed fee,” all of them added scope beyond the originally agreed “fixed scope.” This isn’t to say they were acting in bad faith, it’s just hard for most people to anticipate everything they need up front. Either way, this demonstrates the need to approach “fixed fee” projects with caution.

The estimates for the “original scope” were generally pretty good. There was only one project where I substantially underestimated the work. This was a time and materials project, so this was in the client’s favor. Both of the projects that overshot the estimates were due to additional scope added later.

Data delays

This chart shows the number of calendar days spent waiting on clients to send data, for the six projects with a predefined dataset.

One major source of chaos was waiting for data. Out of the six projects requiring a predefined dataset, three were delayed by ~three or more weeks. One of the projects “finished” several months before the client sent a final dataset.

Invoice delays

This chart shows the number of calendar days spent waiting on clients to pay invoices. The column on the left (generally) corresponds to initial deposits, which were typically Net 7. The column on the right corresponds to final project payments, which were typically Net 28.

A source of emotional chaos was waiting on clients to pay invoices. Out of fourteen invoices, only three were paid on time (i.e., within one day of the due date on the invoice).

Calendar days v.s. work days

This chart compares the number of “calendar days” to “work days” for seven of the projects above, between the start and end of a project (i.e., the dates of the first and last hour worked). A “calendar day” measures time elapsed on the calendar (e.g., Monday to Monday is 7 calendar days). A “work day” is a unit of 8 hours worked and tracked on the project. All projects above were spread out over a longer period of time.

Projects require more calendar days than full, eight-hour work days. Typically a given project required less than ten hours of work each week. Most calendar time for a project is waiting on clients (e.g. for data, or feedback).

Methodology and Data

Timesheet data was collected throughout the course of each project, then tags were reviewed and updated afterward. Each timesheet entry has a duration, calculated by subtracting the start time from the end time. Each entry has one or more activity type tags associated with it. For entries with multiple tags, the time spent on each tag is allocated by dividing the entry duration by the number of tags (excluding “chaos” and “overkill” tags). This ensures that hours for each tag sum to the total hours in the project.

The dataset for the 10 projects is available here:
Timely Advice: Client Dataviz Timesheet Data

Takeaways

Only 60% of time spent designing and developing dataviz was spent “designing” or “developing.” Research, communication and data wrangling are a significant and important part of the job.
Estimates and time tracking can help set realistic expectations for time and effort required, with clients and yourself. Both of these practices can be enlightening in other ways: Estimates are invaluable for proactively uncovering misalignment and time tracking can help you optimize your time… and minimize wasting precious, finite minutes of your life pixel-pushing in Keynote.
Plan for chaos and scope creep. Chaos can be minimized, but not avoided. You can solve some of this in your discovery process, by trying to get expectations out of stakeholders’ heads and onto paper. However the best place to handle this is in your contract, either by setting clear boundaries on scope and provisions for overages, or by favoring time & materials when there’s significant uncertainty.
Working time !== calendar time. The projects above were delivered on time (or early!), but it was rare for any one project to require more than thirty hours of work in a given week. Most calendar time for a project is waiting on clients (usually for feedback or data). The benefit is you can either weave together multiple projects in parallel… or leave yourself slack time for silly projects like this!

Americans Struggle with Graphs

Mon, 26 May 3000 00:00:00 GMT

tldr:

Baseline numeracy in the United States is not great. There are surprisingly basic interpretations that many struggle with.
Graph comprehension depends on both viewers’ numeracy and charts’ “read” complexity.
To reach more people, don’t just plot, write.

A few months into the pandemic, Alessandro Romano and others surveyed 2,000 people to demonstrate that ”the public do not understand logarithmic graphs used to portray COVID-19.” They found that just 41% of participants correctly answered basic questions about log-scaled graphs, compared to 84% for linear.

But the problem is bigger than log scales. As we’ll see below, much of “the public” struggle with even the most basic charts and graphs, let alone complex visualizations.

The Curse of Knowledge

A note in the comments stands out:

“As a former infographics editor at a major newspaper, I always thought one of my strengths was a lack of math skills. If I could understand a chart, perhaps readers could, too. And yeah, I never used a log chart.” - Robert B.

This raises more questions: How many data journalists think like Robert B.? Or, as Romano & friends suggest, do people in the mass-media “routinely” assume log scale axes are widely comprehensible? If it’s the latter, and they’re overestimating the world’s quantitative abilities, how many other important data stories are lost on general audiences?

For data folks, this is an easy mistake to make. If a big chunk of your day is spent in a python notebook or your lunch conversations often veer toward what’s new on arXiv, you might be in the same boat.

The Heath brothers call this the “Curse of Knowledge.”

“Once we know something, we find it hard to imagine what it was like not to know it. Our knowledge has ‘cursed’ us. And it becomes difficult for us to share our knowledge with others, because we can’t readily re-create our listeners’ state of mind.” - Chip & Dan Heath, Made to Stick

That is, if you’re highly numerate, it’s often difficult communicating with people who are not.

To cure the “Curse of Quantitative Knowledge” and see the world of data through the eyes of a more typical, less savvy audience, we’ll look at 3 different studies exploring numeracy and graph literacy at scale.

Then to make it concrete, we’ll also look at 10 specific questions from those studies and I’ll provide estimates for a) how accurately a typical user might interpret them and b) what % of US Adults would be able to reliably interpret them correctly.

These benchmarks will hopefully give a more intuitive sense of the needs for wider audiences, and the importance of pairing dataviz with other modes of communication. For example, if you want to make your (hard-earned) insights approachable, good writing is just as important as good charts.

The Numeracy Problem

Numeracy isn’t innate.

Apparently babies and rodents are both able to differentiate simple quantities. Given a choice between two stacks of crackers, babies know to choose the bigger stack (src). Rats can learn to press a bar 8 or 16 times to receive snacks (src).

But that’s roughly the extent of our innate abilities with numbers. The rest, including basic concepts like ratios and negative numbers, are learned.

The PIAAC studies “numeracy” across 38 countries.

Every few years, the OECD runs a large study called “The Program for the International Assessment of Adult Competencies” (PIAAC). It examines basic skills of adults around the world, one of which is numeracy.

The researchers sit down with ~245k people across 38 countries, for about an hour each and quiz them. They calculate their scores on a scale of 1–500, where 500 is a perfect score. Those scores are then bucketed into one of five levels, where Level 1 is least proficient and Level 5 is most proficient.

Level 3 seems to be an important threshold for proficiency. A typical “level 3” person scores between 276–326 points (src, pg 71) and they can answer “level 3” questions 67% of the time (src, pg 64). We’ll explore examples questions later, but the PIAAC describes Level 3 questions as:

Tasks at this level require the respondent to understand mathematical information that may be less explicit, embedded in contexts that are not always familiar and represented in more complex ways. Tasks require several steps and may involve the choice of problem-solving strategies and relevant processes. Tasks tend to require the application of number sense and spatial sense; recognizing and working with mathematical relationships, patterns, and proportions expressed in verbal or numerical form; and interpretation and basic analysis of data and statistics in texts, tables and graphs. ( src , pg 71)

How numerate are US Adults according to the PIAAC?

How to read this chart: The first row shows the combined distribution of PIAAC numeracy scores for the top 3 scoring countries (Japan, Finland, Netherlands). The second row shows the same distribution for the United States. Each column shows how many people, per 100, achieved scores within the given column range. For example, in benchmark countries, 41/100 people had Level 3 scores, falling between 276-326. For the United States, just 29 people achieved this same level. Data is from the NCES PIAAC Data Explorer.

Numeracy rates in the United States are middling compared to other countries surveyed, and much lower than numeracy leaders like Japan, Finland, and the Netherlands (“Benchmark Countries” above, per src). For 2012–2014 results, a typical US Adult’s score was 257 (src), putting them solidly in the Level 2 range (226–276 points, src, pg 71). Just 39% of US adults tested as proficient (level 3 or higher), compared to 61% for the benchmark countries (src).

So if just four in ten US adults perform above Level 3, then six in ten struggle to “recognize and work with mathematical relationships, patterns, and proportions expressed in verbal or numerical form; and can interpret and perform basic analyses of data and statistics in texts, tables and graphs.”

“These results are another signal that many Americans struggle with the most basic of math skills,” says NCES Associate Commissioner Peggy Carr (src).

“But my audience is smart”

Assuming audiences are smart and well-intentioned is a good practice. It’s important to treat viewers respectfully. But even “smart” audiences aren’t always quantitatively savvy.

There is a strong relationship between numeracy and education, but there are exceptions. For example, even among those with more than a high school education, 47% still performed at Level 2 or below on the PIAAC (src).

In 2008, Hawley & friends found that even among participants with at least a bachelor’s degree, 33% were classified as low numeracy (src). In a 2001 study of the “highly educated,” Lipkus & friends found that 16–20% of participants incorrectly answered very basic questions related to risk magnitudes (e.g., “Which represents the larger risk: 1%, 5%, or 10%?”) (src).

Even doctors struggle. Rao’s 2008 review highlights (src) a survey of family physicians, showing that despite 95% of participants affirming the importance of understanding biostatistics, only 25% reported confidence in the subject. Based on the test results, their lack of confidence was well-founded: They averaged just 41% correct answers. Granted, biostatistics is a higher bar, but hopefully this illustrates that even advanced audiences aren’t always as advanced as they’d like to be.

Graph Comprehension

How does numeracy relate to communicating data?

Galesic and Garcia-Retamero’s work suggests that, not only does low-numeracy limit a person’s math capabilities, it also correlates strongly with their “graph literacy,” or their ability to interpret charts and graphs (src).

According to their study: “The same meta-cognitive abilities that lead to high numeracy scores also foster good graphical literacy skills.” And the reverse is true: Of the 261 “low numeracy” US Adult participants, only 89 (34%) exhibited high graph literacy.

How will users “read” the data?

Another insight from Galesic and Garcia-Retamero: Graph comprehension isn’t solely based on the reader’s abilities, it also depends on interpretation task. They suggest 3 ways people “read” a graph. A user can:

“Read the data,” identifying specific values on a graph
“Read between the data,” finding relationships in the graph’s data
“Read beyond the data,” making inferences from the graph’s data

Each of these levels is successively harder, and this is reflected in their results. US participants could correctly “read the data” in 86% of responses, “read between” in 67% of responses, and “read beyond” in 63% of responses.

Note, these results appear more positive than the PIAAC suggests. To get a better sense of what data-readers can actually handle, let’s look at some of the underlying questions from the 2 studies.

How much complexity can people handle?

Let’s look at some graph comprehension questions & results from the PIAAC, the National Adult Literacy Survey Questions and Galesic & Garcia-Retamero’s “Graph Literacy” study.

PIAAC Sample Questions

We’ll start at Level 3, the “medium difficulty:”

A "Level 3" Question. Only 37% of US Adults will regularly answer this correctly. (Image from NCES, src)

A Level 3 Question: For a time series line graph: “During which period(s) was there a decline in the number of births?”

The typical PIAAC numeracy score for a US adult was 257/500.

Therefore the average US adult has a ~26% chance of answering a Level 3 question correctly (src, pg 72).
A “Level 3” person, whose scored between 276–326 points, would answer this correctly 50–80% of the time.
Since 37% of US Adults scored at Level ≥3, we can say that just 4 in 10 US Adults can reliably answer a question like this.

A "Level 1" Question. 92% of US Adults will regularly answer this correctly. (Image from NCES, src)

Questions for a dial thermometer:

Level 1: “What is the temperature shown on the thermometer in degrees Fahrenheit (F)?”
Level 2: “If the temperature shown decreases by 30 degrees Celsius, what would the temperature be in degrees Celsius?”

These appear to be Level 1 and Level 2 questions.

A typical US adult will answer a Level 1 question correctly 89% of the time (92% of US Adults are Level ≥1 and will answer this correctly most of the time).
They’ll answer a Level 2 question correctly 66% of the time (70% of US Adults are Level ≥2 and will answer this correctly most of the time).

(Note: NCES site lists these as Level 3, but the reader companion lists similar questions as “low difficulty” or Levels 1/2)

A "Level 2" Question. 70% of US Adults will answer this correctly (via National Center for Education Research)

For a table and bar graph: “Which two bars are incorrect?”

This is a “level 2” Question. A typical US adult will answer this correctly 66% of the time (70% of US Adults will answer this correctly most of the time).

The National Adult Literacy Survey Questions

An earlier study in the United States, “the National Adult Literacy Survey,” suggests that the typical US adult would be able to “identify information from a bar graph depicting source of energy and year” only ~50% of the time. They’d be able to “use a table of information to determine patterns in oil exports across years” only ~25% of the time (src).

Prompt to a "Level 2" Question. (Image from NCES, src)

Level 2 Question: “You are a marketing manager for a small manufacturing firm. This graph shows your company’s sales over the last three years. Given the seasonal pattern shown on the graph, predict the sales for Spring 1985 (in thousands) by putting an ‘x’ on the graph.”

An average US adult answers this correctly ~60–80% of the time. (src, pg 102)

Prompt to a Level 3 Question. (Image from NCES, src)

Level 3 Question: “Suppose that you took the 12:45 p.m. bus from U.A.L.R. Student Union to 17th and Main on a Saturday. According to the schedule, how many minutes is the bus ride?”

An average US adult answers this correctly ~35–65% of the time (src, pg 102).

“Graph Literacy: A Cross-Cultural Comparison” Questions

In “Graph Literacy: A Cross-Cultural Comparison” Galesic and Garcia-Retamero tell us “even the simplest graphs may be difficult to understand for many people” (src).

3 different charts, prompting questions below from Galesic and Garcia-Retamero "Graph Literacy" study. (Reproduced from src).

A few example questions and expected results:

Reading off a point on a bar chart (left chart): “What percentage of patients recovered after chemotherapy?” - 85% US adults answered correctly.
Determining difference between 2 bars (left chart): “What is the difference between the percentage of patients who recovered after a surgery and the percentage of patients who recovered after radiation therapy?” - 70% US adults answered correctly.
Comparing slopes 2 intervals of a line (middle chart): “When was the increase in the percentage of people with Adeolitis higher? (1) From 1975 to 1980, (2) From 2000 To 2005, (3) Increase was the same in both intervals, (4) Don’t Know” - 62% US adults answered correctly.
Determining difference between 2 groups of icons (right chart): “How many more men than women are there among 100 patients with disease X?” - 59% US adults answered correctly.

Takeaways

When communicating data, the questions above offer useful benchmarks for determining your addressable audience size given the complexity of your data story:
If it’s roughly as complex as identifying and subtracting two values (e.g. “What is the difference between the percentage of patients who recovered after a surgery and the percentage of patients who recovered after radiation therapy?”), then you’re speaking to ~7 in 10 people.
If it’s roughly as complex as identifying trends on a line-graph (e.g. “During which period(s) was there a decline in the number of births?”), then you’re only speaking to ~4 in 10 people

Based on these, you can adjust your presentation of the data accordingly. If you know you’re only speaking to an advanced audience, you’re good to go. But if you’d like to reach a wider audience, there may be chart choices that are more accessible, but you’ll also need to supplement the data with other modes of communication like interpretive text or narrative storytelling.

What can we do better?

Keeping our audiences in mind matters now more than ever. Covid-19 is a tornado of numerical concepts and conditions that people struggle with (e.g. large numbers, exponential curves, politics / emotion, etc). Further, the communities that are most impacted by the virus are also the most underserved in terms of numeracy education. Both of these issues raise the bar for communicators to make their insights more accessible.

So what can we do to solve the “Curse of [Quantitative] Knowledge?”

Don’t assume widespread numeracy. Be conscious of your audience’s appetite for complexity.
User test your work on real people. There’s nothing like user feedback to surface areas that can be further clarified.
When data needs to be accessible to the majority of the population (at least of US Adults), ask yourself: Is this more or less complex than subtracting two values from a bar chart? If so, charts alone won’t be enough.
Annotate everything. Whenever possible, provide written instructions on how to interpret your visualizations and supplement visualizations with narrative descriptions on key takeaways.
If you know something like log-scale axes won’t be widely understood, do it anyway. Many folks argue that exposure to more difficult graphs actually helps improve graph-literacy, so maybe take one for the team?

Workplace Analytics Benchmark Report

Fri, 20 Jun 2025 00:00:00 GMT

Context

Client

Worklytics is an industry leader in workplace analytics, helping organizations measure & continuously improve the way they work.

Prompt

How might we visualize workplace benchmarks, with humanity, clarity, and style?

Background

Data becomes “actionable” when it’s misaligned with our expectations. For example, knowing that employees spend 20 hours each week in meetings might be an interesting “fun fact,” but what can you do with it?

Is 20 hours good or bad? If bad, how bad? How urgently does it require your attention?

To answer this, you need to know what’s “normal.”

Benchmarks help you pair “20 hours per week of meeting time” with context like “compared to hundreds of thousands of similar office workers, 20 hours a week is an extreme outlier, higher than XX% of the population.” This makes it clear that 20 hours is a crazy amount of time and that your teams’ meeting habits might need some attention. This realization is the first step toward actions like improving meeting hygiene, better collaboration tooling to encourage asynchronous communication, or adopting practices like “no meeting Wednesdays.”

Worklytics’ benchmarks provide this important extra context and make up one of the world’s leading datasets on “what’s normal?” at work. The Worklytics Benchmarks Report showcases these benchmarks and educates customers on how to use them in their own analysis.

Design

Challenge #1: What does “normal” look like?

When designing the main benchmark visualization, we needed to balance clarity and approachability, while playing nicely with Worklytics’ existing visual language. Fortunately Worklytics makes this easy. They’re not shy about leaning into more expressive charts like jitter plots, which feature prominently in their app and in their reporting.

Worklytics reports already do a great job at “showing the data.” Not only do these more expressive charts create visual interest, they’re effective data design and support better decision making. The Benchmarks report needed to build on this foundation.

Plots like these are powerful for visualizing people analytics data because they a) have an easy, concrete visual metaphor (1 dot = 1 person), b) they promote better business decisions in a variety of scenarios since they show the full range of outcomes, and c) they can be quite compact and easy to lay out in dense reporting.

Jitter plots have an important drawback though: Because they’re compact, they tend to hide the shape of the underlying data. Since the benchmarks are detailed enough to show the shape, and the shape is an important part of the story, we wanted to show it off. It’s also a great source of visual variety, which goes a long way toward differentiating metrics in a long report.

Insights

Show more data: For the benchmarks, showing the full distributions of data is important.

Wilmer & Kerns 2022 What’s Really Wrong With Bar Graphs of Mean Values: Variable and Inaccurate Communication of Evidence on Three Key Dimensions"

Holder & Xiong 2022 Dispersion vs Disparity: Hiding Variability Can Encourage Stereotyping When Visualizing Social Outcomes

Hofman et al 2020 "How Visualizing Inferential Uncertainty Can Mislead Readers About Treatment Effects in Scientific Results"

While conventional business reporting favors simple-seeming charts (e.g. bar charts of averages), these overly simplistic visualizations are often misleading (Wilmer & Kerns 2022). Hiding outcome variability encourages misjudgements about the causal stories behind the data (Holder & Xiong 2022).

Related biases can also impact a variety of other business decisions, like overpaying for programs that only offer marginal improvements (Hofman 2020). More expressive charts like jitter plots or quantile dot plots avoid these issues by showing the full range of data.

This shows the benchmark population as if it were 500 people and each dot is a single person (1 dot = 0.2% of the population). The blue vertical bar shows the median value (50th percentile). Darker dots are within the normal range, representing the middle 50% of the population (between the 25th - 75th percentiles). In a typical week, the median person eats 9.0 M&Ms per day. For most people, this ranges between between 7 and 11 M&Ms eaten (the 25th and 75th percentiles).

Kale et al 2020 Visual Reasoning Strategies for Effect Size Judgments and Decisions

Lead with the familiar: While averages and medians are only a small part of the story, they’re still the first thing people look for. By overlaying the bright blue median bar (p50=9.0) and giving it the most visual weight, we can ensure the charts keep viewers in their comfort zones, meeting their immediate expectations and even minimizing certain types of decision bias (Kale 2020). Because the prominent median gets noticed first, the additional detail of the distribution is purely additive—adding context—without sacrificing the immediacy of a more familiar plot, like a bar chart. Detail doesn’t have to be distracting.

Holder & Xiong 2023 Polarizing Political Polls: How Visualization Design Choices Can Shape Public Opinion and Increase Political Polarization

Milkman et al 2021 Megastudies improve the impact of applied behavioural science

Allcott & Mullainathan 2010 Behavior and energy policy: Investment in scalable, non–price-based behavioral interventions and research may prove valuable in improving energy efficiency.

Scott & Nowlis 2013 The Effect of Goal Specificity on Consumer Goal Reengagement

Benchmark ranges. The charts also highlight the interquartile range of the benchmark distributions, with blue dots and shading from the 25th to the 75th percentiles. Benchmarks aren’t goals, at least not necessarily. However, our research and others’ show that charts like these can be highly influential (Holder & Xiong 2023, Milkman et al 2021, Allcott & Mullainathan 2010): It’s human nature to shift our attitudes and behaviors to align with perceived social norms.

Presenting the benchmarks as a range of outcomes avoids being overly assertive about the importance of any particular point on the spectrum, which is a judgement best left to customers. At the same time, to the extent that organizations would like to shift toward the benchmarks, targeting a range of acceptable outcomes can be more motivating toward longer term perseverance in behavior change, relative to a goal defined as a point value (Scott 2013).

Kay et al 2016 When (ish) is My Bus? User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems

Proven approachability: Quantile Dot Plots help data-shy audiences understand variability and uncertainty. These charts are well-studied within the VIS community and reliably effective, even for helping random people at a bus stop to predict uncertain bus arrivals (Kay 2016). This works because the individual dots are concrete and (potentially) countable. This affords a simple, but powerful visual metaphor: You can read the chart by imagining each dot is a person, and they’re all lined up based on their outcome on the chart.

Kong et al 2019 Trust and Recall of Information across Varying Degrees of Title-Visualization Misalignment

Good dataviz is good writing. Even with proven charts, data-literacy issues can put insights out of reach for some audiences. For this reason, it’s always good to provide detailed “how to read this chart” explainers. It’s also good to tell the same story in multiple ways, both visually and in writing. This has an added bonus of assisting memorability, as people remember soundbites from chart titles more than the charts themselves (Kong 2019).

Challenge #2: Information Overload.

In addition to benchmarking the overall population, Worklytics also provides benchmarks for eight specific subgroups like frontline managers, software engineers, or people who work at huge corporations. While this enables customers to make more “apples to apples” comparisons, it adds quite a bit of density.

Insights

Small multiples. Aligning the plots vertically into small multiples gives viewers enough space to consider each subpopulation individually, while also making it easy to compare between rows, to see how metrics differ between groups.

Franconeri et al 2021 The Science of Visual Data Communication: What Works

Blue normal range anchor. To further facilitate between-row comparisons, we extended the normal range for the overall population all the way down the page (and onto the second page) as the soft blue band in the background. This makes it easier to compare subpopulations to the overall norm, without having to bounce your eyes up-and-down the page (Franconeri 2021). As an added bonus, it also serves as a pleasant structural element on each page, guiding your eye toward the most critical content.

Full two-page spread for the (fake) metric: # Cookies Eaten Daily. The first row is the Overall Population, representing 100% of people in the benchmark dataset. The large blue-gray band in the background shows the normal range of this overall population (you can see it aligns with the p25 and p75 values for the “Overall Population”). Each following row shows the distribution for a specific sub-population. For example, individual contributors are 80% of the population, so that row shows dots for 400 people (excluding a few outliers who fall outside the plot range). The final two plots (right page, bottom) include a timeseries for understanding seasonality in cookie-eating, and the table at the bottom serves the reference use-case, for viewers needing to cite some specific benchmark value.

Dot counts as differentiation. To reinforce the idea that each row represents a distinct subgroup of the overall population, and convey how these subgroups can differ dramatically in size, the number of dots on each row is proportionate to the size of the subgroup. For example, you can see there are only a handful of dots on the senior leaders row, because senior leaders are only a small proportion of people within a typical organization.

Strict Baseline Grid. To minimize visual noise, each chart and all text elements were carefully aligned against a consistent grid, both vertically and against a text baseline. This gave us room to pack in more information while avoiding a cluttered feeling.

The bottom row of dots aligns with the top of the plot, and the baselines for each of the percentile annotations. Matplotlib put up a good fight to prevent this, but we made it happen!

Challenge #3: Where’s the action?

While the metric pages were designed to minimize clutter and overload, the scope of the report added another challenge: It’s 89 pages long and covers 35 metrics. And each metric includes eight profiles and 12 charts.

Benchmarks make data actionable, but with this much material, how do we guide viewers toward “the action?” How can we use the report to demonstrate the types of comparisons that make this data valuable?

Insights

Always be educating. Introductory material sets up the rest of the report for success. We expect that most viewers will quickly flip past this on their way to the main content, but even scrolling through they can gain a gut sense of what to expect from the report and gain some exposure to the visual language. And, as questions pop up, they’ll know exactly where to look first.

In addition to the explainer text that shows up on each metric page, introductory material like the “How to use this report” can help viewers get the most out of the report.

Follow the blue path. The blue band through each page represents the benchmark range for the overall population. This element does a lot of work within each page (e.g. it’s a visual anchor, as well as a reference for the charts), but it also works between pages. As viewers navigate from section to section and metric to metric, the blue band shifts positions horizontally, giving each section a unique fingerprint, while indicating transitions between metric sets and previewing their distributions.

The blue benchmark range served as both a reference for comparing vertically stacked plots, as well as a source of visual differentiation across each page, giving each metric a subtle visual fingerprint.

Action in the outliers. Benchmarks are actionable because they highlight data that doesn’t match expectations. They reveal outliers in the organization, which represent the biggest opportunities either for improving stale processes (e.g. senior leaders getting first dibs on cookies) or finding exceptional teams worth emulating (e.g. the lower tail of senior leaders who eat a reasonable amount of cookies).

So the best way for analysts to use the benchmark data is looking for places where their organization and the benchmarks are misaligned, then digging deeper to figure out why. Because the overall population acts as a “benchmark of benchmarks,” we’re able to use this approach in the report by showing each groups’ normal ranges in high contrast blue, making it easy to spot parts of their curves that are misaligned with the overall population, demonstrating this as a technique that Worklytics’ customers can do in their own analysis.

The designs invite viewers to look for misaligned outcome distributions and attend to outliers. For example, in the chart above we can see that the median (p50) Senior Leader eats an exceptional number of cookies. But that’s not the case for every Senior Leader: The left tail shows ten of the Senior Leader dots fall within the normal range. They are exceptional for Senior Leaders, but normal for everyone else. Why is that? Is this left tail of senior leaders not eating enough cookies? Or maybe, their outcomes suggest that excessive cookie consumption isn’t strictly necessary for effective leadership? Spotting misalignment and outliers enables more incisive snacking analysis, and therefore deeper insights overall.

Results

"Benchmarks drive real interest from and value for our customers, but they can be challenging to navigate across so many metrics and personas. That's why we're so glad to work with Eli on projects like this. He has a reliable talent for distilling that complexity into data stories that are both sophisticated and accessible, even for our customers' busy executives. He also kept us well-informed every step of the way and clearly articulated his thought process at key decision points. Our customers loved the results, we loved the process. Amazing work all around!"
Catherine Coppinger, Head of Customer Insights at Worklytics

The report is live here:

The Worklytics Benchmark Report, Version 2.

How to visualize social outcome disparities with jitter plots and Google Gemini

Thu, 19 Jun 2025 00:00:00 GMT

Bar charts are a bad idea when visualizing social outcomes like public health, student test scores, or economic inequality. Reducing entire populations of people down to a single average tends to be misleading; you might even argue it’s “deadly!”

Instead, when visualizing people, we should aim for more expressive charts that show the full range of outcomes, like jitter plots, histograms, or quantile dots. Seeing variation helps audiences understand the systemic forces that affect people at a population level.

When I talk about this in workshops, I see lots of nodding heads and knowing looks, but there’s almost always the same pushback:

“This is great, but I don’t know R or Python.”

A fair point! Tools like Excel and Google Sheets don’t make this easy. Even the more modern viz tools like Datawrapper require a bit of wrangling to show something like ranges.

What can we do instead?

The most recent wave of AI models are surprisingly adept at generating charts. With the right prompting, they can punch out workable javascript, using D3 or vegalite to render a dataset into a remarkably pleasant chart.

In this post we’ll walk through a series of prompts to demonstrate how to visualize inequality with jitter plots, step-by-step, using Google Gemini and regular language.

We’ll use county-level income inequality in North Carolina as our example. This helps demonstrate that, at a population level, how much you earn is less about how smart you are or how hard you work, it’s more about where you live.

Step 1: Pulling the Data

First, we need data. The US Census Bureau’s American Community Survey (ACS) is a great resource for this. We can even ask the model to generate the exact URL to get the data from the Census API.

Prompt:

Could you give me an API request for ACS data for 2023 in north carolina at the county level for median household income, county population size, and number of households?

Gemini returns a well-formatted API request URL:

https://api.census.gov/data/2023/acs/acs5?get=NAME,B19013_001E,B01003_001E,B25003_001E&for=county:*&in=state:37

You can try it yourself here.

It returns a bunch of JSON rows that look like this:

[["NAME","B19013_001E","B01003_001E","B25003_001E","state","county"],
["Alamance County, North Carolina","64445","174286","68441","37","001"],
["Alexander County, North Carolina","65268","36440","13895","37","003"],
...
["Yancey County, North Carolina","54961","18676","8188","37","199"]]

Step 2: A First Rough Chart

Now, let’s get an initial visualization. We can give the API URL directly to Gemini and ask it to build a basic chart.

Prompt:

Create an HTML page with a D3 chart.

The chart should show horizontal jitter plot (also called a strip plot). The dot values should be based on Median Household Income. The dots should be light gray and have 0.8 opacity and have blending set to multiply.

The x domain (horizontal scale) should range from [lowest value floored to nearest 10000, highest value ceilinged to nearest 10000]

The x axis should include a horizontal line and 3 ticks for the start, middle, and end of the domain. The axis line and tick marks should all be the same dark gray color. The start tick label should be left aligned and the end tick label should be right aligned, so the whole label stays on screen.

Everything should be framed inside the same box and left aligned. The height of the jitter plot row should be no more than 120px. The xaxis should be just below the plot with minimal spacing.

The box should be 800px wide and 450px tall. It’s okay if it hangs off the screen.

Using this data… https://api.census.gov/data/2023/acs/acs5?get=NAME,B19013_001E,B01003_001E&for=county:*&in=state:37

The result:

Gemini's first attempt at a jitter plot.

It’s a bit rough, but a great starting point.

Chart types have ambiguous names, so providing a couple ways to identify the chart might help the model understand your intent (e.g. both “jitter plot” and “strip plot”)
You might not always need to specify an x-axis domain, but sometimes it needs some help.
Being specific on things like styling and layout should help get more predictable results.
Specifying exact dimensions (800x450px) can be helpful when you’re plugging charts into a report or slide deck.

Step 3: Titles and Text

Good dataviz is good writing. Specific chart titles are one of the best ways to make sure charts communicate clear takeaways for wide audiences. In fact, research shows titles are often more memorable than the visual itself (Kong et al 2019).

Let’s add a clear title and a dynamic subtitle that automatically highlights the range of outcomes.

Prompt additions:

The chart title should be: “In North Carolina, median household income varies widely by geography.”

Subtitle should be: “[county with highest income] has the highest income in NC ([highest income value]), while [county with lowest income] has the lowest ([lowest income value]).”

The axis title should be Median Household Income.

Underneath the x-axis should be explainer text that says “How to read this chart: Each dot is 1 county in NC. Dots are positioned horizontally based on the county’s median household income. Dots are randomly positioned vertically within the row for visual separation.” Make sure to respect the line breaks.

Everything should be framed inside the same box and left aligned. The order should be the title, then the subtitle, then a line break, then the plot, then the x axis, then the explainer text.

The result:

Good dataviz is good writing. So we've added a title, subtitle, and explainer text.

Getting better!

Note how it handles the brackets like [county with highest income] and [highest income value] and writes the code behind the scenes to fill in the correct values.

Step 4: A Reference Line

While we want people to take in the whole distribution, reference lines provide a familiar anchor for audiences who might be skittish about less conventional charts. It bridges the gap between a conventional bar chart and a jitter plot, making the new format more approachable.

Prompt additions:

In the middle of the row should be a black vertical tick line that shows the average value for all data points in that row. The line should be solid and 4px thick, Above the tick line should be the actual average value.

There should be a legend that shows the following elements on the same row. They should be separated by 8 spaces worth of horizontal spacing. * a matching gray circle icon with text “1 dot = 1 county” * a matching vertical black line icon with text “state average”. The “black line icon” needs to be vertical to visually match the vertical tick used to signify the average.

The order should be the title, then the subtitle, then the legend, then a line break, then the plot, then the x axis, then the explainer text.

The result:

To help orient viewers, we've added a reference line in the middle, showing the average value across all counties.

This is meant as a simple demo, but in “real life” you might consider other options for the reference lines like the statewide average, instead of the average of averages currently shown.

Step 5: Layout Cleanups

The chart is coming together, but as you saw above, it’s still a bit of a mess. Sometimes it requires more handholding on details like layouts or writing.

Here, we’ll give more specific guidance to create a cleaner, more professional look.

Prompt additions:

In the subtitle, the county names should just include the county name, not “north carolina” (e.g. “Wake County” not “Wake County, North Carolina”). In the subtitle use “NC” not “North Carolina”

There should be 48px of padding on all sides of the chart. The vertical spacing between the title and subtitle should match the vertical spacing between the subtitle and the legend. The vertical spacing between the legend and the plot should be 48px. The xaxis should be just below the plot, with 24px of spacing between the dots and the axis. The axis title should be below the axis, with 12px of spacing between the axis and the axis title.

The subtitle, legend text, value label, axis ticks, and axis title should all have font-size = 14px. The explainer text should be font-size=12px.

The result:

We've added some padding, more consistent spacing, and tweaked the fonts.

This still has some annoying spacing issues, but it’s pretty close for a robot that can’t actually see the chart…

Step 6: Adding Mouseovers for Interactivity

Because this chart is just a web page, we can also make it interactive.

Adding tooltips that appear on hover allows viewers to explore individual data points to get a better sense of which counties make up the extremes.

Prompt additions:

When you mouseover the dots, it should show a tooltip like "[county name]: $[household income value]"

The result:

A screenshot of mousing over the dot for Wake County.

What’s particularly handy, Gemini’s canvases come with share links, so you can share the whole thing:

https://gemini.google.com/share/b3023a7417f8.

Step 7: Download Button

To make the chart portable, let’s add an “Export to SVG” button. This lets you download a high-quality vector graphic that can drop into PowerPoint, Keynote, Figma, or whatever you’re working with.

Prompt additions:

All CSS styles should be included inline in style attributes. Do not split the CSS into a style sheet or reference styles through classes. This includes the dot styles, the axes, font families for all text elements set to “sans-serif”. The title, subtitle, legend, and explainer text should be included as SVG, not HTML.

Add a button underneath the box that says “Export to SVG.” This button should download the chart as an SVG file.

The result:

Added an 'Export to SVG' button.

And there it is! This button was actually quite a time saver for making this post.

Step 8: Split by region

Okay so we’ve gotten to a fairly nice, polished chart. But just like in real life, sometimes you have to see something to realize it’s not exactly what you want.

The previous chart shows county variability, but it doesn’t tell us much about the counties themselves.

Let’s add another dimension. We’ll divide up the dots into regions, so we can compare Eastern, Central, and Western North Carolina.

This will require region labels which aren’t in the original Census data. With a bit of googling I found the following table from North Carolina’s state government, mapping county names to regions. Conveniently, it also has an API which gives results like this…

{"total_count": 100, "results": [
    {"county": "Perquimans", "region_name": "Eastern North Carolina"}, 
    {"county": "Chowan", "region_name": "Eastern North Carolina"}, 
    {"county": "Halifax", "region_name": "Eastern North Carolina"}, 
    ...
    {"county": "Mitchell", "region_name": "Western North Carolina"}, 
    {"county": "Haywood", "region_name": "Western North Carolina"}, 
    {"county": "Cherokee", "region_name": "Western North Carolina"}
]}

Now we’ll see if the model can not only add more rows to the chart, but also join together two datasets to make this happen.

Prompt additions:

Can you use the following data to divide the jitter plot into rows based on the county’s region, using the following data? e.g. so there’d be one row with Eastern North Carolina where all the dots are counties in Eastern NC

https://demography.osbm.nc.gov/api/explore/v2.1/catalog/datasets/north-carolina-geographic-regions/records?select=county%2C%20region_name&limit=100&refine=region_type%3A%22Other%20%22

Note in the census data counties will be labeled “XXX county” but in the API it will just be listed as “XXX”

Can you make each jitter row 33% the height, so everything fits? And add left padding to the jitter plots and the x axis so that the labels on the left don’t overlap with the plots? Needs probably 200px The row labels and explainer text should be aligned to the left, so they’re left aligned with the chart title.

The result:

The original jitter plot is now split into 3 rows, one for each region.

Done!

Now we can see there’s variability within the state, but there are also differences between the regions.

We can also see that Western North Carolina, near the mountains, is generally lower income, while higher income is more concentrated in Central North Carolina, which makes sense, since this includes the state’s major cities like Raleigh, Durham, and Charlotte.

There’s certainly more we can do with this chart, for example hinting at county populations or exploring how factors like income relate to education or segregation. But for just a few minutes of prompting, this is a great start for descriptive reporting on income levels in NC.

You no longer need to R or Python to move beyond the bar chart. With a bit of prodding, more equitable, effective, expressive charts (like the jitter plots) can be “vibe coded” with generative AI models like Gemini, Claude and ChatGPT.

Next steps:

Check your results closely! AI models require a close eye to verify their output.
If you or your team want to learn more about equitable data design, please check out 3iap’s workshops like The Equitable Dataviz Primer or Equitable Epidemiology.
Any questions, comments, or feedback? Feel free to email hi@3iap.com.

Codesigning Effective Score Reporting

Thu, 01 May 2025 00:00:00 GMT

Context

Client

The ETS Research Institute.

Prompt

How might we redesign The Nation’s Report Card reporting to clearly, effectively visualize student achievement across the United States?

Background

ETS researchers approached 3iap with an ambitious goal: Redesigning reporting for “The Nation’s Report Card.”

The Nation’s Report Card is the US federal government’s gold-standard data covering student achievement. The “report card” is how the US National Center for Educational Statistics (NCES) publishes the highly-anticipated National Assessment of Educational Progress (NAEP) student test results.

NAEP reporting is an important test case, informing an even broader ambition to ultimately improve reporting for any large-scale student assessments. Making these results more useful for informing policies is key to improving learning outcomes for all students.

About NAEP and The Nation’s Report Card

NAEP data is the United States’ most authoritative signal on student achievement, for the country as a whole and across all fifty states.

It’s not just foundational for education research, it’s also how state-level education leaders keep tabs on each other. A state’s rise (or fall) in NAEP rankings is a closely-watched signal for validating their education policies. Similarly, a drop in NAEP scores can be a motivating force to rally political support for changes and improvements.

NAEP data is also crucial for understanding group-level educational outcomes in the United States.

Design

Challenge

Because NAEP data disaggregates outcomes across student subpopulations, it gives us visibility into how students’ differences in opportunity manifest as differences in outcomes.

The challenge is that when charts show “differences in outcomes,” it makes “differences in opportunities” easy to overlook. And this can lead viewers toward some surprising misjudgments about the data.

Reproductions of two misleading charts that appear on the US Department of Education’s Nation’s Report Card platform. These show eighth grade reading and math scores respectively. The plot on the left is a conventional bar chart, showing average scores for students across racial groups (with actual group labels hidden). The plot on the right is a timeseries chart emphasizing the “achievement gap” between two student groups. These charts are risky because they encourage misplaced blame for outcome differences and harmful social misbeliefs about groups with lower outcomes.

Group-level achievement data are often shown using conventional charts like the above. Sometimes they’re bar charts, contrasting average outcomes between groups (left). Other times they’re timeseries, highlighting the contrast between two groups (right).

By fixating viewers on the differences between groups, charts like these can be dangerously misleading, misrepresenting the underlying causes of outcome differences and the student groups being visualized (Holder & Xiong 2022, Gutiérrez 2008). These misbeliefs have troubling implications not only for students, but also in undermining successful education policy (Holder & Blakely, Metzyl 2019).

3iap’s research was actually the first to empirically demonstrate this biased interpretation for data visualizations, which you can read more about here: Unfair Comparisons.

Holder & Padilla 2024 Must Be A Tuesday: Affect, Attribution, and Geographic Variability in Equity-Oriented Visualizations of Population Health Disparities

Our research shows how these biases are influenced by design choices. For example, bar charts like the above, which hide variability within groups, tend to make these biases worse. On the other hand, charts like jitter plots and histograms show full distributions of data, emphasizing within-group variability and between group overlap; these more expressive charts can actually correct some of the bias, reducing the risk of misattributions and misrepresentations (Holder & Padilla 2024).

Goals

3iap’s collaboration with ETS focused on uncovering new charts, intentionally designed for visualizing group-level education outcomes while minimizing social-cognitive biases.

Specifically, we were exploring alternative designs aligned with the following goals:

NAEP Score charts should meet basic standards of clarity and approachability for The Report Card’s original communication goals. They need to clearly present group-level test scores.
The charts should minimize misleading biases by applying 3iap and others’ research into clearer social outcome reporting.
The charts need to be practically useful for education analysts, school administrators, and education policymakers.

Approach

Guided Codesign

While our research-backed design principles are a great first step, there’s still a big gap between principles and producing viable designs that are ready for a big platform like The Nation’s Report Card.

Blurred results of the codesign group’s “crazy 8s” sketching session, diverging on design ideas, followed by dot-voting results to converge around the highest priority concepts.

Resilient Designs: A guided codesign process can be a reliable way to develop designs that not only build on research and best practices, but that also meet practitioners’ other communication goals and adapt to their specific contexts. The key is how this collaborative process helps surface a) a wider set of constraints and considerations, b) from a more diverse set of perspectives, c) as early in the process as possible. Getting as many of these ideas out of people’s heads and onto paper makes it easier to manage critical design tradeoffs and ultimately reach more optimal solutions.

Learning By Doing: In addition to producing better designs, codesign also helps teams learn and adopt more effective design practices. While workshops are helpful for introducing research and design guidelines, there’s nothing like hands-on experience for learning something new. This is particularly true when the material is unintuitive or challenges entrenched conventional wisdom.

Decision Alignment: Codesign also promotes organizational alignment and cohesive decisionmaking. By actively participating in the design process, everyone involved has a more intimate understanding of important tradeoffs, making it easier to ultimately align around important decisions.

3iap partnered with ETS researchers on three main areas:

Design training and data-design expertise. This ranged from early training sessions for participants, to targeted insights related to specific codesign sessions, to ongoing feedback and consulting on more general best practices for effective data communication.
Facilitated design exercises. These helped codesign participants to articulate a diverse set of needs, as well as to brainstorm and prioritize viable alternative solutions.
Design research consulting. As ETS researchers planned further studies, 3iap supported ongoing research efforts, contributing to literature reviews, instrument exploration, and experiment design tradeoffs for empirically validating candidate designs.

Codesign Insights

ETS designers’ mockups of the group’s seven final design concepts, each demonstrating one of the following social outcome reporting design principles: Emphasizing within-group variability, minimizing inter-group comparisons, causal explainability. Mockups by ETS.

What was remarkable about the codesign process was how, even though it was open-ended and group-driven, the final designs converged around a set of powerful, research-backed design principles, each leaning into complementary theories of action.

Emphasizing within-group variability. This was a key insight from 3iap’s studies: Outcome charts can minimize misattributions and misrepresentations by emphasizing within-group variability (and highlighting between-group overlap).
Minimizing inter-group comparisons. A number of research threads suggest that any amount of inter-group comparisons might be problematic. And from a design perspective, they’re also not necessary for communication goals. For example, to report on outcomes for rural students, there’s actually no hard reason to compare their outcomes to those of urban or suburban students: Comparisons to population means or universal goals convey the same information, with less risk of side effects.
Causal explainability. Every analyst knows that “correlation doesn’t imply causation.” But the “illusion of causality” is a surprisingly common bias when interpreting data, and it’s closely related to issues like misattribution. When we mistakenly blame individuals for bad outcomes, it’s because we’re overlooking the more complex and diffuse upstream causes like school funding differences and neighborhood-effects. Many of the designs attempt to correct for this by subtly priming other plausible influences for the visualized outcomes.

Results

The latest charts used on The Nation’s Report Card site to present 2024 NAEP group-level test scores. The ETS codesign project influenced this updated, more effective design approach.

We changed reporting The Nation’s Report Card. The National Center for Educational Statistics (NCES) heard what we were doing and adopted many of our most critical ideas. In their most recent launch of The Nation’s Report Card website, they present 2024 NAEP group-level test scores with small multiple histograms that emphasize within-group variability, minimize inter-group comparisons, and include group identity in line with a variety of other (more relevant) factors like socioeconomic status. They’ve also removed references to “achievement gaps” and related plots.

Our codesign study was accepted at NCME. The brilliant team at ETS submitted the codesign study to the National Council on Measurement in Education (NCME). We were accepted and presented together as a panel at the 2025 Annual Meeting in Denver.

The (data) stories we tell ourselves.

Fri, 26 Apr 2024 00:00:00 GMT

Can geometric shapes be jerks? What can this tell us about interpreting visualizations?

This is a video clip from a famous experiment, published in 1944, by psychologists Fritz Heider and Marianne Simmel. Take ~47 seconds to watch it. It’s worth it, we promise! There’s no sound (just background noise), it’s safe for work, you can play it wherever.

Pause!

Ask yourself: What did you see happening in this video?

…

If you’re like most people, you imagined a fairly elaborate, perhaps dramatic story about these little shapes, as live characters, moving about the screen. For example, one participant in the original study described the scene in great detail:

“A man has planned to meet a girl and the girl comes along with another man. The first man tells the second to go; the second tells the first, and he shakes his head. Then the two men have a fight, and the girl starts to go into the room to get out of the way and hesitates and finally goes in. She apparently does not want to be with the first man. The first man follows her into the room after having left the second in a rather weakened condition leaning on the wall outside the room. The girl gets worried and races from one corner to the other in the far part of the room. Man number one, after being rather silent for a while, makes several approaches at her; but she gets to the corner across from the door, just as man number two is trying to open it. He evidently got banged around and is still weak from his efforts to open the door. The girl gets out of the room in a sudden dash just as man number two gets the door open. The two chase around the outside of the room together, followed by man number one. But they finally elude him and get away. The first man goes back and tries to open his door, but he is so blinded by rage and frustration that he can not open it. So he butts it open and in a really mad dash around the room he breaks in first one wall and then another."

Note that in the video:

The shapes are anthropomorphic. We see them as living things.
There’s cause and effect. We see the big triangle push the little triangle.
There’s motive and intent. The triangles are fighting for a reason.
There’s a very real feeling of menace, as the big triangle approaches the circle.

Now consider: What did you actually see in this video?

In a very strict sense, not much at all. According to the authors’ description, all you actually saw was:

“three geometrical figures (a large triangle, a small triangle and a disc, also called a circle)... moving in various directions and at various speeds.”

All the other details you added yourself. They were from your own experiences, expectations, and imagination.

And you wouldn’t be alone! In their first study, only one single participant (out of 34) described the video in purely geometric terms. Everyone else imagined elaborate, personal narratives like the one above.

In a second study, they used the same video and asked participants to describe more about the characters and their motives. In this follow-up, 100% of participants imagined a fight, thought the little triangle and circle belonged together, and that they were collectively opposed to the big triangle.

The reasons for the fight were more divided:

50% of participants, of course, saw a damsel in distress. They gendered the shapes and defined the story as two triangle men fighting over a circle woman.
30% of participants described the big triangle as a bully. They attributed the fight purely to the big triangle’s aggressive provocation.

Participants also described the shapes’ character in surprisingly consistent terms:

97% described the big triangle as aggressive, possessive, and belligerent.
47% described the little triangle as heroic, valiant, and courageous
75% described the circle as frightened, meek, and unsure of herself (61% described the circle as feminine)

Again, none of these explanations are strictly supported by the video. But this translation, from meaningless moving shapes to an elaborate interpersonal struggle, is deeply baked into how we see the world. Our brains are pattern-matching machines. We’re quick to convert coincidental signals into causal narratives.

Further still, the tendency to imagine motivation, agency, and intent, without any actual evidence, is deeply baked into how we see other people.

It turns out, we also do this with data. We need constant reminders that “correlation isn’t causation” because when we see correlation, we almost can’t help but imagine causation. When we hear that White people live longer than Black people, we disproportionately (and unreasonably) attribute the disparity to genetic differences. We see economic outlook driving presidential approval, instead of approval driving outlook (or, perhaps more realistically, a side effect of increasingly galvanized partisan loyalties).

What does this mean for dataviz?

As humans, we’re quick to see stories where we shouldn’t. This means there’s no guarantee that the stories that are supported by a dataset will be the same as the stories people imagine when they’re viewing a chart. There’s a difference!

When we’re looking at a visualization, especially if we’re evaluating it critically, two questions should be top of mind:

What does the data say, strictly?
What might the data suggest?

These two questions are also important for making visualizations. As communicators, it’s our job to make sure that what the data says and what the chart suggests are as closely aligned as possible.

This post by Eli Holder originally appeared on effaff.com here.

Philanthropic Conversation Analysis

Mon, 01 Apr 2024 00:00:00 GMT

Context

Client

Strategic communication team for a major U.S. philanthropic foundation

Prompt

How might we help a major philanthropic foundation’s leadership understand and leverage public conversations around their programs?

Background

One of the largest U.S. philanthropic foundations needed to better understand their public impact and communication effectiveness. While their primary focus is creating positive change, they recognized that strategic communication can amplify social impact by:

Attracting potential partners and collaborators
Using their platform to amplify others’ impactful work
Ensuring funding opportunities reach a wide, diverse pool of potential grantees

The foundation’s leadership team was preparing a comprehensive listening study for their board. With a dataset containing thousands of media mentions (e.g. exports from tools like Sprinklr, Cision, etc), they needed help transforming raw data into memorable data storytelling, actionable insights, and board-ready visualizations.

Key Challenge

Distill vast amounts of semi-structured social listening data into clear, compelling narratives that would resonate with board members while maintaining the analytical depth necessary for strategic decision-making.

Insights

Leveraging AI for Scale. Manual analysis of tens-of-thousands of records would have been prohibitively time-intensive. While we approach AI technologies thoughtfully, natural language processing has proven particularly effective for text classification tasks. And some research suggests that large language models can match or exceed human accuracy in detecting key themes and sentiment. By implementing rigorous validation processes, we could harness these capabilities to dramatically accelerate data preparation without sacrificing analytical rigor.

Example concept brainstorming results, paired with inspiring visualizations for style references.

Insightful data stories start with insightful questions. The 3iap approach to collaborative data storytelling pairs qualitative discovery with quantitative exploratory analysis:

Facilitated, collaborative discovery sessions help surface the organization’s key knowledge gaps and identify compelling narrative elements that resonate with stakeholders.
Rigorous exploratory data analysis ensures all insights are grounded in empirical evidence.
This dual approach enables crafting stories that are analytically sound, organizationally relevant, and culturally compelling.

Executive Communication Design. Presenting to board members requires a careful balance between accessibility and depth:

Primary visualizations need to communicate key insights at a glance
Supporting materials should anticipate and address most likely follow-up questions
Documentation and training equip presenting executives with contextual knowledge for deeper discussions

Solutions

Analysis Pipeline

Developed automated data engineering workflows for processing raw media mentions
Integrated machine learning models for topic classification and sentiment analysis
Created validation frameworks to ensure accuracy of automated systems

Data Storytelling Consulting

Facilitated structured discovery sessions with key stakeholders
Mapped organizational knowledge gaps to data-driven insights
Iteratively refined narrative approaches based on stakeholder feedback

Data Design Services

Designed primary board deck emphasizing clarity and immediate comprehension
Developed visualization guidelines and an automated visualization suite for rapid iteration
Supported detailed follow-up discussions with supplementary materials and ongoing analysis

Example slides for four key story points. These mockups use fake, but similar data to the actual project.

Results

The automated analysis pipeline and strategic visualization approach have become integral to the foundation’s communication strategy, enabling them to make more informed decisions about communication priorities, better understand their impact in different focus areas, and more effectively engage with stakeholders and potential partners.

Impact Highlights:

Automated processes reduced data preparation time from months to minutes
Established reproducible framework for future media analysis
Client successfully communicated key insights to board members and the broader organizational communication strategy via “road show” presentations

Polarizing Political Polls

Fri, 01 Dec 2023 00:00:00 GMT

TLDR

3iap’s peer-reviewed research was accepted to IEEE VIS 2023
Public opinion visualizations can unreasonably influence viewers’ attitudes toward public policy
Visualizing polarized political attitudes can increase polarization
Popular polling charts in the media are epistemically sketchy and should be reconsidered to minimize harmful side effects (3iap is happy to help).

Effective dataviz means more than accurate, readable charts. If our goal as data designers is encouraging more reasonable beliefs and decisions, then designing effective dataviz requires understanding the many ways that even good data can lead to bad conclusions.

It also means recognizing that dataviz can actually make this worse. Even clean, honest charts can lead to surprisingly unreasonable beliefs about the stories behind the data.

In the United States, our politics are an overwhelming source of unreasonableness. It’s well-reported that political polarization is bad for governance, bad for relationships, and bad for the social ties that bring a society together.

What’s maybe less understood is how polarization actually makes us unreasonable. As we’ll see, it’s associated with a variety of cognitive impairments like rigid thinking, overconfidence, intolerance, and motivated reasoning. Partisanship can also unreasonably shape our attitudes about weighty topics like public policy.

A blue party, a red party, and their polarized attitudes toward arming household pets with deadly lasers.

Political opinion polls are another area where dataviz can backfire and make viewers less reasonable. For example, it turns out that popular ways of visualizing polarization can actually make it worse.

In our new research we explored some surprising impacts of visualing public opinion polling results, a popular use of dataviz in political journalism.

3iap designed and ran three different experiments, testing nine different ways to visualize political attitudes, looking at six different topics of public policy, with a pool of thousands of research participants. We collaborated with Georgia Tech’s Cindy Xiong to analyze the findings and publish the results. Our paper passed multiple review rounds from a panel of academic data visualization experts and was accepted for publication by the highly-selective IEEE VIS conference.

For all this fuss, we’re able to offer two important findings. Our research is the first to demonstrate:

Public opinion visualizations have a social conformity effect. Charts showing that an idea is popular can make the idea more popular. When viewers saw that certain policies were popular with their groups, the policies became significantly more popular with the viewers themselves.
Visualizing polarization can make it worse. For specific types of charts, these social influences can take the shape of polarization. When viewers see that attitudes are polarized by party, their own attitudes become more polarized across parties.

In this post we’ll attempt to unpack the findings in a way that’s bearable for designers, analysts, and data journalists who don’t want to suffer through the paper itself (we don’t blame you!). We’ll also add some additional context and color that we couldn’t fit into the paper itself and offer some more concrete takeaways for practioners.

Pre-recording of our IEEE VIS talk, designed in collaboration with the brilliant Gabrielle Merite

Research Questions

The work is organized below, by question. You can click the links below to navigate within the page.

Theory: Is this even plausible? How could innocent public opinion charts possibly increase political polarization?
Hypotheses: What, precisely, are we expecting?
Establishing an effect: Can realistic public opinion visualizations impact viewers’ attitudes toward a polarized topic like gun control?
Social Conformity: Are visualized attitudes contagious?
Polarization: Can chart-induced social conformity impact polarization?
Broader Impact: What are the stakes? Is this a problem?
Design Implications: What are the practical implications? What should journalists, analysts, and visualization designers do differently?

Theory

How does political polarization work? How could dataviz possibly impact something so big and nebulous as political polarization?

What is political polarization?

When people talk about polarization, they’re intuitively describing a situation where two different groups who disagree with each other so much that they can’t work together. They disagree on so many things, or with such force, that there’s no common ground where they can meet and build any kind of consensus. More specifically, polarization typically means one of two popular technical definitions:

Issue polarization is how we feel about ideas. Ideas can be either broad ideologies or specific policy proposals. For example, U.S. attitudes toward issues like abortion and guns are strongly divided across party lines and “compromise” solutions to either issue are essentially non-existent. People also diagree on broader issues like “the size of government” or “personal responsibility.”
Affective polarization is how we feel about people. How much do we like people in our own party? Or, more telling, how much do we dislike people from the other party? By this definition, we’re more polarized because we increasingly dislike people from the other party.

In the United States, we’re increasingly polarized across both of these dimensions and they’re actually self-reinforcing. It’s intuitive to imagine that disagreeing on policy issues might make us grumpier toward the people we disagree with. Some research suggests it can go in the other direction as well (i.e. affective polarization can drive issue polarization).

For this study, we focused on issue polarization, exploring viewers’ attitudes toward specific policies, because this type of polarization shows up so frequently as charts in the news.

Why is polarization a big deal? Who cares?

Polarization makes it harder for opposing groups to work together. Intuitively you can imagine the tense feeling of sitting down to Thanksgiving dinner between your always-aggro, MAGA-loving uncle and your too-smart-for-her-own-good, DSA-supporting niece. Given a little bit of jet-lag and a few glasses of wine, any remotely interesting topic (e.g. climate change, gun control, health care) is sure to ignite some sparks.

This tension can have far-reaching effects in every part of society.

Social cohesion. The awkward thanksgiving effect plays out in larger society. People divide themselves geographically based on political preferences. They increasingly refuse to date across party lines (and marriages fail at a higher rate when they cross party lines). People stereotype and demonize their fellow citizens, describing them as “closed-minded, dishonest, immoral, and unintelligent.”
Governance. Imagine that same tension in the halls of Congress, between the 535 elected officials we trust to run our government. Congress is like one big, awkward Thanksgiving dinner where you can only talk politics. Polarization makes it difficult for the government to adapt to change with big policy positions. More recently, we’ve seen it can also make more routine procedures painful (e.g. raising the debt ceiling or funding the government).
Information resistance. Partisanship can make us resistant to facts. Researchers suggest it gives us a sort of tunnel vision and is associated with a variety of cognitive impairments like rigid thinking, over-confidence, and intolerance. It can raise blind spots to very basic information. Politically motivated analysis can also affect people who are otherwise analytically savvy. It’s worth noting that this isn’t some impenetrable veil. Facts are still influential, they just have more hoops to jump through.
Life-or-death personal decisions. Covid-19 offered dramatic tales of unreasonable partisan decision-making. For example, researchers found, that the more someone disliked Democrats, the less likely they were to follow public health guidance. There’s a tiny bit more nuance than “Catching Covid to Own The Libs,” but not much: People in hot spots were more likely to adopt “an accuracy motivation” and wear a mask, but even then conservative adherence lagged behind expectations. These individual decisions might explain wider impacts, like the mortality disparities between “red” and “blue” states.

What’s the psychology of polarization?

Polarization is complex. There are lots of interconnected chains of causes and effects. Social psychology offers a revealing lens into the tangle, which we cover here: Political psychology primer for information designers.

What is public opinion polling? Why does it matter for politics?

Public opinion polls are a popular topic of political data journalism and some of the most prominent examples of dataviz in the media. We’ll unpack two versions of this: election vs issue polling.

Election Polling

Election polls survey people about candidates in upcoming elections, loosely attempting to forecast the final results. Presumably this might offer some public good, as candidates could use polling results as a feedback loop to calibrate their platform to the needs of the electorate. In reality this is the data-journalism version of “horse race” political reporting.

Election polls have created a lot of fuss in the last few years. In addition to promoting the “horse race” view of politics (and eroding the pillars of democracy), they also suffer perceived accuracy issues and might even distort voter turnout. For example, in 2016 Hilary Clinton was predicted to win the popular vote, and she did. However, Clinton’s commanding lead in the polls also reportedly led some of her supporters to complacency. They saw her as the inevitable victor and concluded their votes weren’t needed for her to win.

Venerated pollsters like Pew and Gallup have given up on this type of polling entirely, citing opportunity costs. The subtext being that even though these two highly-respected institutions might do a fine job at election forecasting, they don’t think it’s worth the effort (or their reputations). Instead, they’ve shifted their focus to the issues.

Issue Polling

Eight different ways to visualize public opinion polling results. Data in the charts are fake.

It doesn’t always feel like it, but apparently politicians are occasionally responsive to their constituents. How do policymakers know what their constituents want though?

Citizens are typically invited to contact to their elected officials. This is a somewhat biased signal from politicians’ perspective though, since it tends to favor older or more educated people who have the time and inclination to reach out. There are also, of course, ~12,000 lobbyists that policymakers can turn to, in case they’re curious to hear what’s important to people with money. For everyone else though, there are opinion polls.

According to Pew: “The basic goal of a good public opinion poll is to give everyone in the population, regardless of their wealth, age, education, race, knowledge of politics… an equal voice about the issues of the day.”

At their best, public opinion polls give voice to the voiceless, guiding policymakers with an unbiased window into what’s important to their constituents. Through the miracle of random sampling, public opinion surveys reach a more representative set of constituents than policymakers might otherwise hear from. Presumably, policymakers can then use these survey results to align their policies with their constituents’ priorities.

Polling results also make for popular stories in the news, as part of an overall trend of increased media coverage of polarization. In particular they’re often used to highlight political polarization. That is, instead of simply reporting whether an idea is popular or unpopular nationally, results are split out by political party to highlight the partisan gaps between Democrats and Republicans. While it’s not clear how the “red vs blue” framing advances pollsters’ “equal voice” aspirations, decomposing results by party can help explain the underlying politics of a particular idea.

Despite their popularity, these visualizations can have toxic side effects. By highlighting polarization, they can make it worse.

What’s the pathway from public opinion to polarization?

Our attitudes are heavily influenced by the people around us. This is sometimes referred to as “social conformity” and it can very much effect our political attitudes.

This chart shows that "Banning canned food from Canada" is unpopular, supported by just 42% of pets in the United States. If your cat could read a dot plot, how would this chart influence their attitude toward this controversial policy?

For example, this chart shows pretend-results from a public opinion poll. The poll asked American Dogs and Cats (“All U.S. Pets”) how they feel about a policy proposing that the United States ban imports of canned pet food from Canada.

Specifically, the chart shows that this policy is generally unpopular. It’s supported by only 42% of the population. Based on social psychology, we might expect that if our pets were browsing the internet and saw a chart like this, they’d identify with their fellow pet-citizens and adjust their own attitudes to match the social norm shown in the chart. For viewers who were previously supportive of the policy, theory suggests that they’d decrease their support. On the other hand, for viewers who were already against the policy, they might actually increase their support since they see that others are more relatively ambivalent.

This example highlights our first research question: If a chart shows that an idea is popular, can the chart make the idea more popular?

This chart shows that "Banning canned food from Canada" is unpopular with dogs (supported by just 27%) and popular with cats (supported by 72%). If your dog could read a dot plot, how would this chart influence their attitude toward this controversial policy?

We might expect a similar social conformity effect toward political parties. We’ve reviewed several ‘partisan cue’ studies where highlighting a party’s endorsement of a policy can dramatically change a viewer’s attitudes toward that same policy. Showing polling results split by political party should have a similar effect.

For example, this chart shows the same fictional pet-survey results, now split by political party (supposing, of course, that Dogs and Cats are political adversaries). We can see the ban is popular with Cats (supported by 72%) and unpopular with Dogs (supported by just 27%). These are effectively party endorsements, they’re just quantified and visualized.

Since partisan cues can change viewers’ attitudes toward endorsed policies, we’d expect charts like these to have similar social conformity effects. So if a moderate Dog sees this chart, we’d expect them to decrease their support (toward 27). If a moderate Cat sees the chart, we’d expect them to increase their support (toward 72). If a bunch of moderate Dogs and Cats all see this chart, we’d expect their attitudes to diverge away from each other.

This example highlights a potential downstream effect of attitudes spreading through charts like these. For partisan-split polling charts, we might expect viewers’ attitudes to diverge across party lines and become more polarized.

This raises our next research question: Can visualizing issue polarization increase polarization?

Hypotheses:

What were we expecting from the experiment? The research questions we just outlined led to our basic hypotheses. Here they’re just slightly more formalized, similar to the paper.

H1: We suspected that public opinion charts do more than just communicate opinions, they can actually shape them.
H2: We expected public opinion charts to have a social normative influence, or a social conformity effect, where people align their attitudes with the attitudes they see from their social or political groups.
H3: We suspected that some of these charts might actually influence polarization, either increasing it by shifting group attitudes away from each other, or decreasing it by shifting them closer.
H4: We expected these polarization influences are related to the way charts are framed. Charts showing the national consensus should cause people’s attitudes to converge toward the middle. Charts showing polarized partisan attitudes should cause people’s attitudes to diverge away from their out-party.

Experiments

Experiment #1:
Establishing an Effect

Can realistic public opinion visualizations impact viewers’ attitudes toward a polarized topic like gun control?

Motivation

In our first experiment, we wanted to rule out neutrality. Specifically, we wanted to see if realistic charts of a hot topic issue (i.e. gun policy) could actually influence viewers’ attitudes. If all these charts did was passively relay information, we wouldn’t expect them to have any influence on viewers’ attitudes toward gun policies. (And then we’d have to go find a new chart to pick on.)

Setup

We’ll describe the experiment below, but the quickest way to grok is trying it yourself. We have a demo version here: Experiment #1 demo.

Experiment 1 stimuli. Note this data is realistic but fake.

We tested three different charts, designed to look like visualizations you’d see from one of the big-name pollsters. Each chart shows a stack of policy proposals (or baked goods) and how popular they are with various groups.

These charts are fairly dense. They’d require viewers to slow down a bit to take them in. And since they cover nine different policies they would be tough for people to remember any specific value. So we generally expected these to be read as a broad gists like “Democrats support policies like federal gun databases and banning high-capacity magazines. Republicans don’t support these policies. The gap is pretty wide.”

The partisan dot plot chart on the left includes a dot for the overall national popularity AND a dot for either political party. We’d expect viewers to anchor to some combination of their political party and the national consensus dot.
The consensus dot plot in the middle is identical to the previous chart, except it drops the partisan dots and just shows the national consensus. We’d expect participants, regardless of their party, to anchor to this common-ground, national identity.
Finally, the control matches the consensus dot plot, except instead of gun policies it talks about baked goods. As pervasive as politics has become in the United States, we were assuming that donuts and cookies wouldn’t be politically triggering for most people (hopefully), so this shouldn’t impact their attitudes towards guns.

We asked participants to first answer a few comprehension questions about the chart (e.g. “What is the average favorability of ‘Banning assault-style weapons?’”). This was a sneaky way to ensure they engaged with the middle six policies on the chart. After that, on the next page, participants reported their own attitudes toward those same six policies. This setup was maybe a bit overwhelming in hindsight, but it ensured that any effects we saw would have to be at least durable enough to influence participants after a few minutes had passed.

What might influence look like?

Our analysis approach was complicated, but given the number of variables it kind of had to be. Participants’ responses could reasonably depend on several different factors like the policy itself, their perception of the policy’s political lean, the policy’s actual popularity, the popularity we showed to people, and the participant’s personal politics. Each of those factors are independent of which chart we actually showed them. We’ll walk through a couple of examples to make this more concrete, but first let’s talk about the results we might expect.

These four upstanding citizens represent a range of baseline attitudes toward a gun control policy from Experiment #1.

To start, let’s consider four different participants, with different levels of support for a policy like “Banning high capacity ammunition magazines holding 10 or more rounds.” We can see each participant with their hypothetical support level (from 0—100, representing least to most support). This includes the following:

A extremely conservative Cat strongly opposed the policy (10),
A moderate Cat moderately opposed (45)
A moderate Dog moderately supportive (55)
An extremely liberal Dog, strongly supportive (90)

In this example we’re assuming that political alignment and support for gun control are closely aligned, which they actually are in reality (though not as much as you might expect…).

The example stimulus chart shows that the policy is relatively popular nationally, supported by 59% of U.S. Adults. How would this chart influence our four example citizens? We'd expect their attitudes to shift toward the reference value (the dot with 59, representing "All U.S. Adults"). The arrows represent the direction of their expected attitude change. For example, we'd expect the white cat on the far left to shift its attitude from 10 toward 59, becoming more supportive.

Let’s assume these four participants are influenced by the chart above. The chart shows that the policy is somewhat popular, supported by 59% of U.S. Adults. Three participants are originally less supportive than the norm shown in the chart (10, 45, 55 are less than 59). One participant is more supportive than the norm shown in the chart (59 < 90). If our theory holds up, we’d expect the three participants with less baseline support to increase their support after seeing the chart (right arrows). This is fairly intuitive; seeing that an idea is relatively popular can make it more popular.

We’d also expect the participant with high baseline support (90) to reduce her support (left arrow). This is maybe less intuitive; it implies that a relatively extreme liberal is reducing her support for a popular liberal policy, but in this case conforming with the norm of “All U.S. Adults” means moderating her relatively strong belief.

The example stimulus chart shows that the policy is popular with Democrats (with 85% support) and unpopular with Republicans (with 33% support). How would this chart influence our four example citizens? We'd expect their attitudes to shift toward the reference values for their respective parties. The liberal dogs would shift toward the Democrat reference point (the blue dot) and the conservative cats would shift toward the Republican reference point (the red dot).

What happens with this “partisan split” chart? Instead of just showing overall national support, this chart also shows that the policy is popular with Democrats (with 85% support) and unpopular with Republicans (with 33% support). If our participants had seen this partisan chart, we’d expect slightly different effects.

Notice that the moderate conservative cat’s (45) attitude is now shifting in the opposite direction, decreasing from his baseline support (left arrow). This is because he sees that policy is unpopular with his political group.

For the other three participants, their directions of attitude change are the same as if they’d viewed the consensus chart (converging toward the middle). The extreme conservative (10) becomes more supportive, the moderate liberal (55) becomes more supportive, and the extreme liberal (90), moving toward the middle, decreases her overall support because his baseline attitudes are so extreme that he’s presumably surprised at how low the policies support is with fellow democrats.

This gets complicated because this single chart could lead to possibly alternating directions of attitude change, depending on participants’ political alignment and the values shown in the chart. Notice that the arrows above alternate directions (right, left, right, left).

These charts show expected attitudes to bias across the political spectrum. The x-axis shows participants' political alignment (0 = most liberal, 100 = most conservative) and the y-axis shows their expected attitude bias toward the various policies we surveyed (average responses for people in the treatment vs the control). For example, when participants view the consensus chart showing a liberal policy like banning high-capacity magazines, we might expect the most extreme liberals to reduce their support for the policy (negative bias) as they adopt a relatively weaker consensus attitude. We'd expect the moderate liberal and two conservatives to increase their support, to match the national consensus.

Since we can use participants’ political alignment as a proxy for their baseline attitudes towards gun policies, this is another way we could look at the previous two examples we just walked through. Above the x-axis represents participants’ political alignment, and the y-axis represents the direction of attitude change. The dotted lines show the rates of change we might expect if our theories were true. Notice the directions of the arrows match our previous examples, but they’re flipped and turned 90 degrees.

So what results did we actually see?

Results

The results for Experiment #1 are messy. They suggest some kind of social conformity effect, where participants indeed changed their attitudes in response to their identity groups, but we can’t yet speak to polarization.

Partisan charts increased conservative's support for both conservative and liberal policies, shifts that are both polarizing and depolarizing. Consensus charts increased moderate's support for popular liberal policies, shifts that are depolarizing. Plots of reported attitude bias toward different sets of policies (y-axis: the difference between reported attitudes for the treatment, minus control, using estimated marginal means) as a function of their political alignment (x-axis:0 = most liberal, 100 = most conservative). The dashed lines show loosely interpreted biases we might expect based on the theory. Uncertainty bands indicate 95% CIs. Positive bias indicates higher than expected support, negative indicates lower than expected. The left panels show responses to liberal policies (e.g. banning assault weapons), the right panels show responses to conservative policies (e.g. expanding concealed-carry). Stars indicate significance at p<0.05.

The plots above show participants’ response curves (color lines) compared to our theoretically expected results (dashed lines). The stars indicate places where the bias was significantly larger than zero, indicating a subset of participants who were significantly influenced by the charts. This happened in three different places, each of which are roughly where we’d expect, given a social conformity effect. This suggests that participants perhaps updated their attitudes to better align with their social groups. However, the curves don’t line up exactly with expectations, or show clear separation from zero for other political segments.

Even though 2/3 of these results could be interpreted as polarizing or depolarizing influences (if you squint hard enough), we shouldn’t read too much into them except to say “Hey look, these charts affect viewers attitudes, even though they shouldn’t.”

After running this experiment we realized quite a few things about our setup that might have muddied the results. So in Experiment 2 we made some adjustments and were able to get a much clearer signal.

Experiment #2a:
Social Conformity

Are visualized attitudes contagious?

Motivation

Our results from Experiment 1 suggested a social conformity effect, but they weren’t particularly clear cut. In Experiment 2 we made a few changes to better suss out the effect. As a reminder, the social conformity effect would be interesting for two reasons:

On its own, it implies that an important dynamic in social psychology translates into charts. It would mean that charts showing something is popular have the power to make it more popular.
The effect is also interesting because it’s a prerequisite for how charts like these might influence political attitude polarization.

Setup

Again, we have a demo here that you can walk through: Experiment #2 demo. You can change the assignmentId value in the URL to generate different variations of the stimuli.

Experiment 2 stimuli. Note, this data is plausible, but again fake. In the experiment, the charts were based on randomly generated data.

We tested four different conditions this time. The partisan and consensus charts are similar to last time, except instead of viewing a stack of policies like Experiment #1, participants only saw one policy at a time.

The partisan chart also changed slightly: It no longer shows the overall national popularity, only the popularity for each political party. We also added a text condition, similar to political science “partisan cue” experiments. And we changed the control to a stock photo.

One of the challenges with measuring conformity is that it depends on the relative positions of participants’ prior attitudes and the position of the reference group attitude,shown on the stimulus chart. The bigger the gap between your starting attitude and a proposed attitude, the more the proposed attitude will influence you. (This is based on an interesting social psychology theory of attitude change called Social Judgment Theory).

In Experiment #1, since we used a static chart based on realistic public opinion results, the gaps between participants’ starting attitudes and what they saw on the chart were quite small (since Experiment #1 was essentially repeating the original survey used to generate the stimulus charts).

So in Experiment #2, we needed a way to test larger gaps between participants’ starting attitudes and what we showed them in the charts. We made two changes to handle this:

We switched away from gun control to policy topics that weren’t as well known, so we could make up random values without raising suspicion.
We generated the charts dynamically, so we could test a distribution of stimuli values. This would let us look for a relationship between the stimuli values we showed and the response biases from participants. If that relationship exists we could conclude the values in the charts were the cause of the bias.

What are we looking for?

These charts show theoretically expected attitudes bias for each stimulus chart type, across the range of values that participants might have seen in their stimulus charts. The x-axis shows the value participants saw in their stimulus chart (<50 is generally opposition, >50 is generally support) and the y-axis shows their expected attitude bias toward the policy shown in the stimulus chart. For example, in the middle plot, if a liberal participant sees that 69% of democrats support a policy, we'd expect them to also show increased support for that policy, relative to participants in the control condition who saw no information about other liberal's support for the policy.

If there’s a social conformity effect, we’d expect to see a bias in participants’ responses that’s proportionate to the popularity shown in the stimulus charts. So we’d expect responses to fall along the dashed line in the charts above.

Results

Experiment 2 results: Visualized attitudes influenced reported attitudes. For all three treatment conditions, participants’ attitudes biased toward their visualized in-groups’ attitudes. These plots show participants’ bias in their reported attitudes toward various policies (y-axis: the mean difference between reported attitudes for the treatment, minus control, using estimated marginal means) as a function of their in-group’s visualized attitude (x-axis, 0 = in-group opposes, 100 = in-group supports). Positive bias values indicate higher than expected support, negative indicate lower than expected support. The uncertainty ranges indicate 95% confidence intervals. Stars indicate significant differences at p<0.05.

We found a fairly clear relationship between the values generated for the stimulus charts and the bias in participants’ responses. When participants saw that their group supports a policy, they increased their support, and vice-versa.

The results for the consensus chart (left, above) show that if a policy is perceived as nationally popular, then it was more supported by participants. Note this has a more compact x-axis than the other two (from 30—70). This was for the sake of realism. By definition, even for hotly contested policies like gun control, the national consensus will still fall somewhere in the middle of the range.

The results for the partisan chart (middle, above) show that if a policy is perceived as popular with a participant’s political party, then it gained more support from participants in that party. Note that the x-axis here corresponds to the value we generated for the participant’s party.

The partisan text chart (above, right) has the same upward slope indicating that seeing party support is also associated with higher participant support. This is expected based on prior political science research. Here the curve tapers off more aggressively though. This is presumably because the text condition doesn’t actually quantify popularity with either party, it just says that it’s “supported” or “opposed.” So if we read into that chart, it might mean that the word “supports” in the text condition corresponds with 75-ish percent support for the population?

What this suggests, independent of polarization, is that public opinion charts like these have a social conformity effect and the attitudes they show can be contagious. If a chart shows that something is popular, then it becomes more popular with people who see the chart.

Experiment #2b:
Polarization

Can these charts influence polarization? Here we analyze our results from Experiment #2, testing for polarization.

Motivation

The example stimulus chart shows that the policy is popular with Democrats (with 66% support) and unpopular with Republicans (with 33% support). How would this chart influence our two moderate citizens? We'd expect their attitudes to shift toward the reference values for their respective parties, and diverge away from each other.

We’ve covered why polarization is a big deal. In the previous analysis we also showed that partisan charts were influential in ways that should lead to polarization.

If these charts nudge viewers’ attitudes toward their parties, and the two parties happen to take opposite positions on an issue, then it seems fairly reasonable to assume that people in those parties would diverge away from each other. Presumably if the above happens for two people, we should see it for whole groups of people.

The example stimulus chart shows that the policy has mixed popularity, with support from 51% of U.S. Adults. How would this chart influence our two moderate citizens? We might expect their attitudes to shift toward the more moderate reference value in the middle (but we'd be wrong!).

On the other hand, in the previous analysis we saw that the national consensus charts can also be influential. Since these charts represent the national consensus (by definition), we might assume that if people from different parties shift toward the consensus view then their attitudes would converge toward each other and become depolarized.

Setup

We’re still using Experiment 2 data, we’re just analyzing it a different way.

What are we looking for?

The example stimulus chart shows that the policy is popular with Democrats (with 66% support) and unpopular with Republicans (with 33% support). How would this chart influence a whole population of citizens? We'd expect the average attitudes for each party to shift toward the reference values for their respective parties, and diverge away from each other. That is, we'd expect the gap between their typical attitudes to grow wider.

We talked about different types of polarization, but to get more specific: We’re looking for issue polarization, where two opposing groups’ aggregate attitudes diverge away from each other (for some set of issues).

To be clear, this doesn’t necessarily imply a dynamic like “extremism.” It doesn’t even imply that the parties are necessarily moving toward more galvanized views. All we’re looking for are the groups moving away from each other, so it would still count as “polarization” if one of the groups happens to shift toward a more moderate or ambivalent position.

For example, one of the policies we tested was “requiring self-driving cars to always have a person in the driver’s seat, who can take control in an emergency.” This policy is oddly quite popular with both parties, but when we showed that it was unpopular with democrats, then liberal participants showed less support for the policy. That is, they became more ambivalent toward the policy. But since this movement was away from the Republican position, we’d still consider it polarization because it implies the two parties are increasingly misaligned.

The example stimulus chart shows that the policy has mixed overall support with U.S. Adults. How would this chart influence a whole population of citizens? We might expect the average attitudes for each party to shift toward the national reference value. Since this value is moderate, by definition, we'd expect their attitudes to converge toward each other. We'd expect the gap between their typical attitudes to shrink.

To show depolarization, we’d want to see the opposite shift, where the average attitudes converge toward each other. Here we’re looking for typical group attitudes to converge toward each other. The gap between the parties should shrink.

Results

Experiment 2 results: The partisan range chart led to significantly more divergent polarization than the other three conditions. The horizontal bars show the mean inter-party attitude distance (gap) between left- and right-leaning participants. The symmetric distributions on the ends show the bootstrapped samples of how wide the bars could be. Bars are centered horizontally to avoid implying changes in absolute attitude positions for one particular party. Stars indicate significant differences-in-gaps from control based on non-overlapping CIs (* = 95%, *** = 99.9%).

The results were quite stark. The chart above shows the average attitude gap that we calculated across all 5 policies (using a procedure called bootstrapping). The attitude gap jumped 69% from the control to the partisan range plot (from 11.7 to 19.8 points). The results show quite clearly that visualizing polarized partisan polling results can be divisive.

The gap for the partisan range plot was not only significantly wider than the control condition, it was even wider than the partisan text condition, implying the chart was uniquely polarizing. We have to be careful how we interpret this, because the partisan text and partisan range plot conditions aren’t exactly apples to apples. The text condition is not only a different medium (text), it also only says whether the parties support or oppose the policy without conveying the magnitude of support. So the difference might be the chart, or it might be the fact that the chart gives more granular detail. Either way, this shows that the chart has an even stronger effect compared to how these things are usually captured in political science survey studies.

Disappointingly, the consensus chart didn’t lead to depolarization. This might be because the consensus chart somehow had a weaker effect, or some aspect of the experiment design (e.g. a side effect of randomly varying the consensus value shown on the chart). Having said that, the consensus chart at least didn’t make polarization worse, so it’s arguably the least risky option we’ve found in terms of polarization.

Impact: What are the stakes? Is this a problem?

In this study, we show that public opinion charts can unreasonably influence viewers’ attitudes. And when these charts show polarization, they can influence attitudes in a way that increases polarization.

We’d still love more research to say how widespread these effects are in practice, but we can be reasonably confident they’re “real.” The significant (and substantial) attitude shifts would be hard to explain otherwise.

The charts showing that a policy was popular made it roughly five points more (or less) popular on a 100-point scale.
In terms of polarization, the partisan charts increased the gap between parties by over half (around 68 percent).

These effects suggest a number of social risks.

Risk: Increased Polarization

The first obvious risk is increased issue polarization, which we show directly. This applies to the partisan charts that increased issue polarization between people from opposing political parties.

Increased “polarization” isn’t intrinsically a bad thing. There are reasonable arguments that polarization is okay, or at least acceptable. So we’re not saying these charts are evil, just that they have risks we shouldn’t overlook. We might be okay with those risks. For example, if these charts increase polarization but also increase overall support for needlessly controversial issues like climate change or vaccination, perhaps they’re net beneficial.

Another caveat is that “issue” polarization doesn’t necessarily imply other types of polarization. For example, it’s not clear from our results how seeing policy disagreements impacts viewers’ feelings toward people from other parties (i.e. affective polarization). Other research suggests these are distinct phenomena and that different styles of reporting polarization can increase affective polarization while actually decreasing issue polarization; when reporting on individual “exemplars” of Democrats or Republicans who are particularly unreasonable, viewers become more reasonable in their own policy judgments but dislike the other party even more.

Even with the caveats, increasing issue polarization is still risky. To the extent that politicians’ and the media’s issue positions reflect their constituents, a more issue-polarized electorate can lead to more issue-polarized elites who are beholden to extremists and struggle to effectively govern. Or, even if “issue” polarization is a downstream consequence of “affective” polarization, accelerating issue polarization can still have dramatic consequences.

To the extent that we’re wary of increasing political polarization, we should also be wary of producing or publicizing partisan polling charts.

Risk: Laundering Misinformation

Polling charts can be risky without increasing polarization. For example, even though our consensus charts didn’t impact polarization, they were still influential. In all three of our experiments, regardless of chart condition, we found that if a chart showed that some policy is popular, the chart made that policy more popular.

Very few Venusaur fans believe in human-made climate change. How would this charge impact other Venusaur-sympathetic Poké Trainers?

Since political and social psychology suggest that the substance of an issue matters less than which groups endorse it, we might assume that this social contagion effect applies just as easily to bad ideas as good ideas. Since some perniciously silly ideas (e.g. climate change denial, vaccine conspiracies, wearing crocs in public) actually can be quite popular with certain subsets of the population, visualizing those groups’ attitudes might legitimaize these ideas and make them even more popular.

Risk: Undermining public health (equity)

Attitude contagion effects might also apply when charts unintentionally convey popularity. For public health, this might apply to charts that show intervention disparities between different social groups.

Young people are boosted at a much lower rate than older people.

For example, this chart is intended to show a failure in the healthcare system to effectively reach younger demographics with Covid boosters. One way they could be misread though is that boosters are somehow less popular with young people (e.g. “boosters are only for old people”).

Since we know that popularity is contagious, charts like these might actually backfire and lead younger people toward further reluctance about getting boosted. Other research supports this argument, showing that behaviors can indeed be influenced by visualized social norms.

Risk: Undermining public knowledge

This last risk takes some unpacking. Visualizing public opinion polls is arguably unethical. These charts can promote epistemically irresponsible attitudes and beliefs. They can increase our convictions without increasing our knowledge.

To unpack this, let’s quickly recap ~2400 years of information ethics. Going all the way back to Plato, epistemologists (philosophers of knowledge) have argued that we can’t really know something unless we have good reasons to believe that it’s true. In 1877, W.K. Clifford, an epistemological ethicist (a judgy philosopher of knowledge) went a step further, arguing that belief without reason is actually unethical, because our beliefs affect everyone around us, at least indirectly. More recently in 2013, psychologists describe these unjustified beliefs as an “illusion of explanatory depth,” corroborating Clifford’s arguments with a relevant example of how one person’s attitudes can affect everyone else, suggesting that Illusory knowledge is itself a source of political extremism.

Based on Clifford, since the charts we’ve studied can influence someone’s support for a particular idea without justifying why that idea deserves their support, the charts are unethical because they promote unjustified beliefs.

Practical Implications

For survey researchers, political data journalists, and data visualization designers, what should we do differently?

Expect harm. There’s a journalistic norm that, as long as information is “true” and “important,” publishers are absolved of responsibility for any harmful outcomes from their reporting. The Times’ publisher advises journalists to be ”profoundly skeptical” that harms are anything more than the “subjective views” of sneaky political actors trying to undermine independent journalism. Our work suggests this consequential skepticism is misplaced. Even ostensibly neutral information like polling results can carry social risks that are worth conscious consideration. Further, since both election and issue polling are now empirically linked to democratically distortive consequences, political journalists may be better served by assuming political polling results are intrinsically toxic.

Weigh the (social) risks. Polling charts don’t just passively convey survey results, instead they actively influence the attitudes they visualize. This implies that producing or publicizing these charts is inherently risky. It can incur social costs like increased polarization or spreading silly, unjustified ideas. It’s not enough for charts like these to be clear and accurate. Instead they need to at least provide enough information value to offset their harm.

Don’t show that something is popular unless you’re okay with making it more popular. Public opinion doesn’t have to be “right” to be influential. Some people think the earth is flat. There may be polling results showing that this silly idea is quite popular within the surveyed group. If we were to highlight that finding, we’d risk making the silly idea even more popular. We don’t have to censor results like these, but to the extent that we reasonably believe that the earth is round, we should think twice about amplifying opinions that say otherwise.

Explain why attitudes are popular. Even though the non-partisan, consensus charts don’t increase polarization, they’re still epistemically sketchy since they can spread beliefs without justifying them. One obvious way to address this is offering justification. That is, when sharing polling results, it’s insufficient to just show that something is popular. There also needs to be context explaining why an idea deserves to be popular (or not). Like most data, pairing results with the appropriate context will help viewers to form their own attitudes, ideally based on stronger justifications than “all my friends are doing it.”

Resources and Next Steps

We’ll present our results at the 2023 VIS Conference in Melbourne this October and post the video here. In the meantime, feel free to dive into any of the following links.

Our pre-print is available here
Data and analysis code are here
You can see our talk teaser trailer here on YouTube. This presentation was designed in collaboration with the brilliant Gabrielle Merite
There’s also a brief background post here: A political psychology primer for information designers.
For inspiration on depolarization, check out the Strengthening Democracy Megastudy

Huge thank yous to Gabrielle Mérite for art-directing the presentation and curating stylishly liberal dogs. And Cindy Xiong for advice and guidance throughout the project.

Research Results Presentation Design

Fri, 01 Dec 2023 00:00:00 GMT

Talk Recording

Context

Collaborators

This was a collaboration between Eli Holder (3iap), Cindy Xiong Bearfield (Georgia Tech), and Gabrielle Merite (Figures & Figures).

Prompt

How might we present new research about data visualization and political polarization in a way that’s sufficiently contextualized, but still engaging and approachable?

Background

Public opinion polling charts dominate U.S. media coverage. While these visualizations drive engagement for news publishers, our 2023 IEEE VIS study “Polarizing Political Polls” found they may undermine the democratic process by reinforcing misbeliefs, promoting unconsidered policy judgements, and even encouraging political polarization.

We wanted a way to communicate these findings to non-expert audiences (i.e. non social scientists at the 2023 VIS conference in Melbourne), while navigating political sensitivities and conveying the gravity of our research.

Key Challenges

The "Polarizing Political Polling" study explored the impact of partisan charts showing public opinion divided by political party.

The 10 stimuli charts tested during the study.

The presentation needed to solve for three key challenges:

Complex subject matter. Some social psychology concepts like attitude change are deceptively complex. While social conformity might be simplified to “monkey see, monkey do,” understanding our research required an intuitive sense of more precise, dynamic models (i.e. Sherif and Sherif’s Social Judgement Theories on interpersonal influence).

Complex data. The main results were based on 1,962 participants’ responses to 7,560 dynamically generated charts, looking across 5 political policy topics, 6 different charts types, and multiple signals of political ideology. Much to the chagrin of “Reviewer #2,” our models were necessarily complex. While scatter plots and regression lines could straightforwardly show charts’ influences on participants’ attitudes, showing unexpected emergent effects like “polarization” meant capturing aggregate, relative changes between groups.

Politics are exhausting. No one wants to talk about politics at a dataviz conference. Even in 2023, the emotional landscape of political beliefs and polarization were distressing, precarious, and intrinsically divisive. And charged issues risk emotional barriers to learning.

Design

Jake and Jane were two of the characters we introduced to support the overall narrative and demonstrate social conformity effects.

An action shot of Jane influencing Jake. Jake moves along the axis toward Jane, representing his attitudes converging towards Jane's stronger position.

Insights

Narrative. While social influence could be modeled as a complex network of causes and effects, our simple human brains prefer stories. Narratives can be a surprisingly effective way to convey complex causal influences (Sloman 2017, Niederdeppe 2013).

Motion. Change is easy to comprehend when you can see it happening. Animated visualizations can help audiences grasp dynamic systems more intuitively than static charts alone.

Fables are safe. Animals are cute. Following storytellers from George Orwell to Sesame Street, sensitive topics are more easily discussed in the animal kingdom. While divisions between “Democrats vs Republicans” are tired and painful, “Cats vs Dogs” are an even more recognizable rivalry, evoking similar loyalties, but with much lower stakes.

The Presentation

Slides from the Polarizing Political Polls IEEE VIS 2023 presentation.

The talk followed the classic arc of research presentations: Motivating context and key concepts up front, create tension with a knowledge gap, then get to the science.

Motivation And Key Concepts

To motivate the study and cover background concepts, we started with a story about a controversial policy in the animal kingdom (“The Canadian Can Ban Plan”) and then introduced our characters: Jake the moderate dog, Jane the wise influencer (also a dog), and Tom the moderate cat. We used visual metaphor to cover attitude conformity in the model of Sherifs’ Social Judgement Theory: We posed Jake and Jane on an attitude scale, then literally shifted Jake toward Jane, as his attitude converged towards hers.

We used this same narrative world to introduce the stimuli charts and open questions for the research (e.g., “Could charts influence Jake like Jane did?”). And as the questions became more complex we built up more complex diagrams iteratively (i.e. adding more cats and dogs to represent the group dynamics).

Visualizing The Results

Slide showing the results of policy polarization, both in terms of dogs and cats moving away from each other, but also, in parallel, the divergence diagram shows a widening inter-party attitude gap.

Slide showing experimental results for stimuli charts' influence on inter-party attitude gaps. In the presentation this slide appears after stepping through each result individually, to create an impression of the blue and red distributions moving away from each other.

There were two important parts to showing the polarization dynamics.

First, the “attitude gap” chart shows the amount of polarization driven by each stimuli chart, encoded as the distance between the blue and red dots on each row.

This chart could have been a bar chart. Showing these values as left-aligned bars would even make the gap sizes easier to compare. However, given this dataset with such obviously wide gaps for the partisan charts, we chose a non-traditional approach to afford a more immediate gist read.

We split the uncertainty distributions to either side and, even though they aren’t specifically tied to either party, we colored them blue and red to signal that the gap is indeed between Democrats and Republicans.
The plots are also centered horizontally to avoid implying that one party was more susceptible to this influence than the other, a more subtle question not answered by this analysis.
This also parallels the visual metaphor we used for showing influence on Jake and his animal friends.

While these choices preclude conventional affordances (e.g. an x axis scale), we consider them worthy tradeoffs for an easier gist read, making it unavoidably clear that partisan charts increased partisan polarization.

Second, pairing the divergence diagram with our friends from the animal kingdom created a natural way to teach audiences how to read the charts, in real time. We also show the same movement again after showing the results slide, to reinforce the impact of the specific partisan chart we studied.

Results

The combination of narrative elements, illustrations, and atypical data visualizations enabled us to:

Make complex social psychology concepts accessible
Keep audience engagement through emotional safety and humor
Demonstrate complex results through approachable visual metaphors

This let us convey both the depth of our research and its broader implications for data design and democratic discourse.

Through a Partisan Lens: How Politics Overrides Information.

Tue, 26 Sep 2023 00:00:00 GMT

As information designers, we don’t typically think of our work as political. Our first loyalty is the data. Our job is wresting big, complex ideas out of the platonic ether and squeezing them into two or three dimensions, on a screen or maybe a poster, so that people can better understand the world around them. Normally we worry about challenges like information architecture, dimensionality reduction, or weaving seemingly disparate facts into a cohesive narrative.

But for the most important issues of our day, politics are a crucial lens into how people see the world, and this can impact how they see data.

For example, consider an influential study from researchers at Yale, looking at how political alignment can create blind spots even for analytically savvy participants. Participants were shown one of two different data stories: one on the efficacy of a skin cream for curing a rash, the other on the efficacy of gun control policies for stemming gun violence. The trick: Both stories were based on the exact same underlying data. So if participants read the data to say the skin cream was effective, they should rationally also conclude that the gun control policies were effective. But that’s not what they did.

Even for this highly numerate crowd, responses became polarized along participants’ political party lines. Instead of objectively following the data, when participants saw the politically charged topic, they couldn’t help but interpret the data as evidence to support their prior political positions.

To design effectively, it’s important to understand not just how to construct a clear chart, but how people will actually interpret them. Since politics can be so distorting, it’s worth understanding how it shapes our interpretations. To do this, we’ll unpack the social and political psychology that drive our attitudes and beliefs about big political issues.

Why should information designers care about political partisanship?

Effective dataviz means designing for more than just the data on the page. The context that viewers bring to a visualization can shape how they respond to it. In our politically charged culture, the topics that need the most explaining are often the most political.

Whether we like it or not, the information that we present will be consumed through a partisan lens. By understanding these processes, we can at least address them consciously. This can help in a few ways:

Adapt to a “fact-free” universe. Information design is premised on information being helpful. But in the bizarro world of politics, attitudes and beliefs aren’t always strictly tethered to any ground truth. Understanding cases like these, when information isn’t useful, can help us choose our battles and prioritize our time for the biggest impact.
Persuade people with people. When reasons fail, people look to others for guidance. For political issues, we’re heavily influenced by the people around us. Understanding how attitudes can spread through dataviz can help us produce more persuasive visualizations.
Minimize the harmful side effects of well-intended dataviz. Information can do more than just inform. For example, partisan issue polling charts can increase political polarization. Understanding these unexpected risks can help us mitigate them.

In a partisan environment, if our ideas and decisions aren’t strictly based on information, where do they come from? To understand this we’ll dive into social and political psychology.

Political information psychology

Understanding social and political psychology can help clarify the boundaries of information’s influence. As we’ve already suggested, the facts aren’t always as persuasive as they should be.

On the other hand, some information can be influential in ways that it shouldn’t be.

Social Influences

Some city dwellers looking up, reenacting Stanley Milgram's famous "Drawing Power of Crowds" experiment

Social Conformity: If you look up, I look up.

It’s almost a cliche to say that humans are social creatures, but that doesn’t make it untrue. We are comically influenceable by the people around us.

For example, in a famous social psychology experiment from the 1960s, Stanley Milgram sent his research team out onto the city streets of New York City. He instructed them to find a crowded part of town, stop in the middle of the sidewalk, and just look straight up at the sky.

When his team stopped in the middle of the sidewalk and looked straight up, all the busy people walking past not only noticed the researchers’ upward gaze, they also stopped and looked up to see what was up there.

Other silly experiments show similar social conformity effects.

Why are MBAs conservative, and social scientists liberal?

Our social surroundings also influence our theories about how the world works, what we believe in, and what we value.

For example, one study followed 91 students throughout their college careers. 34 of them were business majors and 57 majored in a social science. The researchers wanted to understand how the students’ majors influenced their beliefs, particularly how they explained the causes of poverty and unemployment.

During their first year, students’ majors were uncorrelated with their beliefs, but by the third year, business school students were disproportionately likely to blame poverty on the impoverished while social science students pointed to external, systemic factors.

For these students, the embedded cultural values and beliefs of their course-work and environments influenced their beliefs.

Group Influences

An expressionistic interpretation of youthful tribalism, inspired by Henri Tajfel's classic study.

Expressionism’s divisive influence on our youths

The silliness continues when considering the special privilege we give to people who are like us.

The classic experiment highlighting tribalism showed how a group of adolescent boys, with long histories as classmates, were transformed into opposing factions when researchers assigned them to different groups based on their self-reported affinities for Paul Klee or Wassily Kandinsky paintings. Despite the boys’ shared history, when researchers asked them to divide up their participation rewards amongst their classmates, suddenly their prior friendships meant very little.

Instead the boys shifted their allocations dramatically toward their new-found brothers-in-art. This is not to suggest that the nuances of Kleesian vs Kandinskian expressionism were a hot topic for these high schoolers (behind the scenes the researchers assigned their groups arbitrarily).

Instead, this demonstrates how even the most arbitrarily constructed social groups can produce in-group favoritism or outgroup discrimination. In fact other experiments showed similar results when the groups were based on nothing more than a coin toss.

We like people who are like us, even if all we have in common is mutual disdain for some other group of people.

Common ground beyond politics

These social group effects are presumably stronger for political groups, where party members actually have real things in common. Political psychology research suggests that we share some very primal psychological traits and needs with our fellow partisans.

Political psychologists suggest that Conservatives place great value on feelings of security and certainty (while liberals are comfortable with uncertainty, ambiguity and risk). Conservatives also value uniformity in their social groups, while liberals value differentiating themselves. Perhaps because of these low-level psychological needs, members of today’s political parties have a lot in common with their fellow partisans (e.g. particularly for U.S. Republicans, where this also applies to their white, Christina, rural demographics).

This is the basis for the “identity stacking” theory of polarization. This theory observes that more and more of our identity traits have lined up with our political identity. For example, if you know that someone is a Democrat, that also likely means you’ve got good odds on guessing not just their views on climate change, but also which parts of the country they live in, how long they spent in school, how confident they feel about the economy, or whether or not they’re armed.

If we’re influenceable by people who are like us, and we have more and more in common with other people in our political party, then we’d expect our fellow partisans to be particularly influential.

Political Attitude Formation

Do different parties have different attitudes on Canadian imports?

One thing we all have in common: We’re busy. And we’re tired. (So so tired.) Even if we have the interest, very few people have the time or energy to dive into the guts of tax policies, environmental regulations, or the extended implications of Citizens United. These aren’t necessarily personal failings though. Even for policy makers or people who study public policy, there literally isn’t enough time in the world for a single person to personally, critically research all the issues they might care about. Even if they did, there are rarely clear answers. Policy choices are intrinsically big, complex and multi-faceted.

So, for very practical reasons, people form their attitudes and judgements by listening to other people that they trust.

An example of an influential "partisan cue" used in political science research. Note: This isn't actually true, this policy generally has bipartisan support. What effect do you think this has on viewers who identify as Democrats or Republicans?

In particular, we look to our political parties to tell us which policies we should support and which ones we should oppose. For example, in “partisan cue” studies, researchers will choose some obscure policy that most people won’t have a strong opinion about (e.g. estate taxes, prescription drug imports, etc), then they’ll say something like “Democrats tend to support and Republicans tend to oppose importing prescription drugs from Canada.” Suddenly an otherwise bipartisan policy has a 10 point support gap between liberal and conservative respondents.

Do we choose our parties based on their policies? Or do we choose our policies based on our parties?

One interesting study stretches these partisan cue effects even further. The researchers told participants about one of two possible welfare programs, either a severely “stringent” program that’s far less generous than the ones we have today (e.g. $250 per month) or a “generous” program that’s far more generous than any available today ($800 per month + extra for health insurance, rent, childcare, job training and 2 years of college tuition).

From an ideological perspective, you’d expect conservatives to favor the former and dislike the latter, and liberals the opposite. However, researchers found that the content of the policy itself didn’t matter nearly as much as who endorsed it. For example, conservatives were willing to support either program as long as they were told it was supported by “95% of Republicans and 10% of Democrats.”

Instead of choosing political parties that match our ideas for how to govern, the process actually happens in reverse. We’re flexible on policies as long as they’re supported by our people.

This presumably makes life quite difficult for Trump supporters. Noting the fact that the former President has taken both sides on most issues, researchers tested the effects of Trump’s erratic policy statements on his supporters. For example, some participants saw that Trump favored abortion “penalties” while another cohort saw that he opposed it. The researchers found that, regardless of the issue, self-identified conservatives rallied to Trump’s position and the substance of the policy made very little difference. That is, their attitudes reflected their perceived political norms, not necessarily the underlying ideals of conservatism.

How can attitudes spread through dataviz?

As we’ve seen, our attitudes are influenced by the people around us. This is especially true for political judgements that are difficult to form experientially. It turns out that this same influence can happen through charts. For example, public opinion polling is a popular topic for political data journalism. What influence might we expect from charts like these?

This chart shows 67 percent of Americans oppose camo-crocs. This is fake data, but seems like a reasonable guess?

For example, consider the chart above. This chart shows pretend-results from a hypothetical public opinion poll on Americans views of camo-Crocs. Specifically, it highlights the overall popularity for a policy to ban them. Since this shows that the policy is generally popular, we might expect viewers who see this chart to identify with their fellow citizens and adjust their own attitudes to match the social norm shown in the chart. For people who were previously opposed to the policy, social psychology suggests that they’d increase their support. On the other hand, for people who were already very strong supporters, they might actually decrease their support since they see that others are more relatively ambivalent.

This example highlights an important concept: By showing that an idea is popular, charts can make the idea more popular. And vice versa.

This chart shows that Democrats strongly support and Republicans slightly oppose camo-crocs. This is totally fake data. Republicans surely also agree that camo-Crocs are ridiculous.

This chart shakes things up a bit. Now it shows the results from our hypothetical opinion poll split by political party. We can see the camo-Croc ban is very popular with Democrats and less popular with Republicans. These are effectively party endorsements, they’re just quantified and visualized. In the last section, we covered several experiments where highlighting a party’s endorsement of a policy changed viewers’ attitudes toward the policy, so we’d expect charts like these to have similar effects. So if a moderate Democrat sees this chart, we’d expect them to increase their support. If a moderate Republican sees the chart, we’d expect them to decrease their support. If a bunch of moderate Democrats and Republicans all see this chart, we’d expect their attitudes to diverge away from each other.

This example shows one of the potential consequences of attitude contagion. For partisan-split polling charts like these, we might expect people’s attitudes to become more polarized. To the extent that polarization is bad, it implies that charts like these have an inherent social cost. They may be valuable (or at least entertaining), but they’re not without risk.

Our recent research suggests that both of these scenarios are very real and that political polling charts can very much influence viewers’ political attitudes. When viewers see a chart showing that a policy is popular, that chart can make the policy more popular. When viewers see a chart showing that attitudes are polarized across party lines, that chart can make viewers more polarized.

So what? What should data designers and journalists do with this?

Alberto Cairo offers a useful maxim for ethical data journalism: “The purpose of journalism is to increase knowledge among the public while minimizing the side effects that making that knowledge available might have.” He summarizes the goal as: “Increasing understanding while minimizing harm.”

As we’ve seen, attitudes can spread from person to person, regardless of their actual content. This means that visualizing attitudes from survey results can have the unexpected side effect of promoting those attitudes. This can be risky in the context of political polarization, as visualizing polarized attitudes can increase polarization.

The social conformity effect can also be harmful in and of itself.

For example, imagine an interest group called “Dirty Handed Doctors of America.” Let’s say they survey their unhygienic-but-medically-credentialed members. Their main finding: “94% of physicians in our esteemed organization strongly agree we should stop washing our hands before treating patients.” The survey finding may, in fact, be totally accurate. Their opinion is wrong, but it could be true that 94% of them support it. Our research suggests that visualizing extreme attitudes like these might help them spread further (like the germs on their filthy, filthy hands). So even though their survey results might be technically true, publicizing them may reduce support for hand-washing among other sympathetic physicians.

This means that we can’t just assume, by default, that visualizing things like polling results are a civic good, simply because they’re accurate and informative. We have a stronger duty-of-care than simply conveying technically accurate information. Since visualizing attitudes comes with an implied risk, we need to consciously weigh those risks versus whatever benefits we expect from publicizing them.

Takeaways

Viewers’ politics can influence how they see the world. This, in turn, influences how they absorb new information. This has a few important implications for anyone visualizing social issues or otherwise politically-charged information.

Information isn’t as influential as it should be. If all of our attitudes and decisions were purely rational and information-based, the silly effects we highlight above wouldn’t exist. But in the real world, judgments about identical datasets can flip based on a person’s politics. Attitudes toward public policies are more influenced by endorsements than the policies themselves. Information is still influential, but the surrounding social context should be considered as well.
Information can be influential in ways it shouldn’t be. Since our own political attitudes are so easily influenced by others’, information about attitudes (e.g. polling results) can be quite influential. This influence can happen through simple partisan cues, like whether or not a party supports or opposes a policy. This also means that popular political data-journalism, such as election forecasts or issue polling, can have some toxic side effects like increased political polarization. Information designers should take these risks of attitude contagion into account when deciding what to visualize and how to frame their results.
The risks of information may outweigh the benefits. Information designers should take the risks of attitude contagion into account when deciding how to frame polling results, or whether to visualize them at all. In most cases we won’t be able to predict the outcomes, or have clear ethical guidelines on their implications, but by raising the question in the first place we can ensure judgment calls like these are at least made consciously and thoughtfully.

Dive Deeper

This writeup is meant as a primer for 3iap’s latest peer-reviewed visualization research, which we presented at this year’s IEEE VIS conference, in collaboration with Georgia Tech’s Cindy Xiong-Bearfield. If you’d like to better understand the pathway from polling charts to polarization please check out our deep dive on the research project.

Dive Deeper: Polarizing Political Polls Design Research Project

How to create Jitter Plots in Microsoft Excel to visualize social outcome disparities

Fri, 28 Apr 2023 00:00:00 GMT

Screenshot of a jitter plot created in Microsoft Excel, showing regional differences in new HIV diagnoses

After almost every Equity Dataviz workshop, participants inevitably say something like: “Great! We’re sold on equitable charts. Jitter plots seem fun! But how do we do this in Excel? Or Google Sheets? Or R? Or Python?” This post answers that question for spreadsheets like Excel or Microsoft Excel.

Why use Jitter Plots to visualize social inequality?

Visualizing outcomes for different social groups (e.g. race, gender, age, income, etc) is a common first step in analyzing social inequity and holding policy-makers accountable for the decisions behind these disparities.

3iap’s peer-reviewed research shows that, when visualizing social outcomes, certain types of visualizations can increase harmful stereotypes about the people being visualized.

Some visualizations (e.g. bar charts, dot plots) make it seem like differences between groups are much larger than they really are. This triggers unconscious social biases that can lead to unfairly blaming the outcome differences on the people themselves.

On the other hand, charts like jitter-plots or range-plots make it clear that there are also wide outcome-differences within groups. This makes it difficult to stereotype a particular group because viewers can see that, even though there are differences between groups, a person’s group identity only explains a small part of the differences.

In this post, we’ll look at how to visualize between- and within-group-differences using Jitter Plots.

Example dataset

To make this realistic, we’ll look at differences in new HIV diagnoses between different regions in the United States, using a publicly available dataset from AIDSVu. The United States has a national goal of reducing HIV incidence by 75% before 2030. They’re making progress, but for a number of reasons, progress in southern states has been slow.

Walkthrough

First we’ll prepare the data. Above is the original spreadsheet from AidsVu, but I’ve added a column Region (see E:E) using a vlookup from the state-region-lookup sheet.

Next we’ll choose just the columns we want and filter the data using this formula: CHOOSECOLS(FILTER(CHOOSECOLS(Data!C:H,3,2,4,5,6), Data!H:H="Y"),1,2,3,4)

With CHOOSECOLS(Data!C:H,3,2,4,5,6), we’ve selected the columns for Region, State, County, New Diagnoses Rate, and CHOOSECOLS(Data!C:H,3,2,4,5,6)
With the filter Data!H:H="Y", we’re choosing only rows that are listed as rate-stable. AIDSVu discusses rate-stability in their methodology documentation.

Since Excel won’t let us use text values in a scatter plot, we create a column that represents each region as a value from 1-4.

We use a vlookup in column E:E to give each row an index value: =VLOOKUP(A2,$H$2:$I$5,2,FALSE). This looks up the region’s index from the table on the right.

We’ve also renamed the columns to clarify which columns drive which axes, for when we create the chart.

Next we’ll add a chart.

Select columns D:E
In the top menu, choose Insert > Scatter
You’ll see Excel lays out a rough scatter plot
To tidy things up, we’ll delete the chart title and x-axis title. We’ll also adjust the formatting for the Vertical Axis, setting the Bounds to 0-5 and the units to 1.

How do we make the dots jitter?

Jittering helps make the underlying distribution more apparent.
First, we’ll add a constant to K2 to control the amount of vertical jitter on each line
Then we’ll modify our formula for column E, changing the formula to =VLOOKUP(A2,$H$2:$I$5,2,FALSE)+RAND()*$K$2-0.5*$K$2
- Remember =VLOOKUP(A2,$H$2:$I$5,2,FALSE) is how we look up the row index from the table on the right
- We’ve added +RAND()*$K$2-0.5*$K$2 which generates a random number from 0—1, then multiplies that number by our jitter amount constant K2. We subtract half of K2 to center it on the line.
Finally we drag the formula down column E to apply it to each row
You can see the dots on each row are now vertically jittered.

With jitter plots, people generally still expect to see what the “typical” value is for the dots (either an average or ideally a median). To show this using Excel we’ll have to get a bit clever.

Our plan here is to prepend the average values as part of the overall dataset shown on the scatter plot, but we’ll differentiate the average values by putting them in another “series”

First we claim column F to contain data for the average series that we can use in the chart
Then we’ll move the whole dataset down 4 rows
In column A we put the 4 region names
In column F we do a similar vlookup as we used in column E, to look up the row index for the region, i.e. setting F2 to: =VLOOKUP(A2,$H$2:$I$5,2,FALSE). Then we drag that down for the 4 rows.
In column D, we use averageifs to get the averge value for each region, i.e. setting D2 to: =AVERAGEIFS(D$6:D$366,A$6:A$366,"="&A2)
Now we’ve got the data in place to show the averages. Next we’ll update the chart…

To show the averages on the chart:

We open the chart settings
We change the source by highlighting from D1 down to F366
Then adjust the Y Values if excel doesn’t do this automatically
You can see the average values show up as orange dots.
You can see this has all the parts we need, but we need to do some cleanup to make it clearer.

The most important part of any dataviz is the way we label it. So let’s add the region names. To add labels to the rows, we just insert a textbox using Insert > Text Box. Then adjust the size of the chart so the rows line up with the text.

For extra credit we can use shapes to overlay a goal range and a small star icon, indicating the U.S. goal to reduce HIV incidence by 75%.

Next steps:

Hopefully this helps you show outcome variance when visualizing social outcome disparities (or any kind of group outcome differences).

A demo version of this spreadsheet is available here. Please feel free to copy and use as a template.
There may certainly be easier ways to accomplish the above. If you’ve got other good spreadsheet hacks, please send us a note at hi@3isapattern.com.
You can also use Google’s Gemini AI tool to generate similar charts: How to visualize social outcome disparities with jitter plots and Google Gemini
If you or your teams are interested in learning about more equitable dataviz in general, please check out 3iap’s Equity Dataviz workshop, where we’ll cover our latest research on the topic, unpack the underlying psychology of data-driven stereotyping, and cover alternative ways to visualize social outcome disparities (without making them worse).

How to make Jitter Plots in Google Sheets to visualize social inequality

Fri, 07 Apr 2023 00:00:00 GMT

Screenshot of a jitter plot created in Google sheets, showing regional differences in PrEP coverage

After almost every Equity Dataviz workshop, the first response from participants is inevitably something like: “Great! We’re sold on equitable chart design… but how do we do this in Excel? Or Google Sheets? Or R? Or Python?” This post answers that question for spreadsheets like Excel or Google Sheets.

Why use Jitter Plots to visualize social inequality?

Visualizing differences in outcomes between social groups (e.g. race, gender, age, income, etc) is an important part of analyzing social inequity and holding policy-makers accountable for the decisions that drive these disparities.

3iap’s peer-reviewed research shows that, when visualizing social outcomes, certain types of visualizations can increase harmful stereotypes about the people being visualized.

In this post, we’ll look at how to visualize between- and within-group-differences using Jitter Plots.

Example dataset

To make this realistic, we’ll look at differences in PrEP coverage between different regions in the United States, using a publicly available dataset from AIDSVu. For context PrEP stands for Pre-Exposure Prophylaxis. It’s a prescription that drastically reduces people’s chances of getting HIV. The United States has a national goal of covering 50% of at-risk populations by 2030. They’re making progress, but for a number of reasons, progress in southern states has been slow.

Walkthrough

First we’ll prepare the data.

We’ve copied the columns for State Abbreviation, State and State PrEP Rate from the raw data
We’ve assigned each state to a Region (see C:C)
On the right we’ve given each region an index that we’ll use later to assign regions to rows in the plot. We use a vlookup in column D:D to give each row an index value: =VLOOKUP(C2,$H$2:$I$5,2,FALSE). This looks up the index from the table on the right.

Next we’ll add a chart.

Select columns D:E
In the top menu, choose Insert > Chart
You’ll see Google Sheets defaults to some sort of histogram

Then we’ll update the chart settings on the right.

Change Chart type to Scatter chart
Change X-axis to State PrEP Rate
Change the first Series to Region Index
You can see we’ve got the start of a jitter plot, with each dot representing an individual state’s “PrEP rate”
We have a fairly extreme outlier though. This is Washington D.C.

To handle the Washington D.C. outlier:

We don’t want to clip the data point entirely but we also don’t want every other point to be so tightly packed you can’t see it.
We’ll “clamp” the value and say the max value is 250, then we’ll clarify what’s happening with annotations.
You can see this shrinks the x-axis and makes the other states’ values easier to see.

We want to make our treatment of the Washington D.C. outlier transparent:

We’ll change the shape of the marker to a star, to differentiate it from the other points (double click on the Washington D.C. point to edit the appearance of that single point)
We’ll also add a text annotation in the subtitle.
Note: We’ve also reversed the ordering of the region indexes so that the rows are in alphabetical order.

How do we make the dots jitter?

This isn’t a big deal for this dataset, since it’s only ~50 total dots. But for larger datasets jittering helps make the underlying distribution more apparent.
First, we’ll add a constant to K2 to control the amount of vertical jitter on each line
Then we’ll modify our formula for column D, changing the formula to =VLOOKUP(C2,$H$2:$I$5,2,FALSE)+RAND()*$K$2-0.5*$K$2
- Remember =VLOOKUP(C2,$H$2:$I$5,2,FALSE) is how we look up the row index from the table on the right
- We’ve added +RAND()*$K$2-0.5*$K$2 which generates a random number from 0—1, then multiplies that number by our jitter amount constant K2. We subtract half of K2 to center it on the line.
Finally we drag the formula down column D to apply it to each row
You can see the dots on each row are now vertically jittered.

With jitter plots, people generally still expect to see what the “typical” value is for the dots (either an average or ideally a median). To show this with Google Sheets we’ll have to get a bit clever.

Our plan here is to append the average values as part of the overall dataset shown on the scatter plot, but we’ll differentiate the average values by putting them in another “series”

First we claim column F to contain data for another series that we can use in the chart
Then we scroll down to the bottom
In column C we put the 4 region names
In column F we do a similar vlookup as we used in column D, to look up the row index for the region, i.e. setting F54 to: =VLOOKUP(C54,$H$2:$I$5,2,FALSE). Then we drag that down for the 4 rows.
In column E, we use averageifs to get the averge value for each region, i.e. setting E54 to: =AVERAGEIFS(E$2:E$53,C$2:C$53,"="&C54)
Now we’ve got the data in place to show the averages. Next we’ll update the chart…

To show the averages on the chart:

We open the chart settings on the right
We add another Series: This is column F Region (Y, Avg Series)`
You can see the average values show up as red dots.
You can see this has all the parts we need, but we need to do some cleanup to make it clearer.

To make this a bit clearer, let’s rename our columns (again!):

We’ll change column D to “State Avg”, since each blue dot represents an individual state’s average value.
We’ll change column F to “Region Avg”, since the average markers represent the average of averages for all states in the region.

To differentiate the average markers further:

We’ll go into the Customize > Series and change the shape of the marker to someting different than the circle. Outside of Google Sheets these should be big vertical lines, but we’ll settle for diamond shapes.

You can see it’s taking shape, but the row numbers are super confusing.

The most important part of any dataviz is the way we label it. So let’s replace the confusing row numbers with the actual region names.

Unforutnately Google doesn’t make this easy. There’s no clear way to use text as a y-value for their scatter charts.
Instead, we’ll just fake it by inserting a drawing. In the top menu, choose Insert > Drawing. In the drawing you’ll create a text box and type the name of each region on each line, then save and close the drawing.
To make room for our new drawing, double click on the y-axis, then change the font color to white. Then select the plot area of the chart and drag the left side over, to create a margin on the left.
Finally drag the drawing text over the chart. This is certainly not a glamorous solution to getting readable labels on the chart, but worthwhile given how much clearer it makes the story.

Next steps:

Hopefully this helps you show outcome variance when visualizing social outcome disparities (or any kind of group outcome differences).

A demo version of this spreadsheet is available here. Please feel free to copy and use as a template.
There may certainly be easier ways to accomplish the above. If you’ve got other good spreadsheet hacks, please send us a note at hi@3isapattern.com.
You can also use Google’s Gemini AI tool to generate similar charts: How to visualize social outcome disparities with jitter plots and Google Gemini
If you or your teams are interested in learning about more equitable dataviz in general, please check out 3iap’s Equity Dataviz workshop, where we’ll cover our latest research on the topic, unpack the underlying psychology of data-driven stereotyping, and cover alternative ways to visualize social outcome disparities (without making them worse).

Absurd Print Editorial Visualizations

Wed, 01 Feb 2023 00:00:00 GMT

Context

Prompt

How might we illustrate examples of “motivational dataviz” in a way that’s entertaining and works for print?

Background

Nightingale Magazine is the Data Visualization Society’s beautifully crafted print publication, covering new ideas and compelling work in the data visualization community. They asked us to contribute an article and illustrations about the non-informational, motivational impacts that data visualizations can have on viewers.

Challenge

Designing for print is a unique, refreshing challenge. To work in this medium, each visualization needed to work as a standalone artifact and be relatively constrained in size and color. They also needed to be approachable and quick to consume to catch readers’ attention as they flip through the magazine.

Key Insights

Big, Absurd Ideas

Keep it light. While each visualization illustrates a unique psychological mechanism, each one of which could fill a textbook chapter on data psychology, most readers would be new to the overall concept of affective visualization. So it was more important to keep things light and memorable (and maybe a bit silly) than attempt to capture each concept in realistic detail. Instead we used “stylized” charts showing the application of these mechanics based on different absurd scenarios.

Arias-Bolzmann et al, 2013: Effects of Absurdity in Advertising: The Moderating Role of Product Category Attitude and the Mediating Role of Cognitive Responses

Productive Absurdity. Absurdity can be a useful communication tactic for winning over skeptics and increasing memorability (Arias-Bolzmann et al, 2013). Given that dataviz is traditionally valued for its rational, sense-making benefits, but the article instead focuses on non-cognitive impacts that readers may be unfamiliar with, a more disarming approach should make these new concepts more palatable.

Design

Viz #1

Allcott & Mullainathan 2010: Behavior and Energy Policy: Investment in scalable, non–price-based behavioral interventions and research may prove valuable in improving energy efficiency.

Holder 2020: Motivational Data Visualization: Exploring the motivational impacts of 7 user-data visualizations in everyday products and services.

A silly reproduction of Sacramento Municipal Utility District’s electricity usage letter (via Allcott & Mullainathan 2010). The horizontal bar chart shows a customer’s consumption compared to their peers. Charts that compare a person’s outcomes to social norms are effective through a social-psychology process called “social normative conformity,” which is a fancy way of saying “peer pressure.” You can learn more about the power of social-normative influence in dataviz here.

Viz #2

Milkman et al 2021: Megastudies improve the impact of applied behavioural science

An absurd reproduction of Robert Cialdini’s social norms intervention for 24-Hour Fitness customers (via Milkman et al 2021). Charts like this also rely on social-normative influences.

Viz #3

Matthaei et al 2014: Consensus recommendations for the use of Ambulatory Glucose Profile in clinical practice

A silly reproduction of Matthaei et al’s Ambulatory Glucose Profile, showing a distribution of blood glucose levels compared to a target range over a 24 hour period (via Matthaei 2014). Charts like these are effective because of the psychology of feedback. The benchmark range makes this particularly effective through the psychology of goal setting.

Viz #4

This lineup of creatures was from a popular notch.me visualization (designed in collaboration with the amazing illustrator Andra Popovici), aimed at long-distance runners. The critters were ordered by the longest-documented distances that each animal actually travels in the wild. When users crossed a milestone, they “outran” the animal. For example, after running 10 miles, a user would see, “at 10 miles, you outran a GIANT PANDA (8.7 miles),” with an illustration of a grumpy, defeated panda bear.

Then to motivate the next milestone, users were informed of their untimely demise: “Unfortunately, 10 isn’t far enough. Grizzly Bears can last 15.9 miles… And this one ate you,” with an adorably bloody, highly-satiated grizzly bear.

Milestones like this are effective by breaking big goals down into more proximate, achievable goals enhancing a viewer’s sense of efficacy. Goal proximity is an influential component of the psychology of feedback.

Viz #5

A stacked ice cream icon array, comparing ice cream sales (scoops) to warm weather (suns), to put sales in the context of fair expectations. Quantiative feedback loops can be risky because they can create unhealthy fixations and expectations. Humanistic metric design can help frame outcomes against fair expectations.

Results

Nightingale Magazine

Digital and print copies of Nightingale Magazine are available from the Data Visualization Society here.

Twenty-Nine Years of Hot Air

Tue, 31 Jan 2023 00:00:00 GMT

Context

Prompt

The World Government Summit asks: “What Just Happened? What’s improved? What’s broken through? What’s gone supernova? Charting our development across many different metrics over a 10 29 year period to highlight the successes - and the bottlenecks.”

To reduce harmful emissions, and encourage a net-zero world, it’s important to understand where these emissions come from. To that end, we thought we’d use the WGS retrospective prompt to look back at sources of Greenhouse Gas emissions.

Click here to download the poster.

Challenge

How might we…?

How might we understand emissions in a way that promotes climate justice? While climate change is everyone’s problem, some countries have contributed to it disproportionately.
While absolute magnitudes of emissions are important for prioritizing opportunities, a country’s responsibility for the current climate crisis doesn’t necessarily scale with their overall emissions.
For example, India is one of the world’s largest emitters of Greenhouse Gases. But as the second largest country in the world, this reflects their size, not necessarily their relative responsibility.

Insights

In “Quantifying national responsibility for climate breakdown: An equality-based attribution approach for carbon dioxide emissions in excess of the planetary boundary,” Jason Hickel offers an insightful framework for determining a country’s “fair share” of greenhouse emissions.

If we can quantify how much a country should emit, we can draw comparisons to how much they actually emit.

This will allow us to explore absolute emission magnitudes, but with the needed context to avoid painting an overly harsh view of countries like India (with large populations, but low per-capita emissions).

Design

The Poster

This poster explores global cumulative greenhouse gas emissions from 1990 - 2019. Emissions for each country are shown with small, white rings filled with colorful puffs of air. Each country has 1 puff per 10M people. Puff sizes show typical emissions for those 10M people, over all 29 years, such that the total area covered by all puffs represents the total emissions for the country (each pixel ~=4Gt of CO2e).

The larger, color rings represent each country’s “fair share” of emissions, based on Hickel, 2020. Comparing total “puff area” to the area of each “fair share” ring shows how much excess CO2e a country has emitted (e.g. US / EU / Canada “overshoot” their budget, while India has low emissions relative to its size).

For comparison, the 30 biggest rings in the background represent total global emissions for each of the 29 years (also scaled to 1px ~=4Gt of CO2e).

Methodology

Data and Analysis

Data is from ClimateWatch, as cited by The World Bank which was originally referenced in WGS’ sample data.
“Fair Share” emissions were calculated as a proportion of average population, using Jason Henkel’s framework in “Quantifying national responsibility for climate breakdown”

Technology:

Javascript
Three.js / GLSL
Figma

What Can Go Wrong? Deficit Thinking in Dataviz

Sat, 23 Jul 2022 00:00:00 GMT

Above: An example of a “deficit framed” chart showing hypothetical racial outcome disparities in education. Avoid charts like this!

This article was co-authored by Eli and Pieta Blakely. It’s also posted on Pieta’s site here.

Numbers don’t lie, but they’re under no obligation to tell the truth. There are also no guarantees that audiences will look at our data and reach the conclusions we might expect. With data visualization, like all other forms of communication, it’s important to recognize the difference between what the data says and what audiences actually hear.

Nowhere is this distinction more meaningful than visualizing racial inequality.

When data represents groups of people it’s inescapably interpreted through the audience’s prior knowledge, assumptions, biases and stereotypes about the groups being visualized. This makes visualizing social data unique from, say, visualizing rocket trajectories, stock prices, widget sales, or turtle migrations.

Social biases are especially important to consider when visualizing data concerning people from minoritized communities. Generally the people consuming these visualizations won’t be members of the groups being visualized, they’ll be members of majority groups for whom the visualized groups are psychologically distant “others.” (e.g. In the US, white people are the majority, so even though a visualization may feature asian, black, hispanic or indigenous groups, it will primarily be viewed by white people.)

Interpreting information about “other” groups can have some surprising consequences. For example, in a study published earlier this year, Skinner-Dorkenoo et al. found that the more white people in the US are aware of racial disparities in Covid-19 outcomes, the less empathy they showed for Covid sufferers and the less willing they were to support safety precautions. That is, the more people believed that problems only affect others, the less willing they were to solve them. In this case, highlighting racial disparities in Covid-19 outcomes actually made them worse.

As equity-minded data workers, the burden falls to us to present this data in ways that help audiences look past their biases. Raising awareness of social disparities is a worthy goal, but we have to do it in ways that avoid the negative side effects.

The Power of Context

For anti-racist dataviz, our most effective tool is context. The way that data is framed can make a very real impact on how it’s interpreted. For example, this case study from the New York Times shows two different framings of the same economic data and how, depending on where the author starts the X-Axis, it can tell 2 very different — but both accurate — stories about the subject.

As Pieta previously highlighted, dataviz in spaces that address race / ethnicity are sensitive to “deficit framing.” That is, when it’s presented in a way that over-emphasizes differences between groups (while hiding the diversity of outcomes within groups), it promotes deficit thinking (see below) and can reinforce stereotypes about the (often minoritized) groups in focus.

In a follow up study, Eli and Cindy Xiong (of UMass’ HCI-VIS Lab) confirmed Pieta’s arguments, showing that even “neutral” data visualizations of outcome disparities can lead to deficit thinking (and therefore stereotyping) and that the way visualizations are designed can significantly impact these harmful tendencies.

What is deficit thinking in the context of dataviz? Why is it a problem?

A multiline chart showing 10th grader test scores disaggregated by race. Plotting each line on the same chart risks a "deficit framing;" by inviting direct comparisons between groups, the chart invites viewers to conclude that lower-scoring groups are individually deficient to higher-scoring groups. Chart redrawn from Blakely's "Presenting data for a Targeted Universalist approach" (src).

The chart above might seem innocuous. It’s not necessarily wrong or inaccurate. It’s not intentionally misleading. But when people read charts like these, many of them will reach harmful, false conclusions (e.g. “Black and hispanic people did worse on tests because they don’t value education.”).

Why? The chart above has an implicit deficit frame. By emphasizing direct comparisons between minoritized and dominant groups, the chart encourages viewers to consider the groups with the worst outcomes (often marginalized groups) as deficient, relative to the groups with the best outcomes (often majority groups).

“How could someone conclude that from a simple chart?”

People interpret data through the lens of their pre-existing beliefs and biases. Deficit-framed charts exacerbate 2 troublesome social-cognitive biases that underpin our tendencies to stereotype: 1) outgroup homogeneity is the tendency to overestimate the similarity of people from other social groups and 2) attribution biases (like the “fundamental attribution error”) are the tendencies to attribute others’ successes (or failures) to personal qualities, even when the outcomes are obviously outside of their control.

Left: Bar chart showing pay disparities between 4 groups of restaurant workers. Right: Jitter plot showing the same data.

Consider the two charts above, each showing the same data. The bar chart exaggerates group homogeneity because it gives no indication of the outcome variation within the groups. It only shows the outcome differences between the groups. This creates the false impression that all Asian people earn higher wages than all White, Hispanic and Black people, when in reality there’s considerable overlap (as you can see in the Jitter plot). Research shows that when people underestimate the variation of outcomes within a category, they overestimate the impact of the category on the outcome. That is, they’re more likely to falsely conclude “being Black causes lower wages.”

Attribution biases can mislead audiences further: When considering the correlation between a certain race and their income, people can falsely conclude it’s caused by personal characteristics (e.g. “Black people don’t value education, therefore they get lower test scores,” a false stereotype) instead of environmental reasons (“being Black in an environment of systemic oppression leads to lower test scores”). Even though we know the latter is true, the former false stereotype is constantly repeated and reinforced - this requires diligence on our part, to reassert the latter truth, explicitly and frequently.

The Risks of Deficit Thinking

These biased, faulty readings highlight two central harms of deficit thinking: 1) “victim blaming” and 2) distraction.

Davis & Museus offer 2 examples of “victim blaming:” “For example, conversations about students who are at risk imply that they are likely to fail, and discourse around grit suggests that students’ individual deficiency (i.e., lack of grit) is responsible for the challenges that they experience in education.”

This tendency to blame perpetuates harmful stereotypes, creating a vicious cycle that can actually further entrench the disparities in question. According to social psychologist Charles Stangor “stereotypes become self-fulfilling prophecies, such that our expectations about the group members make the stereotypes come true.”

These lowered expectations can be detrimental in a number of ways. As an example, Stangor points to studies showing that teachers’ expectations for their students can significantly influence students’ academic performance. This may also explain the well-known “gifted gap” where gifted programs are disproportionately made up of white, wealthy children. Lowered expectations also affect the person being stereotyped. For example, in studies on stereotype threat, researchers have shown repeatedly that priming subjects with stereotypes (or even reminders of their own race) can impact scores on standardized tests.

Deficit thinking and blaming tendencies also potentially create a distraction effect. Again, Davis & Museus describe this well:

“The emphasis on individual and cultural deficiencies perpetuates assumptions that our system should seek a quick fix to remedy disparate experiences and outcomes rather than focus on addressing core systems of oppression and systemic inequities that permeate social and educational institutions. In doing so, deficit thinking prevents policy makers, educators, and communities from focusing on the actual root causes of the challenges that people of color, low-income populations, and other minoritized groups face.”

This makes sense. Since personal attributions (victim blaming) are a cognitively easier explanation, and people stop digging deeper once they’ve found an explanation that fits, victim blaming can obscure external causes, leaving more complex, systemic problems unaddressed.

Charts like the above may add a different kind of distraction: Since they present each group as a monolith, they also erase the diverse strengths and needs for each group, and therefore encourage inappropriate one-size-fits-all solutions. A focus on race itself as a cause prevents more nuanced, deeper thinking about how to help individual members of the group reach their full potential.

Conclusions

As presenters of data, we have a responsibility to present it in ways that encourage readers to extract useful information. There are years and years of negative racist stereotypes in literature and news. If we don’t think carefully about doing better, our charts and graphics can be read through those harmful lenses - perpetuating the problem instead of solving it. As data handlers and presenters, we have a responsibility to make careful and thoughtful decisions about how to present our data to support deep thinking and positive results.

About the authors:

Pieta Blakely is an expert evaluator, who has helped countless mission-based organizations measure and maximize their impact. Her nonprofit program evaluation practice, Blakely Consulting, helps clients build organizational cultures that thrive on joyful accountability and doing important work well.
Eli Holder is a data visualization designer and the founder of 3iap (3 is a pattern). 3iap, a data, design, and analytics consulting firm, specializes in psychologically effective data visualization, product design, and custom data product development.

Deliberative Bodies: Number of Constituents Shot Dead, per U.S. Senator

Fri, 08 Jul 2022 00:00:00 GMT

Motivation

How many deaths are our Senators responsible for?

In the United States, the 2nd Amendment grants “the right of the people to keep and bear Arms.” Like other rights granted by the constitution, it’s not without limitations. For example, it means that almost anyone can own a grenade launcher, but it does not mean that Chick-fil-A can sell them through the drive-through (“my pleasure!”). Grenade launchers are controlled; you can have one, but first there’s some paperwork.

Similarly, the 2nd Amendment doesn’t protect the current trigger-happy free-for-all. That’s a political choice made, largely, by members of the U.S. Senate. They have the option to pass laws that could save lives, but most choose not to.

Gun violence is valuable to politicians because it’s a “wedge issue.” They use issues like gun safety to turn us against each other because it makes it easier to get re-elected. When we’re busy fighting about whether teachers need guns or schools need smaller front doors, we’re more likely to overlook how little our elected officials do for us.

Their political strategy isn’t without consequences and the states that continuously elect these do-nothings are the ones that suffer the most. For example, in 2020, 500% more people per million died by firearms in Alabama than Massachusets.

Details

This graphic shows how many people from each Senator’s state were killed by guns, during each year they served in office, including homicides, suicides, or other gun-related deaths. The height of each column shows the combination of their inaction (time in office) and how relatively deadly their states are in terms of gun violence. Values are normalized to deaths per million citizens to support comparisons between states.

Data is based on the CDC Wonder database, which gives death counts by state and cause, from 1999 to 2020. Gun-related deaths are based on these ICD-10 codes: X93, X94, X95, X72, X73, X74, W32, W33, W34, Y22, Y23, Y24, Y35, U01.4.

Values before 1999 and after 2020 are estimated. Three Senators above served before 1999. Values for these years were estimated based on a combined model of the CDC data and published reports of homicide and suicide rates attributed to firearm for Alabama, Iowa, and Oklahoma. Values for 2021 and 2022 are linearly extrapolated from the 1999-2020 CDC data.

Dataset and script for estimates are available here:
Deliberative Bodies Dataset

What is a data visualization app?

Tue, 05 Jul 2022 00:00:00 GMT

Column 1: Temple University’s LawAtlas Health Policy exploration tool, 3iap’s Doom Haikus explorer, Jessica Nordell’s “NormCorp” simulator. Column 2: Survey research exploration app, Betterment’s retirement saving simulator, The Pudding’s Hip Hop Vocab Explorer. Column 3: Clinical trial interactive storyteller, Schema Design’s Garden of Health installation, Wu / Osserman’s People of the Pandemic simulator, St. Lous Fed’s FRED econometrics visualization tool. Column 4: NYTimes’ Rent or Buy calculator, PwC’s Perform Dashboard, Holders’ Radical Dots simulator

What are data visualization applications?

There are 3 types of dataviz applications:

1. Data storytelling apps

Interactive storytelling example apps

Storytelling dataviz apps, typically focused on a narrative, aimed at educating and raising awareness around the most important findings or insights in a dataset. These often come in the form of web-based, linear scrollytelling experiences, such as the projects featured on The Pudding. These can also be interactive, for example see 3iap’s recent work visualizing clinical trial data, which includes narrative elements and interactive filtering.

2. Data exploration tools

Data exploration example apps

Data exploration apps focus on flexibility, helping audiences use dataset(s) to answer a potentially infinite, long-tail of questions. This turns dense data into an resource that non-savvy consumers can use for general reference, or very advanced users can use for complex analysis. These are typically web-based and let users slice, dice and visualize data in a variety of ways. The St Louis Federal Reserve’s FRED tool is the quintessential example of a data exploration tool. 3iap’s recent work visualizing health policy datasets includes both dataset search and discovery, as well as custom visualizations.

3. Simulators and calculators

Simulation visualization example apps

Calculators and simulators help audiences “experience” datasets or complex models. This approach is particularly valuable when helping users make analytically difficult decisions, such as Betterment’s calculator for retirement planning or The New York Times’ famous mortgage calculator. This can also help everyday readers visualize complex social phenomenon, such as Jessica Nordell’s gender discrimination in the workplace simulator, Shirly Wu and Stephen Osserman’s People of the Pandemic, The Washington Post’s animated Covid Simulations, or Eli’s Social Polarization Simulator.

What tools and technology are used to build data visualization apps?

Web Applications / Javascript: Custom dataviz apps are most frequently built for the web, with tools like D3, React, Svelte, or Three.js.
Mobile Android / iOS Apps: In product visualizations, especially those targeted at creating consumer feedback loops, will be built directly into the products’ apps.
Native Code / Unreal / Unity: For museum or corporate office installations, data visualization — and occasionally data art — can be built using common web stacks, but often use lower-level technology like C++ or game engines like Unreal or Unity.

How much do dataviz apps cost to design and build?

Learn about the cost to develop data visualization apps here.

Data Visualization Consulting

Sun, 05 Jun 2022 00:00:00 GMT

Table of Contents:

What services do data visualization consultants typically offer?
What are the benefits of effective data visualization?
What type of data visualization consulting do different types of clients need?

What services do data visualization consultants typically offer?

Data Storytelling. Data-backed insights can influence everything from purchasing decisions to public policy. Expert data designers help transform these findings into an approachable, memorable narrative that’s visually appealing enough to break through the noise in our newsfeeds and inboxes, to reach, engage and educate busy and distracted audiences.

Custom Visualization Tools. Unique datasets answer unique questions. For example, large, opendata datasets can inform a wide variety of research and analysis, but the way the data will be used is difficult to predict ahead of time. Experienced data visualization developers can design and build interactive data exploration tools and custom applications, purpose-built to support common questions for a dataset, while also supporting open-ended insight discovery.

Data Product Design. All software products revolve around data, whether it’s an iPhone app targeted at consumers, or a b2b SaaS platform. Embedded analytics and well-designed feedback loops can create a win-win for software startups: Not only can they help users get more value out of services and motivate them to achieve their most important outcomes, they can also drive key metrics for the apps themselves like retention, conversion, and LTV.

Approachable Analytics. For Business Intelligence tools (like Looker, Tableau, PowerBI, etc) to support decision making, they need to be designed in a way that’s understandable and accessible to employees, even those who aren’t data-savvy.

Dashboard Consulting. Good dashboards inform, great dashboards align. Effective dashboards rest on a foundation of well-designed metrics and KPIs around which the entire organization can rally. Executing this requires expertise at the intersection of management consulting and data visualization design.

Training. Even in professional settings, data-literacy rates can be surprisingly low. To develop a data-driven culture, it’s crucial for internal researchers, analysts, and data-scientists to learn the most effective, accessible data visualization techniques. Training can help data professionals present their findings in a way that even executives can understand.

Designs for the LawAtlas policy-mapping dataset explorer. See case study here.

What are the benefits of effective data visualization?

Awareness, reach and engagement for complex topics (data storytelling).
Supporting new discoveries (interactive data exploration tools).
Smarter, faster decision-making (visual analytics).
Improving business performance and organizational alignment (OKRs or KPI dashboards).
Encourage targeted behavioral outcomes (visual feedback loops).

Designs for clinical trial results data storytelling. See case study here.

How to hire the right data visualization consultant?

Non-profit, NGO, Marketer, or Academic Researchers with profound insights to share.

Data Storytelling
Custom Visualization Tools

Software Startup, SaaS company, or App, with underutilized app data to unlock.

Data Product Design
Custom Visualization Tools

Business Leaders, creating a work culture that’s metrics-driven, data-informed.

Approachable Analytics
Dashboard Consulting
Training

Data Visualization App Development Costs

Sun, 05 Jun 2022 00:00:00 GMT

Contents:

What is a data visualization app?
How long does it take to build a custom data visualization app?
What are typical rates for data visualization service providers?
What is a typical cost to build a dataviz app?

What is a data visualization app?

There are 3 types of dataviz applications:

Storytelling dataviz apps, typically focused on a narrative, aimed at educating and raising awareness around the most important findings or insights in a dataset. These often come in the form of web-based, linear scrollytelling experiences.
Data exploration apps focus on flexibility, helping audiences use dataset(s) to answer a potentially infinite, long-tail of questions. This turns dense data into a resource that non-savvy consumers can use for general reference, or very advanced users can use for complex analysis. These are typically web-based and let users slice, dice, and visualize data in a variety of ways.
Calculators and simulators help audiences “experience” datasets or complex models. This approach is particularly valuable when helping users make analytically difficult decisions or understand complicated social or natural phenomena.

You can learn more about different types of dataviz apps here.

How long does it take to build a custom data visualization app?

Distribution of 1,550 hours spent on 3iap client projects, split by activity type.

In a recent study published in DVS Nightingale, 3iap analyzed the time required for 10 major dataviz projects that we designed and / or developed for clients. Design and engineering take up most of the time (>60%), with the remainder going to research, analysis, and client communication.

Design time for a few representative examples of dataviz applications:

Complex Dataset Exploration Tool: 14 days of design
Interactive Scientific Storytelling App: 3 days
Analytics Product Design System: 6 days
Complex Survey Exploration Tool: 8 days

Engineering time for a few representative examples were:

In-Product Chart Component: 8 days of engineering
Scrollytelling Interactive Infographic: 11 days
Interactive Scientific Storytelling App: 13 days
Interactive Content Explorer: 7 days

In this study, overall project time requirements varied between 7 and 26 days. This range represents small to medium sized projects, while larger projects might be scoped more like traditional software product development (e.g. 2-3 people, working 2-3 months, or 80 - 160 days on a tightly focused minimum viable product).

What are typical rates for data visualization design and development service providers?

Whether the contract is time and materials, a project fee, or even value-based pricing, the underlying math is still based on a target daily rate (or hourly, or weekly).

According to the Data Visualization Society’s recent industry survey, data visualization freelancers charge an average rate of $800 / day (or roughly $100 / hour). This average represents members across a wide spectrum of experience and expertise. While freelancers might seem like an economical choice, this comes at a risk to the overall timeline and often requires working across multiple contributors to meet the full set of capabilities (analysis, design, and engineering are typically distinct disciplines).

For comparison, more established firms typically charge 2-3x this rate, presumably for increased certainty around quality, timeline, and delivery. While generalist software design / development agencies (e.g. Carbon Five, Thoughtbot, etc) should generally be avoided for data visualization projects (dataviz, as a discipline, can be deceptively difficult to get right), they’re a reasonable benchmark for fees you might find from the industry’s best data visualization firms, who often charge between $2,000 - $2,400 per day.

At $1,400 - $2,000 per day, 3iap’s rates are somewhere in the middle. This reflects similar quality and capabilities as established firms, but as a small practice, 3iap has minimal overhead and the flexibility to work with startups and smaller, mission-oriented organizations.

What is a typical cost to build a dataviz app?

For small to medium sized dataviz projects (e.g. see work samples), we might expect between 7 and 40 days of work, so overall project costs might be:

At consulting firm rates (e.g. McKinsey, Bain, BCG, Deloitte) rates: $28k - $160k
For a typical dataviz design and development firm: $18k - $96k
For a team of freelancers: $5.6k - $32k (note: this might seem like a steal, but remember the risks mentioned above)

For larger projects, combining dataviz and general software development, we might expect 80 - 160 days of work across a small team, with cost estimates as follows:

At giant consulting shop rates: $300k - $700k
For a typical dataviz design and development firm: $200k - $400k

“Dispersion & Disparity” Research Project Results

Wed, 01 Jun 2022 00:00:00 GMT

Which of these two charts leads viewers to “blame the person“ (e.g. “Group C had worse results because of who they are“)? Which encourages blaming the environment (e.g. “Group C had worse results because of their circumstances“)?

TLDR

3iap’s peer-reviewed research was accepted to IEEE VIS 2022
Social cognitive biases can significantly impact dataviz interpretation
Charts showing social outcome disparities can promote harmful stereotypes about the people being visualized
Showing within-group outcome variability can mitigate these risks
Popular visualizations for showing social inequality can backfire and make it worse. News publishers, public health agencies, and social advocates should consider alternative approaches to minimize harm to marginalized communities (3iap is happy to help).

Last summer, two different clients approached 3iap for help designing and developing DEI-related (diversity, equity, inclusion) data visualizations: one showing outcome disparities in higher education, the other analyzing organizational behavior for signs of systemic racism / sexism in the workplace. These are heavy topics. They require fidelity not just to the data, but to the people behind the data.

And, as we discovered, addressing these specific design challenges also required answering a larger, open research question at the heart of equity-centered dataviz: How do you visualize inequality without promoting it?

Collage of deficit-framed data visualizations of racial disparities

The problem seems simple. To raise awareness about inequality, simply highlight the (stark) differences in outcomes between groups (e.g. income, life expectancy, educational attainment, etc). You might try a chart like one of the above and simply decompose the outcome in question into averages for each group of interest (e.g. splitting the data by race, gender, income, age, etc). As you can see, this is a popular approach!

However, as first noted by Pieta Blakely, when people see charts like the above, they can reach some surprisingly toxic conclusions: They blame the outcomes on the groups themselves, mistakenly assuming that groups with better outcomes are somehow better people (and groups with worse outcomes are somehow personally deficient). In their attempts to highlight outcome disparities, these charts reinforce the stereotypes that make disparities possible.

In their attempts to highlight outcome disparities, these charts reinforce the stereotypes that make disparities possible.

This seemed like a bold claim, especially since charts like the above feel so familiar and intuitive (e.g. “how could bar charts be racist?!”). Others in the dataviz community were skeptical (including myself and at least one of our clients). However, Jon Schwabish and Alice Feng lent support in their authoritative “Do No Harm” guide. And Pieta’s argument is consistent with a wide body of research across social equity, social psychology, and data visualization.

Given the uncertainty, the high stakes, and the ubiquity of these potentially troublesome charts, the issue begged further investigation.

So over the last year, 3iap designed and ran a series of experiments to understand the question empirically. Then in collaboration with Cindy Xiong, of UMass’ HCI-VIS Lab, we analyzed the results and uncovered some compelling findings. We’ve submitted the study for peer review and publication, but we’ll cover the main questions and results here.

Eli and Cindy’s paper was peer-reviewed and accepted to the IEEE VIS2022 conference. It’s available here: “Dispersion vs Disparity: Hiding Uncertainty Can Encourage Stereotyping When Visualizing Social Outcomes.”
All of the code, data, stimuli, analysis, etc. to reproduce the experiment are available here on OSF: Dispersion vs Disparity Materials.
Eli is also available for workshops and training sessions to help other data communicators apply these findings. If you’re part of a data, design, or analyst team interested in this topic, please get in touch!

Research Questions

The results are summarized below, organized by question. Click a link to navigate within the page.

Theory: How could visualizing social outcome disparities reinforce harmful stereotypes? Is there a plausible pathway between “bar chart” and “perpetuating systemic oppression?”
Measurement: How do we measure a chart’s impact on stereotyping? How do we operationalize “deficit thinking?” What would an experiment look like?
Impact: Is this a big problem? Do people in the wild actually misinterpret dataviz in ways that are consistent with deficit thinking?
Design Impact: Can dataviz design choices impact stereotyping? Can the way data is presented help (or hurt) audiences’ tendencies toward stereotyping?
Generalizability: Are these results generalizable outside the experiment?
Design Implications: What should data visualization designers do differently?

Q1: Theory: How could visualizing social outcome disparities reinforce harmful stereotypes?

Is there a plausible pathway between “bar chart” and “perpetuating systemic oppression?”

Dataviz & Deficit Thinking

Practitioners have previously argued (src, src) that charts like the above encourage a form of “deficit thinking” — by emphasizing direct comparisons between groups, they create the impression that the groups with the worst outcomes (often marginalized groups) are personally deficient relative to the groups with the best outcomes (often majority groups).

According to equity scholars Lori Patton Davis and Samuel D. Museus, “Deficit thinking” encourages “victim blaming;” it favors explanations that hold group members personally responsible for outcomes (e.g. “It’s because of who they are”), as opposed to explanations related to external causes (e.g. “It’s because of systemic racism”) (src).

Victim blaming can lead to two further harms:

Since it’s a cognitively easier explanation (src, src), victim blaming potentially obscures external causes, leaving widespread, systemic problems unconsidered and unaddressed (src).
It also reinforces harmful stereotypes, setting lower expectations for minoritized groups that become self-fulfilling prophecies (src).

Stereotyping & Social Psychology

Stereotypes are over-generalizations about a group of people (src). Stereotypes are harmful when we let them determine our expectations about individual members of a group (src).

Stereotyping is facilitated by perceptions of group homogeneity. When we overestimate the similarity of people within a group, it’s easier to apply stereotypes to individual members of the group (src). Unfortunately we’re predisposed toward overestimating group homogeneity for other groups than our own (src, src).

Our faulty judgements about other people reinforce stereotypes. For example, we often attribute others’ successes or failures to personal qualities, even when the outcomes are obviously outside their control (src). (This leads to blaming.) Compounding this, when we observe something negative about a person from another group, we tend to associate similar negative attributes with other members of the group (reinforcing stereotypes) (src, src).

These biases help explain the risks of deficit-framing. By emphasizing between-group differences:

perceptions of within-group homogeneity become exaggerated,
the lower-outcome groups (subconsciously) take the blame, and
the faulty personal attributions are then amplified to the entire group.

Prejudicial tendencies can be overcome. The more exposure we have to people from other groups the less likely we are to stereotype them (exposure helps us appreciate group heterogeneity) (src, src).

Dataviz & Uncertainty

Summary statistics are like stereotypes for numbers. Like stereotypes, summary statistics can be harmful when they lead us to overlook underlying variability (i.e. discounting outcome uncertainty) (src, src, src). People already tend to discount uncertainty (src), but dataviz design choices also make a difference (src, src).

In the same way exposure alleviates stereotyping, exposure to the underlying distribution helps viewers see past point estimates and appreciate uncertainty.

Several studies suggest that a visualization’s expressivity of uncertainty impacts understanding (src, src). For example, gradient and violin plots outperform bar charts (even with error bars) (src), Hypothetical Outcome Plots (HOPs) outperform violins (src). Even showing predicted hurricane paths (as ensemble plots) improves understanding compared to monolithic visualizations (src, src).

Xiong et al show a link between granularity and causation (src). More granular charts (less aggregated) reduce tendencies to mistake correlation for causation. Since illusions of causality stem from oversensitivity to co-occurrence, perhaps more granular charts work by exposing users to counterexamples (src, src).

Finally, Hofman et al show that when viewers underestimate outcome variability (by mistaking 95% confidence intervals for 95% prediction intervals), they overestimate the effects of certain treatments (src). They also speculate that a similar effect might contribute to stereotyping. When viewers underestimate within-group outcome variability (overestimate group homogeneity), they overestimate the effect of group membership on individuals’ outcomes.

Dataviz & Stereotyping

Ignoring or deemphasizing uncertainty in dataviz can create false impressions of group homogeneity (low outcome variance). If stereotypes stem from false impressions of group homogeneity, then the way visualizations represent uncertainty (or choose to ignore it) could exacerbate these false impressions of homogeneity and mislead viewers toward stereotyping.

The same dataset, visualized two different ways. The left fixates on between-group differences, which can encourage stereotyping. The right shows both between and within group differences, which may discourage viewers' tendencies to stereotype the groups being visualized.

If this is the case, then social-outcome-disparity visualizations that hide within-group variability (e.g. a bar chart without error bars) would elicit more harmful stereotyping than visualizations that emphasize within-group variance (e.g. a jitter plot).

Q2: Measurement: How do we measure a chart’s impact on stereotyping?

How do we operationalize “deficit thinking?” What would an experiment look like?

Example charts used in the experiments. The full set of stimuli are available on OSF.

Each participant saw a chart like one of the above, showing realistic outcome disparities between 3 - 4 hypothetical groups of people. Then they answered several questions about why they thought the visualized disparities existed.

Half of the questions offered a “personal attribution” - they implicitly blamed the people themselves for the disparities (e.g. “Based on the graph, Group A likely works harder than Group D.”).
The other half of questions offered an “external” attribution - they suggested that the group’s environment or circumstances caused the disparities (e.g. “Based on the graph, Group A likely works in a more expensive restaurant than Group D.”).

Since the participants weren’t given any other information about the groups of people in the charts, the only “correct” response was to disagree or say “I don’t know, there’s not enough information.” That is, any amount of agreement indicates bias. (As we’ll see below, these “correct” responses were relatively rare).

To evaluate stereotype-related beliefs, we’re mainly interested in participants’ personal attribution agreement. Personal attributions, in this case, indicate deficit thinking, or a negative belief about some of the groups in the chart. For example, agreeing that “Group A did better than Group B because they work harder” implies a belief that people in Group B aren’t hard workers.

So if a chart like the above leads viewers to agree with personal attributions for the outcome disparities, then it supports harmful stereotypes about the groups being visualized. Charts that lead to stronger personal attribution indicate stronger encouragement for stereotypes.

Interpreting external attribution agreement

External attributions for these charts are equally “incorrect” readings of the data. These would only indicate stereotyping if personal and external attributions were mutually exclusive. Some scholars implicitly argue that this is true, and some studies on correspondence bias assume they’re diametrically opposed, but at least based on our data, this doesn’t seem to be the case. For our purposes, external attributions might indicate a more empathetic outlook (e.g. seeing yourself in another person’s situation) but we mainly use them as a baseline for comparing personal attributions.

Q3: Impact: Is this a big problem?

Do people in the wild actually misinterpret dataviz in ways that are consistent with deficit thinking?

The good news: Most participants’ responses indicated more empathy and understanding and less personal blame.

In our last 3 experiments, across all conditions, 77% (542/709) of participants agreed with external attributions to explain the visualized disparities (i.e. they agreed that the results above were at least partially caused by external factors outside group members’ control. e.g. “Based on the graph, Group A likely works in a more expensive restaurant than Group D.”)

The bad news: In those same 3 experiments, 53% (377/709) of participants agreed with personal attributions to explain the visualized disparities (i.e. they agreed that the results above were at least partially caused by personal characteristics of the people in the groups e.g. “Based on the graph, Group A likely works harder than Group B.”).

This supports the hypotheses that visualizing social outcome disparities can encourage deficit thinking. The charts offered no information as to why the outcomes occurred, but still just over half of participants agreed with explanations that “blame” the outcomes on the people being visualized.

Interpreting overlapping personal and external attribution agreement

Note: Many participants agreed with both external and personal attributions. This implies that the two beliefs aren’t mutually exclusive (e.g. it’s possible to believe that restaurant workers from Group A are harder workers and they work in nicer restaurants). It’s also worth noting again that neither of these conclusions are rational without more information, which the charts don’t provide.

Q4: Design Impact: Can dataviz design choices impact audience stereotyping?

Can the way data is presented mitigate (or exacerbate) audiences’ tendencies to stereotype?

In 3 out of 4 experiments, we found statistically significant differences in personal attribution agreement between the designs we tested. This indicates that some designs are better than others for mitigating the “blame” associated with stereotyping.

Experiment 1: Bar Charts vs Jitter Plots

In Experiment 1 we found that Jitter Plots reduced personal attribution (i.e. blame) by 7.0 points, relative to Bar Charts (p=0.0011).

This experiment looked at Bar Charts v.s. Jitter Plots. This comparison is important because bar charts (as designed here, without any indication of uncertainty) represent the status quo for how this type of data is commonly presented. So, if we can show that another type of chart gets better results, we can establish that there’s room for improvement. Jitter plots are a good candidate for a “better” design for a few different reasons:

They still prominently feature group averages (the vertical bars), but strongly emphasize outcome uncertainty (the dots, in aggregate, show the rough shape of the outcome distribution).
They’re symmetric. Bar charts are asymmetric and have a “within the bar” bias, because all of their “ink” is between 0 and the average outcome, people mistakenly assume that individual outcomes fall below the mean. Jitter plots, however, are symmetric, and show the full distribution on either side of the average marker.
They’re “unit” encoded and possibly anthropomorphic. Because each dot represents a “real” person’s outcome, the results are less hidden behind an abstract calculation (e.g. the group average). And, because the dots are people, it might help viewers feel more empathy (though other studies show anthropomorphism’s effects on empathy are very limited).
They communicate sample size. In the Jitter plot above, Group D shows more dots because there are more people in Group D.

So, for these reasons, Jitter Plots should perform better than Bar Charts. And, based on our results, they do. The next question is why? Which of these factors (e.g. outcome uncertainty, symmetry, anthropomorphism) is most influential? We isolate these in the next experiment.

Experiment 2: Dot Plots vs Prediction Intervals vs Jitter

In Experiment 2 we found that Prediction Intervals reduced personal attribution (i.e. blame) by 7.3 points, relative to Dot Plots (p=0.0045).

This experiment looked at Dot Plots vs Prediction Intervals vs Jitter Plots. Whereas Experiment 1 tried to “stack the deck” to test for any difference, Experiment 2 was designed to isolate why some charts perform better than others.

All three charts are symmetrical and prominently feature “dots” as encodings.
Dot Plots are similar to Experiment 1’s Bar Charts because they both encode the group average, but neither communicates uncertainty.
Prediction Intervals are similar to Jitter Plots, but they encode outcome uncertainty (the range of possible outcomes) with a monolithic interval, so they don’t benefit from unit-encoding or anthropomorphism. (Note that “Prediction Intervals” are distinct from “Confidence Intervals” which we explore in the next experiment.)
The Jitter Plot in this experiment is slightly different from Experiment 1. In Experiment 1, the Jitter plot showed unequal sample sizes between rows. In Experiment 2 the Jitter plot shows an equal number of dots per row. So none of the 3 charts in this experiment communicate sample size differences, letting us rule that out as a factor (we explored this further in Experiment 4).
So if we see differences between Dot Plots and either of the other two designs, we’d know the improvement is caused by communicating uncertainty.
If we see differences between Jitter Plots and Prediction Intervals, we’d know the improvement is caused by anthropomorphism.

Since there were no significant differences between Prediction Intervals and Jitter Plots (and Prediction Intervals actually did slightly better), we can rule out anthropomorphism and unit encoding as the influential factor.

Since Prediction Intervals and Jitter Plots performed better than Dot Plots (though only Prediction Intervals v.s. Dot Plot differences were significant), we can reasonably conclude that the improvement is related to visualizing uncertainty.

The next question, then, is what kind of “uncertainty” makes a difference? We unpack this in Experiment 3.

Experiment 3: Confidence vs Prediction Intervals

In Experiment 3 we found that Prediction Intervals reduced personal attribution (i.e. blame) by 5.1 points, relative to Confidence Intervals (p=0.0424).

In the previous 2 experiments, we show that visualizing uncertainty reduces personal attribution (e.g. Prediction Intervals and Jitter Plots show outcome uncertainty, while Dot Plots and Bar Charts don’t). “Prediction Intervals,” however, represent a different “type” of uncertainty than the more conventional “Confidence Intervals.” Since other research shows that Confidence Intervals bias users in ways that might affect perceptions of group homogeneity, we need to isolate the type of uncertainty that helps mitigate personal attribution.

The Prediction Interval charts in this experiment are identical to the previous experiment. Their bars represent “outcome uncertainty” - that is, the range of outcomes that a real person might experience.
Confidence Intervals, however, represent “inferential uncertainty” - that is, their bars represent uncertainty in estimating the group average. They only show outcome uncertainty indirectly.
Importantly, Confidence Intervals are based on standard error of the mean and can be made arbitrarily small with a large enough sample size. Whereas Prediction Intervals are based on standard deviation of the outcomes, they’re not sensitive to sample size, and will generally appear wider. This means that when viewers mistake a Confidence Interval for a Prediction Interval, they’d get the false impression that outcome variability is quite low (i.e. that group homogeneity is high). That is, Confidence intervals hide the fact that there is so much overlap in outcomes between groups.

Since Prediction Intervals performed better than Confidence Intervals, we can conclude that the type of uncertainty visualized makes a difference. The improvement is related to visualizing Outcome Uncertainty. This implies that the improvement is related to viewers’ impressions of group homogeneity, as we’d expect from prior work.

Q5: Generalizability: Are these results generalizable outside the experiment?

We took several steps to make this more robust:

Experiment #1 tested charts with 2 different group types: one was race, the other was arbitrary letters. The results were actually stronger when the groups were defined with meaningless letters. An optimistic read of this: If charts are actually about racial groups, people are less likely to “blame” the groups. More likely though, we suspect that the meaningless letters condition showed larger differences because racial groups triggered participants’ social desirability biases, meaning the letters case is closer to reality. In either case, this implies designs’ effects on personal attribution aren’t strictly tied to charts about racial disparities (e.g. could apply to gender, age, class, income, etc).
Across the 4 experiments, we tested 4 different “topics” with unique datasets. Experiment 1 covered hypothetical waitstaff pay disparities. Experiments 2-4 randomly assigned one of three topics: household income, life expectancy or literacy test scores. We based the data in each chart based on actual disparities documented for similar outcomes.
Our participants were drawn from Mechanical Turk and screened based on their ability to correctly read a graph. This helps control for results related to misunderstanding the data. This implies a selection bias toward participants who are potentially more educated (and liberal) than the general population of the United States. Education levels and politics are correlated with reduced stereotyping, so our audience might be less likely to stereotype regardless of the charts.

Q6: Design Implications: What should data visualization designers do differently?

When visualizing data about social outcome disparities, how can designers (significantly) reduce dataviz audiences’ biases towards stereotyping?

Even if the purpose of the visualization is to raise awareness about disparities between groups, designs should still make it obvious that there are also wide differences in outcomes within groups. This disrupts false conclusions of group homogeneity and therefore tendencies to stereotype.
Favor visualizations that emphasize within-group variability, like Jitter Plots or Prediction Intervals.
Avoid charts that only show (or over-emphasize) groups’ average outcomes, like Bar Charts or Dot Plots. Even if these include error bars (e.g. as confidence intervals), they’ll likely still encourage more personal attribution than alternatives that emphasize outcome uncertainty.

Research Roundup: Dataviz & Deficit Thinking: Could visualizing social inequality reinforce it?

Mon, 04 Apr 2022 00:00:00 GMT

moved

Ugandan Social Survey Data Explorer

Fri, 01 Apr 2022 00:00:00 GMT

Context

Client

A large, multinational market and public opinion research firm.

Prompt

How might we help non-profit and NGO analysts understand the landscape of diverse social support groups in Uganda?

Background

The client recently finished a multi-year research project, surveying different populations across nearly every district in Uganda. The client asked us to help find the stories in the data and design an interactive tool for exploration.

Goal / Challenge

Many-to-many storytelling. The tool would be used by many different groups of users, each with different interests and priorities. The data also contained many unique and interesting stories. So how do we help the right users find the right stories?

Interactive functionality like filtering and scrubbing, combined with benchmarking, help users answer a broader range of questions from each component.

Design

Full mockup of the geo-explorer functionality, with lipsum text added for anonymity.

Insights

Break the stories up into chapters. By grouping insights into a number of different themes, we can offer an information hierarchy that is easily navigable while still allowing related insights to build on each other.
Interactivity to functionality without expanding scope. With tools like universal filtering, the same charts and graphs can be viewed through a variety of different lenses, answering more questions with fewer charts.
Keep charts simple. While some projects benefit from fine-tuned, custom visualizations, for this project, with this much content, the story is best told by composing many simple charts that work together as a group.

Solutions

Geographic Explorer: To understand the densities of particular groups, we designed a 2-layer geographic explorer. The first layer was a choropleth map, showing group density by district. Users could then click into certain districts to see how groups are dispersed within the region.
Group Directory: To help analysts identify specific groups that meet their theories-of-action, groups could be sliced and diced within the group directory.
Live Narrative Reports: To help new users (or exceptionally busy users) grasp key takeaways, a series of “live” reports could be read through linearly, or combined with filtering tools, for exploring various themes in greater depth.

Public Health Policy Dataset Discovery and Exploration

Tue, 01 Mar 2022 00:00:00 GMT

Context

Client

Temple University’s Center for Public Health Law Research, in partnership with Graphicacy

Prompt

How might we increase LawAtlas adoption, democratize policy surveillance data, and help public health analysts advocate for more effective health policy?

Background

LawAtlas rests on a profound insight: With careful analysis, the gray areas of the law can be distilled into simple sets of yes / no questions.

Even though the text of a law can vary from state to state, experts in the field are more than capable of judging their most important attributes. This allows comparison across jurisdictions, supports a wide range of public policy analyses, and ultimately promotes smarter policy decisions.

Key Challenge

With 137+ datasets, and more added every year, it’s difficult for researchers and analysts to discover the datasets that are most relevant for their open-ended research questions.

Designs often begin with pen and paper, as a way to quickly brainstorm visualization approaches.

Design Goals

Burning Questions:

The first priority of any data discovery tool is answering users’ questions. While LawAtlas data supports a long tail of analyses, a few question templates jumped out as common and important:

How widespread is a particular policy? e.g. “How many countries ban or restrict abortion?”
How do specific policies overlap and intersect? e.g. “During Covid, how many states closed public OR private schools AND required indoor masking?”
How has adoption changed over time? e.g. “Which areas adopted Good Samaritan laws first?”

Other Considerations:

The product design centered around answering these key questions, while also:

Enabling frictionless search & discovery. With 100+ datasets, across a number of different topics, finding data of interest can be hunting for a needle-in-a-haystack.
Supporting a variety of jurisdictional levels, including cities, states, territories and countries.
Handling heterogeneous dataset. While datasets are similarly structured, they’re not identical, so any visualization needs to gracefully handle outliers.

Results from brainstorming different ways to overlay multiple patterns on tiles. When overlapping patterns on many different tiles, visual design affects not just aesthetics, but also legibility.

Insights

What's a "choropleth" map?

Data Visualization

Would a choropleth work? While choropleth maps are a popular choice for similar clients, they’re limited to displaying only a single variable. To show policy groupings and co-occurrence, we’d need to show multiple variables at once.

Some research suggests as many as 6 or 7 different variables can be overlapped on a single surface, so we explored overlapping patterns to represent different (combinations of) policy attributes.

Benefits of Tile Maps: Geographic maps benefit from viewers’ spatial memory when looking to specific areas, however for this use case, the priority was visualizing an overall tally of adoption and supporting multiple variables, both of which benefit from tile maps’ fixed size per jurisdiction.

Interactivity can help manage complexity. Multi-dimensional datasets are inherently complex and quickly exhaust viewers’ cognitive capacity. The tool should offload this burden to the user’s mouse: additional context and reminders can be just a click (or hover) away.

Examples of three different tile map layouts. On the left is a fixed map of the united states. In the middle is a force-directed layout. On the right is a dense map of the world.

Search and Discovery

All roads (searches) lead to Rome (data directory). Depending on users’ needs, they might favor different ways to identify datasets of interest, therefore search and filtering mechanics should support multiple paths (without introducing undue clutter or UI complexity).

Design

Final mockups, covering the dataset visualization tool (top row), the surrounding website, and the dataset search and discovery functionality (bottom row).

Solutions

4-D Policy Tile Map Visualization. This tilemap approach makes it easy for users to see both overall adoption of a single policy and the tapestry of policy combinations across states by overlapping 4 distinct markers on each tile. The tilemap also supports the full range of geographies found in the datasets.

Interactive Dataset Explorer Tool. The Policy Map Visualization is manipulated through a 4-variable toggle for each policy attribute, where each variable corresponds to a color that appears on the corresponding tiles.

Universal Instant Search. To find datasets of interest, users can search for a specific dataset by keyword or filter datasets by topic. From the dataset directory page, users can filter by further criteria identified by the team.

Results

Live Site

The redesigned, relaunched LawAtlas explorer tool is available here.

Feedback

“This is fantastic. It is so intuitive. It took me no time at all to generate a map that can so easily be described to an audience"
CDC Analyst

“This site is truly a revolution in the way we share and communicate legal data."
Client Director of Communications

Clinical Trials Visualization Tool

Sun, 01 Aug 2021 00:00:00 GMT

Context

Prompt

How might we help a Fortune 500 pharmaceutical company educate healthcare professionals on the efficacy of a breakthrough treatment?

Background

The client recently developed a treatment for a condition that affects millions of people. In clinical trials, their treatment outperformed alternatives across a variety of dimensions: It provided more relief, to more people, faster and for a longer duration. As part of their campaign to educate healthcare providers on this new treatment, the client needed a way to visualize this clinical trial data.

Design Challenges

Visual impact matching treatment impact. The standard metric for reporting treatment efficacy for this condition is the proportion of people who achieve a certain threshold of relief. This new treatment, however, is so effective that the majority of subjects achieved almost total relief, which results in a very dull graph!
Humanizing clinical trial data. Healthcare professionals are inundated with data. This makes it easy to lose sight of the fact that, behind the data, there are real patients whose lives are impacted.
Staying true to the science. Without exception, the visualization must strictly match the data from the trials.
Slicing and dicing. To show efficacy across a variety of patient criteria, the final product needed interactive filtering functionality.

Insight

Show everything. Under the right conditions, ensemble visualizations help viewers identify general trends from many, many data points. Since the trials were so successful, most of the 100s of patients had similar outcome trajectories (up and to the right). Plotting each patient’s history as its own line would help viewers 1) identify the overall trend through the ensemble effect and 2) appreciate that they’re seeing hundreds of subjects’ results. Each person’s results act as a texture that supports the overall story.

Design

Solutions

Through research and client workshops, developed a story about what made the treatment uniquely successful (compared to alternatives) and, importantly, how the data can be quantified in a way that’s familiar to specialist physicians.
Aligned client teams on design approach through a series of brainstorming and concepting exercises.
Designed high-fidelity mockups and prototypes to show how the final visualizations would look with realistic data (this is particularly important for data-as-texture visualization approaches) and how the final product would reflect the company’s brand.
Developed custom javascript pipeline to dynamically calculate metrics and render each patient’s individual line to an HTML canvas. This was optimized to ensure performance and support smooth animations across browsers, even on older iPads.
Developed a custom React app, packaging the visualization into a tool that the client’s education teams could use when presenting to healthcare professionals.
Worked with client analysts through rigorous audits to make sure the underlying data and visualizations were true to previously published research.

Results

Successfully launched the visualization tool.
Achieved a compelling visual story (“I thought [the visualizations] were absolutely stunning”) and stayed true to the science (approved by client’s science and medical communications teams).

“Absolutely stunning!”
Client Consultant

Visualizing 'Small Multiples' Charts with Observable Plot

Wed, 19 May 2021 00:00:00 GMT

This post has moved

Visualizing 'Small Multiples' Charts with Observable Plot

Wed, 19 May 2021 00:00:00 GMT

Let’s explore 2 things in parallel:

Observable’s new Plot library for quick data visualizations and exploratory data analysis.
The minor plague that is parking sprawl.

A few reasons why Observable Plot is great:

It’s super quick and relatively mindless to crank out “good enough” charts and graphs. If you need something fancy, d3 is still a reasonable bet, but for basic bar graphs, line charts, distributions, etc., it’s does the trick with minimal fuss.
The API is intuitive, minimal and uses the conventions that most d3 data visualization developers have come to rely on for custom dataviz.
The faceting concept, which we’ll explore here, makes it easy to visualize many different dimensions of the same dataset in parallel, as small multiple charts.

A few reasons why parking lots are the worst:

Car accidents. 20% of car accidents happen in parking lots (leading to 60k injuries each year, src).
Housing prices. More parking → less housing. In NYC, a 10% increase in minimum parking requirements leads to a 6% reduction in housing density (src).
Pollution. More parking → more auto emissions (src).
They’re so, so ugly.

Land Use for Parking Dataset

Let’s start with a dataset. Note that Plot is built with “Tidy Data” in mind, which is another way of saying it’s clean and tabular. Observable’s definition:

Each variable must have its own column.
Each observation must have its own row.
Each value must have its own cell.

So I’ve put together a County Parking Area Dataset here. It’s a combination of the results of this study, which models parking lot land use for the United States and the US Census National Counties Gazetteer File, which has basic facts about counties like population size and land area. It’s ~16k rows, each with 6 fields:

geoid: The FIPS state + county code for the county
countyName: A human readable name for a county
landAreaMSq: Land area in meters squared
parkingLandAreaMSq: Parking lot land area in meters squared
year: The year associated with the parking lot measurement estimation.

We can pull down the data with:

const countyDataTidy = d3.json("https://gist.githubusercontent.com/elibryan/0bc177106babf67c1bf446d81fc6e5c9/raw/cdd7a8e8e6e52502630246f5aabe8beae4115a6e/parking-area-dataset.json")

Then let’s make some charts!

How much have parking lots spread in single city?

A simple area chart in Observable Plot

Let’s start simple and just look at growth for one city. Let’s say Raleigh NC.

First let’s pull out just the Raleigh related rows:

  // The Geoid for Wake County, NC
  const raleighGeoid = "37183";
  // Filter the dataset for just Raleigh data
  const raleighTidyData = countyDataTidy.filter(
    record => record.geoid === raleighGeoid
  );

Then we’ll create a simple area chart showing just the Raleigh time series.

We get the plot above from the following snippet:

Plot.plot({
  marks: [
    Plot.areaY(raleighTidyData, {
      x: "year",
      y: "parkingLandAreaMSq"
    })
  ]
})

This loosely translates to “given this tidy data, show me a sane area chart where X is the “year” field and Y is the “parkingLandAreaMSq.” Granted, the result is ugly, but this is a single, straightforward function call.

This introduces Plot’s concept of “marks.” In this context, a “mark” is an abstract term describing any visual encoding of data. Plot offers built in marks for all your favorite data visualizations (e.g. bars, lines, dots, areas, etc).

Let’s clean it up a bit:

We get the chart above from the following snippet:

Plot.plot({
    // Set formatting for the y axis
    y: {
      label: "Parking Lot Area (km^2)",
      tickFormat: (d) => d3.format(",.2r")(d / 1000000)
    },
    // Set the overall chart height
    height: 200,
    // Add "marks" to the plot
    marks: [
      // Define an area...
      Plot.areaY(raleighTidyData, {
        // Where X is year
        x: "year",
        // Y is parking lot area
        y: "parkingLandAreaMSq",
        // Color it a gross orange, to remind us that parking lots are gross
        fill: "#D46C25"
      })
    ]
  });

Conclusions:

Plot gives (nearly) 1-liner graphs for visualizing (silly) data in Javascript
Since 1974, Raleigh’s has more than doubled its surface area devoted to ugly parking lots

How much have parking lots spread across multiple cities?

Small multiples charts in Observable Plot

Let’s plot the 20 counties with the largest land-use area devoted to parking lots.

We get the graphs above from the following code snippet:

// The dataset includes observations for 5 different years
  const pointsPerCounty = 5;

  // Let's get the 20 counties with the largest (ever) parking lot areas
  let largestCountyIds = d3
    .groupSort(
      countyDataTidy,
      (records) => -d3.max(records, (record) => record.parkingLandAreaMSq),
      (record) => record.geoid
    )
    .slice(0, 20);

  // Filter a subset of the data for the selected counties
  const countyIdsToPlotSet = new Set(largestCountyIds);
  let countyDataTidySubset = countyDataTidy.filter((record) =>
    countyIdsToPlotSet.has(record.geoid)
  );

  // Let's add indicies to each row based on the county (a hack for later)
  // It doesn't matter what the indices are, so long as they're sequential
  countyDataTidySubset = countyDataTidySubset.map((record) => ({
    ...record,
    index: largestCountyIds.indexOf(record.geoid)
  }));
  // return countyDataTidySubset;

  // Extract the largest Y value (another hack for later)
  const yMax = _.max(
    countyDataTidySubset.map((record) => record.parkingLandAreaMSq)
  );

  return Plot.plot({
    // Draw a grid on the plot
    grid: true,
    // Set width to 800
    width: 800,
    // Slightly abusing facets to just show a grid of arbitrary charts
    y: {
      label: "Parking Lot Area (km^2)",
      tickFormat: (d) => d3.format(",.2r")(d / 1000000)
    },
    facet: {
      data: countyDataTidySubset,
      x: (record) => Math.round(record.index % 5),
      y: (record) => Math.floor(record.index / 5)
    },

    marks: [
      // Show borders around each chart
      Plot.frame(),
      // Show the area chart for the county with the matching index
      Plot.areaY(countyDataTidySubset, {
        x: "year",
        y: "parkingLandAreaMSq",
        fill: "#D46C25"
      }),
      // Show a label with the name of each county
      Plot.text(countyDataTidySubset, {
        filter: (d, i) => i % pointsPerCounty === 0,
        x: () => "1992",
        // Add the title to the top of the chart
        y: yMax,
        text: "countyName",
        dy: "1em"
      })
    ]
  });

We’re doing a couple things here:

First we’re extracting the 20 counties with the largest parking lot areas
Then we’re plotting them by slightly hacking Plot’s faceting system

Conclusions:

LA County has a crazy amount of parking lot. As of 2012 it’s 290km² (111 sq mi). That is, LA county has about 5x more area for parking than Manhattan has for everything.
Plot’s Facets are great for showing small multiples charts of datasets split by dimension.
Parking lots are the worst.

Changing minds with data

Fri, 14 May 2021 00:00:00 GMT

Sometimes even very good data can fall flat on unreceptive audiences.

When communicating data, it’s important to start with the right expectations. Here are 2 surprising barriers to consider:

The majority of the population struggles with numeracy and graphicacy. This isn’t just people who claim they’re “bad at math,” it’s a problem for even well-educated professionals (doctors included). In some studies, roughly 6 in 10 people struggled to identify basic trends in a line graph src.
When we look at data, we see what we want to see. In a 2017 study, Dan Kahan and friends recruited a group of highly-numerate participants and presented cohorts with (hypothetical) data describing the efficacy of a (fictional) skin cream. As expected, the group found easy consensus and interpreted the data “correctly.” They then took the same dataset and gave it a different narrative. Instead of describing the efficacy of skin cream for solving rashes, the experiment cohort was told the data described the efficacy of various gun violence interventions. Their findings: Even for this highly numerate crowd, responses to the data became polarized along particpants’ party lines. That is, the data told them what they already believed. Another study found that the higher an individual’s IQ, the better they are at coming up with reasons to support a position—but only a position that they agree with.

Visualizing agent based models

Thu, 13 May 2021 00:00:00 GMT

Agent based models help us simulate systems with many differnet interacting players. Because these systems are chaotic and nonlinear, it’s hard to capture their dynamics with simple equations (no matter how complex your spreadsheet).

Here are 4 examples of agent-based models, visualized:

Conway’s Game of Life shows how very simple agents (cells) can have wildly emergent behaviors from even just a few simple rules.
The Washington Post’s Covid-town simulator shows how Covid-19 spreads through a small town of dot-people, helping users see the consequences of exponential growth for themselves.
The Radical Dots simulator shows how beliefs spread through a small population of 100 people, to help users understand the dynamics of political echo chambers.
Sim City, the classic city simulator, shows how all the interacting people and parts of a stylized city can result in different urban planning outcomes. While agent-based modeling has more serious urban-planning applications, Sim City also reminds us that models can be quite engaging.

Dashboard Psychology: Effective Feedback in Data Design

Wed, 07 Apr 2021 00:00:00 GMT

“What you measure, you improve.” We’ve heard this a million times. It sounds nice. It seems plausible. There’s a bunch of evidence supporting it.

But how does this actually work?

What is it about seeing numbers that pushes people to action? What separates an admirable, “actionable” dashboard from all the B.I. “data vomit?”

To understand effective, motivational data design, you need to understand the psychology of feedback. So let’s look at a few examples of (quantitative) feedback in information design.

Tufte v.s. Robinhood. Two very different charts demonstrate the opposing forces of feedback.
Indiegogo and Fundraising Progress. Is positive or negative feedback better? Depends on the audience.
Dynamic Speedometers. How just two numbers create contrast (and safe drivers).
Atom’s Meditation Forrest. Why counting things feels good.
Withings’ Weight Graph. Balancing contrast and commitment for difficult health behavior interventions.
Benchmarking Worklytics. Why do business people love benchmarks?

Tufte, Robinhood, and the fundamental forces of feedback

Left: Screenshot from the Robinhood app, showing my rapidly growing personal fortune. Right: A segment from Tufte and Powsner's "Graphical Summary of Patient Status," showing a patient's blood glucose levels compared to a normal range.

Let’s start with two examples of quantitative feedback: 1) Robinhood’s investment portfolio chart, and 2) Tufte’s patient status chart.

On the left: A line graph from the Robinhood app. As a testament to my investing savvy, you can see that I’ve grown my portfolio 8.75 percent (to a whopping $25). As charts go, this one’s a dumpster fire and is Fox-News-level manipulative. But it’s quite encouraging!

On the right: A segment of Tufte and Powsner’s “Graphical Summary of Patient Status,” showing that a patient’s blood glucose is elevated above the expected range. Though Tufte now discourages this design (he recommends sparklines), it’s a powerful example of critical feedback.

These charts have more in common than you might expect. They both forgo y-axes. They’re both data-ink efficient. They both give quantitative feedback.

Where they differ: Robinhood’s chart uses feedback to create commitment and motivate users to carry on. Tufte and Powsner’s chart uses feedback to offer contrast, enabling users to adapt and change course.

Robinhood → Commitment

Robinhood’s chart demonstrates how feedback influences our commitment to a goal (e.g., day-trading Gamestonks and Dogecoin until you’re super rich).

Efficacy / Expectancy

Three screenshots from Robinhood, showing 3 different snapshots of my investment portfolio (and 3 different time scales). Notice the green and the vertical upward movement.

For new Robinhood users, their graphs shout, “OMG, you’re great at this!”

For example, by blurring the lines between users’ deposits and their investment returns and starting the plot from y=0, users are never more than a few taps from a bright-green graph of their portfolio, showing a sharp, satisfying uptick. Rosy feedback like this builds users’ self-confidence (efficacy) and encourages higher expectations for future returns (expectancy), thereby increasing their commitment to continued trading.

Feedback’s effect on efficacy applies outside of Robinhood. Positive feedback increases goal pursuit for students (src) and logistics employees (src); negative feedback, when it erodes confidence, can knock people completely off the wagon (src); it might even explain why some progress bars feel more satisfying than others (src).

Reinforcement

Five screenshots from Robinhood, cycling between positive and negative feedback.

While Robinhood’s overall impression is rosy, the day-to-day experience feels more like a rollercoaster. The first thing users see is their portfolio’s performance today. The chart itself is tall, with a zoomed-in Y-axis to maximize the distance between the plot’s min and max. When markets are open, it updates in real-time (to mesmerizing effect). And, regardless of how much your portfolio is up or down, the whole app is either bright green or bright red based on the direction of change.

A positive spin on this: it makes day-trading more visceral, fun, and emotional! It encourages trading the same way a “runner’s high” encourages marathon training - it just feels good! Slightly darker: it’s using the random walk of the stock market as variable reinforcement, hijacking user’s anticipatory responses, and gently nudging them toward addiction (like a slot machine).

While Robinhood pushes it to a predatory extreme, the principle remains: when feedback itself is rewarding, humans learn to associate those positive experiences with pursuing the goal, therefore reinforcing motivation toward the goal (src).

Expectancy + Reinforcement → Commitment

By creating an early impression of confidence and progress, then drawing users into a realtime rollercoaster, Robinhood’s charts use feedback to increase users’ commitment to day trading.

Tufte → Contrast

Powsner and Tufte’s ”Graphical Summary of Patient Status” demonstrates feedback as a source of contrast and a signal to change course.

Discrepancy

Powsner and Tufte's blood glucose chart, annotated.

The chart above says, “Uh oh, the patient’s blood glucose is higher than it should be.” It does this by comparing two values: 1) the dots are the patient’s blood glucose measurements, and 2) the series of vertical lines are the target range for those measurements. Ideally the dots fall inside the lines, but for this patient, they’re just above. Separately, neither the dots nor the target range provide much useful information. What matters is the contrast (discrepancy) between the current and goal states.

This contrast is what makes a chart “actionable.” Specifically, the size of the discrepancy supports one of two possible actions:

If the discrepancy is small, things are good, so the implied action is, “Keep doing what you’re doing.”
If the discrepancy is large, things are bad, so the implied action is, “Do something different.”

Charts can’t tell you what to do next. That’s not their job. What they can tell you is when action is required (and how urgently).

Contrastive feedback highlights the gap between current and goal states. To the extent that you’re committed to achieving the goal, you respond by adapting your approach toward closing the gap (src, src, src, src, src).

Attention

Tufte and Powsner's charts applied to 22 dimensions of a patient's health status (slightly modified into a horizontal layout, src).

The blood-glucose chart is one of many. A holistic view of a patient requires considering many similar indicators in parallel. This highlights another important aspect of feedback: no metric stands alone. So, in a sea of competing goals, which ones need your attention right now?

Contrast plays a role here as well. Assuming equally important metrics, the metrics with the largest discrepancies have the most potential for improvement, and are likely worth prioritizing.

Powsner and Tufte’s design amplify this in two ways: 1) the individual charts convey contrast at a glance, making it quick to determine if additional attention is required, and 2) each graph’s y-axis is scaled so that the normal range is a constant height, making the magnitude of discrepancy comparable between the charts, so the (globally) extreme values will look the most extreme on the page.

Contrast → Change

By comparing metrics’ current states and target states, Tufte and Powsner’s charts use contrastive feedback to direct physicians’ gaze toward areas that most need their attention. This contrast creates change by alerting physicians (and patients) to the need for action, enabling them to close the gaps.

Indiegogo and Fundraising Progress

Is positive or negative feedback better? Yes.

The tricky part: “commitment” and “contrast” are often at odds. Feedback that improves commitment can relieve the tension created by contrast, whereas feedback that highlights contrast can damage our commitment.

Balancing commitment and contrast matters when choosing between positive and negative feedback. For example:

If you’re teaching a child to play piano, you might lean toward positive feedback. Encouraging and rewarding their successes builds their commitment, whereas focusing on mistakes may cause them to give up before they have a chance to improve (children, dolphins and husbands have this in common).
However, if your student already plays for the philharmonic, negative feedback might be more effective. If they’re already committed and confident in the instrument, highlighting their mistakes surfaces the gap between their current playing and latent virtuosity, helping them improve.

Let’s look at another example. Consider the following common wisdom about fundraising…

Fundraising and the “Green Bar Effect”

"The Green Bar Effect" -- The psychological impact on contributors when you don't set a realistic goal. #IndiegogoTips pic.twitter.com/SDdhsImx
— Indiegogo Reads 🚀 (@Indiegogo) November 28, 2012

If you’ve ever considered crowdfunding, you might know about the “green bar effect.” The non-profit accelerator Fast Forward offers the following advice for fundraisers:

“Set an attainable goal: people want to fund a project that has made significant progress toward its goal. If you set your initial goal too high and haven’t fundraised enough, strangers are less likely to donate (this is the “green bar effect” or the “bandwagon effect”: a progress bar showing 40 percent project funding versus 15 percent is more successful).” (src)

Koo and Fishbach tested the green bar effect in a related experiment. They found that when sending fundraising letters to low-commitment donors, they could increase donations by emphasizing how much money had already been raised (src). Because other people have put up money, it signals that the goal must be important. This perception of importance then, increases donors’ commitment, leading to more donations.

But, what works for low-commitment donors has the opposite effect with high-commitment donors. The most effective letter to the latter cohort highlights how far they still need to go.

The folks at Fast Forward address this indirectly with another tip:

“Pro-tips: aim to have about ⅓ of your goal locked down via commitments from your network before you even open up the campaign. Planning prior to campaign launch is crucial — give yourself at least a month of prep time.”

That is, by getting early donations from your network — presumably from those who are most committed to you and your goal — you’re solving the “green bar effect” problem, but you’re also asking these early donors at exactly the right time to maximize their contributions, when the discrepancy between the current and goal state is largest.

Fundraising Thermometers

A handful of fundraising thermometers (from the zillions on Pinterest)

The fundraising thermometer is one of the more common visualizations you’ll see implemented with construction paper. But don’t let their humble execution fool you! There are two aspects of this that make it powerful for commitment building:

The metaphor allows small, incremental notches toward a larger goal. When tracking progress toward a larger goal, these intermediate goals give users a focal point that’s closer and easier to achieve. This proximity boosts confidence. Then, as users accomplish more of these smaller goals over time, they act as a record of past commitment, further propelling their efforts forward.
The size of these is also an influential factor. By giving this such a large physical presence, it reinforces the importance of the fundraiser to the organization, further building commitment.

Dynamic Speedometers

Feedback can be surprisingly simple and still effective. Not all instances of feedback need to balance between commitment and contrast.

A dynamic radar speed sign, contrasting the road’s speed limit with drivers’ current speed (src).

My favorite example of contrastive feedback is the dynamic radar speed sign. Using just two numbers (hardly a visualization), these clever signs significantly improve safe driving. You can see an example above. The number on top shows the current speed limit. The number at the bottom shows the passing driver’s speed (as determined by a radar gun attached to the sign).

Notably, neither of these signs offer drivers new information. All cars have speedometers built into their dashboards. All (most?) roads have speed limit signs. But, by putting the two numbers side by side, it invites drivers to compare their speed with the speed limit. It creates contrast, encouraging drivers to slow down.

According to Thomas Goetz’s reporting in Wired, officials in Garden Grove, California, used these signs to great success in reducing speeds near school zones. After failed attempts at more heavy-handed approaches (e.g., writing lots of tickets), city officials deployed these driver feedback signs across five different school zones and saw speeds drop 14 percent in nearby areas. According to Goetz, dynamic speedometers have similar effects elsewhere, showing 10 percent speed reductions overall.

A few years later, Stanford researchers Kumar and Kim demonstrated similar results by inverting the roadside dynamic speedometer: they moved the speed limit sign inside the car. Their prototype was a small display, mounted near the car’s dashboard, showing the speed limit for the car’s current location right beside the car’s speedometer. In their (simulator) experiments, their test subjects drove 13 mph slower.

Today speed limit signs are increasingly built into cars’ dashboard displays. Not to be outdone, many roadside signs have upgraded to emoji displays, presumably seeking to add a small dose of guilt to the mix (other evidence suggests this is actually effective).

Atom Trees and why counting is encouraging

Screenshots from the Atom meditation app, showing 1 tree for each meditation session.

Data visualizations don’t need to be actionable to be influential. Similarly, feedback doesn’t need to be contrastive to encourage positive change. Instead, you can help users strengthen commitment toward a goal by visualizing their past progress.

Screenshot from the Atom meditation app, showing a weekly timeline with a check for each day I meditated (and nothing for the days I skipped).

The meditation app Atom offers a recent example of this. In addition to offering guided meditations (ala Headspace), Atom offers two simple ways for users to visualize their progress: a weekly timeline with a simple checkbox for days you’ve meditated (left) and a small, growing forest that adds a new tree for every completed meditation session (above).

Above you can see my tiny grove expanding to nine trees for the nine sessions I’ve meditated. Whereas the timeline shows the gaps in my meditation practice, the trees are purely a count of my successes.

A number of studies have explored metaphorical visualizations for progress tracking (e.g., with fish, gardens, monsters) but there are even more examples where simply counting something is quite encouraging (consider Fitbits and other pedometers, Wii Fit stamps, Snapchat streaks, bullet journals).

One possible explanation: people like to see themselves as consistent (src), so seeing our past actions might remind us of our past goals which we’d like to carry forward.
Another reason: it’s fun. It feels good. Atom’s little trees are pleasant and seeing a new one pop up is satisfying. It works the same way chocolate motivates you to open the next door on an Advent calendar. When the feedback itself is rewarding, you associate that positive affect with pursuing the goal.

Feedback can reinforce commitment by memorializing previous activity and by offering a generally pleasant experience.

A Forgiving Weight Graph

Contrast creates change. Highlighting the gap between the current state and a goal state creates awareness and tension that you can then work to resolve. But if closing the gap feels unattainable, people can lose their commitment and give up entirely. This dynamic is critical in the context of weight loss.

Maintaining Commitment

Three weight tracking graphs from popular health tracking apps (Noom, Withings, and Google Fit). Note the prominent trendline v.s. the more granular weigh-in measurements.

Weight graphs are common in personal fitness apps. They’re typically a line graph, plotting the measurements of users’ weigh ins, helping them visualize progress toward a long-term weight goal. Above you can see three different examples from Noom, Withings, and Google Fit.

Despite the ubiquity of this feature, weight tracking is somewhat controversial. Even though it’s associated with improved outcomes (src, src), many worry that simply stepping on the scale might discourage people enough to give up entirely.

This is called the “what the hell” effect. When you’re trying to do something new and difficult, all it takes is a few examples of discouraging feedback for us to lose our confidence and give up on the journey all together. (This is particularly challenging with weight loss. Our weight varies naturally throughout the day, but this natural variation can look a lot like failure. You haven’t gained weight, you might just need to poop!)

This highlights the tension between commitment and contrast-oriented feedback. The contrast between a user’s current weight and target weight makes weight tracking a useful intervention. It helps build intuition about the relationship between actions (e.g., what we’re eating and doing) and outcomes (e.g., changes in weight). But in the context of weight loss, commitment can be especially fragile.

Presenting contrast is straightforward. The graphs present users’ weight measurements relative to their target weight (e.g., on the Withings chart, middle, you’ll notice a horizontal goal line).

They also preserve commitment in a few subtle ways:

To build in forgiveness, the graph only shows weigh-in results as small points on a desaturated line. They give the most visual emphasis to a moving average line, designed to rise and fall more slowly, reflecting a more stoic view of users’ weight.
The graphs all have somewhat exaggerated y-axes for the given values. This further dampens the vertical distance a user would see for any given weigh in.
At least in Noom, after every weigh in, the app offers positive, process-oriented text feedback, to help users maintain a sense of efficacy and commitment.

Based on user comments, the design seems to be appreciated:

“Another reason that I weigh in every day, quite honestly, I love the way the daily weight line has been changed. I love how it irons out the outliers of the highs and lows and helps you feel not quite so bad about a day where you weighed a few pounds more than before. I really like that.”

Contrastive Feedback at Work

Benchmarks create contrast

An example slide from Worklytics WFH report. (ACME is Worklytics’ fictional demo company.)

Worklytics helps “people analytics” teams understand what’s happening within their firms by quantifying traditionally hard-to-measure organizational dynamics (e.g., collaboration, communication, employee experience, etc).

For example, following the pandemic, every big company in the world wanted to know: “Are we doing this ‘remote’ thing right?!” Worklytics’ “Remote Work Analysis” answers that by giving clients visibility into how surprisingly-consequential behaviors like “emails after 6 pm” changed throughout the pandemic (and how they can affect employees’ work/life balance).

(Full disclosure: I work with Worklytics, designing reports like this.)

Previous versions of this slide only included the blue trendline, but not the yellow benchmark range. As you might expect for a B2B data product, when we showed charts like this to clients, the first thing they’d ask — inevitably — was, “Okay, so it’s up, but is that normal?!” Before including the benchmarks, Worklytics’ co-founder, Phil Arkcoll, spent half of every client presentation answering some variation of that same question for every slide.

Is this because business people love benchmarks? Yes. 100%.

What drives this affection? Benchmarks’ popularity stems from contrast. They give users a reference point for comparison, turning a lonely metric into contrastive feedback, therefore making it actionable.

Specifically, benchmarks trigger one of three responses:

If the metric is outside (worse than) the target range, we’re doing poorly (e.g., worse than the majority of peer firms). The gap between the trendline and the benchmark range highlights: a) there’s room for improvement, and b) how much you can improve. Awareness of this gap creates tension, which leads to conversation, which leads to change.
If the metric is within the target range, you’re doing okay and this is one less thing you need to worry about. (This might seem trivial, but it’s actually huge. More on this below.)
If the metric is beyond (better than) the target range, not only do you not need to worry, you have something to brag about! (When “actionable” ⇒ “getting users promoted” you’re on the right track!)

Feedback Determines Focus

Slides from Worklytics’ “Remote Analysis” report.

Benchmarks help us prioritize.

When presenting performance feedback, it’s easy to forget that you’re not just answering the obvious question (“Are we good/okay/bad?”). There’s also an implied question for every metric: “Do I need to worry about this more or less than the 20 other KPIs I’m tracking?!”

This is especially true for executive teams. When even attention is a scarce resource, it’s actually quite useful for a graph to say, “Nothing to see here!” It lets viewers quickly (and safely) put the question to rest and move on to the next thing (i.e., the other areas with larger opportunities for improvement). Research suggests this phenomenon applies to non-execs as well: When you hear you’re doing well at one goal, you tend to shift your focus to other goals.

This is another area where Worklytics’ benchmarks are effective. Their reports, as you can see above, are comprehensive. Similar to Tufte and Powsner’s “Patient Status” viz, Worklytics’ reports use a common visual language to denote: a) the measurements (blue-ish lines), and b) the benchmarks (yellow areas). So on every slide, users can answer that first implied question at a glance (i.e., “Do I even need to worry about this?”), then quickly move on and identify the areas that need their attention the most.

Since adding benchmarks to their WFH reports, not only has Phil been saved from endlessly answering “is that good or bad?,” clients have also remarked on how the additional contrast helps exec teams make faster, more confident decisions about where to focus their attention.

All organizations do silly things. The bigger they are, the sillier they can get. Given an infinitely-long list of silly things to improve, aligning leaderships’ focus on the right problems is a crucial first step toward positive change.

Recap

“What you measure you improve” is backed by the psychology of feedback. Understanding this can make us more persuasive data designers.

Effective dashboards provide effective feedback and rely on two dynamics: 1) they increase our commitment to a goal, ensuring we’ll stick with it, and 2) they draw contrast between our current state and goal state, helping us see how far we need to go and our biggest opportunities for improvement.

When designing dashboards (or other quantitative feedback systems), there are a lot of different tactics for influencing commitment and contrast:

You can build commitment by highlighting past progress (e.g., Atom, fundraising progress bars), making the feedback experience itself rewarding or fascinating (e.g., Atom, Robinhood), downplaying failures (e.g., weight graphs), and shrinking the perceived distance to a larger goal (e.g., fundraising thermometers).
You can draw contrast with as little as two numbers (e.g., speed limit signs), long distance goals (e.g., weight graphs, fundraising progress bars), social comparisons, and various benchmarks (e.g., Tufte and Powsner, Worklytics).

Radical Dots Simulator: Visualizing Agent Belief Change

Thu, 04 Mar 2021 00:00:00 GMT

The world is full of people who are immune to data and reason. We’ve got Q-anons, anti-vaxxers, flat-earthers, and many millions of delusional people who think cats make fine pets (which is exactly what the whiskered illuminati want you to think).

But what can we do? If you come at people — guns blazing — with your opposing beliefs, the best case you can hope for is not being persuasive, worst case you’ll actually drive them further away (src, src, src, src).

What’s the psychology behind belief formation? Where do our deeply-rooted, occasionally-pernicious beliefs come from? How are they influenced by the people around us? What role does evidence play in what we believe?

A few theories offer compelling explanations for how individuals develop beliefs, but how does this play out with lots of people interacting in real time?

To test this out, let’s try a small simulator, using a rough model of 100 agents, who walk around, talk to each other, and attempt to influence each other, loosely based on Social Judgement Theory and Selective Exposure.

Fast forward a couple weeks later, we’ll find we’ve spent a stupid amount of time watching little swarms of small dots dance around and argue with each other, but we’ve found a few cool things.

Cool Things:

With just a few simple rules, agent-based models can show complex (and weirdly hypnotic) emergent behavior. (The dots are dancing for a reason!)
To the extent that a) the theories represent reality and b) the model represents the theories (two big qualifiers), it seems like Social Judgement Theory + Selective Exposure explain some interesting real life behaviors (e.g., groupthink, echo chambers, the persistence of strong beliefs).
Visualizing agent-based models, and all their different variables, was a struggle (i.e., how to show 100 people × their 10 beliefs, all changing over time?), but eventually I stumbled onto a technique that worked surprisingly well.

Before we dive in, let’s talk about some theory (and cats (who are the worst)).

Here we have three people with different attitudes towards cats. As you’d expect, the Cat Lover loves cats, Cat-Neutral person is ambivalent, and Cat Hater hates cats.

The Theories

Social Judgement Theory of Attitude Change

This project was inspired by a theory of attitude formation called Social Judgement Theory, which was first proposed in the 1960s by three social psychologists: Carolyn Sherif, Muzafer Sherif, and Carl Hovland.

They proposed that people accept (or reject) new ideas based on their pre-existing “latitudes” of rejection or acceptance. The short version:

If we hear an idea that we agree with, we’ll strengthen our existing belief (toward the new idea).
If we hear an idea that we’re neutral toward, we’ll change our weakly-held belief (toward the new idea).
If we hear an idea that we disagree with, we’ll strengthen our existing belief (away from the new idea).

(Note: Both the first and last scenarios result in the person doubling-down on their existing beliefs.)

What happens when you tell a Cat Lover that “cats are the worst?” Hint: It does not persuade them of feline duplicity, instead the cat lover strengthens their love of cats; specifically, their latitude of rejection grows and their latitude of acceptance shrinks.

What does it mean to “strengthen” our attitudes? The more “involved” we become with a topic, the stronger our attitudes become. These “strongly held” attitudes are defined by narrow latitudes of acceptance and wider latitudes of rejection. That is, once we’re involved in an issue, we only accept a narrow range of new ideas that confirm our existing beliefs, and we reject a wider range of ideas that (we perceive to) contradict our beliefs.

In the example above, the man in the purple shirt tells the cat lady, “Cats are the worst.” Instead of convincing her, his statement has the opposite effect and she becomes more extreme in her original belief; she becomes more ego-involved, widens her latitude of rejection, and shrinks her latitude of acceptance. After this, the only acceptable ideas about cats are those acknowledging absolute feline exceptionalism.

One important takeaway for the data community: sometimes it doesn’t matter how good your data is, or how well it’s presented — changing minds, even with data, is anchored by pre-existing beliefs (src).

What happens when you tell three people, each with different initial attitudes toward cats, that, “Cats can be furry little jerks?” The Cat Lover and Cat Hater entrench their pre-existing beliefs. The Cat Neutral is perhaps persuaded and becomes more anti-cat.

For example, if you tell someone, “Cats can be furry little jerks because they vindictively knock things off the counter,” they won’t necessarily stop to consider the merits (or how much glassware they keep on their countertops). Instead, their attitude will change based on their pre-existing opinion of cats.

Cat haters (aka realists) might say “True! Cats are the worst.” Their anti-cat convictions will become stronger and eventually even neutral ideas like “cats are fine” become unacceptably pro-cat.
Cat-neutral people might say “I suppose.” They may actually stop to consider the argument and, upon consideration, will grow to dislike cats a bit more. In the example above, the cat-netral person flips their scale.
Cat lovers might say “No way!” and strengthen their resolve that cats are the best. They shrink their latitude of acceptance and grow their latitude of rejection, until even neutral ideas like “cats are fine” become unacceptably anti-cat.

Belief formation is obviously more complicated than this, but Social Judgement Theory provides a compelling theoretical foundation for why it’s counterproductive to confront strong believers with opposing beliefs.

This has a few interesting implications:

Arguing directly against a strongly-held belief is counterproductive.
It is easiest to change our minds when we’re neutral toward a topic. Once beliefs become strongly held, they’re nearly impossible to change.
Cats are, in fact, the worst (Sherif, Sherif, and Hovland have not spoken to this, directly, but we can assume they’d also reach this inevitable conclusion).

Selective Exposure Bias

Cat people like to hang out with other cat people. Dog people like to hang out with other dog people. (Metaphorically.)

Another dynamic to consider: our attitudes are driven by what we’re exposed to, but we tend to only expose ourselves to information that confirms our existing beliefs. We choose confirmatory information in lab settings (src, src). We surround ourselves with people who look like us and share our values (src). And Facebook / Twitter / Youtube (or any other personalized feed) exacerbate this exposure bias by feeding us content optimized for our engagement (e.g., things that either confirm our pre-existing beliefs or lead to internet arguments that entrench our beliefs even further).

The Simulation

Sherif, Sherif, and Hovland explored attitude latitudes in the context of an individual’s response to new ideas, but what happens when we scale up to a bunch of people, with a bunch of different beliefs? How might it play out when we consider exposure dynamics? Would we see some of the polarization and extremes we see in real life?

In the simulation, each person (dot) has 10 beliefs and two position coordinates.

Setup:

Let’s start with a person (from here on, a dot). Let’s say each person (dot) has 10 beliefs, ranging from -1.0 to +1.0 (e.g., -1.0 might agree with, “Cats are the worst,” +1.0 might agree with, “Cats are the best”). Let’s also give the people (dots) physical locations (x,y), so they can move around and speak with other people nearby.

100 people = 100 dots

Then let’s say we have 100 people. We’ll initialize them with 10 random beliefs and random positions, and then apply the following set of rules over several turns.

Rules:

Each turn, one or more random dots “speak” to their neighbors about a random belief.
Dots that “agree,” step closer to the speaker and adjust their attitudes toward the speaker’s belief. Dots that are neutral also step closer and adjust their belief toward the speaker’s belief.
Dots that “disagree,” step away from the speaker and adjust their belief away from the speaker’s belief.

That is, beliefs are updated based on each dot’s latitude of acceptance (rejection) and as dots move to (from) speakers, they’re exposing themselves to more (fewer) other beliefs that they agree with.

Dots (people) forming small groups based on their common / opposing beliefs.

In this animation, you can see small groups of people (dots) banding together, presumably based on their common beliefs. We’ll discuss more of these behaviors below, but first let’s figure out what’s actually happening.

The problem: From the visual, it’s tough to tell why the dots are sticking together. Do the grouped dots actually share beliefs? Is it some random effect based on the movement rules? Or maybe it’s a bug in my embarrassingly-long Observable notebook?

Assuming the simulation has something meaningful to say, what’s the right way to visualize the underlying variables and see what’s actually driving the behavior?

Visualizing 100 dots × 10 beliefs × location, over time

Location, location, location.

Dots (people) forming small groups based on their common / opposing beliefs. The dashed lines indicate where dots are walking. The large, light gray circles indicate dots’ range of speech.

First, the easy part, the dots are arranged in x,y based on their simulated physical location. You can imagine these dots milling around in some big, open space. Occasionally the dots sprout a dashed line, showing where they intend to go and then they walk toward that point. In the example above, you can see that as they form groups, the dots in each group try to walk toward the center of their group.

Importantly, the dots only speak to other dots nearby. The large, very-light gray rings around the dots show how far they can speak (and which other dots can hear them).

The physical metaphor makes for easy visualization (and it vaguely looks like the Senate floor during a big vote), but it’s also analogous to other behavior. The dots walk toward and away from speakers based on whether or not they like what they’re hearing, in the same way we follow / unfollow (friend / unfriend) people on the internet, or move to states / cities / neighborhoods that we think will have other people who reflect our values. All of this “movement” impacts which other people and which information make it into our exposure bubbles.

What do most dots believe?

What do the dots believe as a community? How about an opinion poll?

This viz shows the distributions of all 100 dots across all 10 beliefs. Each row represents a belief. Each dot’s x-position represents their belief value for that belief. The left end is -1.0 (disagree) the right end is +1.0 (agree).

This is a stacked beeswarm visualization, showing where each dot stands on a particular belief. There are 10 rows, one for each belief. Each of the 100 dots is repeated on each row, positioned in x by the value of the dot’s corresponding belief.

In this example you can see the dots start with wide variations in belief values, then converge toward consensus in the middle. In the top two rows, you can see two-to-three distinct groups attempting to break away from each other.

This version shows two views of the dots. On the left is the map, showing the dots’ location. On the right are the distributions of their belief values. Dots can be selected, turning them orange. This lets us compare dots’ locations with their specific beliefs.

Rendering the dots individually in a beeswarm also makes it possible to select them individually. This allows highlighting dots across both views, making it easier to follow belief changes for individual (or small groups of) dots.

Two Views → One View

There are two problems with the map + beeswarm visualizations:

Moving our eyes back and forth is a pain, especially if we’re trying to watch a single dot evolve over time. It’s hard to follow macro-changes.
It still doesn’t answer our original question: are these dots hanging out with each other because they have shared beliefs?

Is there a way to get some sense of dots’ beliefs inline, on the original map? It doesn’t need to be precise, a “gist” read is good enough. A combined visualization would ideally accomplish the following design goals:

Dots with similar beliefs look similar at a glance. Dots that are different look different. (These aren’t necessarily the same thing!)
Show all 10 beliefs on each tiny dot, with some sense of magnitude for each belief value. It’s not as important to distinguish specific beliefs.
When similar dots gather closely together, they “blend” and look like a cohesive group. Similarly, heterogeneous groups’ outliers stand out clearly.
Micro and macro changes are discernible as the dots evolve over time.

Viz Attempt #1: Single Belief, Diverging Color Scale

Here we can see dots colored based on their values for belief #1. Orange is -1.0 and purple is +1.0. Watch the top groups’ colors (attitudes) change as they merge.

This animation shows each dot colored along a diverging color scale, based on the value of a single, selected belief (orange = -1, purple = +1.0).

In this example, we’re looking at how each dot feels about belief #1. The dots in the bottom-left are orange, because they oppose the belief. The top-right group is purple, because they’re in favor.

Eventually the top-left and top-right groups merge together, causing the top-left group to change from orange to purple, indicating they changed their belief to match the (purple) top-right group.

This view accomplishes some of our design goals, namely it makes differentiating dots easier along a single belief dimension. But, since the dots have many beliefs, it’s hard to get an overall sense of change.

Viz Attempt #2: Stars / Spokes

Four attempts at displaying dots’ 10 beliefs radially. The dots in each of the four variations are identical.

In Oriented Texture Slivers, Weigle and friends suggest that people can differentiate up to 15 different angles of overlapping lines. I thought a similar approach might work here, given that we only need to represent 10 beliefs. For this exploration, each dot has 10 angled spokes, representing each of the dots’ 10 beliefs. The spokes grow longer as the beliefs become stronger (and invert when belief values are negative).

I thought the spokes looked like creepy little spider legs, so I also tried this with radial segments (which turned into pseudo-pie charts), chevrons (which turned into stars), and small arcs (which turned into sea shells).

While this made it possible to differentiate some of the dots’ beliefs, none of these achieved the desired glanceability. I’d hoped that seeing many parallel spokes would form a semi-continuous “texture” across nearby, similar dots, but often it just looked like random cross-hatching.

Worse still, it makes dots with exactly opposite beliefs look almost identical (i.e., a dot with even beliefs = +1, odd beliefs = -1 looks very similar to a dot with even beliefs = -1, odd beliefs = +1)

Viz Attempt #3: Back to Colors

These dots are colored based on all 10 of their beliefs, by grouping their belief values into three groups, then mapping the average of each group to RGB values. Here you can see the dots forming into five different groups, with distinct beliefs per group and therefore different colors.

Here we’re coloring dots by mapping their 10 belief values into RGB space (by chunking the 10 belief values into three groups, taking the average of each group, scaling each average to 255, then assigning it to R,G or B).

I dismissed this approach initially, assuming the beliefs would cancel each other out, resulting in a bunch of indistinguishable beige and gray dots. But when I actually tried it, I was pleasantly surprised.

This approach finally answers our initial question, showing that groups of dots actually do share similar beliefs. It also shows when heretical interlopers don’t belong (before the group assimilates or kicks them out). It’s not great at indicating the magnitude of beliefs, but overlaying the spider legs provides the best of both worlds.

An unexpected side effect: The two extremist dots resolve to pure black and pure white, for a nice visual morality pun!

Emergent Behaviors

Now that we can see what the dots are doing (and thinking!), let’s get back to exploring the attitude formation theories.

Dots tend toward conformity with their neighbors.

Here each dot can speak to every other dot. In this scenario dots quickly conform their beliefs. Even though they have matching beliefs, they never actually settle (as indicated by the continuously changing color of the group).

Moderate dots within speaking distance show group-think and converge to a common consensus. (This isn’t unconditional though, as we’ll see below).

Here, each dot has a megaphone and can talk to every other dot on the map. You can see they quickly come together, both physically and in terms of matching beliefs. You can also see that the spokes stay relatively short, indicating more moderate beliefs.

If you view this scenario in the simulator, you can see that their overall belief distributions hover perpetually in the middle.

Isolated dots form diverging echo chambers.

Here the dots form three different groups. Within those groups the dots conform to each other, but between the groups the beliefs are very different. Because there are fewer voices per group, extreme dots can have an outsize influence.

Similar to real-life echo chambers, once the dots divide into bubbles, they quickly sync up their beliefs. Each group’s beliefs diverge from other groups and they tend to be more extreme.

This is actually an extension of the group-think effect. The dots conform with their immediate neighbors, but because each group starts with different members, they end up in different places.

Echo chambers occur naturally, but in this scenario we’re forcing it by initializing the dots to be more physically distant and giving them a slightly larger speech radius to make sure they form into two-to-three neat groups and develop their beliefs before inter-group communication can happen.

Once beliefs become extreme, they stay that way.

This is the map and the belief distributions of a later-stage simulation. You can see on the right that all of the dots have settled into strong and extreme beliefs (see splits between left and right).

This is the map and corresponding belief distributions for a later-stage simulation.

In scenarios where echo chambers form, beliefs tend toward polarized extremes. You can see on the right that 8-of-10 beliefs are split (e.g., a bunch on the left, a bunch on the right).

Just like in real life, extreme beliefs are hard to change. The model is perhaps more extreme than we’d see in real life because once it reaches this state, it’s basically stuck for the rest of the simulation; the dots may continue to change their locations, but their beliefs are static. In the current model, the only way to pull a dot out of an extreme is for a critical mass of other dots with more moderate beliefs to approach it and win it over, but typically all the dots become extreme together.

You can see this emerge faster in the simulator by turning up the play speed.

In real life, beliefs develop in response to more than just the people around us. We, at least, have personal experience and objective reality to anchor (some of) us to reasonable beliefs, so it’s less likely an entire population descends into total madness.

Dots join / disband based on common beliefs.

Above the top-left two groups merge together, but retain their different beliefs.

If you watch these long enough, you’ll eventually see unexpected behavior like this. At the top of the map, we see two groups of heterogeneous dots merge into a single group.

Their beliefs are already fully formed (and locked into extremes), so they won’t converge on common beliefs (you can see they retain their original colors) but they seem to stick together — at least temporarily — because they agree on more things than they disagree (e.g., for every four speeches that repel the group, six draw them back together). This isn’t usually a stable formation though; typically the groups will split up again and find their own separate homes.

You can (usually) see this play out in the simulator with these settings.

Extremist dots end up lonely.

Notice the lonely white (top-right) and black (bottom-right) dots.

You might have noticed that in each of our groups there’s always a black and white dot, with opposing extreme beliefs. As we’ve learned above, once extreme beliefs set in, they don’t really change. Most other dots develop their beliefs together, so they tend to reach consensus, but the extremist dots stick purely to their beliefs, and as a result end up being pushed out by the others.

Preachy dots drive each other apart.

These dots may look colorful and friendly, but they are, in fact, little jerks.

The dots in this scenario operate under a different rule than the others we’ve seen. Dots in previous sections choose which belief to “speak” about randomly, so they discuss an even distribution of beliefs. Dots in this scenario, however, are more narrowly focused and only speak about their most strongly held beliefs.

You can try this scenario in the simulator with these settings.

When dots only speak to their most extreme beliefs, they push each other to extremes more quickly, leading to more fragmented sets of strong beliefs, causing each dot to have less in common with its neighbors, ultimately driving them apart. You can see here that the dots all have different beliefs (i.e. there are 10+ colors here v.s. 2–3 colors in other scenarios). You can also see that most dots are hovering around the perimeter — that’s because they’re (literally) trying to escape each other.

An innocuous interpretation: imagine all the dots are recent vegetarians, CrossFit converts, or Tesla owners. They’re excited, but mostly harmless. (“We get it Dave, 400 miles on a single charge really is impressive. Thank you for sharing. Again.”). We all have our soapboxes — annoying our friends is part of the fun!

Slightly more troubling: this looks a lot like the environment created by engagement-optimized newsfeeds (e.g., if Facebook’s algorithm mistakes heated arguments for “engagement,” people see more arguments in their feeds and are more exposed to others’ extreme beliefs). To the extent that the real world looks like dot world, engagement-optimized feeds would not only accelerate extreme belief formation, but they’d also allow our opposing beliefs to drive us apart faster than our common beliefs keep us together.

What behavior do you see?

Try the simulator for yourself here: Radical Dots Simulator Notebook

The notebook has both the dots “map” and the beeswarm belief distribution, with controls for tweaking the parameters.

Recap

Good charts aren’t always enough! Social Judgement Theory suggests that others’ attempts to influence us only strengthen our pre-existing attitudes. This explains why even strong evidence often fails at persuading others of opposing beliefs.
Our tendency towards Selective Exposure means we tend to avoid information that doesn’t confirm our pre-existing beliefs.
The simulator attempts to model Social Judgement Theory and Selective Exposure with three relatively simple rules. Despite this naive approach, it shows surprisingly realistic emergent behaviors like groupthink, echo chambers, and the stubborn persistence of extreme beliefs.
Neither the theory nor the model offer silver bullets for neutralizing extreme beliefs, but they both suggest it’s pretty useless to be too pushy about our own beliefs.
From a purely viz perspective, we looked at a few different ways to visualize dots’ beliefs and found that color can be super effective for visualizing belief convergence/divergence for groups of dots. (Perhaps this is useful for visualizing other agent attributes in agent-based simulations?!).

Designing Approachable, Self-Serve People Analytics

Mon, 01 Feb 2021 00:00:00 GMT

Context

Client

A leading startup in the workplace analytics space. Client gives “people analytics” teams visibility into their firms by quantifying traditionally hard-to-measure organizational dynamics (e.g., collaboration, communication, employee experience, etc).

Prompt

How can we communicate a mountain of insights in a way that’s easy for executives to digest and take action on?

Background

As Covid-19 emptied out offices, every organization made a hurried transition toward working remote. For many firms, particularly the very large companies that partnered with the client, there was already a struggle to understand and optimize their organizations. Without line-of-sight between managers and employees, the struggle became even more acute and many turned to workplace analytics. This created significant demand for the client’s data and insights, particularly their “Remote Analysis” reports.

Challenges

Low-level data can’t stand alone. Client’s reports relied heavily on the presenter’s narration. However, for their insights to be impactful beyond the initial presentation, the data needed to also be easily understood by other stakeholders outside the room.
“Hand crafted” reports don’t scale. Reports would often take several days of manual wrangling between Data Studio, Google Sheets and Powerpoint.

Design

Insight

With purpose-designed data visualization, the client’s reports could more easily reach executive teams beyond the initial presentation, supporting more informed decision-making throughout their organizations. By automating the busywork, we could free up time for more value-add analysis.

Services

Redesigned charts, graphs and slides for improved understandability and engagement (matching client’s brand and visual language). Developed story and structure for deck, making it easier for readers to navigate and understand all 50+ slides as a whole.
Worked with leadership team to translate their context, experience and overall insights into the deck’s commentary, annotations and benchmarks, making analysis more standalone, approachable and self-serve.
Automated and templatized labor-intensive report-creation process.
Designed Google Data Studio dashboard templates to support ongoing, end-user monitoring of underlying workforce dynamics.

Results

Client’s clients report that executive teams make faster, more confident workforce decisions.
Reduced busywork to generate presentation decks from days to minutes.

Polling Dataviz Component Development

Fri, 01 Jan 2021 00:00:00 GMT

Context

Clients

Graphicacy + client (a survey analytics product platform, used by the world’s leading polling & market research firms).

Prompt

How might we build robust, product-ready polling data visualizations?

Background

The client’s product is a purpose-built analytics tool for pollsters and market researchers to visualize and analyze large-scale survey results. Because surveys unfold over time (e.g. for longitudinal studies, research panels), time is an important component in comprehending survey data. Given the importance of time series data and growing pains with their off-the-shelf charting library, the client saw time series use-cases as an opportunity for improving both comprehension and UX.

Challenges

Time series analysis was an important use-case for the client’s users, but Plotly (their off-the-shelf charting library) didn’t provide the flexibility or interactivity to deliver on the needed UX.
Graphicacy wanted expertise on delivering their designs as embedded, in-product data visualizations that played nicely with the client’s existing react codebase.
Survey data, like all data, can be messy. The visualization needed to be robust against missing or misformatted data and a variety of users.

Design

Insight

Embrace the chaos! In-product dataviz is only as good as the test dataset. In addition to developing the visualization components, automated tests can ensure the dataviz renders properly in even the weirdest conditions, so edge-cases won’t creep through to end-users,

Solutions

Developed a custom, dynamic time series graph as configurable, maintainable React components, fully integrated into their existing codebase.
Developed a test framework and full coverage of automated tests so issues are spotted early and effortlessly and their engineers can maintain the code with confidence.

Results

“We were looking for help developing an in-product data visualization for one of our clients. We needed to make sure it was well-architected, robustly tested, and performant. We're very happy we worked with 3iap. Eli understood exactly what we needed, kept us updated at every stage of the development process, and even helped us clarify the open UX questions. The results were great, the final product worked well, looked clean, and gave us a happy client."
Chris Lanoue, Director of Engineering and Innovation, Graphicacy

Doom Haikus: Machine Learning, Poetry and Despair

Tue, 01 Dec 2020 00:00:00 GMT

Context

Prompt

How might we develop a training dataset for machine summarization (and cope with the worst year in modern human history)?

Background

The project started with a seemingly simple goal: teaching Tensorflow to write haikus about current events. Since poetry is subjective, there aren’t straightforward objective functions to quantify good haikus, so we’d need some actual haikus as training data (i.e. self-supervised is out).

Challenges

2020 had no shortage of news to seed moody poetry, but at the time no such news → haiku datasets existed.
There are a number of ways to gather up pre-written text in the shape of a haiku (5,7 and 5 syllables), but these would surely lack the spirit of a real haiku (much less the kireji).

Approach

Insights

When you need repetitive and weird things done by humans on the internet, there’s only one place to turn: Mechanical Turk.

Design & Build

Developed a data pipeline to extract top posts from popular news-related subreddits, scrape the referenced URL to develop a prompt, submit custom “human intelligence tasks” (HITs) to Mechanical Turk, then pull results, clean and store them. (Technology was Google Cloud Run, Firestore, Clojure)
Developed a supplemental database of word word syllables that don’t appear in canonical sources like CMU’s Pronunciation Dictionary
Automated the review process for the majority of submissions, built tooling to facilitate quick manual review of exceptions
Data Product Prototyping. Developed a javascript prototype app to explore haikus, combining instant, client-side search with a custom canvas data visualization, allowing users to read individual haikus but also get a sense for overall news trends. (Technology was React, D3, Canvas)

Results

From March - December 2020, ~2,000 people on Mechanica Turk submitted ~2,700 hand-written haikus about the news events of the day.
Launched https://doomhaikus.3iap.com in December 2020.
Project / dataset were featured on Data Is Plural, Nightingale, Product Hunt (home page), Boing Boing and AVClub.

Visualizing Expectations

Sun, 15 Nov 2020 00:00:00 GMT

It’s hard to do hard things. Continuing to do hard things is even harder.

American culture glorifies persistence. When we hire, we hire for “grit.” We lionize the leaders who saw us through our longest, most difficult times (e.g. MLK, Washington, Churchill, FDR, Moses, Frodo). Resiliency inspiration saturates social media (e.g. Einstein: “It’s not that I’m so smart, it’s just that I stay with problems longer.”). And we all remember Mitch McConnel’s compliment to Elizabeth Warren’s tenacity: “Nevertheless, she persisted!”

But what if persistence isn’t some innate super power?

As we’ll see, our stick-to-it-iveness depends on our expectations. And our expectations depend on our available information. This makes expectation-setting a powerful use case for information design.

So let’s explore the psychology of information and expectations, then look to some effective examples of visualizing expectations in the wild.

Informational Grit

Consider Jeff Bezos’s anecdote in his letter to Amazon shareholders (src):

“A close friend recently decided to learn to do a perfect free-standing handstand… She then practiced for a while but wasn’t getting the results she wanted. So, she hired a handstand coach… In the very first lesson, the coach gave her some wonderful advice. “Most people,” he said, “think that if they work hard, they should be able to master a handstand in about two weeks. The reality is that it takes about six months of daily practice. If you think you should be able to do it in two weeks, you’re just going to end up quitting.” Unrealistic beliefs on scope - often hidden and undiscussed - kill high standards. To achieve high standards yourself or as part of a team, you need to form and proactively communicate realistic beliefs about how hard something is going to be.”

One of the perks of being a billionaire is friends who can afford handstand coaches, because these coaches are full of surprisingly practical wisdom. There’s quite a bit of evidence supporting this wisdom:

In 2002, Oettingen & Mayer showed that healthy expectations can also improve our chances for success in 1) job hunting, 2) dating, 3) test-taking and 4) recovering from hip-replacement surgery (src).
In 2010, Swift & Callahan showed the power of expectation setting for encouraging clients to remain in psychotherapy (src). By telling new clients, up front “this process usually takes 13–18 sessions” they improved client retention significantly, boosting therapy sessions by 80% (from 5.9 to 9.9 sessions).
In 2019, Chu & friends spent 130 hours interviewing patients about their experiences waiting to see a doctor (src). Their #1 suggestion: “Proactively informing patients of delays.” Setting expectations about delays improves “willingness to wait” by reducing uncertainty and increasing tolerance.
Earlier this year (2020), Briscese & friends explored how expectations impact willingness to comply with Covid-19 stay-at-home orders (src). They asked 2,697 participants what they’d do if quarantine orders continued for X more weeks / months. When X was “much longer than expected,” noncompliance intentions more than doubled.

https://xkcd.com/612/

What makes expectation setting powerful?

Reducing Uncertainty. There’s an intrinsic reason for valuing clear expectations: Uncertainty hurts. Actually, in some cases (src), uncertainty is worse than physical pain! Some researchers suggest that anxiety itself is just our body responding to uncertainty (src, src). Others suggest that the reason we seek any information is reducing this uncertainty-driven anxiety (src, src).

Agency. Weather apps have earned a spot on all of our home screens (and the #2 use case of Alexa / Google Home, src) because, before we walk outside, we want to know if it’s hot or cold and what’s falling out of the sky. There’s obviously nothing we can do to change the weather, but the information gives us some agency in how we prepare for it (e.g. flip-flops or snowshoes?).

Preempting Trouble. When doing a hard thing, there will always be bad days. Every diet will have a few unintended cheat days. Quitting smoking often means multiple “last cigarettes.” Every Poké Trainer will lose a few battles on the road to Poké Mastery. These slip-ups are only a problem when they spiral into self doubt. Preempting helps us avoid downward spirals by normalizing the inevitable setbacks and putting them in context of the larger journey, helping us preserve our sense of efficacy.

Information Design Examples

Our willingness to endure depends on realistic expectations about the road ahead. This might explain why expectation setting is so prominent in information design.

A collage of countdown clocks, from various subway stations across NYC.

Countdown Clocks

If you’ve ridden the subway in NYC in the past 10 years, you’ve likely run into what the MTA calls a “Countdown Clock.” When you’re waiting on the platform, you can look up at these big displays to see an ETA for the next train’s arrival. The MTA loves these, so they spent 11 years installing them in all 471 subway stations.

What’s so great about countdown clocks? They imbue patience in subway riders. According to Zou & Sha, this leads to very positive outcomes (src):

Countdown clocks reduce riders’ perceived wait time
They improve rider satisfaction rates (& perception of the MTA)
And, they actually improved weekly ridership (1783 riders per station)

A screenshot from Noom's onboarding flow, unfortunately reflecting the author's actual distance to a reasonable weight.

Weight Loss

Losing weight in a healthy way is a long term journey, but every day can feel like a struggle. For people trying to lose weight, it’s very easy to lose sight of the goal and let a few instances of stress-eating slip into a downward spiral (see: the “what the hell” effect).

When I worked at Noom in 2014, we confirmed this in our own user research, indicating how easy it was for users to feel lost. Retaining users in Noom’s program meant helping them visualize how their struggles today would pay off tomorrow.

Noom’s weight graphs help users visualize not just their past and current weigh-ins, but also their projected weight in the future. Both the in-app weight graph and the onboarding visualization (above) show the user’s current weight, projected forward, decreasing roughly ~2lbs / week until the day they hit their target weight. Not only does this set realistic expectations about weight loss as a long term commitment, it gives users something to look forward to. From personal experience, I can tell you that Noom users are very attached to this feature.

A screenshot of Google Maps' recommended routes from NYC's Upper West Side to Union Square.

Route-Planning

Google Maps has 2 goals when showing you routes: 1) helping you get to your destination (safely, quickly, etc) and 2) encouraging you to continue using Google Maps.

The product’s predicted ETAs set expectations about travel time. The traffic density indicators preempt trouble ahead. Both of these make the road ahead more bearable for users. They also inform the user’s decision about alternative routes / travel methods, improving agency.

Toward Goal #2, the product devotes a fair amount of UI to CYA (“cover your ass”) expectation setting. Route-planning is difficult, particularly so because the user’s travel experience is out of the app’s control; at rush hour, even a perfect algorithm may only have bad options to offer. In those situations, letting users know that even the “fastest route” will still have traffic (and exactly when / where to expect it) preempts these unexpected gotchas. As an added bonus, it’s also a clever way to say “Don’t blame us when you’re stuck in traffic, blame the cars in front of you!”

A kindle, showing estimated reading time for the entire book.

Estimated Reading Time

According to Arienne Holland, estimated read times first appeared on longreads.com around 2010 (src). Since then, Amazon and Kindle have also adopted this pattern. These services have a shared interest in encouraging users to read more and Holland offers some anecdotal evidence that the indicator is effective (i.e. for improving time-on-site and reducing bounce rates).

In her listicle defending listicles (src), New Yorker’s Maria Konnikova argues that lists appeal to readers because they set clear expectations about reading time, helping us decide where to devote our attention.

“Within the context of a Web page or Facebook stream, with their many choices, a list is the easy pick, in part because it promises a definite ending: we think we know what we’re in for, and the certainty is both alluring and reassuring. The more we know about something - including precisely how much time it will consume - the greater the chance we will commit to it.

Konnikova cites 2 studies suggesting that this effect is due to reducing cognitive effort in decision making (src) and reducing uncertainty (src).

Finally, in his “Ode” to Kindle’s read time estimates, Dan Safer describes its value in terms of agency, specifically how it helps him manage his time (src):

“Do I want to finish this chapter before bed? Oh, it’ll only take me 2 more minutes, so sure.”

Excessively pleasant CSS progress bars (src).

Progress Bars

There is a crazy amount of research on the UX impact of progress bars. It turns out that progress bars can be quite effective… sometimes (src).

The Nielsen Norman Group recommends progress bars (or “percent done indicators”) for UX wait times longer than 10 seconds (src). According to Nielsen, these help “users develop an expectation for how fast the action is being processed.”

Progress bars “convey important information about (approximately) how long the wait time is. This information gives users control (that is, they can decide whether to wait or not); it decreases uncertainty about the length of the process and may reduce the perceived wait time.”

As waiting indicators, progress bars are powerful because of the expectations they set. They have an end state and they show you the distance until you reach that end state. Knowing that distance decreases uncertainty and increases agency.

There are, of course, other progress indicators like spinners or “throbbers,” that don’t show an end state, they just indicate that progress is happening. Some research suggests these can actually triple a user’s willingness to wait for a page to load, from 13 seconds to 38 seconds (src).

However, according to Cao & friends’ (src), “indeterminate” progress indicators (e.g. spinners) aren’t as effective in reducing perception of wait time. In their experiment they asked participants to wait up to 12 seconds for various media to load. When they asked “What is the maximum delay you have experienced during the test?” people in the spinner group said ~11 seconds, while people in the progress bar group said just 6 seconds.

In another study, Hohenstein & friends tested 3 different loading screens to see which designs had the biggest effect on perceived wait time: 1) a passive animation (e.g. a spinner), 2) an interactive animation (e.g. a mini-game) and 3) a progress bar (src). Consistent with Cao & friends, progress bars outperformed passive animations. But the mini-game was the best performer by far. It seems expectation-setting is powerful, but nothing beats distraction!

(One important note: Progress bars are not a silver bullet. For example, when attached to lengthy surveys they can actually increase respondents’ drop-off rates. In this context, they’re only effective when users perceive their progress as initially fast, then slower towards the end (src). And this seems to be driven more by the goal gradient effect, rather than expectation setting.)

#boatgoals

Financial Calculators

Betterment’s “Goal Forecaster” helps set realistic expectations about personal finances. Presenting the projected savings in a fan-chart helps inoculate users against inevitable economic bumps in any financial journey, while still conveying the benefits of investing over the long term.

Simulators like these help users connect the cause and effect of their actions now (auto-deposit) with likely future outcomes (goal). Making these relationships concrete helps people grasp complex concepts like exponential growth and motivates them to save more (src, src).

You can learn more about Betterment’s motivations for this feature here: Motivational Visualizations.

An example Gantt Chart from Asana.

Anti-Pattern: Gantt Charts

Detailed project plans inspire a sense of false confidence (src). Kahneman and Tversky coined the term “planning fallacy” 41 years ago (src), Fred Brooks told us definitively “The Waterfall Model Is Wrong” in 1995 (src) and even the Pentagon has recently adopted an “agile” approach for the $428B, F-35 fighter jet (src).

And yet the Gantt Chart lives on! Their continued existence, I think, speaks more to our desire for expectations rather than Gantt charts’ ability to deliver.

The problem begins at the Gantt chart’s fundamental unit: Each block represents a chunk of work, with a discrete start and end time. Since it’s so rare that actual work matches a predicted duration, these blocks mask important uncertainty. This problem compounds as the blocks stack to the right, by presenting an entire project’s completion date as a single point in time rather than a gradient of possible dates.

Instead of reducing uncertainty, Gantt charts only mask it. Combined with the planning fallacy, this leads to overly optimistic, but very precise estimates. It sets expectations that are sure to be violated.

Google Flights price prediction UX.

Google Flights - Price Prediction

Expectation setting isn’t just about enduring long journeys. Google Flights uses expectation setting to make your plane-ticket search as short as possible.

According to Google’s Vyacheslav Polonski (src), users face overbearing uncertainty when booking flights, particularly around pricing. “Buying a plane ticket is nothing like buying a cappuccino,” he says. This pricing anxiety meant users would “wait for as long as possible before booking… This process could take anywhere from a couple of days to several months.”

This is no bueno for the user. It’s also bad for Google. Waiting can easily turn into never booking or booking with a different service.

Google Flight’s price predictor solves this hesitation by setting realistic expectations about price changes in the future. When users feel confident the price won’t change in their favor, they’re more likely to book now.

Since ML models themselves can be unpredictable, the Google Flights team also put considerable effort into setting expectations about the model itself. Polonski cites Google’s Explainability & Trust PAIR Guidelines (and extensive user testing) as helpful tools for this.

According to Polonski, the results were positive: “This feature performed really well in usability tests… When people saw the price history graph and expressed satisfaction in what they were seeing, we knew we were on the right track.”

Google also (briefly) offered a price guarantee, signaling further confidence in their approach and, perhaps, that their solution to the hesitation problem was worth at least as much money as they’d lose to bad predictions.

Takeaways:

Consider expectation-setting as an important use case for information design because clear expectations give people the confidence to persevere.
Expectation-setting works by reducing uncertainty, increasing our sense of agency and preempting the inevitable setbacks of any long journey.
When possible, don’t just offer audiences data from the past. Instead, paint a path forward. By helping people visualize their future, you can: Improve patience (and satisfaction) while waiting for doctors, trains, traffic and slow websites; Encourage long-term retention in weight-loss programs and mental health treatment; Grant confidence to move forward with small decisions, like what to read, and big decisions, like booking airline tickets and investing for the long term.

Good Dashboards Inform. Great Dashboards Align.

Tue, 29 Sep 2020 00:00:00 GMT

When we think of dataviz in the workplace, we’re usually thinking of Tableau and Looker dashboards, designed to make everyone smart and support big, strategic decisions. But there’s another overlooked benefit of visualizing an organization’s data: the power to align.

Every organization struggles to keep members aligned around a common purpose. Brilliant strategy and customer insights are useless if everyone is rowing in different directions. Goal setting and OKRs go a long way toward accomplishing this, but this is where the humble dashboard can really shine.

One of the more common uses for dashboards is visualizing a team’s recent performance.

At their best, these team performance dashboards aren’t just about measurement, they’re a monument to what matters to the organization. The presence of a metric on a dashboard signals “this metric is important to the organization,” which in turn signals “this metric should probably be important to you, too.”

In this way, dashboards can do more than visualize performance, they can actually drive it by reinforcing the organization’s values, nudging the culture and focusing teams’ efforts on the most important outcomes.

In this post we’ll cover 3 examples of impactful dashboards in the workplace that were designed to help teams align their everyday behaviors.

1. Chartbeat

A screenshot of Chartbeat's real-time dashboard. The middle section, with the headlines, updates in real time to show what articles have the most viewers, right now.

Though I still cringe when I see the goofy gauge dial, Chartbeat is one of my favorite examples of how dashboards can align an organization.

When Chartbeat started in 2009, news publishers weren’t exactly enthusiastic about reader analytics. Some editors were openly hostile. For example, The New York Times’ executive editor claimed: “We believe readers come to us for our judgment, not the judgment of the crowd. We’re not American Idol.”

This initial resistance made Chartbeat’s eventual dominance that much more impressive. By 2015, the top 80% of news publishers in the United States all used Chartbeat to track audience engagement (including the previously reluctant New York Times).

One of the overlooked aspects of Chartbeat’s success was that the dashboard was so much fun to look at. One of the privileges of working for Chartbeat was visiting these top-tier news publishers for user research trips. On almost every visit you’d see Chartbeat’s bigboard, mounted front-and-center for all to see.

“Sometimes one sees writers just standing before [the big dashboard], like early hominids in front of a monolith.”

While you might debate the merits of journalists glued to their audience metrics, one thing that isn’t disputed is that many were, indeed, glued.

Nick Denton, founder of Gawker (RIP), described reactions to Chartbeat’s bigboard to the The New York Times as: “Sometimes one sees writers just standing before it, like early hominids in front of a monolith.”

What made the dashboard so engaging? Chartbeat broke ground in two ways:

The Data: Instead of relying on editorial instincts about audience interests, Chartbeat gave newsrooms hard evidence on what was popular and what wasn’t, helping writers & editors adapt stories to their audiences.
The Viz: Chartbeat’s visualizations showed user behavior in real time. Even as an observer it was mesmerizing to watch stories shuffle up and down the leaderboard. According to CJR’s Caitlin Petre, in her epic ethnography of Chartbeat, Gawker & NYTimes, users described this effect as “sanity-ruining,” “addictive,” and “the crack cocaine of web data.”

Chartbeat’s mission, of course, wasn’t to create newsrooms full of glassy-eyed journalists staring at dashboards. Aligning audiences around more meaningful journalistic metrics was both part of the mission and existential to their business. So for every hypnotic realtime dial, there were also indicators for more consequential metrics like user engagement and retention.

By commanding attention and redirecting it toward audience behavior, Chartbeat elevated the importance of the audience itself, transforming many of the world’s largest newsrooms in the process.

Focusing on the right metrics.

For Chartbeat users, the real time visualizations were powerful, but sometimes the effect could be overwhelming.

Even though Gawker’s employees were some of the most engaged, Petre found that “many writers and editors largely ignore those Chartbeat metrics designed to reward high-quality content in favor of the [real time] concurrents dial, which is the closest thing Chartbeat has to more typical metrics like page views and uniques.”

In Gawker’s case, at the time of Petre’s interviews, there were strong incentives for writers to favor more traditional traffic metrics, but it’s important to remember that any workplace intervention is sensitive to the pre-existing culture and it may take some calibration to get it right.

Effective dashboard design considers not just which metrics to prioritize or how to visualize them, but also the cultural context of the organization and how the data will be used. Or, as Petre notes: “This data is simply too powerful to implement on the fly. Newsrooms should create opportunities - whether internally or by partnering with outside researchers - for reflective, deliberate thinking… about how best to use analytics.”

Takeaway

To jumpstart a data-driven culture, one tactic is making the data more enjoyable to consume. At first, users might only engage because it’s fun to look at, but over time the things that capture our attention actually become more important to us. Beware, though: Like any gateway drug, visualization mesmerization should be used with care to avoid unwanted side effects!

2. PwC Perform

A screen from a customer testimonial video for PerformPlus. The team leader on the left is reviewing metrics with his team.

PwC UK’s PerformPlus program has a similar engaging, aligning effect.

“It’s not just about the metrics for us. You can really feel the buzz and energy on the floor.”

The program is a 12-ish week engagement where PwC coaches (real humans, non-accountants) give teams the tools and tactics to rally around their metrics. The program draws teams’ attention to the metrics that matter and teaches leaders how to interact with their teams through the language of data. This drives significant results for a wide variety of organizations, consistently improving target productivity KPIs by double-digit percentages.

“It’s not just about the metrics for us. You can really feel the buzz and energy on the floor. All of that has led to a significant improvement and change in culture. And the results have been amazing,” says Chris Rauch, EVP, Customer Success at Sage, a PerformPlus client.

Perform’s central artifact is the team’s “Information Centre” (British for “dashboard”). Because the Information Centre has a prominent physical presence, it has a similar effect to Chartbeat’s Big Boards; it creates a focal point, drawing teams’ attention to their numbers, creating alignment by elevating the perceived importance of those metrics.

One of our goals when designing the Information Centre was making people the stars of the show. We wanted team members to see themselves in the data and feel proud. So the main stories on each team’s dashboard are 1) the people on the team, 2) their individual performance metrics and 3) how their metrics cascade upward into the team team’s metrics. This helps team members frame their performance in terms of the bigger picture. As an added bonus, since everyone’s favorite word is their own name, for team members to see their name in big letters feels good, even if it’s a dashboard at work.

Team Data → Team Conversation

Perform’s secret weapon wasn’t the data or the dashboards, it was their emphasis on tying the data to team rituals. I think this offers important lessons for data-designers in the workplace.

Visualizations should always be considered in the context of the conversations they’re likely to create. I think this is true in general, but it’s especially important when communicating people’s performance data at work.

On the positive side, rituals like daily “huddles” (standups) create alignment by putting team metrics at the center of the conversation. They also serve as a starting point for proactively, collaboratively solving problems as a team (because everyone’s looking at the same data everyday, it’s hard to avoid any issues that may come up).

When done well, this creates an environment of not just intra-team accountability, but also psychological safety and mutual support.

But there are risks to visualizing team performance data at work. If it’s implemented poorly, displaying team or individual performance metrics out in the open can have surprising effects (src). Like anything in the workplace, it takes care to maintain a culture of safety, risk-taking and problem solving (src).

Consciously pairing data with regular, structured team rituals ensures the benefits of a data-driven culture, with less potential for toxicity.

Takeaway

For data to make a difference, we usually expect it leads to some action. At work, that first action is often a conversation. These conversations reinforce the importance of the metrics and nudge teams toward collective problem solving. Seeing the data helps trigger and ground the conversation, but the dynamic that actually improves performance is teams working together to solve their problems.

3. The Garden of Health

A photo of the Garden of Health installation. Each flower represents a patient at one of Adventist's locations. Each butterfly represents a nurse, physician or other provider tending to a patient.

This last example looks more like data art than data visualization, but it’s equally powerful.

The image above shows the “Garden of Health,” an installation by Schema Design Studiofor their client Adventist Health. It’s currently placed prominently in the lobby of Adventist HQ, at their main campus in Roseville, CA. Each flower represents a patient at one of Adventist’s locations. Each butterfly represents a nurse, physician or other provider tending to a patient.

According to Schema: “The goal of the project was to illustrate the diversity of the Adventist Health patients, connections and coordination between the organization’s teams, and productivity throughout the system.”

The final result was a virtual garden, acting as a visual metaphor for the patients and providers across the Adventist Health ecosystem.

“Impactful” doesn’t require “Actionable”

This installation has the same aligning effect as the other dashboards we’ve reviewed, but it’s unique because it’s powerful without being actionable. There is information being conveyed, but the main effect is motivational.

By choosing patients as the main characters in this data story, and by giving their flower avatars the most detail and emphasis, this visualization signals the same priorities to onlookers; patients are the most important thing in the visualization because they’re the most important thing to the organization.

When I spoke with Sergei Larionov, Schema’s creative director behind the project, he said that even the garden metaphor was meant to convey Adventist values, in particular their focus on “holistic” care. According to Sergei:

“The garden metaphor supports the ‘holistic care’ story beautifully … There are basically two perspectives of the human body. The Western view sees it as a machine, to be fixed when it’s broken. The eastern perspective views the body as a garden - it’s a thing to cultivate. This resonated. They got it right away.”

Alignment + Affirmation

At first you might assume the project was meant for patients’ benefit. Many studies suggest that artwork in healthcare settings can have a positive effect on patient outcomes and experiences (src, src, src, src, src, flowers too! src).

But the installation actually wasn’t for the patients at all. The HQ building where it’s installed is an administrative office. According to Sergei:

“One of the needs we uncovered through conversations with [the client] - the people in the system’s administration, who don’t directly deal with patients, they’re so removed from the patient experience.”

This adds another interesting layer to the work. The visualization became “a means for people in the administration to connect with the population of patients and providers.” By visualizing “the totality of the network” they were helping the staff feel closer to the patients, nurses and physicians whom they don’t get to see everyday.

So the visualization aligns passersby around the primacy of patients and it also gives Adventist employees something to feel proud of, by reflecting their hard work in a way that’s more immediate and visceral than their real-life feedback loops. Further still, by showing all patients across the ecosystem, it reinforces the way each person’s contributions add up to the greater good that the organization strives for.

Takeaway

Data displays don’t necessarily need to convey information to be a potent force in the workplace. Visualizations can align viewers around what’s important to the organization and create an emotional connection to what matters to us as humans - helping other people.

Or, as Sergei says, “You can think of data as any other raw material, like clay or ink… Not to communicate information, but to communicate feelings and attitudes.”

Conclusions

Dashboards at work can be so much more than a firehose of data. The higher calling for the humble dashboard isn’t greater dissemination of information, it’s greater alignment.

All it takes is a slight change in perspective: Dashboards, when done well, aren’t just a visual data feed, they’re a reflection of the teams they represent and all of their hard work toward common goals. This is easier said than done, but as Chartbeat, PerformPlus and the Garden of Health show us, armed with a little bit of psychology and a humanistic perspective, the humble dashboard can give teams unifying superpowers.

Motivational Data Visualization

Fri, 07 Aug 2020 00:00:00 GMT

John Snow, “the father of epidemiology,” is famous for his cholera maps. These maps represent so many of our aspirations as data-designers.

By obsessively (heroically!) diving into London’s 1854 cholera outbreak, laboriously gathering an infection dataset (at his own peril!), then overlaying the data around the infamous pump on Broad Street, John Snow not only revealed a fundamental truth about London’s cholera outbreak, he also made it inescapably obvious to stubborn public officials. John Snow and his visualization saved a city. Or so the legend goes (src, src).

John Snow's legendary Broad Street maps

True or not, the Guardian describes John Snow’s maps as “a model of how to work today” (src). We aspire toward visualizations like this because they enlighten. We judge their quality by “knowledge gain,” or how much insight a person gains for having experienced them.

These singular acts of visualization brilliance are certainly worthy of admiration. But data visualization doesn’t need to reveal the secrets of the universe to make the world a better place.

Does data-viz need to be insightful to be influential?

A more common experience with data, I suspect, is much more personal even if more mundane. It’s looking at our checking accounts, glancing at our car’s (literal) dashboards, watching step counts climb on Fitbits, checking our blood-glucose monitors, tracking our periods, etc.

None of these experiences necessarily offer life-changing analysis. But because these data interactions reach so many people, so frequently - and, often, they’re designed to nudge us in the right direction- I suspect their aggregate impact is at least comparable to legendary works such as Snow’s.

A developing branch of data-viz…

Data-viz doesn’t need to reveal the secrets of the universe to be impactful.

As more of everything we do leaves a trail of data, there are growing opportunities to surface this data in ways that positively impact users’ lives. (And, conversely, there is greater risk of doing harm.)

The growing ocean of personal data means new opportunities to make the world a better place by focusing on smaller, more intimate, everyday visualizations. In particular, I think there are significant, untapped and underappreciated opportunities helping people, as users of various products and services, to visualize their own data.

This type of design is worth highlighting because, even though it draws heavily from two well-established disciplines (dataviz + UX), the intent is different. Instead of optimizing for revelation (as with dataviz) or utility/delight (as with UX), user-data visualization centers more on personal reflection. It’s more about motivation than information, and this brings unique considerations, opportunities and risks (as we’ll see below!).

In this post, we’ll examine 7 examples of “user visualization” (i.e. helping people visualize themselves). I hope this accomplishes the following:

Draw attention to the practice of user visualization to help seed further conversation within the Dataviz community.
By highlighting the methods behind these examples, help dashboard and product designers realize a few superpowers they might not be tapping into.
Inspire other practitioners to explore new ways of conveying smaller, more personal truths.

Visualizing Power Consumption

A reproduction of Sacramento Municipal Utility District's electricity usage letter (via "Behavior and Energy Policy"). The horizontal bar chart on the left shows the customer's consumption compared to their peers. The chart on the right shows "injunctive feedback," conveying approval or disapproval of the customer's behavior.

In 2008, in an effort to reduce electricity consumption, the Sacramento Municipal Utility District tried a new technique. They sent 35,000 customers a letter with a few simple charts, showing customers their electricity consumption compared to their neighbors.

In the charts above you can see a bar graph with 3 bars, representing power consumption for 3 different groups: “efficient” neighbors, all neighbors and the recipient’s household. This showed customers if they were consuming more or less electricity than their neighbors.

Customers who received these letters reduced their power consumption by 2% on average (src). While this seems small in terms of an individual’s power bill, across all the customers this amounts to terawatts of electricity and millions of dollars saved collectively. Since then, other utilities around the country have adopted this approach.

Superpower:

Dataviz is a powerful tool for feedback. But what makes these charts impactful isn’t just the feedback (“you consumed X of electricity”), it’s the context the feedback is presented in (“you consumed X, most of your neighbors consumed Y”). As people, we look to others’ behaviors as norms for our own. So by seeing others’ power consumption, people change their own consumption habits to be more consistent with the social norm.

In a related experiment, Schultz & friends found that the same message with an additional smiley or frowny face amplified the effect even further by reinforcing the behaviors of households that were already doing a good job (src).

Takeaway:

Nothing about the electricity usage charts is profoundly insightful, at least not in the way we might think of John Snow’s cholera maps. But even simple visualizations, when paired with an understanding of human and social psychology, can be impactful to large numbers of people. These types of visualizations won’t win a Pulitzer, but in aggregate they can make a big difference in our lives and our communities.

Visualizing Blood Glucose

Tweets of people's blood-glucose levels and the stories behind them. The line graphs show users' blood glucose measurements, plotted overtime throughout the day, relative to a healthy target zone. If users' blood sugar gets too high or low it can have serious health consequences.

For the 1.6M people in the US with Type1 Diabetes, graphs like these are never far from top-of-mind. As Continuous Glucose Monitoring (CGM) becomes widespread, these charts have become part of daily life.

Often, the plot of your blood glucose is the story of your day.

If you search Twitter for #T1D you’ll find example after example like the tweets above. What makes these so compelling is that they’re lived data. They’re a reflection of the user’s life. The value is not just the information, it’s visualization as a satisfying memento.

Of course, these charts are also effective. In multiple studies, giving subjects with Type1 Diabetes a CGM helped improve blood sugar control significantly (src, src). For people with Type1, the CGM and these simple plots, are life changing (and occasionally life saving).

Matthaei & friends' Ambulatory Glucose Profile (AGP), showing a distribution of blood glucose levels compared to a target range over a 24 hour period.

Superpower:

Like Tufte and Powsner’s ”Graphical Summary of Patient Status” (src), which show data in the context of a historic normal range, the chart above from Matthaei & friends presents a target range for blood glucose (because levels shouldn’t be too high or too low). This helps users quickly identify dangerous deviations, so they can take appropriate action (e.g. administering insulin).

While presenting goals as a range is necessary for glucose monitoring, there’s also evidence that representing any goal as a range is more effective than a point value. Targeting a range of acceptable outcomes is simultaneously more forgiving and ambitious, encouraging users to persevere over the long-term (src, src).

Takeaway:

User data is lived data. This enables visualizations that are functional, but can also involve unique dimensions of nostalgia, intimacy and affirmation. Even without revealing new insights (e.g. users already know what grandma’s cooking is likely to do to their levels), there’s still value in visualization as memento.

Visualizing Screen-Time

A bear avatar progressing through 6 states of Chow's "Time Off" experiment (src). As user's screen-time metric increases, avatars become progressively sicker.

Older millennials may remember the Gigapet invasion of 1997. Suddenly, everyone in 6th grade had a colorful plastic ornament hanging off a belt loop or tucked in a baggy pocket. Several researchers have shown that the attachment we felt toward these digital creatures is transferable to other parts of our lives that actually deserve our vigilance.

One challenge many of us face is limiting screen time. In 2019, Kenny Chow showed that representing screen time as an infliction on a “lively” animated avatar was a promising intervention toward reducing participants’ screen time (src). The more time users spent with devices open, the sicklier the avatar becomes.

This model of creature-as-data has worked in other contexts. Lin & friends showed that representing physical activity as a virtual fish tank helped participants become more physically active (src). Consolvo and friends showed a similar effect for visualizations of flowers and butterflies standing in for exercise events and goals (src).

At first glance, these representations seem more like “data art” than “data visualization.” But I think the effect is the same. Instead of encoding data for preattentive processing via our brains’ spatial wiring, the avatars encode the data for the parts of our brains that make us social and supportive. We feel the data through our empathy for the “living” character on the screen.

Superpowers:

This probably doesn’t make sense for a Tableau dashboard at work. For many use-cases (e.g. journalism, BI, reporting, etc), immediacy, precision and depth are supreme. For personal-viz, though, some of these constraints can be relaxed:

If you can assume the user will interact with the visualization multiple times, a longer learning curve is sometimes acceptable (and to the extent that learning the viz can instill feelings of mastery, a steeper learning curve might be desirable).
Precision is sometimes undesirable (e.g. for weight loss, users can overly fixate on natural ups-and-downs). Representing data on a vague scale like “sick avatar” to “healthy avatar” can help users gain sensitivity to the metric without fixation.
If the underlying data is relatively simple and a shallower representation creates a sense of emotional attachment, forgoing depth for extended engagement is an easy tradeoff to make.

Takeaway:

The goals and constraints for motivational viz can be quite different from traditional dataviz, making very non-traditional visualizations like Avatars, Fish and Flowers potentially very effective. When the goal is not just conveying information, but creating an emotional connection, tapping into users’ empathy through tactics like anthropomorphization can help form an emotional bond to the underlying data.

Visualizing Financial Goals

Betterment's "Goal Forecaster." The fan chart shows that, if I deposit $254.14 / month, I have a high-probability of cruising the Caribbean in a new 26' sailboat in just 20 short years.

Most brokerage services encourage bad investing habits. From the moment users login, they see charts & graphs of their portfolio performance in the past or the market’s performance right now.

When I spoke with Dan Egan (Head of Behavioral Finance at Betterment), he said this backward-looking focus leads users to reactionary decision making, ultimately hurting their long-term financial outcomes (e.g. “Oh! The market just went down, I have to sell out because I don’t want it to go down more”).

Egan’s research suggests that the more users fixate on portfolio performance, the more likely they are to fidget, and fidgeting lowers users’ expected returns (src).

According to Dan: “In investing, people spend a lot of time looking at history. But you just can’t change history. You can’t change that outcome.” Instead, “the only thing that matters is what you decide to do with today and how that sets you up for the future.”

So Betterment encourages users to look forward by offering visualizations like the Goal Forecaster, designed to “focus you on the future and where you want to end up.” It helps you see “how your actions today put you in a better position in the future.”

Research backs this up. Simulating various saving schedules helps users connect the cause and effect of their actions now (auto-deposit) with likely future outcomes (boat). Making these relationships concrete helps people grasp the effects of complex concepts like exponential growth and motivates users to save more (src, src). Another 2015 study suggests that highlighting the difference between savings goals and savings projections helps participants make better decisions, using the endowment effect to their advantage (src).

Finally, the “Goal Forecaster” helps set realistic expectations. Visualizing realistic expectations about future goals makes us more likely to achieve them (src). Presenting the projected savings in a fan-chart helps reinforce the message that there’s uncertainty in any financial journey, while still conveying the benefits of investing over the long term.

Superpower:

Dan says, consider “showing not telling:”

“If you let people simulate their outcomes - if you let them experience parallel universes - that is more effective in getting them to understand what’s going to happen than if you describe it. Allowing people to really experience it and see the variations allows them to internalize it as real.”

Takeaway:

For Betterment, letting users look ahead is motivational and informational. Simulating short term actions playing out over the long term helps educate users about difficult-to-imagine cause and effect relationships, giving them confidence to make smarter choices. Having realistic expectations about the future helps users prepare themselves for a long journey.

Visualizing Fitness Activity

Two screenshots of Jawbone UP's activity graphs. The graph on the left shows a user who has not met their 10,000 step goal. The graph on the right shows a user who has exceeded their goal.

Tracking physical activity helps many people stay active (src). The challenge is maintaining users’ attention and getting them to stick with the intervention. In this context, visualization can be a tool for both conveying data, and rewarding users for collecting the data in the first place.

Jawbone UP

Jawbone’s UP app (circa ~2012) was one of my favorite examples of this. In the screenshots above, there are two bar charts of users’ step counts. For most of the day, you’re seeing a chart like the left, but by the end of the day, once you’ve crossed your goal threshold (e.g. 10,000 steps), the background explodes into a happy sunburst, helping you celebrate the accomplishment!

With wanton disregard for data-ink ratios, Jawbone’s designers were masters at conjuring up visual joy. The effect was users who felt supported and encouraged to keep hitting their goals (and, implicitly, to continue tracking and interacting with the data).

Notch.me

Examples of notch.me visualizations based on users' fitness tracking activity. Top-left shows a template comparing user distance traveled to stacked Danny Devitos. Top-right, from 2012, compares a user's running activity to then-presidential-candidate Mitt Romney's exercise regime. Bottom-left shows user steps for a year as density of tiny feet accumulating inside a larger foot. Bottom-right shows user's typical activity levels per weekday as a stream graph.

With Notch.me, we did similar work using emailed “postcard visualizations” to heighten user’s sense of accomplishment at various milestones (and create a sense of anticipation for future milestones). The data was sourced from user’s personal activity tracking (e.g. via Fitbit) and presented back to users in various encouraging visualizations like the ones above.

The two top postcards above, use visual metaphor to compare a user’s activity to something unexpected and silly; they’re extremely low data-density, but still quite effective in directing recipients’ attention toward their data.

The two bottom postcards show users’ activity for a set time period (left: 1 year, right: 1 week); this encouraged users to track more consistently during those periods in anticipation of a more elaborate, pleasant visualization (the more you track, the cooler it looks).

In various experiments, visualizations like the above captured user attention and boosted engagement with the data. And, as you might expect from “sugary,” chart-junky visualizations, users loved them.

Superpowers:

Remember Nigel Holmes and the power of humor (src): “My feeling then, and now, is that humor is a good way to get people’s attention. Get a reader to smile, or recognize a visual reference and they’ll surely read on.”

In addition to capturing attention, Holmes’ style of visualization has also been shown to increase memorability of the content (src). Perhaps an overlooked aspect of Holmesian visualizations is that they feel good. This is powerful in its own way: Positive feedback and positive affect (emotion) help people make positive changes.

Takeaway:

For information to make a difference, it has to be seen. Humor, visual metaphor and even pleasant visual flourishes not only serve to draw users’ attention, they’re also emotional and viscerally rewarding methods to reinforce engaging with the data. For Jawbone’s visual flourishes and, pretty much everything with Notch.me, the goal for each visualization was as much about information as it was affirmation.

Visualizing Reading Activities

Screenshots of reading activity from Amazon Kindle's "Reading Insights" tool. The left chart shows books I've read to my oblivious 3-month-old daughter. The chart in the middle shows # of days I've read with the Kindle. The charts on the right show reading activity "streaks."

Kindle’s “Reading Insights” is a relatively new addition to the Kindle iOS / Android apps. I think there’s a lot to praise with this. Reading is another of life’s many activities that, despite wanting to do more of it, we struggle to make it happen as often as we’d like. So offering users a way to track and reflect on their reading could see a meaningful increase in time spent reading.

I think it’s also emotionally satisfying. In the same way a physical book shelf is like a trophy case for nerds, seeing the books you’ve read front and center feels good. Like stickers, it’s a visual representation of your accomplishments that you can feel proud of.

The downside, and this is true of many similar visualizations, is it gives Amazon a justification for even further surveillance (src). This serves as an important reminder that, with this new frontier of data there are ways that it can be legitimately useful to end-users, but that’s not without tradeoffs. As makers and designers we should keep the ethics of personal data use top of mind. (As a silver lining, thanks to CCPA, you can now request all of Amazon’s data on your reading habits to visualize for yourself!)

Takeaway:

Visualizing user data creates exciting opportunities for both users and designers, but it also brings new responsibilities. For every thoughtful blood-glucose visualization attempting to empower users, there’s an Uber out there using behavioral data to manipulate. As we rightly question the role of data and algorithms in a just world, we shouldn’t forget that visualizations, like any non-neutral technology, can be used for harm and for good.

Takeaways

Epic, John Snow-eque, Pulitzer-worthy dataviz is amazing, but visualization can also be impactful in everyday use. This seems worth celebrating.
Instead of revealing the secrets of the universe, user visualization is about offering lenses on a subject that people are already intimately familiar with: Themselves.
While the medium is the same, design goals for “User Visualization” differ from traditional data-viz. Instead of singular dedication to information gain and/or impacting big decisions, user visualizations tend to benefit everyday activities, impacting more frequent, micro-decisions.
User visualizations can help in a variety of ways: Encouraging adoption of social initiatives, managing health conditions, helping people persevere toward long-term goals or developing new habits, etc.
If you’re a dashboard designer, a (data) product designer or anyone else helping users visualize their own data, you’ve got superpowers to inform and inspire.

Goal Setting: 'Measurable' shouldn't mean 'Miserable'

Fri, 17 Jul 2020 00:00:00 GMT

By now we’re all familiar with OKRs. Objectives and Key Results were invented by Intel’s Andy Grove, passed onto Kleiner Perkins’ John Doerr, who then spread the gospel to his portfolio companies, most notably Google. Larry Page credits OKRs as the managerial secret-sauce behind their rapid growth.

As a testament to the power (and the pain) that comes in defining OKRs, Google’s CEO Sundar Pichai describes the process as “agonizing.”

“There are single OKR lines on which you can spend an hour and a half thinking, to make sure we are focused on doing something better for the user.” — Sundar Pichai to John Doerr in “Measure What Matters”

(Fun Fact: Sundar’s annual comp from Google is ~$80M / year (src), so that comes out to a cost of ~$60k per Key Result).

Key Results (KRs) are “measurable”

Assuming your organization has also imbibed the OKR Kool Aid, you’ve probably run into this “agonizing” guideline: Key Results are how we “prove” that our team has accomplished the objective, so they should be measurable and indisputable.

“The key result has to be measurable. But at the end you can look, and without any arguments: Did I do that or did I not do it?” — John Doerr (src)

In “How to set good OKRs,” Weekdone offers even stricter guidelines, advising that Key Results should NOT be:

“Binary, they should be numeric and measurable”
“Tasks to be achieved. Key results are metrics!”

Are K.R.s just B.S.?

At first I thought strict measurability was just productivity dogma, another thing an agile coach preaches, because their own agile guru passed it down to them. But measurable KRs can really help avoid common goal-setting traps:

If our KRs become a todo list, we’re incentivized to just check the boxes rather than accomplish the intent of the objective. For example, if our objective is: “Bake the greatest pizza in the world,” it’s easy to a) buy great ingredients, b) find a brick oven, c) bake the pizza, etc. but there is a lot of wiggle room between a, b, c and “the greatest pizza in the world.”

If we dictate tasks, we rob our teams of the chance to develop their own solutions for accomplishing the objective. Giving teams space to solve problems creatively and independently is one of the biggest benefits of human-centered performance metrics.

The Challenge

Defining “measurable” Key Results (KRs) is easier said than done. Even after years of practice, I still struggle against the impulse to think “stuff we need to do” instead of “measurable outcomes we want to accomplish.”

Plenty of my clients and teammates struggle with this as well. The root cause: Our brains are better at articulating things that are immediate and familiar. In the case of performance goals, the tasks that we do all-day-every-day are more immediate and familiar, therefore they come to mind first. But metrics are more abstract and removed, so it takes effort to think in those terms.

This struggle is totally normal. The “tasks” we think of first are just our subconscious pointing the way to the “key results” we actually care about.

And just because “measurable” key results don’t come naturally doesn’t mean we shouldn’t strive for them. As Doerr describes it, OKRs are a muscle that people need to exercise to develop.

The Hacks

One day, the brilliant Liz DeLuca and I were working on OKRs and, unsurprisingly, we noticed many of the teams’ goals weren’t goals at all, they were tasks. The conversation turned introspective (as it often would). Why is it so hard to break this habit? Why isn’t this more intuitive?

During this particular session I happened to be reading Douglas Hubbard’s “How to Measure Anything.” In addition to offering an entire philosophy of measurement and decision-making, the book offers a mountain of useful tactics for thinking about measurement.

As an experiment, we decided to apply some of Hubbard’s approaches to our goal-setting exercises — and after some trial and error — it works! We used these to successfully define OKRs across a variety of initiatives and it’s been my goto approach ever since.

When defining Key Results, instead of fighting against the instinct to generate Tasks, use them as a starting point, then do the following steps to dive deeper:

Ask “Why?” First, ask “Why might this task be important?” What would the task accomplish? Repeat this until you arrive at a root Outcome. This helps decompose the Objective into the Outcomes that really matter.
Ask “What will we observe?” Second, for each Outcome from #1, ask “What might we observe in the world if this were true?” Brainstorm different possible changes you might observe in the world around you.

The Steps: Consider an Objective → Brainstorm Tasks → Ask “Why?” → Outcomes → Ask “What will we observe?” → Observations → Good KRs.

Let’s explore a pizza-based example.

Objective: “Bake the greatest pizza in the world”

Let’s give ourselves a vague, but inspiring objective to test out these techniques. Let’s say we want to “bake the greatest pizza in the world.”

Like most people, when we think “What does it mean to bake the greatest pizza in the world?” we immediately think of the tasks we need to do:

Task: Buy really fresh ingredients
Task: Watch 200 pizza related shows on Netflix
Task: Find a brick oven
Task: Settle on a recipe

When we bake the greatest pizza in the world, we might in fact do all of the Tasks listed above, but none of these tasks prove that we’ve baked the greatest pizza in the world. They also don’t leave room for alternative solutions (e.g. there are surely better ways to learn than “watching Netflix”).

These tasks aren’t yet good KRs, but they’re a fair starting point. We just need to dig a bit deeper with some some “why?” laddering.

Hack #1: Asking “Why?”

Let’s unpack 2 of the Tasks we generated above as examples.

Example Task # 1: Why would we “buy fresh ingredients?”

It seems intuitive that the Task “buy fresh ingredients” is important, but what’s the impact we’re expecting as a result of fresh ingredients? A few possibilities:

Fresh ingredients will make the pizza taste better,
Fresh ingredients might make it healthier (or healthy-ish?)
And maybe fresh ingredients help with texture and structure?

These are a great start. By looking at the potential impacts of the Task “Buy fresh ingredients,” we’ve identified 3 qualities that are important to our objective: “Great pizza” means “good taste, healthy-ish, well-structured.”

Example Task #2: Why “watch 200 pizza-related shows on Netflix?”

While Netflix surely has 200 full episodes of celebrity / comedian chefs sampling the best pizza from Antartica to Atlantis, spending 100+ hours watching Netflix seems like a circuitous route to baking the greatest pizza in the world.

But let’s go with it. Remember, the tasks we first think of are just our subconscious’ weird way to point us to the Outcomes that matter. So what are the positive impacts we might expect from binge-watching cooking shows?

Cooking shows help us identify “unknown unknowns” in making pizza
Shows help us understand the different varieties of pizza (and nuances to the craft of pizza making)
It’ll help us learn new techniques without re-inventing the wheel

We’re getting closer, but these still feel distant. So let’s ask “why?” again.

How will “identifying ‘unknown unknowns’” help us bake the greatest pizza in the world? So that we’re not surprised when we try to make the pizza for ourselves. If we invite our friends over for pizza, we want to feel confident we can serve something edible. We want a reliable process for baking the pizza.
How will “understanding the nuance of pizza” help us bake the greatest pizza in the world? Maybe because it helps us bake tastier pizza (this is the tasty Outcome from our first example).
How will “not reinventing the wheel” help us bake the greatest pizza in the world? It means we don’t need to spend 30 years learning the tricks for ourselves, we can just learn from the great pizza chefs who came before us. We don’t want to spend a million years learning to do this.

Aha! Now we’ve got some real Outcomes… the Task “Watch 200 pizza cooking shows on Netflix” is really about: learning a reliable process, baking tasty pizza and learning quickly.

Recapping Example #1 and #2

Now we’ll know we accomplished our Objective (bake the greatest pizza in the world) if the following Outcomes are true:

The pizza is tasty
It’s healthy
It’s well-structured (not soggy)
We have a reliable process to prepare the pizza
We didn’t spend a million years on this. We learned quickly

These are good Outcomes. We’ve decomposed bake the greatest pizza in the world into the 5 components that we care about; at the very least these will help us align our team around impact (v.s. dictating tasks).

But these Outcomes still aren’t measurable KRs. How do we quantify these?

Hack #2: Ask “What might we observe?”

The next trick comes from “How to Measure Anything,” where Douglas Hubbard argues that if an outcome is worth pursuing, then it must have some observable effect.

“First, we recognize that if X is something we care about, then X, by definition, must be detectable in some way. How could we care about things like “quality,” “risk,” “security,” or “public image” if these things were totally undetectable, in any way, directly or indirectly? If we have reason to care about some unknown quantity, it is because we think it corresponds to desirable or undesirable results in some way. Second, if this thing is detectable, then it must be detectable in some amount. If you can observe a thing at all, you can observe more or less of it… If we can observe it in some amount, then it must be measurable.” - Douglas Hubbard, “How to Measure Anything”

If our KRs correspond to some desirable results, we should be able to observe some effects of those results. So what might we observe in the world that indicates the Key Results were accomplished?

If you get stuck, Hubbard offers a thought experiment you can try:

“Imagine you are an alien scientist who can clone… entire organizations. Let’s say you were investigating a particular fast food chain and studying the effect of a particular intangible ‘employee empowerment.’ You create a pair of the same organization calling one the ‘test’ group and one the ‘control’ group. Now imagine that you give the test group a little bit more ‘employee empowerment’… What do you imagine you would actually observe — in any way, directly or indirectly — that would change for the test organization?”

Let’s try this for each Outcome from above.

tasty: If we’ve baked a truly tasty pizza, then we might observe… People tell us it’s good. Even pizza snobs say they like it. We see people choose our pizza over other pizzas.
healthy: If the pizza is healthy, then we might observe… We don’t feel like falling into a food coma after we eat it. We calculate that the # calories in our ingredients are lower than calories for other pizzas.
well-structured: If the pizza is well-structured, then we might observe… When people hold a slice, we don’t see the cheese immediately slide off. We don’t see a bunch of greasy plates after serving a group.
reliable process: If we have a reliable process, then… When cooking test pizzas, our trash can doesn’t fill up with burned pizza. Each pizza requires the same amount of time and ingredients. We never feel surprised.
learned quickly: If we learned quickly then… When we calculate the time we spent on this project, it’s not too many days.

For each KR, we’ve now got a few options for things we might observe. So which observations should we use to define the KR?

One consideration: How hard / expensive would it be to actually measure the observation? For example, to measure the tasty Objective, we could spend $50k hiring a research firm to double-blind-placebo-proof-focus-group the pizza with n=300 participants. Or we could sponsor a few meetups, send them our pizza + 2 competitive pizzas and see which pies get eaten first. Since we’re probably not trying to get these results published in a peer-reviewed journal of pizza, the former is probably overkill. The latter is maybe less precise, but it’s much cheaper and still a fair signal.

Recap

We originally started with a list of top-of-mind Tasks for the Objective bake the greatest pizza in the world. With Hack #1, we asked “Why?” to transform those Tasks into the Outcomes we care about (tasty, healthy, well-structured, reliable process, learn quickly). Then with Hack #2, we brainstormed and chose Observations we might see if the Outcomes were true. The final step is putting it all together and assigning targets for the Observations. This might look like…

Objective: “`Bake the greatest pizza in the world`”

Key Result: At 8/10 meetups, our pizza is eaten before 2 distinctly marked, but unbranded competitive pizza choices
Key Result: <200 calories when we add up the ingredients in a slice
Key Result: For 100% of test pizzas, the cheese does not slide off when held at a 45° angle
Key Result: We bake 3 test pizzas in a row in 30 minutes, w/ total ingredients costing < $5 / pizza
Key Result: We spend less than 5 days learning to do this

Takeaways

Measurable Key Results require some extra thinking-pain. But they’re worthwhile because they keep us outcome focused and leave teams space to apply their own creative solutions toward the Objective.
When defining KRs for an Objective, it’s okay to start with Tasks. This is your subconscious pointing you to the Outcomes that really matter.
To turn a Task into an Outcome, work backwards by asking “Why do this? What do I think this will accomplish?”
To turn an Outcome into a quantifiable Key Result, ask “What would I observe in the world if this were true?” Choose an Observation that’s reliable and efficient to track ask your KR.

Winning at @ratemyskyperoom

Fri, 10 Jul 2020 00:00:00 GMT

In these uncertain times, the world needs hard-hitting data journalism now more than ever. This is not that.

Instead, I’ve analyzed 1,321 Tweets to answer a question many of us pandemic-bound remote-workers have wondered since Zoom became part of our daily lives: Do people like my room?!

Unlike Animal Crossing, there’s no authoritative raccoon we can rely on for objective feedback about our decoration skills.

Instead, here in the real world, the closest thing we’ve got is Room Rater (@ratemyskyperoom). As more and more (famous) people are revealing their homes via the laptop lens, Room Rater has stepped up to judge them, publicly and quantitatively.

Not all of our homes will be broadcast on national TV. At least not in the near future. But we can all agree, when that day comes, we want the world to see our rooms (and, by extension, our very beings) as worthy of a 10/10.

So I ask: “What does it take to get a 10/10 rating for my room?!”

To find out, I pulled down all of @ratemyskyperoom’s 1,321 room rating tweets from May 2020 to July 2020, parsed out the ratings, then looked at the content of both the images and text for each of their tweets.

Below are the critical, profound and stirring insights I’ve found in the data.

% Distribution of All Room Ratings per Score (src). The average rating is 7.5/10.

Insight #1: It’s not terribly difficult to get a good rating.

The good news: It’s apparently not hard to get a high score. The average rating is 7.5/10 and they hand out 8/10s like candy. In fact, most of the ratings are at least 8/10.

Room Rater talks a tough game, but deep down they’re softies.

This silly rendering shows changes in room ratings for individual twitter users, between their first rating and their last ratting (src). The left column shows speakers who have improved their room's rating. The right column shows speakers who have fallen from grace.

Insight #2: Second chances offer a path to redemption (or not)

Even if your first rating is low, you’ve always got another chance on your next TV appearance. At least 83 people’s rooms have been rated on more than one occasion.

Whether you come from the hard streets of Scranton or Sesame (@JoeBiden +1, @elmo +1), whether you’re a politician, press, pollster or professor (@RepKarenBass +2; @marycjordan +3, @FrankLuntz +3; @j_g_allen +4), Room Rater is willing to give your room a second chance. Above, you can see the 14 people on the left who improved their rating by at least 3 points between their first and last appearances. The awards for most improvement go to @anitakumar01 and @RosenJeffrey (+5 each).

But beware! What Room Rater giveth, Room Rater taketh away. At least 6 people scored worse between their first and last ratings. Sorry @pattonoswalt!

What’s the secret to Room Rating success?

To understand this, I looked at 2 sources: The room images and the text of each tweet. To analyze the images, I ran each one through the AWS “Rekognition” image recognition APIs. To analyze the text, I looked at single word usage (e.g. “wu” and “tang”, not “wu tang”). Neither of these methods are particularly robust, but there were still some interesting findings.

On the left are 3 example room images posted by Room Rater of @jheil, @todrick and @Judgenap. On the right is how Amazon's AWS Rekognition service labels each image.

What can image recognition tell us about high quality skype rooms?

Not much. The AWS Rekognition algorithm seems well-tuned for differentiating broad categories of things, but this isn’t super helpful when the domain of images is already somewhat narrow. Above you can see the results for a few sample images. I was hoping for results closer to “Hey look, @todrick has ice cream on the walls!” But at least it’s good at spotting people with human faces…

That’s not to say it was completely devoid of insight. Let’s look at one example.

Rating distributions of rooms with or without animals (src).

Insight #3: Puppies make for Great Skype Rooms.

Our first fun fact: the Room Rater judges are quite fond of rooms with animals. On average, rooms with animals were rated 1.2 points higher than rooms without animals. This could include artwork with animals, sculptures of horses or actual pets lounging around in the person’s background.

Above we have 3 graphs. The graphs on the far left and far right, labeled No Animals and Animals show the distribution of ratings for rooms that do not have Animals in them v.s. rooms that do have Animals in them (at least as far as AWS is concerned). The images overlaid on the bars include a sample of the images from the original tweets. (If you go to the Notebook here you can click the images on the graphs and see the animals)

The graph in the middle, labeled No Animals v.s. Animals, shows the same data from the other two graphs, overlaid on top of each other. The middle graph also includes little notches in the bottom showing the average for each distribution (in this case No Animals averaged 7.4 and Animals 8.6).

Rating distributions of rooms with or without books, art or plants (src).

Insight #4: Confirmed: Plants, Art and Books make for nicer Rooms.

This is nothing groundbreaking, but certainly reassuring. The photos where AWS’s algorithm could identify Plants, Art or Books got higher ratings than the room photos without Plants, Art or Books.

Rating distributions of rooms where the speaker is either male or female (src).

Odd: Male speakers’ rooms are rated more often, but Female speakers’ rooms are better.

Male speakers appear in the ratings almost 2x more than female speakers (842 ratings for men, 425 for women), but on average the women’s rooms are 0.3 points nicer. (*Caveat: Here “male” and “female” are based on AWS’s prediction of the person’s gender from the image.)

What does @ratemyskyperoom’s written feedback tell us about earning a high-scoring room?

There were no life-altering revelations from the images, but maybe we can look to Room Rater’s written feedback directly for some insights…

Rating distributions comparing rooms where feedback includes language about camera angles or lighting (src).

Insight #5. First, get the basics right.

It’s important to get the basics right. That is, frame the camera correctly and check your lighting. When Room Rater’s written feedback mentions reframing the camera (words like “reframe,” “crop,” “camera,” “ceiling,” etc.) it’s usually for lower scoring rooms. While creating a sense of ”depth” is a sure win.

Good lighting is similar. Feedback including words like “dark” or “backlit” are a bad sign, so make sure you’re not sitting in the dark.

Rating distributions comparing rooms where feedback includes words for succulents or orchids (src).

Insight #6. Succulents are good. Orchids are better.

While Room Rater is quick to suggest getting a Succulent for your room, or any plant in general, they actually seem much more fond of Orchids.

Rating distributions comparing rooms where feedback includes words for Cory Booker or Books (src).

Insight #7. Clarifying Books v.s. Booker.

Room Rater approves of rooms containing “Books.” They do not approve of rooms that contain Cory Booker. (Subtle difference.)

Rating distributions comparing rooms where feedback includes words for Hostage or Historic (src).

Insight #8. “Historic” is better than “Hostage.”

Perhaps the largest success factor is learning the difference between ”Hostage” and ”Historic.” As if being abducted isn’t bad enough, Room Rater is quite judgy about seeing these victims on the airwaves. To their credit, they do offer followers an extensive set of “Historic Skype Rooms” for viewers to emulate when designing their own rooms.

Rating distributions comparing all ratings to ratings where feedback includes words for pineapples, Elmo, Wu Tang or Mid Century (src).

Insight #9. Easy wins for boosting your Room Rating.

And, finally, if you need a quick boost before you go on air, the Room Raters are quite fond of Pineapples, Elmo and the Wu Tang Clan, so you may consider working those themes into your Mid Century modern decor and you’ll be scoring 10/10 in no time.

Takeaways

If you’re feeling stressed because the world is ending, elaborate and pointless data exploration can be a fantastic coping mechanism.
The secrets to @ratemyskyperoom success: Good lighting, good camera framing, plants, art, books, animals, pineapples, Elmo and Wu Tang. It also helps if you’re not Cory Booker and you have not been taken hostage.

Settling the Debate: Bars vs Lollipops (vs Dot Plots)

Tue, 23 Jun 2020 00:00:00 GMT

In 2017, some people were arguing on the internet. Notably, a few of them were thoughtful characters in the data visualization community, such as Stephen Few, Andy Cotgreave, Alberto Cairo, and Jeffrey Shaffer, to name a few.

The topic: What’s the value of a Lollipop chart? Is there something to the aesthetic (per Andy’s point)? Is the extra white space between symbols easier on our eyes (per Alberto)? Or are Lollipop charts just a “less effective version of a bar graph,” inspired by “the same thing that has inspired so many silly graphs: a desire for cuteness and novelty” (per Stephen)?

The discussion takes place in the comments on Stephen’s blog. Andy responds inline and with a post of his own. And, recently, Hicham Bou Habib summarized the discussion on Twitter, which is how I first discovered the controversy. (Thanks, Hicham!)

The debate unfolded over 24 days, from May 17 to June 10, 2017. The comments totaled about 10,000 words in response to a 400 word blog post.

Why not just test it?

What stood out to me: Why not just test it out? What’s the opportunity cost for these smart folks to hash out the pros-and-cons of two very similar visualizations? Is it more or less than the $100 it costs to push a test to a crowd-source, micro-task platform like Mechanical Turk? (Actually, yes, probably, because it’s a great discussion, but in practice …).

Stephen addressed this indirectly in the comments:

“There is still much about data visualization that we don’t understand. Shouldn’t we be spending most of our time exploring the unknown where the greatest potential discoveries exist rather than wasting our time covering the same well-known territory over and over again?”

To which I say: no, no, let’s test the silly dataviz … for fun?!

So, can we settle this high-stakes, life-or-death, “bar vs. lollipop” controversy with Mechanical Turk? This is an attempt to find out. As an added bonus, per Hicham’s and Alberto’s suggestions on Twitter, I also added a last minute variant for dot plots, in hopes of actually learning something useful.

Read below for the results of my grand experiment.

What are we hoping to learn here?

Some questions raised by the discussion:

To Stephen’s main objection: How do lollipop charts affect readers’ comprehension? How much of our epistemic responsibility are we sacrificing for a playful aesthetic?
Is there any significant difference at all?
Is one of the designs more “glanceable” than the other? (i.e. can it be comprehended more quickly?)
For which use cases might bars outperform lollipops (or vice versa)?
Might dot plots be a more reasonable solution than either of the above?

Examples of data from one of the surveys question sets, represented as the 3 chart variants. Actual visualizations did not include the color highlights. You can see how these looked in the actual study here.

Experimental Setup

To test the above, I gave ~150 Mechanical Turkers a simple quiz, asking them to answer questions based on different graphs.

(When I’ve looked at similar studies, seeing the prompts and questions for myself always makes the experiment clearer, so I’ve put an example of the “quiz” in this Observable notebook: Bars vs. Lollipops vs. Dots: Sample Prompt Questions).

The goal of the quiz was not to test the participants, of course, but to test the visualizations within the quiz, to see if one of the three different chart types results in more accurate or faster comprehension.

Each participant was shown a quiz consisting of the same 21 questions. The questions were divided into three different sections, where each section had a different chart that the questions referred to (the first was about bake sale results, the second was student grades on a quiz and the last was teacher salaries).

Behind the scenes, each participant was randomly assigned to one of three experimental groups, one for each of the three chart types: Bar Graphs, Lollipop Charts, and Dot Plots. You can see the three variants in the image above. All the charts presented in a given quiz were the same type (i.e. a participant would see all bar graphs, or all dot plots, or all lollipops, and not a mixture of different chart types).

Each section’s chart was based on a randomly generated, normally distributed dataset to control for possible advantages that a given chart type might have with different shapes of data (e.g. in the original blog post’s comments, someone hypothesized that lollipop charts might have an advantage when the underlying data was uniformly far from 0).

To measure comprehension time, timestamps were recorded for when participants answered each question, to give a rough sense of the time spent on each answer. This was not a perfect way to measure timing, but should still give some relative signal between the variants.

The Questions

To evaluate different use cases, workers were asked a variety of questions across five different question types. Similar to Saket & friends’ 2015 paper, the question types covered a different tasks ranging from simply retrieving values (e.g. “How many Donuts did Mr. Brown’s class sell during the bake sale?”) through summarizing the “gist” of the presented data (e.g. “How many cookies did the teams sell on average?”).

Example “question types” and questions:

Retrieval: “How many Donuts did Mr. Brown’s class sell?”
Extremes: “For the class that sold the least cookies, how many cookies did they sell?”
Gist: “What was the average quiz score for all students?”
Differences: “How much higher (or lower) was Fran’s vs. Freddy’s score?”
Filtering: “How many students scored at least 90% on the quiz?”

To minimize issues with numeracy or graph literacy, the questions were kept very simple. “Harder” questions like prediction or inference were avoided (src). To explore differences in answer types, some questions were multiple choice and some were numeric.

The Measures

To evaluate accuracy, I used two different measures: “Error” for numeric questions and binary “Correctness” for multiple choice questions. To evaluate comprehension time, I measured “Time” between answers. For each of these, see more details on calculations in the “Analysis Notes” below.

Error: For numeric question accuracy (e.g. “How many Cookies sold?“) I calculated each answer’s “error,” or the difference between the correct answer and the user’s answer (similar to Heer & Bostock, Cleveland & McGill). This ensures more granular variation across answers, making it easier to detect smaller differences in accuracy.

Correctness: For multiple-choice question accuracy (e.g. “Which class sold the most Cookies?”), I assigned correct answers a “1” and incorrect answers a “0,” so that the mean of 1s and 0s is the percent correct (similar to Saket & friends). While this allows less granular differences in answers, it fits more naturally with categorical answers for “gist” questions (e.g. “which class?”, “which teacher?”, etc).

Time: To measure response times, I calculated the difference between an answer’s timestamp and the timestamp of the previous answer.

How to read: Each dark circle represents the average error (or time) from all responses (e.g. when presented with the dot plot variant, users typically answered with less than 2% error). The "wings" around each dot represent the 95% confidence interval of the true average (e.g. if everyone on earth answered the same questions with a dot plot, there's a 95% chance that their average error would fall between the wings).

The Results

After all the fuss, the results show no significant differences between Bar and Lollipop graphs. They led to roughly equal accuracy and equal response times.

There were, however, significant differences between Dot Plots and the other two chart types.

Participants answered more accurately, or with significantly less error, on numerical questions when presented with Dot Plots vs. Bar Graphs (p < 0.001). Even though there were significant differences between the “best” (dot plots) and the “worst” (bar graphs), the intermediate differences were not significant (i.e. there were no significant differences between Dot Plots and Lollipops, or between Lollipops and Bar Graphs). There were also no significant error differences between any of the three charts on the multiple choice questions.

Testers also answered significantly faster when presented with Lollipop Charts vs. Dot Plots (p<0.05). They answered questions most quickly when presented with Lollipops, but there were no significant response-time differences between Lollipops and Bar Graphs, or between Bar Graphs and Dot Plots.

But the effect sizes …

One important note: Just because a difference is significant, doesn’t mean that it’s meaningful. Between the three chart variants, the best one was only ~1% more accurate on average than the worst (e.g. on a chart where the Y-Axis is 0–100%, this is equivalent to mistaking 71 for 72). And the fastest chart was ~1 second faster than the slowest.

How to read: Each set of confidence intervals represents typical error (or time) when users were asked to answer different types of questions, represented by the examples on the far left (e.g. Retrieval, Extremes, etc). The top row, Overall, shows performance across all question types (duplicating the 2 charts earlier in this post).

Task-Specific Results

Task-specific results were consistent with overall results.

For accuracy, there were statistically significant effects for two question types: “Retrieval” (F(2,543)=4.871; p=0.008; η2=0.018) and “Difference” (F(2,393)=5.264; p=0.006; η2=0.026). For Retrieval, Dot Plots showed significantly less error than Bars or Lollipops (p<0.01). For Differences, Dot Plots led to significantly less error than Bar Graphs, but there were no significant differences between Dots and Lollipops or Lollipops and Bars.

For speed, there was a statistically significant effect only for “Gist” question types (F(2,813)=3.454; p=0.032; η2=0.008). However, post-hoc tests did not confirm this, showing no significant, task-related differences between any pair of variants (Tukey’s HSD test, or ”Honestly Significant Difference test,” is known to be “conservative,” which may explain the discrepancy).

The chart on the left shows the distribution of choices for participants favorite snack. The chart on the right shows the confidence intervals for users perceived effectiveness for the charts in each variant.

User Preferences

Finally, the experiment looked at two other questions: 1) Which of the chart variants do people prefer? And 2), perhaps most critically, which of the listed bake sale treats do participants prefer?

When asked “how effective were the above charts in presenting the data?” users rated Lollipop charts as the most effective on a 5-point scale. This difference was not significant though.

Troublingly, a plurality of respondents (42%) reported that Brownies were their favorite treat. Most concerning: Cupcakes, the obviously superior snack, were the least frequently preferred treat. Unfortunately, this departure from objective snack reality calls into question the validity of the entire study. As such, I encourage readers to examine the Experimental Setup and Analysis Notes carefully and tell me where I went wrong.

Analysis Notes

Of the 150 workers who submitted answers, I omitted nine. Even though the Mechanical Turk Worker selection criteria was set to 95% HIT acceptance (or 95% of their previous ”Human Intelligence Tasks” were accepted by previous requestors), not all submissions seemed to be done in good faith. I did not require a screening exercise (per Heer and Bostock or Saket & friends), instead I removed workers after-the-fact based on one or more of the following conditions: average numeric errors were more than 20% incorrect, whose multiple choice answers were less than 25% correct, or who had previously participated in practice runs of the survey. I also omitted 52 answers (1.7% of 2,898) where error was greater than 50%, assuming mistakes of this magnitude are due to something other than the graphs themselves.

Error was calculated as:

log2(| 100 * (expected_value - answer_value) / max_value | + 1/8)

where expected_value is the correct answer, answer_value is the worker’s answer and max_value is the largest value shown on the randomly generated graph. This is based on Heer & Bostock, Cleveland & McGill’s error formulas, with additional terms for scaling to 100%, since the 3 question-sets’ graphs have different ranges of values (see the question sets here, for context). Taking the log of the absolute error helps transform the data into a more normal distribution, friendlier for ANOVA / F-Testing. The ⅛ term is used because log2 gets squirrely at zero.

Timing was calculated as:

log2(answer_timestamp - previous_answer_timestamp)

where answer_timestamp is the last time a user changed the related answer field on the survey form and previous_answer_timestamp is the previously most-recent time a user answered any other question (since some users answer questions out of order). Again, log2 transformation makes the values more normal for significance tests.

Both measures were evaluated at log scale, but confidence intervals above are reported after back-transforming the results into the more readable original scale. All means are mid-means.

Takeaways:

“Why not dot plots?” This was my first time working with a dot plot in practice. As the most accurate variant, clearly they have some benefits …
There were no significant differences between Bar and Lollipop Charts.
Don’t stress about “best.” These three charts are certainly different visually, but in front of users that didn’t amount to much. If the difference between the “best” and the “worst” is only 1–2%, and you’re not working on medical devices or launching rockets, then it’s probably not worth stressing too much about which chart is “best.” If Lollipop charts fit the aesthetic you’re aiming for, then enjoy the lollipops.
If you are working on medical devices or launching rockets, you should be wary of theory in a slightly different way. As Heer and Bostock note in “Crowdsourcing Graphical Perception” because visualizations are a combination of many parts that can interact in sometimes unpredictable ways, there’s only so much you can extrapolate from first-principles (or even other people’s empirical evidence). There’s no substitute for testing data visualizations with real users.

Lemonade & Human-Centered KPIs

Thu, 18 Jun 2020 00:00:00 GMT

Meet Lola. She’s 9 years old. She has two siblings — twins, age 7 — Jack and Jane. They have a neighborhood friend, Murray, 8½. Unlike Jack, Jane and Murray, Lola is ambitious.

Lola was never one for dress-up or playtime. Instead of princess costumes, she favors pantsuits. She owns 7 of them, one for each day of her work week. She’s never asked her parents for an iPhone. Instead, she bought herself a Blackberry two years ago — she claims the physical keyboard is faster for email.

Lola’s latest endeavor is selling lemonade.

Jack, Jane and Murray are dutifully onboard. She’s also recruited other children from the neighborhood and divided her helpers into teams, so they can cover multiple locations across the neighborhood.

Unfortunately…

After the first week, as you may expect, Lola’s helpers didn’t live up to her high expectations.

As an aspiring operator, Lola remembered a quote in John Doerr’s “Measure What Matters” where Larry Page attributes Google’s success to OKRs:

“OKRs have helped lead us to 10x growth, many times over… They’ve kept me and the rest of the company on time and on track when it mattered the most.”

Like many who start on their metrics journey, and, because at 9, Lola has not yet fully-developed abstract reasoning, she’s not sure how to select smart KPIs her team can rally around. She’s also heard the horror stories about Wells Fargo’s $3B in fines and litigation, a consequence of the bank’s singular focus on “# accounts opened.”

Believing in the power of metrics, but wary of negative side effects, Lola starts with first principles. She asks: “What makes a good metric in the first place?”

What makes a good metric?

Good metrics are controllable.
Good metrics support fair comparisons.
Good metrics are hard to game.
Good metrics are concrete and familiar.
Good metrics instill a sense of pride.

1. Good metrics are controllable.

Teams must have agency and ability to impact their metrics.

As an example, let’s look at Lola’s metrics for tracking customer satisfaction. Lola knows that happy customers are key for creating word-of-mouth growth. And she knows that her customers’ satisfaction primarily comes from 2 factors:

The quality of the lemonade
The customer experience (when interacting with her team)

Lola could measure overall customer satisfaction by surveying customers with NPS or PMF surveys, but those measures are only partially driven by team performance, and therefore only partially within her teams’ control. Instead, she needs a metric determined by customer experience, independent of product quality.

One possible way to observe a happy customer is looking at the tip jar. Lola reasons that people will tip after the transaction, but before they’ve tasted the lemonade, so it’s a clearer measure of their experience, independent of the product itself.

So she decides to measure the amount of tips received by each stand. We’ll call this Absolute $ Tips. She knows this still isn’t a very robust metric, but it’s something the team can influence directly, so it’s a good start.

Why does this matter?

Self-efficacy helps determine motivation (src). If the target outcome is outside a team’s control, trying to optimize against it can feel disheartening (src). Beyond that, it also makes the metric noisy and ambiguous, and therefore difficult to track and optimize against.

2. Good metrics support fair comparisons.

Metrics should fairly represent the underlying, often-heterogeneous teams being measured. They shouldn’t advantage some groups over others.

In Lola’s case, tracking Absolute $ Tips is hardly fair…

In an unfortunate bout of nepotism, Lola gave her siblings the prime location (close to the Whole Foods AND the Soul Cycle), whereas Murray and the other children were assigned to lower-traffic stands.

Not only are Jack and Jane seeing more traffic, their location’s demographic is also likely more wealthy and able to give larger tips. Even on their worst days, Jack and Jane will pull in more tips than Murray. This means measuring Absolute $ Tips gives Jack and Jane a measurement advantage over Murray.

Absolute $ Tips is okay for tracking an individual team’s customer service, but it’s not fair or comparable when comparing teams. To correct this, Lola needs to account for demographic and traffic differences between stands.

Instead of Absolute $ Tips, changing the metric to # Tips might account for differences in tip sizes (e.g. instead of “$100 in tips,” it’s “Customers tipped 50 times”). But # Tips is still susceptible to traffic differences between locations, so a further refinement might be to normalize the metric, defining it relative to overall # of Customers.

This gives us: (# Tips) / (# Customers). This is a more fair metric for lemonade stand customer service, across multiple locations.

This is looking better! But there’s another concern…

Why does this matter?

For metrics (or any workplace intervention) to be successful, employees need to “buy in” and trust that it represents their interests (src).

Perceptions of organizational justice can significantly impact employee motivation and acceptance of organizational change (src, src). If team members perceive a metric as unfair, they’ll take it less seriously and it’ll be less influential.

3. Good metrics are hard to game.

It’s good that a single metric can be accomplished in a variety of ways. This leaves room for teams to apply their own creative solutions. But, without care, this can incentivize the wrong things. (For example, tracking employee workstation activity often leads to deceptive “mouse moving.”)

Lola is concerned that even the new normalized customer satisfaction metric is still possible to game.

For example, Murray’s partner Don Jr. has always gotten a little extra help from his parents, and Lola suspects they may be inflating Don’s numbers by tipping multiple times per order.

So she revises her metric again by changing # Tips in the numerator to # Tippers (i.e. number of customers who tipped). This minimizes the influence of any one customer on the ratio and brings the metric closer to the intended outcome: Happy customers.

This gives us: (# Tippers) / (# Customers).

As a further guard, Lola could also “qualify” the metric to only count “real” tips from “real” customers (where “real” is non-employee-family customers). She might offer a friends-and-family discount with a sign-in sheet, then subtract the # F&F of Signatures from # Tippers.

This would give us: (# Tippers - #F&Fs ) / (# Customers - # F&FS).

This is getting relatively abstract though…

Why does this matter?

The biggest challenge with any performance metric: “You’ll get what you ask for, but not necessarily what you want” (src). To avoid incentivizing sketchy behavior, metrics must be “hard to game,” typically by aligning them more closely with the intended result, qualifying them or considering them alongside some counterbalance metric (src).

4. Good metrics are concrete and familiar.

Good KPIs are easy to relate to the underlying work and behaviors. They use language that’s familiar to the team. They’re no more abstract or complex than necessary.

Lola feels good about her “tipper rate” metric as an indicator of team contributions to customer satisfaction, but now she’s worried it’s too abstract, especially considering her team is 100% children. She wants to make sure that, day-to-day, the team isn’t distracted by the particulars of how a metric is calculated. So she takes a step back. What #s should her team focus on “in the moment?” That’s easy. Back to Absolute $ Tips. Seeing a customer put a $ into the jar is a visceral experience. And the tip jar itself is a visual indicator of that metric - the fuller the tip jar, the better you’re doing.

Why does this matter?

Presenting information concretely makes it easier to remember and stick in our minds (src, src). For metrics to have an impact, people must be able to easily relate the numbers to the underlying behaviors and actions.

5. Good metrics instill a sense of pride.

Well-designed metrics memorialize a team’s hard work. They demonstrate how individual contributions build to some greater good for the business, or ideally, for our fellow humans. They look good on a resume or a humble-brag on LinkedIn.

Lola’s last concern with her “tip rate” metric is that it sets a selfish tone. Her team obviously benefits from the tips, but how does that relate to their overall mission? Lola has, of course, articulated their mission statement: “Hydrating and refreshing the world with organic, citrus beverages.”

So, after a busy week, when the team reflects back on their work, how can they keep the focus on the mission and avoid fixating only on the tips they’ve earned for themselves (or Lola’s questionable labor practices)?

Lola, thinking back to a case study she’d read on Ray Kroc, remembers the #s on Golden Arches around the world: “500 billion served.” Lola adapts this for her enterprise: ### Customers Refreshed.

After calculating each day’s sales, she updates the “customers refreshed” metric and sends it out to her team. This helps align her budding organization on the people they’re helping (v.s. the money they’ve earned). And, at the end of the summer, when the number has accumulated a few extra zeros, they can reflect and feel proud of keeping their community hydrated.

Why does this matter?

We all want to be part of something bigger than ourselves. This is a fundamental part of what motivates us as people (src). The metrics we choose should reflect that. Further, as humans, we’re terrible at weighing “value now” v.s. “value later” (src), so the more visible you can make long term metrics, the easier it is to relate current actions to long-term outcomes.

Takeaway

It’s true that “what you measure, you’ll improve.” But it’s also true that “getting what you ask for” doesn’t mean “getting what you want,” and your efforts at performance tracking might do more harm than good. To solve for this, design your metrics intentionally. Ensure that each KPI is actionable, fair, robust, concrete and inspiring.

"What is this chart trying to tell you?"

Sun, 10 May 2020 00:00:00 GMT

TLDR

User-testing dataviz is uniquely important (and uniquely challenging) relative to other types of design
After hundreds of hours user-testing dataviz, a few high-leverage questions emerge as revealing and widely applicable
These questions will lead to more effective designs and, over time, more effective data design teams
3iap is happy to help with training workshops or custom design research.

After hundreds and hundreds of hours watching users misunderstand data visualizations, you might find user testing strangely addictive.

It’s fascinating how even the simplest charts and graphs can be so frequently misinterpreted. This isn’t just Fox News’ crimes against y-axes, which are purposefully designed to mislead. This can include basic bar charts, composed by smart designers, that still lead otherwise smart users to unexpected interpretations of the underlying data.

The first challenge: The tiniest design elements can unexpectedly impact users’ mental models and cause irrecoverable confusion. Many talented practitioners offer principles for presenting data visually, but with so many moving parts to consider, it’s impossible to predict the combined effect without actually testing it.

The second challenge: Data visualization is fundamentally different from other types of design. So it follows that it requires a different approach to test it and validate that it works.

If we want data to be impactful, it has to be understood. If we want it to be understood, it must be tested with real users. If we want to user test data visualization, we need to consider more than just usability.

What to look for when user testing data visualizations?

In testing visualizations, we’re typically checking for:

Comprehension: The visualization communicates what we think it does
Explainability: Users can relate the visualization to the underlying phenomena being measured (e.g. what’s happening in the real world?)
Affect: Users understand conclusions intellectually and they can feel it in their gut. It instills an appropriate sense of urgency.
Actionability: It nudges users towards a specific course of action.

The questions below are 3iap’s toolkit for validating these criteria.

1. “What do you see here?”

A controversial "tornado plot" highlighting cyclical change rates of Covid-19 deaths, from Danny Dorling. You can see several bewildered onlookers answer "What are you seeing here?" on Chart Chat, Twitter and Reddit.

“People often ask me: ‘What’s the most important thing I should do if I want to make sure my Web site is easy to use?’ The answer is simple. It’s not “Nothing important should ever be more than two clicks away,” or “Speak the user’s language,” or even “Be consistent.” It’s… ‘Don’t make me think!‘” (Steve Krug, “Don’t Make Me Think”)

What was true for Steve Krug in 2000 is still true for design today: To make something accessible, optimize for reducing cognitive load.

This may seem unintuitive for tools that only exist for navigating complexity. You might even say “but this data is meant to be thought-provoking!” But just because the story is complex, doesn’t mean the presentation should be.

To validate this, you want to test what users conclude from a visualization if you weren’t sitting there with them. (Steve calls this “get it” testing.)

So, just like you would in a typical user test, start with open-ended questions like “What are you seeing here?” or “What is this graph trying to tell you?” It’s a gentle nudge to see how users respond, free of any prompting, priming or bias.

This gives you a signal of what captures their attention and the conclusions they might draw on their own.

What to look for?

Good: They quickly recite the intended conclusions, in priority order. They’re not squinting. “I did a good job today, I walked 10k steps”
Trouble: They’re distracted by minutia. They imagine spurious contradictions. “This says I walked 2km. Is that today or this week? I only burned 500 calories, but it’s green so I guess that’s good? Also, how long is a km again?”
Tip: 50% of people will recite literally what they’re seeing on the screen (e.g. “I see a blue chart that says ”# of steps today. Then a green graph that says ”# calories. I like the colors!”). Just re-ask a different way to nudge toward interpretations (e.g. “What is this screen trying to tell you?”).

2. “What was the best [x]? What was the worst [x]?”

Potential user responses to various graphs. Left: Step Counts (Google Fit); Middle: Screen Unlocks (Android); Right: Ambulatory Glucose Profile (AGP)

To be truthful and revealing, data graphics must bear on the question at the heart of quantitative thinking: “Compared to what?” (Edward Tufte, “The Visual Display of Quantitative Information”)

The whole point of visualization is comparing quantities visually. Our brains understand geometry better than abstract numbers, so we visualize.

Testing is simple: Ask users questions about the data. One of the more reliable: ask them to identify the extremes (e.g. “What day was your healthiest day? What day was your least healthy?” or “When did you make the most sales? When was the slowest?”).

You might also consider phrasing this in terms of value judgements (e.g. “best, worst”, not “largest, smallest”), to make sure they’re able to differentiate quantities and draw appropriate conclusions. This is particularly important when “good” isn’t necessarily the highest or lowest values presented (e.g. blood-glucose levels).

You can also follow this up with similar quiz-like questions, such that, if they understand the data, they should be able to easily answer the question (e.g. “During which period(s) was there a decline in the number of births?”). See this writeup on questions used in data-literacy studies for more examples.

Hearst & friends explore this further and find that having people answer questions about complex visualizations helps make designers “aware of the problems that might arise in actual use of their design.” (src)

Answers to these questions might be obvious when testing simple representations, like a single bar chart. But the more you stray from simple representations, the more they’ll struggle. For example, see NYTimes’ exploration of Elizabeth Warren’s 2019 policy proposal costs. You could expect people to struggle with determining if “Housing” or “Other” are the smallest portions. That’s not always a bad thing, it just depends on the level of precision required to tell the story.

What to look for?

Good: They should be able to answer these quickly. They can spot ambiguity, if it exists. There’s little trouble relating quantities to labels.
Trouble: More squinting. Mistaking highs for lows. Tracing with fingers.

3. “What caused the data to have this shape? What happened in real life that affected the numbers you’re seeing here?”

Hypothetical user responses to 2 variations of New US Covid Cases per Day (7-Day Avg), from March 14 to May 14

“Abstraction makes it harder to understand an idea and to remember it. It also makes it harder to coordinate our activities with others, who may interpret the abstraction in very different ways.” (Chip & Dan Heath, “Made to Stick”)

Data visualization is, by nature, abstract. Becoming too abstract is a persistent risk. This makes it difficult for people to relate the visualization with what they might experience in real life, ultimately limiting the impact of the data.

For example, the Covid Tracking Project recommends against abstracting away death counts that mute the emotional impact of Covid-19 deaths.

“We recommend using total numbers for plotting deaths to compare one US state or territory against another. In this case, adjusting per capita adds a layer of abstraction to the graphic. This reduces the data’s power and the reader’s comprehension.” (Covid Tracking Project “Visualization Guide”)

But it’s not just abstraction that can mask the underlying phenomenon.

Sometimes telling the complete story requires additional context. For example, charts from St Louis Fed’s FRED show overlays for recessions, giving additional context to properly interpret the data.

Or, sometimes the complete story can be hidden under a homogenous curve. For example, if you were to look at the curve of US Covid-19 cases, you might think the case-count is declining across the country. In reality, most of that drop is due to just 2 states (New York and New Jersey), while case-counts across the rest of the country remain relatively flat.

By asking people to give an explanation for the underlying phenomenon, you can determine how easily they relate the data to their own lives. This can indicate the need to simplify, decompose or supplement the visualization with added context.

What to look for?

Good: They relate the data back to personal experiences (if personal data), they can offer plausible explanations (e.g. “This week was probably bad for Snow Cone sales because it’s December and it’s 27 degrees outside?”), or at least they can interrogate the data to learn more (e.g. tapping a bar on a bar chart to decompose it).
Trouble: Non-answers. They say “well it depends…” (indicating ambiguity). They hunt around for context clues but come up short.

4. “Overall, is this good or bad?”

Hypothetical user responses to Jawbone's classic "UP Coffee" App, tracking users' caffeine consumption.

“Because visceral design is about initial reactions, it can be studied quite simply by putting people in front of a design and waiting for reactions.” (Donald Norman, “Emotional Design: Why We Love (or Hate) Everyday Things”)

To persuade, it’s important to appeal to audiences both rationally and emotionally. Data, when well-presented, can accomplish both. To do this, the visualization should create a sense of tension by highlighting the difference between “this is how things are now” and “this is how much better they could be.”

This works for both for personal, performance visualizations (e.g. Fitbit creates tension between your current step count and the magical 10,000 steps) and for more objective, journalistic work (e.g. looking at this graph, Americans might feel tension: “Denmark, Italy and Canada are beating the US in testing?”).

When this is well-executed, you can expect an almost visceral reaction from your users. Not only do they intellectually understand the “gap,” they feel it.

To verify this, simply ask your test subjects to judge for themselves (e.g. “Do you think this is good or bad?”).

What to look for?

Good: At the very least, they should be able to identify the “gap” and say whether the data presents a scenario that is good or bad. Ideally, they can speak to the magnitude of the difference. Even better, their own emotional arousal matches that magnitude.
Trouble: Hesitation. Guessing. Mistakes. Apathy.

5. “What might you do differently to change these numbers next [week/month/year]? How would this information influence your approach?”

Actionability is in the eye of the beholder. A non-savvy user may only see noise, but an expert like Harry Nicholls could tell you how these scenarios represent "7 Popular Technical Indicators" and translate them into action (to Increase Your Trading Profits).

“There are really only three basic reasons why information ever has value to a business: 1. Information reduces uncertainty about decisions that have economic consequences. 2. Information affects the behavior of others, which has economic consequences. 3. Information sometimes has its own market value.” (Douglas Hubbard, “How to Measure Anything”)

Data is “actionable” if a reasonably savvy user would do something different as a result of consuming it. This is what Hubbard’s 1st and 2nd reasons refer to.

Testing for “actionability” is important for 2 reasons:

You want to make sure the visualization helps users understand the data well-enough to inform their future actions. This is a prerequisite for users to act. (i.e. The steering wheel is useless if you can’t see through the windshield.)
You also want to make sure you’re not overloading people with useless data (i.e. data vomit). If data doesn’t inform some decision or behavior, you probably don’t need to visualize it.

Validating actionability is slightly more tricky. First, identify several scenarios where, given scenarios A, B, and C, a reasonably informed person would tell you to do X, Y, Z (e.g. “Thermometer says 0 degrees” → “Wear a coat”; “Thermometer says 100 degrees” → “Stay inside”).

Second, make sure you’re testing with “reasonably savvy” testers. This is important: Even an objectively effective visualization won’t seem actionable to users who are inexperienced in the problem domain (e.g. the trading graphs above look like nonsense to me, but that doesn’t mean they’re ineffective for technical traders). Unless these non-experts are your audience, they’re only going to give you false negatives. So make sure you’re testing with folks who would know what to do with the information (i.e. users who, when presented with scenarios A, B and C, know to proceed with actions X, Y and Z)

Once you have your scenarios and reasonably informed test users, walk the testers through each scenario, as represented by the visualization, and ask them what they would do. Their answers might differ (e.g. “100 degrees” might mean “stay inside,” “wear sunscreen,” “stay hydrated”), but each of the responses should indicate whether or not the person knows it’s hot enough that they should do something different.

What to look for?

Good: Your experts can translate the scenario you’re presenting to a specific course of action.
Trouble: “I don’t know.” This can indicate a) the person isn’t an expert or b) the data presented isn’t sufficient to determine an action or c) the data wouldn’t actually have any bearing on downstream actions. (C. is the one to watch out for. That’s the sad path to data-vomit.)
Tip: Initial responses will sometimes be vague (e.g. “if the cookies aren’t selling, those lazy girl scouts need to try harder.”), so be ready to follow it up (e.g. “Sure. But based on what you’re seeing here, is there something else they might try?”). Then they might say “Oh, I see all they have left are Trefoils. No one eats that garbage. Get those girls some Thin Mints!”

Takeaway

Design is iterative. Data design is no exception. Ask just a handful of test users these 5 questions and, not only will you learn how to improve your visualizations, you’ll also spark great conversations that help you better relate to your users and ultimately make the work more understandable, relatable, impactful and actionable.

Don't be Creepy. Employee Surveillance Backfires.

Fri, 08 May 2020 00:00:00 GMT

Just because employee spyware exists, doesn’t mean you should use it.

This week, Adam Satariano published an article called “How My Boss Monitors Me While I Work From Home.” He describes a 3-week experiment with HubStaff, an employee surveillance tool. The takeaway: “ick.”

Coincidentally, on the same day, Github announced Github Insights, for tracking engineering team performance. Despite consternation on Hacker News, this is a much saner way to get a pulse on team performance.

Both tools attempt to accomplish the same goal (“team performance visibility,” especially for management). And both are a little creepy. But, the former fails in ways the latter handles quite well.

So what’s the difference?

First, consider 3 principles

If you track something, you’ll see it improve. It works for team OKRs. It works for Fitbits. This is well-documented and generally a good thing.
“Getting what you ask for” !== “getting what you want.” This is a fundamental challenge with any metric. There’s always distance between the measure and the underlying phenomenon. So, just because a metric improves, doesn’t mean underlying behavior has changed for the better.
Nothing good comes from treating people like robots. Grossly violating employee autonomy is a quick way to turn them against your cause.

How does this play out with screenshot tracking?

Satariano’s describes Hubstaff:

Last month, I downloaded employee-monitoring software made by Hubstaff, an Indianapolis company. Every few minutes, it snapped a screenshot of the websites I browsed, the documents I was writing and the social media sites I visited. From my phone, it mapped where I went, including a two-hour bike ride that I took around Battersea Park with my kids in the middle of one workday. (Whoops.)

…

One main feature of Hubstaff is an activity monitor that gives managers a snapshot of what an employee is doing. Broken down in 10-minute increments, the system tallies what percentage of time the worker has been typing or moving the computer mouse. That percentage acts as a productivity score.

It’s true that if you evaluate people based on “10-minute mouse-movements” and “productive looking screenshots”, you will see more mice moving and productive looking screenshots.

What won’t you see? Employees doing more work (or better work).

Even if “it’s a jerk move” doesn’t dissuade you from creeping, be aware that it’s also ineffective.

How does screenshot tracking sabotage productivity?

Screenshot or activity tracking implies that both of these behaviors will be rewarded: a) moving the mouse around while looking at “serious” things, or b) doing actual work. Because “moving the mouse” is easier than “doing actual work”, expect to see more “moving the mouse.”
To the extent that “moving the mouse” conflicts with “doing actual work,” (e.g. reporters get less credit for untracked behavior like calling sources), expect to see more “moving the mouse.”
And, finally, now that employees are pissed off by the now-apparent gaping void of trust, they have even less of an incentive to optimize their performance in good faith, so expect to see more “moving the mouse.”

The worst part: The manager will indeed see more “moving the mouse.” So, in addition to having employees who may-or-may-not be fully committed, the manager now has an even more opaque view of whatever issues are actually causing the team to slip.

Screenshot tracking on Upwork

One engineering lead I spoke with had previously hired programmers on UpWork, a freelance site that offers the option to screen-track workers.

He said: Screen-tracking the freelancers: “almost uniformly led to worse work”

“You have this false sense that you’re measuring their output. It lulls you into a sense of complacency. Plus there’s all these extra screenshots you’re expected to look at. If you can’t measure results by what this person is producing, there’s something deeply wrong.”

He also noted: “But my boss loved it.”

It’s not just screenshots…

Another engineering leader reflected on an early-career experience. He had an overzealous former-manager who insisted on strict time-tracking.

“It was a nightmare. The manager set expectations of 35 hours per week at the keyboard. For every bit of work you do, you track the task, you track the amount of time you spent on it.

He was looking at those numbers and he would bring them up often. So what I found myself doing - this was early in my career - I just fucking lied a lot.

It created stress. It created a bad habit where I had to hit a target number and it wasn’t about delivering code quality. It wasn’t about delivering value. So I just gamed the system.”

It’s not just tracking programmers…

Harvard Business School’s Ethan Bernstein has studied workplace surveillance extensively. In one study (“The Transparency Paradox”), he embeds a team of grad students on a factory floor, and they document even more surprising ways that close supervision can undermine performance:

First the [embedded researchers] were quietly shown ”better ways” of accomplishing tasks by their peers - a ”ton of little tricks” that ”kept production going” or enabled ”faster, easier, and / or safer production.”

Then they were told, ”Whenever the [customers / managers / leaders] come around, don’t do that, because they’ll get mad.” Instead, when under observation, embeds were trained in the art of appearing to perform the task the way it was ”meant” to be done according to the codified process rules posted for each task.

Because many of these performances were not as productive as the ”little tricks,” I observed line performance actually dropping when lines were actively supervised.

Again, it doesn’t pay to creep!

What’s different about GitHub Insights?

It plays nicely with the principles.

If you track something, it’ll improve.
“Getting what you ask for” !== “getting what you want.”
Don’t treat people like robots.

GitHub Insights analyzes engineering teams’ shared code repository to approximate team behaviors that (at least loosely) correlate to team productivity. Because Git repos are essentially big, living records of a teams’ interactions with their codebase, they can be a telling source of data on team behaviors. Other similar “Engineering Intelligence” tools include CodeClimate Velocity, Waydev and SourceLevel.

Unlike invasive methods like Screenshot Tracking, Git-based analytics give teams a way to measure performance that 1) focuses on metrics with a stronger relationship to desired behaviors, 2) is much less creepy and 3) focuses on teams, not individuals.

The metrics being emphasized are more representative of actual productive behaviors worth improving. It’s certainly still possible to game metrics like “code review speed”, but much harder than “moving your mouse.”

It’s much less creepy. Everything in Git is already accessible to the team. If you were so determined, you could calculate the same stats by hand.

The focus of the measurement is the team, not an individual. It’s much easier to buy-into. It’s ego preserving. No one is getting called out individually. And, because this approach preserves trust, people are more likely to optimize behaviors against the target metrics in good faith.

Takeaways

Tracking and measuring are fine. It works. But don’t creep. Favor me that reflect meaningful behaviors. Respect employee autonomy. Focus on improving team performance, not calling out individuals. Be nice. Be human.

Data Product Design for Matching Securities Counterparties

Thu, 01 Aug 2019 00:00:00 GMT

Context

Client

StreetLinx helps leading financial institutions improve connectivity with their trusted counterparties.

Prompt

How might we help buyside and sellside securities professionals better understand their network of counterparties?

Getting to proof

3iap worked with StreetLinx from early whiteboarding and journey mapping (proving the product conceptually), to user research (validating the problem), all the way through raising their first round (investor proof), onboarding their first customers (validating product-market fit) and scaling the product through their first 100 clients and dealers.

Background

Cofounders Gary & Pat left successful careers at Goldman Sachs to chase down an interesting problem, but they needed help to prove out the solution. As sellside veterans, they had extensive experience on the trading floor, talking with the world’s largest hedge funds and asset managers, connecting them on mutually beneficial trades. They were successful because they always knew the right person to call.

StreetLinx's pensive CEO Gary Godshaw, thinking deeply after a long day of brainstorming.

Problem

When you speak with 100s of clients every week, “always knowing who to call” is hard work. Keeping up with the constantly changing strategies and interest for every trader and portfolio manager — at every client — was a herculean task. Even simple tasks like looking up contact information might mean elaborate notebooks and spreadsheets, or worse, delving into the perpetually messy, incomplete data in Salesforce.

Early whiteboarding through (redacted) product designs

Approach

Insight

Clients want to be found. “Getting a look” on a trade helps their business, so they want their dealers to have up-to-date data on their “axe” (interest) of the day and how to get in touch. At the same time, clients wanted similar data on their dealers. As Gary said: “There’s a trade to be made here, I can feel it!”

Services

Product discovery & user research. Facilitating buyside and sellside interviews
Product design & prototyping. Supporting research, sales, fundraising and the final product
Data modeling & architecture. Modeling a diverse set hedge funds / asset managers, their relationships and their complex access controls
MVP design, development and launch. Enabling users to contribute their own data and search the network for matching counterparties

Results

Launched MVP product in 2018
Designs and prototypes helped client raise $MM+ seed round
StreetLinx was acquired in 2021

3D Clothing Rendering Pipeline

Mon, 01 Apr 2019 00:00:00 GMT

Prompt

For the Prickly Clothing Company, what changes about eCommerce when you can offer consumers unlimited SKUs?

Insight

Thanks to print-on-demand and 3d-printing, a growing number of physical products can be efficiently produced as just-in-time one-offs. If merchants can offer a wider variety of products, without the overhead of physical inventory, they can address a longer tail of customer needs.

Solution

Developed pipeline to transform digital pattern images into virtual inventory of women’s athletic leggings, rendering patterns, print files for manufacturing, inventory images and ads, then pushing inventory to multiple storefronts

Results

748 unique leggings generated, 4488 SKUs
New inventory could be generated in seconds

“I was super excited about the prospect of having these leggings in a variety of colors and prints. I couldn’t help myself and had to order more! Who can resist cat leggings!?!”

- Lila, Customer

Productizing Workforce Analytics

Sun, 01 Oct 2017 00:00:00 GMT

Prompt:

How might we help PwC UK translate a successful offline productivity program into an approachable, digital-first data product?

Background:

PwC Perform is a 12 week engagement where PwC coaches work closely with client teams to promote modern, data-driven team management practices and behavioral outcomes. The program draws teams’ attention to the metrics that matter and teaches leaders how to interact with their teams through the language of metrics. This drives significant results for a wide variety of organizations, consistently improving target productivity KPIs by 15-30%.

Problem:

Perform’s program was entirely pen and paper (or, marker and whiteboard). It was successful despite this, but felt anachronistic to clients with increasingly digital workforces. It also meant that any data written to the whiteboard was ephemeral, so opportunities for deeper or longer term analysis were missed.

Insight:

It’s not about the data, it’s about the conversations unlocked by the data. You can read more about this here.

Services:

Embedded with client team to provide ongoing design, research and product direction to client PMs, designers and engineers
Developed product and design strategy, focused on motivational data visualization

Results

Successfully launched PerformPlus MVP
End-user pilot teams engaged daily and significantly outperformed baseline performance goals
Perform program continues rapid growth

Personalized Dataviz: In The Long Run

Tue, 01 Jan 2013 00:00:00 GMT

Context

Prompt

How might we make healthy behavior change more viscerally rewarding and immediate?

Approach

Insights

Self-tracking can change behavior, but becoming healthy is an emotional journey, not just a physical change. Users need more than cold, clinical charts and graphs to stay motivated.

Solutions

Designed, developed and launched Notch.me MVP to encourage healthier behavior through visualization self-tracking data.
Designed a series of shareable, dynamic data visualizations, letting users visualize their behavior and consider different aspects of health.
Developed syncing service to integrate with leading tracking devices (e.g. Fitbit).

“There’s something sort of rock star about a gorgeously designed infographic created about something so personal and banal as how many steps you took today.”
PandoDaily

“Until I saw your graphics and your ‘result stories,’ I really wasn’t that dissatisfied. Now I want to view all of my data this way.”
Notch User

Results

Thousands of users generated and shared hundreds of thousands of personalized data visualizations
Featured on TechCrunch and PandoDaily

Personalized Fitness Dataviz for Consumer Health-Tracking

Tue, 01 Jan 2013 00:00:00 GMT

Context

Prompt:

How might we make healthy behavior change more viscerally rewarding and immediate?

Approach

Insight:

Self-tracking can change behavior, but becoming healthy is an emotional journey, not just a physical change. Users need more than cold, clinical charts and graphs to stay motivated.

Solutions:

Designed, developed and launched Notch.me MVP to encourage healthier behavior through visualization self-tracking data.
Designed a series of shareable, dynamic data visualizations, letting users visualize their behavior and consider different aspects of health.
Developed syncing service to integrate with leading tracking devices (e.g. Fitbit).

Results

Tens of thousands of users generated and shared hundreds of thousands of personalized data visualizations
Featured on TechCrunch and PandoDaily

“There’s something sort of rock star about a gorgeously designed infographic created about something so personal and banal as how many steps you took today.”

PandoDaily

“Until I saw your graphics and your ‘result stories,’ I really wasn’t that dissatisfied. Now I want to view all of my data this way.”
Notch User

3iap

Unfair Comparisons: How Visualizing Social Inequality Can Make It Worse

Introduction

TLDR

Eli’s Personal Backstory

What is “deficit thinking?” What does it mean for dataviz?

Why does this matter? What are the implications?

Previously-fine advice to revisit

Other implications

How could an innocent chart possibly cause stereotyping?

About Stereotypes

How could charts perpetuate stereotypes?

Showing Variability

The general hunch:

The “Dispersion vs Disparity” research project

Experiment design

Experiment Results

Hunch #1: Deficit framing is real. Many people misread “neutral” charts as evidence for personal stereotypes.

Hunch #2: Design choices matter. Charts that hide variability lead to more stereotyping.

Hunch #3: Confidence intervals aren’t enough. Show variability.

Experiment conclusions

What this study does NOT show:

Data design implications

Equity considerations for data designers

Resources and next steps

Experiment Results: The Risks of Visualizing Racial Inequality

Introduction

Sketching Sketchy Bar Charts.

A fun thing

What does this mean for dataviz?

Timely Advice: How Long Does Dataviz Take?

Introduction

3iap’s time-tracking dataset

How much “dataviz” work goes into dataviz?

Design + Engineering

How much “non-dataviz” work goes into dataviz?

Chaos / Overkill

10x 3iap dataviz projects - timing and details

1. Analytics Product Design System (14 Days)

2. In-Product Chart Component (11 Days)

3. Complex Report: Analysis & Presentation Design (24 days)

4. Embedded Reporting Tool Design (12 Days)

5. Scrollytelling Infographic (13 Days)

6. Exploration Dashboard Prototype (7 Days)

7. Complex Survey Exploration Tool Design (18 Days)

8. Interactive Scientific Storytelling (24 Days)

9. Interactive Text Content Explorer (8 Days)

10. Complex Dataset Exploration Tool Design (26 Days)

Other fun facts

Estimation accuracy

Data delays

Invoice delays

Calendar days v.s. work days

Methodology and Data

Takeaways

Americans Struggle with Graphs

Introduction

tldr:

The Curse of Knowledge

The Numeracy Problem

Numeracy isn’t innate.

The PIAAC studies “numeracy” across 38 countries.

How numerate are US Adults according to the PIAAC?

“But my audience is smart”

Graph Comprehension

How does numeracy relate to communicating data?

How will users “read” the data?

How much complexity can people handle?

PIAAC Sample Questions

The National Adult Literacy Survey Questions

“Graph Literacy: A Cross-Cultural Comparison” Questions

Takeaways

What can we do better?

Workplace Analytics Benchmark Report

Context

Client

Prompt

Background

Design

Challenge #1: What does “normal” look like?

Experiment #1:
Establishing an Effect

Experiment #2a:
Social Conformity

Experiment #2b:
Polarization