Misleading charts¶

Exercise misleading-charts-annex.pdf · PDF

Look at the red chart below for two seconds, then answer without thinking: how many times better do critics rate Hit games than Niche games?

Most people say five, six, seven times. Now look at the blue chart. Same table, same query, same three numbers: 67.1, 71.5 and 79.8. Hit games score 19% higher, not 600%.

Nobody changed a number. And yet one of these two charts lies. Since Configuring charts in Superset you are the one holding the knobs, so today you learn the tricks from the inside: to catch them, and to never do them by accident.

Photo mode for data¶

Many video games have a photo mode. The match is the same, but with zoom, crop, angle and filter you can make a lost match look like a win. Nobody edited the match. Somebody only chose what to show.

A misleading chart is a chart in which every number is true and the picture still leads the reader to a wrong conclusion.

It is photo mode for data. We keep the analogy for the whole page: the data is the match, the chart is the photo. There are five classic tricks, and each one is a photo-mode control:

Trick 1: the truncated axis (zoom)¶

A truncated axis is an axis that does not start at zero, so only the top of each bar is drawn.

That is the red chart of the hook. Its Y axis runs from 65 to 80. The numbers behind it:

SELECT owners_tier, ROUND(AVG(metascore)::numeric, 1) AS avg_metascore, COUNT(metascore) AS scored
FROM games_clean GROUP BY owners_tier ORDER BY avg_metascore DESC;

In a bar chart the length of the bar is the number. Cut the axis at 65 and the Hit bar is no longer 79.8 long: it is 79.8 − 65 = 14.8. The Niche bar is 67.1 − 65 = 2.1. The picture now compares 14.8 with 2.1, two numbers that exist nowhere in the data.

Measure the lie: the lie factor¶

In 1983 Edward Tufte, in his book The Visual Display of Quantitative Information, gave the lie a number.

Lie factor = size of the effect shown in the chart ÷ size of the effect in the data.

An honest chart has a lie factor close to 1. Tufte's limit: above 1.05 or below 0.95, the chart distorts. Do it with the two charts of the hook:

Lie factor: 613 ÷ 19 ≈ 32. The picture shows an effect thirty-two times bigger than the one in the data. You can let PostgreSQL do the sum with the unrounded averages:

WITH t AS (SELECT AVG(metascore) FILTER (WHERE owners_tier = 'Hit')   AS hit,
                  AVG(metascore) FILTER (WHERE owners_tier = 'Niche') AS niche FROM games_clean)
SELECT ROUND((hit / niche)::numeric, 2) AS real_ratio,
       ROUND(((hit - 65) / (niche - 65))::numeric, 2) AS drawn_ratio,
       ROUND((((hit - 65) / (niche - 65) - 1) / (hit / niche - 1))::numeric, 1) AS lie_factor FROM t;

It returns 1.19, 7.13 and 32.3.

Trick 2: the cherry-picked range (crop)¶

A cherry-picked range is a start or end date chosen because it produces the headline, not because the question asks for it.

"Game prices in free fall", says the red chart. From 2013 to 2018 the average price of a new Steam game went from 11.21 to 7.12 dollars: a fall of 36.5%. True.

Now the whole match. The data starts in 2004, when the average was 7.27 dollars. Over the full range the change is 7.27 → 7.12: a fall of 2.1%. Prices climbed to a peak in 2013 and came back to where they started.

SELECT release_year, ROUND(AVG(price), 2) AS avg_price FROM games_clean
WHERE release_year IS NOT NULL GROUP BY release_year ORDER BY release_year;

The crop started at the highest point on purpose. Start anywhere else and the headline dies.

Trick 3: dual axes (forced perspective)¶

You know the tourist photo where somebody "holds up" the tower of Pisa. Two things at different distances, lined up so they seem to touch. That is forced perspective.

A chart with dual axes draws two series with two different Y scales, one on the left and one on the right.

Below, the same two series twice: games released per year on the left axis, average price on the right axis. The only difference between the panels is the range of the right-hand axis.

"What happened in 2016, when the two lines crossed?" Nothing happened. Stretch the right axis and the crossing moves two years back. A crossing between two different scales is made by the axis bounds, not by the data. Games and dollars cannot be bigger or smaller than each other.

The honest version costs nothing: two charts, one above the other, sharing the same X axis. The reader still compares the shapes, and no fake crossing appears.

Trick 4: 3D and crowded pies (camera angle)¶

A 3D pie is tilted towards you, so the slices at the front get more ink than the slices at the back for the same percentage. Superset has no 3D pie, but spreadsheets and the press are full of them. The fix is always the same: flat.

The trick you can do in Superset is the crowded pie. This is the Steam catalogue by release year, fifteen slices:

pie
    title The Steam catalogue by release year
    "2004" : 53
    "2005" : 58
    "2006" : 112
    "2007" : 143
    "2008" : 183
    "2009" : 334
    "2010" : 340
    "2011" : 364
    "2012" : 426
    "2013" : 556
    "2014" : 1352
    "2015" : 2544
    "2016" : 4503
    "2017" : 6871
    "2018" : 8845

Without looking at the numbers: what share of the catalogue was released in 2012? And in 2009? (1.6% and 1.3%. Eight of the fifteen slices are thinner than 1.5%.)

There are two mistakes in one here. Fifteen slices are far past the limit of five. And the question was never composition: years are a point in time, so the job is evolution and the tool is a line. You already have the honest chart: UT3 · Games released per year, straight from the decision table of Choosing the right chart.

Trick 5: bad colour (filter)¶

Colour is the filter of photo mode: it changes the mood without changing the scene. Three ways it goes wrong:

On red against green, one fact: about 1 in 12 men and 1 in 200 women have some form of colour blindness, and red-green is by far the most common kind. In a class of 25, expect one or two people for whom your red line and your green line are the same line.

Try it: build the lie, then fix it¶

The best way to recognise a trick is to do it once on purpose. For each case you first take the misleading photo, then name the trick, fix the chart and write the honest title. You need the games_clean dataset in Superset (see Resources).

Case A: "Hit games are in another league".

One more honest sentence for this chart: only 486 of the 17,921 Niche games have a Metascore at all, 2.7%. The Niche bar describes a small, unusual corner of the Niche games.

Case B: "Game prices in free fall".

Case C: "The Steam catalogue by year".

Key takeaways¶

• A misleading chart has true numbers and a false picture. Readers read lengths, slopes and areas, not numbers.
• Truncated axis: bars must start at zero, because the length of the bar is the number. A line may zoom in, if the axis is labelled and the change shown matches the data.
• The lie factor (Tufte, 1983) is the effect shown in the chart divided by the effect in the data. Honest is close to 1; the bar chart of the hook scores 32.
• Cherry-picked range: show the whole range, or say in the title why you cut it.
• Dual axes create crossings that depend on the scales. Stack two charts instead.
• Crowded and 3D pies and bad colour hide the comparison: five slices at most, flat, and colour only when it means something.

Quick check¶

You can now build a chart, choose it, configure it and defend it. One chart answers one question; real decisions need several at once, side by side, on one screen. That is Unit 4: Dashboards and reports.