Histogram and scatter plot¶

Exercise histogram-and-scatter-annex.pdf · PDF

Are most games on Steam cheap or expensive? You have 26,685 prices in games_clean. A bar per exact price would give you hundreds of bars: 0.99, 1.99, 2.49, 2.99... Nobody could read that.

Competitive games solved this problem long ago. In ranked mode, nobody plots one bar per player. The game cuts the rating scale into ranges, calls them Bronze, Silver, Gold, Platinum and Diamond, and counts how many players are in each. One look and you know where most people are, and how rare the top is.

That picture is the first chart of this lesson. The second one answers a different kind of question, also from ranked mode: do players with more hours have a higher rank? One number cannot answer that. You need two, on the same picture.

The histogram: the rank distribution¶

A histogram cuts one numeric column into equal ranges and draws one bar per range. The height of each bar is the number of rows that fall inside it.

Each range is a bin. The shape the bars draw together is the distribution of the column: where the values pile up and where they are rare.

In ranked mode, the numeric column is the rating, the bins are the ranks, and the height is the number of players. With game prices, the column is price, each bin is a price range, and the height is the number of games.

The word is older than video games: the English statistician Karl Pearson introduced the term histogram in 1891.

It looks like a bar chart. It is not¶

Both have bars, and that is where the similarity ends.

In Categorisation you built price_band by hand, with CASE WHEN. Those bands had unequal widths that you chose: Under 5, 5 to 15, 15 to 30. A histogram is different: it takes the minimum and the maximum of the data and cuts everything in between into bins of equal width.

That last detail is about to bite.

One giant bar: what outliers do¶

Ask for a histogram of price with 5 bins and you get the chart on the left.

One enormous bar and almost nothing else: 26,658 products in the first bin, then 22, 2, 2 and 1. The chart is correct. It is also useless.

Why? Five equal bins between the cheapest and the most expensive product. So what is the most expensive?

SELECT game, price FROM games_clean ORDER BY price DESC LIMIT 6;

Look at the names. Most of these are not games: they are design and video software sold on Steam. A handful of rows stretch the axis to 595.99, so each of the 5 bins is 119.2 wide, and nearly every product in the table falls into the first one.

An outlier is a value that sits far away from almost all the others.

The fix has two parts: look only at the range where the games are, and use more bins. Filter to prices up to 60, ask for 12 bins, and you get the chart on the right. The filter leaves out only 83 of the 26,685 rows (0.3 %).

Reading a distribution in words¶

A histogram is finished when you can say it in one sentence. For the chart on the right:

Most games are cheap: 14,707 of 26,602 (55 %) cost less than 5, and that bar includes the 3,094 free games.

Three things to look for in any distribution:

The zigzag has a boring explanation. Shops love prices ending in .99, and 24.99 and 29.99 land in different bins. Some bins contain a popular price and their neighbours do not.

This picture also explains something from Exploratory statistics. The mean price is 7.91 and the median is 4.99. Why so different? The long right tail: a few expensive products pull the mean up, while the median stays where most games are. The histogram is the picture of why they differ.

The scatter plot: one dot per player¶

Back to ranked mode. Do players with more hours have a higher rank? A histogram cannot answer: it looks at one number. Take every player, put their hours on the horizontal axis and their rank on the vertical axis, and draw a dot.

A scatter plot draws one dot per row, placed by two numeric columns: one sets the horizontal position, the other the vertical one. It shows the relationship between the two.

Here is a real one from the class table pokemon. Each of the 949 dots is one Pokémon, placed by its attack and its defense:

You do not read a scatter plot dot by dot. You read the shape of the cloud:

This cloud rises to the right, loosely: Pokémon with more attack tend to have more defense. Tend to. The cloud is wide, and some dots ignore the rule completely. The dot alone in the top-left corner is Shuckle: attack 10, defense 230.

How tight a cloud is has a number.

Correlation measures how closely two numbers move together, from -1 (a perfect falling line) through 0 (a blob) to 1 (a perfect rising line).

In PostgreSQL the function is CORR:

SELECT CORR(attack, defense) FROM pokemon;

It returns 0.44: a positive relationship of moderate strength. The scatter plot is the picture, and the correlation is its one-number summary. Always look at the picture too: in Data do not speak alone you saw four datasets with the same summary numbers and four completely different shapes.

Correlation is not causation¶

A scatter plot shows that two things go together. It never shows that one causes the other.

Causation means that changing one thing makes the other change. A chart alone cannot prove it.

Go back to ranked mode. Suppose players with more hours do have a higher rank. Does playing more make you better? Maybe. Or maybe good players enjoy the game more and so play more: the arrow points the other way. Or a third thing drives both.

The class data have a clean example. Among Steam games that have a review score, pricier games score a little higher. Does raising the price improve the reviews? Of course not. Big studios charge more and have bigger budgets. The budget sits behind both numbers.

flowchart TB
    B["<b>A third thing</b><br>a big studio<br>with a big budget"]
    B --> P["<b>Higher price</b>"]
    B --> M["<b>Better reviews</b>"]
    P -. "the chart shows these two together.<br>It does not show an arrow" .- M

Only the rows that have the value¶

There is one more trap in that price and review example, and it is about which rows are in the chart.

The review score is the column metascore. Only 2,849 of the 26,685 games have one: about 11 %. In Imputation you decided not to invent the missing scores, and you added the column has_metascore to tell the two groups apart.

So a chart of price against metascore describes those 2,849 games, not Steam. And they are not a random 11 %: critics review the games that get attention. Say it in the title.

Here is what Superset draws when you ask for a scatter plot of price and the average metascore:

The cloud rises slightly. But look at which dots you can trust:

The dot at 9.99 is the average of 661 games. The dot at 54.99 is one single game that happened to score 88. Superset draws both with the same size. In the figure above the single-game dots are hollow so that you can tell them apart: 26 of the 58 dots stand for only one game.

Game by game, the relationship is weak: CORR(price, metascore) over the 2,849 reviewed games is 0.16.

Try it: prices, and price against metascore¶

Both charts use the dataset games_clean (see Resources for the data).

Chart 4 — Price distribution. Where are most games, and how long is the tail?

Do not skip the row limit in step 2. A histogram is computed from the rows it receives, and with the default it would read only 10,000 of your 26,685 games.

The twelve bars must match these counts:

Chart 5 — Price vs metascore. Do more expensive games get better reviews?

The dots are the 58 rows of this query:

SELECT price, ROUND(AVG(metascore)::numeric, 1) AS avg_metascore, COUNT(*) AS games
FROM games_clean WHERE has_metascore GROUP BY price ORDER BY price;

Key takeaways¶

• A histogram cuts one numeric column into equal bins and counts the rows in each. It shows a distribution: where the values pile up and how long the tail is.
• A histogram is not a bar chart: its X-axis is a number line, its height is always a count, and you choose the number of bins.
• Outliers stretch the axis and squash everything into one bar. Filter the chart to a sensible range, say so in the title, and do not delete the rows.
• A scatter plot places dots by two numeric columns and shows their relationship: rising, falling or none. Correlation puts a number on it.
• Correlation is not causation. Two things can move together because a third thing drives both.
• Always ask what one mark stands for and which rows are in the chart. Price against metascore describes the 11 % of games with a review, and each dot is a price, not a game.

Quick check¶

You can now show how one number spreads and how two numbers relate. Two famous charts are still missing, and one of them is very easy to get wrong: Pie, heatmap and others.