Big data¶

Is the spreadsheet with your class marks big data? Of course not. And every tap of every travel card on every bus in Mallorca for a whole year? Or every search for a flight and a hotel that a Palma travel website receives in August? Somewhere between those examples there is a line. Where is it?

Big data is one of the enabling technologies you met in Unit 2, and in Unit 4 you saw that AI learns from data. This lesson explains what makes data "big", why "a lot of data" is not the right answer, and how big data, data analysis, machine learning, deep learning and AI fit together, so that you can stop using the five terms as if they were synonyms.

What makes data "big"¶

Big data are datasets that, because of their volume, the speed at which they arrive or their variety, cannot be stored, processed or analysed efficiently with traditional tools (a spreadsheet, a single database on one server) and need a scalable architecture.

In the city-builder game, a small town has a few dozen sensors and you could read them one by one. A city of a million inhabitants has traffic counters, water meters, bus GPS units and citizens complaining on social media, all at once, every second. The overlays still work, but behind them there is no longer one computer: there is a whole data centre.

The first three V's were proposed by the analyst Doug Laney in a 2001 research note for the META Group (now part of Gartner). Most textbooks today add two more.

Some sources add more V's (variability, validity, visualisation...). The US National Institute of Standards and Technology (NIST), for example, adds variability. You do not need to memorise them: volume, velocity and variety are in almost every definition, and veracity and value complete the usual five.

Structured, semi-structured, unstructured¶

Variety becomes clearer when you sort data by how much structure they have.

Who is who: big data, analysis, ML, DL and AI¶

flowchart TB
    BD["<b>Big data</b><br>(the raw material<br>and its warehouse)"] --> DA["<b>Data analysis</b><br>(asking questions<br>of the data)"]
    BD --> ML
    subgraph AI ["<b>Artificial intelligence</b>"]
        subgraph ML ["<b>Machine learning</b>"]
            DL["<b>Deep learning</b><br>(many-layer<br>neural networks)"]
        end
    end
    DA -- "predictive analysis<br>often uses" --> ML

Big data without analysis is a full warehouse that nobody looks at. Analysis without good data gives bad answers. And a lot of the AI that companies actually use is predictive analysis done at scale. That is why the three usually go together.

Four questions, four types of analysis¶

In class: big data in your company's sector¶

Work in pairs, on the sector of your group's project company. This is guided web research: every claim needs a working source. No AI tools.

For your project¶

Steps 3 to 5 are raw material for section B (mini data-science pipeline) of Deliverable 5: section B starts from a question, and the question you write today is a good candidate. The real case you found can also go into the group part as context for the sector. Cite it.

Key takeaways¶

• Big data are data that traditional tools cannot handle efficiently because of their volume, velocity or variety; veracity and value complete the usual five V's.
• Data can be structured, semi-structured or unstructured; variety means mixing them.
• Big data is not "lots of data", and not every company needs it.
• Deep learning is inside machine learning, which is inside AI; big data is the raw material, and data analysis is asking the data questions.
• The four types of analysis answer four questions: what happened, why, what will happen, what should we do.

Quick check¶

You know what big data is and how it feeds analysis and AI. Next you will do the analysis yourself, step by step, on a real dataset, and decide where the data should live: Data science and the cloud.