When working with data, I am usually dealing with something that can be loaded into Python, cleaned with pandas and analysed on a single computer. The Big Data Analytics module in my MSc in Artificial Intelligence and Machine Learning made me think about what happens when that approach is no longer practical.

What if the dataset is too large for one machine? What if new data is arriving every second? What if the information comes from databases, applications, sensors, websites and other sources at the same time?

This module helped me understand the technologies and infrastructure that make it possible to work with data at that scale.

Big Data Is Not Just About Size

One of the first concepts we covered was the 5 Vs of Big Data: Volume, Velocity, Variety, Veracity and Value.

Before studying the module, I mainly associated Big Data with having a very large amount of information. The 5 Vs showed me that size is only part of the problem.

Volume refers to the amount of data being generated and stored.

Velocity is about how quickly that data arrives and needs to be processed.

Variety covers the different forms of data, from structured database records to text, images and logs.

Veracity concerns the quality and reliability of the data.

Finally, Value asks the most important question: can we actually get something useful from all this information?

An organisation can collect huge amounts of data, but that does not automatically make the data valuable.

Understanding the Big Data Pipeline

Another useful part of the module was seeing Big Data as a complete process rather than just a collection of technologies.

Data has to move through several stages:

Collection → Storage → Processing → Analysis → Visualisation

This helped me understand where technologies such as Hadoop, HDFS, MapReduce, YARN, Apache Spark, Kafka and NoSQL databases fit into the picture.

At first, these can look like a long list of tools to learn. Once I understood the problems they were designed to solve, their roles became much clearer.

Some technologies focus on storing very large amounts of data across multiple machines. Others process that data, manage computing resources or deal with information arriving continuously.

The important part for me was not simply remembering what each technology does. It was understanding why we need them in the first place.

Hadoop and Distributed Computing

Hadoop introduced me to the idea of distributing both data and processing across several machines.

Instead of relying on one very powerful computer to handle everything, a large task can be divided into smaller pieces and processed across a cluster.

Within the Hadoop ecosystem, HDFS provides distributed storage, MapReduce handles distributed processing, and YARN helps manage computing resources.

This changed the way I thought about data processing. With a normal dataset, I usually think about how efficiently my code can process it. With Big Data, there is another question to consider: how can the workload be divided across several machines?

That is a very different type of problem.

Why Apache Spark Matters

We also spent time comparing Hadoop's traditional MapReduce approach with Apache Spark.

One of Spark's major advantages is its ability to keep intermediate data in memory for many workloads, rather than repeatedly reading from and writing to disk. This can make certain analytical tasks considerably faster.

Spark can also support different types of work within the same ecosystem, including batch processing, streaming, SQL queries and machine learning.

What I found useful about this comparison was realising that it is not simply a question of Hadoop versus Spark and choosing a winner. They represent different parts and stages in the development of distributed data processing.

Understanding what each technology is designed to do is more useful than simply saying that one is faster than the other.

Working With Data That Never Stops

The module also introduced the challenge of streaming data.

Not every organisation can collect information today, store it and analyse it tomorrow. In some situations, the information needs to be processed almost as soon as it arrives.

Examples could include financial transactions, application logs, website activity, sensors and connected devices.

This is where a platform such as Apache Kafka becomes useful.

It introduced me to another way of thinking about data. Instead of always working with a finished dataset, sometimes we are dealing with a continuous flow of information.

That creates a different set of technical challenges.

Connecting Big Data With AI and Machine Learning

The connection with AI and machine learning was particularly relevant to my MSc.

We often focus on the model itself: choosing an algorithm, training it, evaluating the results and improving its performance.

But a model still needs data.

When AI systems operate at a large scale, collecting, storing and processing that data becomes a major part of the overall system.

Big Data platforms provide the infrastructure that can make this possible.

This gave me a better appreciation of what happens before the machine learning stage. A model might be the visible part of an AI system, but there can be a large data infrastructure working behind it.

More Data Does Not Always Mean Better Results

Another important lesson was that collecting more data is not automatically useful.

Poor-quality data remains poor-quality data, even if you have billions of records.

This is where Veracity becomes particularly important.

The module also covered practical concerns such as privacy, scalability, processing speed and cost.

For example, an organisation might technically be able to collect and process a huge amount of information, but that does not necessarily mean it should. The cost may outweigh the benefit, the quality may be poor, or there may be privacy concerns around how the information is collected and used.

Big Data Analytics therefore involves much more than building the biggest possible data platform.

The system still needs to solve a useful problem.

Seeing Data From a Different Perspective

The biggest change for me was probably the way I started thinking about datasets.

Normally, when I receive a dataset, my first questions are about the analysis. What does the data contain? How should I clean it? Which variables are useful? What can I learn from it?

This module made me think about what happened before the dataset reached me.

Where did the data come from?

How was it collected?

Where is it stored?

What happens if the amount of data grows dramatically?

What happens if thousands of new records arrive every second?

How do several machines work together to process it?

These are questions that an analyst might not always see, but they are an important part of modern data systems.

My Main Takeaway

Big Data Analytics helped me understand that working with data at scale is not simply the same analysis performed on a bigger dataset.

Scale changes the way the entire system has to be designed.

Technologies such as Hadoop, Spark, Kafka and NoSQL databases exist because traditional approaches eventually reach practical limits.

For me, the module helped bridge an important gap between using data for analysis and understanding the infrastructure that makes large-scale analysis possible.

As I continue working with AI and machine learning, that wider view is useful. Before a model can learn from data, someone still has to collect it, store it, process it and make sure it is reliable enough to use.