By Helena Saenz de Juano Ribes, Research Economist at IES.
“Big Data“ emerged as a term to describe the massive datasets that were challenging to process, manage, and analyse using traditional database and software techniques. These datasets were defined by their immense size, diverse formats, rapid generation, and the potential insights they held for businesses. The concept gained prominence alongside advances in computing, the advent of the World Wide Web, and the rapid expansion of the Internet. However, it wasn’t until the 2010s that Big Data became a dominant force, driven by the proliferation of cloud computing, smartphones, and social media platforms, which collectively led to an unprecedented surge in data generation. This explosion of data prompted businesses to realize its immense economic value, particularly through data-driven decision-making and predictive analytics.
Unlocking the potential of Big Data required innovative technologies, including distributed computing, machine learning, and advanced data analytics. As a result, new job opportunities emerged to meet these technological demands. Industries such as healthcare, e-commerce, finance, and urban planning became major adopters of Big Data, using it for tasks like predictive analytics, personalized recommendations, fraud detection, and the development of smart cities. During this time, job titles containing the term “Big Data” were highly sought after, often commanding impressive salaries and becoming a hallmark of technological sophistication.
Using data from the Adzuna Intelligence Platform, we examined the demand for Big Data-related jobs from January 1, 2017, to mid-December 2024. In the years leading up to 2020, the demand for these roles, while modest at around 350 postings per day, followed similar trends to overall job vacancies. Salaries for these positions were consistently competitive, often exceeding £60,000 and peaking near £90,000 in 2019. However, the economic shock caused by the COVID-19 pandemic marked a turning point. While the overall job market began recovering after the initial slowdown, the demand for jobs explicitly labelled as “Big Data” started to decline. The trend that once defined the industry began to fade.
Does this decline in Big Data branding reflect a decrease in the importance of analysing vast datasets? The answer is no, what was happening on the post-2020 period, instead, was a shift in focus and terminology. As AI and automation advanced, Big Data became more integrated into mainstream operations and less commonly identified as a standalone trend. The foundational skills and technologies of Big Data remain critical, powering much of the AI-driven innovation we see today. However, they are now embedded in broader roles that emphasize AI, cloud computing, and data science, with less focus on the “Big Data” label.
The Adzuna Intelligence Platform shows that, for example, job titles such as Data Engineer—often responsible for managing and processing large datasets—grew significantly in demand during the pandemic recovery years of 2021 and 2022. Yet, by 2023, these roles saw a decline in growth relative to the pace of overall job vacancies. This raises important questions: Is this simply a case of rebranding, or has the automation of AI tools reduced the demand for data engineers in the UK?
AI and machine learning tools, while reliant on Big Data, have made it much easier to handle and process large datasets. As these frameworks become more accessible and user-friendly, the focus has shifted from managing data infrastructure to applying AI in practical and innovative ways. This shift has led to changes in job demand, with new positions emerging and traditional ones being redefined. While the term “Big Data” may no longer dominate, the skills and technologies behind it are essential for roles in the AI-driven job market. In fact, much like the rise of Big Data roles in the 2010s, we are likely to see a surge in vacancies featuring “AI” or “LLM” as a central component of their titles. However, will the “AI” and “LLM” have the same fate as “Big Data”?
IES is an independent, apolitical, international centre of research and consultancy in public employment policy and HR management. It works closely with employers in all sectors, government departments, agencies, professional bodies and associations. IES is a focus of knowledge and practical experience in employment and training policy, the operation of labour markets, and HR planning and development. IES is a not-for-profit organisation.
Read more: