The events of 2022 also affected the Russian Big Data market, which faced the inability to use familiar software and purchase some equipment.
You can watch the recording of the discussion .
Moving to the Cloud and Using Open Source Changes in the IT Market Have Affected Companies’ Plans for Working with Big Data
In 2022, Russian companies faced limitations in the use of the Big Data technology stack based on software from foreign vendors, the difficulty of purchasing equipment to scale physical infrastructures, and the need to quickly develop competencies to work with alternative software products.
Participants in the discussion
“New Big Challenges for Big Data. The Impact of Big Challenges on the Russian Big Data Market: Expert Opinions .
The number of companies that have started
The choice of a new technology stack is also, according to the participants of the discussion, an important task facing the IT director today
“Companies built Big Data solutions, in particular Data Warehouse (DWH), with high requirements for compliance with standards, performance, scalability and other criteria. Often, only a few solutions from major vendors met these requirements. After foreign suppliers left the Russian market, the choice of Enterprise platforms for building DWH became smaller. In fact, today companies choose between Greenplum, GaussDB and Clickhouse, which can to a certain extent replace solutions that have become unavailable.”
Increased efficiency of working with data as an effect of migration to new technologies
“Many Russian companies are now in a situation of forced migration, which is impossible without rethinking processes and approaches. But the primary task for all companies faced with the need to migrate to a new stack is to ensure the continuity of processes.”
The participants in the discussion noted that the requirements for the migration process itself have not changed. Here are the main tasks rich people number data that companies face.
- Migrate gradually, transfer small processes sequentially.
Companies will work more actively with clouds
“Our main stack for working with Big Data is built on Apache Kafka, Apache Hadoop, Spark, ClickHouse, Grafana, SuperSet. These products cope well with high loads and petabytes of data and, importantly, they remained available. As a result, we did not have to change the stack, and the market shocks affected us minimally.”
The speakers at the discussion noted that Russian companies currently have several options for continuing to work with big data.
- Transition to open source tools.
- Migrating Big Data to the how to find phone numbers for cell phones cloud is about using tools and infrastructure that the provider is responsible for supporting.
- Working on the Bare Metal server aol email list infrastructure using available tools from Russian vendors.
“The cloud is a universal solution. Depending on the tasks, you can use both public and private clouds without end-to-end Internet access. Thanks to this, companies of any size can access the necessary resources and tools without facing the difficulties of administration and support, since these tasks are the responsibility of the provider.”
Formation of new competencies in teams is the main request of companies
“Developers are now faced with an uncomfortable situation – expert knowledge in some areas is simply no longer needed. Instead, even highly experienced specialists are forced to learn new tools practically from scratch.”
- Move big data work to the cloud to reduce the workload on in-house IT staff.
- Hire a Full Stack Data Engineer who can solve several tasks of different nature, covering the team’s weak points.
“Due to the shortage of specialists, generalists are becoming valuable personnel – the so-called Full Stack Data Engineers, who know all the stages of working with big data and at least at a basic level have mastered the corresponding tools. At the same time, we should not forget about developing new competencies within the team, but we need to study new tools in parallel with their implementation, so that work with big data does not stop.”
Key findings
- Big Data teams will increasingly use cloud infrastructure and cloud-based tools in 2022.
- Companies are forced to look for new specialists and develop new competencies within teams. The market is seeing a growing demand for Full Stack Data Engineers .
VK Cloud and Arenadata conducted a study of the Russian Big Data market. In it, we collected the experience of 150 CIOs, identified the approaches and plans of Russian companies to work with big data in 2022.
