Aggregation of information from diverse networks as the basis for training cyber security specialists on processing ultra large data sets




big data, social networks, cybersecurity, information retrieval systems, data aggregation, data science, information technology


The basic principles of training cybersecurity specialists on processing large data sets to solve complex unstructured tasks in the course of their functional responsibilities based on the achievements of Data Science in the field of cybersecurity, by acquiring the necessary competencies and practical application of the latest information technologies based on methods of aggregation of large amounts of data are substantiated and presented. The most common latest technologies and tools in the field of cybersecurity, the list of which allows getting a fairly holistic view of what is used today by specialists in the field of Data Science, are considered. The tools you need to have to solve complex problems using big data are analyzed. The subject of the study is the fundamental provisions of the concept of “big data”; appropriate data models; architectural concepts of creating information systems for “big data”; big data analytics, as well as the practical application of big data processing results. The theoretical basis of the training, which includes two sections: “Big Data: theoretical principles”, and “Technological applications for big data”, which, in turn, are logically divided into ten, is considered. As a material and technical basis for the acquisition of practical skills by students, a model based on the system “CyberAggregator” was created and described, which operates and is constantly improved in accordance with the expansion of the list of tasks assigned to it. The CyberAggregator system consists of three main parts: a server for collecting and primary processing of information; an information retrieval server (search engine); an interface server from which the service is provided to users and other systems via the API. The system is based on technological components such as the Elasticsearch information retrieval system, the Kibana utility, the Neo4j database graph management system, JavaScript-based results visualization tools (D3.js) and network information scanning modules. The system provides the implementation of such functions as the formation of databases from certain information resources; maintaining full-text databases of information; detection of duplicates similar in content to information messages; full-text search; analysis of text messages, determination of tonality, formation of analytical reports; integration with the geographic information system; data analysis and visualization; research of thematic information flows dynamics; forecasting events based on the analysis of the publications dynamics, etc. The suggested approach allows students to acquire the necessary competencies needed to process effectively large amounts of data from social networks, create systems for monitoring network information on cybersecurity, selection of relevant information from social networks, search engine implementation, analytical research, forecasting.

Author Biographies

Dmytro Lande, Institute for information recording of National academy of science of Ukraine

doctor of technical science, professor, head at the specialized modeling tools department

Oleksandr Puchkov, Institute of special communication and information protection National technical university of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”

candidate of philosophy science, professor, head

Ihor Subach, Institute of special communication and information protection National technical university of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”

doctor of technical science, associate professor, head at the cybersecurity and application of information systems and technologies academic department


B. Franks, and T. Davenport, Taming The Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics. Hoboken, USA: Wiley, 2012.

D. Lande, and E. Shnurko-Tabakova, “OSINT as a part of cyber defense system”, Theoretical and Applied Cybersecurity: scientific journal, vol. 1, no. 1, pp. 103-108, 2019, doi:

D. Cielen, and A. D. B. Maysmen, Introducing Data Science Big Data, Machine Learning, and more using. Python tools. New York, USA: Manning Publications, 2016.

O. G. Dodonov, D. W. Lande, V. V. Pryshchepa, and V. G. Putyatin, Competitive intelligence, Kyiv, Ukraine: LLC “Engineering”, 2021.

P. J. Sadapage, and M. Fowler, NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. Boston, USA: Addison-Wesley Professional, 2013.

T. White, Hadoop: Detailed guide, Newton, USA: O’Reilly Media, 2009.

P. Shukla, and S. Kumar, Elasticsearch, Kibana, Logstash and new generation search engines. Moskow, Russia: Piter, 2019.

B. Azarmi, Learning Kibana 5.0. Exploit the visualization capabilities of Kibana and build powerful interactive dashboards. Birmingham, England: Packt Publishing, 2017.

D. Lande, I. Subach, and A. Puchkov, “System of analysis of big data from social media”, Information & Security, vol. 47, no. 1, pp. 44-61, 2020, doi:

D. Lande, O. Puchkov, and I. Subach, “System for analysing of big data on cybersecurity issues from social media”, Information Technology and Security, vol. 8, iss. 1, pp. 4-18, 2020, doi:

K. Cherven, Mastering Gephi Network Visualization. Produce advanced network graphs in Gephi and gain valuable insights into your network datasets. Birmingham, England: Packt Publishing, 2015.

K. Cherven, Network Graph Analysis and Visualization with Gephi Visualize and analyze your data swiftly using dynamic network graphs built with Gephi. Birmingham, England: Packt Publishing, 2015.

R. V. Bruggen, Learning Neo4j. Run blazingly fast queries on complex graph datasets with the power of the Neo4j graph database. Birmingham, England: Packt Publishing, 2014.

B. М. Gerasimov, О. Y. Sergeev, I. Y. Subach, “Extraction of information phrases from primary electronic documents in information retrieval systems”, Control Systems and Machines, no. 1, pp. 26-29, 2006.

D. W. Lande, and A. N. Grigoriev, “Multilevel classifier-navigator according to the responses of the information retrieval system”, in Proc. Computational linguistics and intelligent technologies: proceedings of the international conference Dialogue, Russia, 2006, pp. 329-331.

S. Murray, Interactive Data Visualization for the Web. An Introduction to Designing with D3. Newton, USA: O’Reilly Media, 2017.

D. Sornette, Why Stock Markets Crash: Critical Events in Complex Financial Systems. Oxford, England: University Press Scholarship, 2017.




How to Cite

Lande, D., Puchkov, O., & Subach, I. (2021). Aggregation of information from diverse networks as the basis for training cyber security specialists on processing ultra large data sets. Collection "Information Technology and Security", 9(1), 4–16.