Topic

TiDB Compute Engine Pipeline Model: Achieving High Efficiency and High Availability in Data Processing

In this talk, we will dive into the architecture and implementation of the Pipeline Model in TiFlash, the TiDB compute engine, a columnar storage engine designed to improve the speed and efficiency of analytical queries in the TiDB ecosystem.The Pipeline Model is one of the core components of TiFlash, which takes a pipeline approach to Pipeline Model is one of the core components of TiFlash, which organises and schedules the execution of query tasks in a pipeline manner, resulting in more efficient data processing and faster query response. We will first introduce the basic concepts of the Pipeline Model, including its design philosophy and main components. Then, we will analyse in detail how this model is implemented in TiFlash, including the processing mechanism of data flow and the specific process of executing queries, as well as how it supports massively parallel computing. Through this presentation, participants will not only be able to gain an in-depth understanding of the TiFlash Pipeline Model, but also gain insights into how it can be used in real-world applications to enhance the performance and efficiency of data analysis. This will be extremely helpful and inspiring for database administrators, data analysts, and professionals interested in distributed computing. Outline: I. Introduction to TiFlash Organisation and Scheduling of Computing Tasks III.Design and Implementation of Pipeline Model 1.CPU Utilisation 2.Task Scheduling 3.Error Handling