Before training a large model, the corpus needs to undergo parsing, cleaning, deduplication, quality scoring, tokenization, and sample assembly. With a scale of PB level, the engineering team faces not only computing power bills every day, but also hundreds of tables, constantly increasing features, and a small number of abnormal data that can cause the entire batch of tasks to be re-run.
This year's VLDB industrial track best paper focuses on a crucial yet often overlooked aspect of large model training: data preparation. The paper "OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration" introduces OmniTable, a unified wide-table system developed by Ant Group.
(Caption: Ant Group's paper "OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration" was named best paper in the industrial track at VLDB 2026, with the award presented at the conference in Boston on Sept. 1.)
The paper link is: https://www.vldb.org/pvldb/vol19/p4276-fu.pdf
According to the paper, OmniTable has managed over 35 petabytes and more than 30.5 billion pieces of large model training data in a production environment, covering data domains such as the web, code, PDF, and SFT. In a real-world SFT data preparation task, the end-to-end cycle was shortened from approximately 14 days to 2.5 days, and manual operation steps were reduced from 45 to 12.
This result is not due to a faster machine. OmniTable has changed the way data engineers organize data and features: the same data domain is presented as a logical wide table at the upper layer, while at the bottom it is still split according to data scale, access method, and computing engine; features have also evolved from temporary calculations in scripts to system assets with definitions, versions, dependencies, and lineage.
A feature led to the creation of 106 tables
Traditional large model data processing typically revolves around physical tables. After a data source is accessed, the parsing results are stored in one table, the cleaning results in another, and quality scores, domain labels, deduplication signatures, and security tags continue to produce new tables or intermediate results. Web, code, PDF, and SFT each maintain their own set of processes.
A single pipeline is not difficult to understand, but as the data source and features continue to increase, the maintenance object expands rapidly. When adding a new quality feature, engineers need to first find all relevant tables, verify fields and versions, and then configure tasks, resources, checkpoints, and failure handling for each dataset. A paper documented a real case where an engineer had to process 106 tables on the task canvas just to supplement one feature.
What's even more troublesome is that the tables only store the results, with rarely a complete record of how the results were calculated. UDFs are scattered across different codebases, and there is a lack of stable associations between input columns, operator versions, running batches, and downstream training tasks. When troubleshooting an abnormal sample, engineers often have to trace across tables and scripts; once the feature version changes, they also need to determine which historical batches need to be recalculated.
OmniTable summarizes these issues into three engineering costs: difficulty in locating data, difficulty in backtracking features, and difficulty in tracing results. The system's design starting point is also straightforward - making data batches and feature columns first-class objects, and relegating physical tables to the storage implementation layer.
Figure 1: Heterogeneous data forms fragmented datasets after passing through decentralized pipelines. Source: Figure 1 of the paper.
"A table" is located at the logical layer
OmniTable's core principle is "logical unity, physical separation".
At the logical layer, each row represents a traceable data entity, and each column stores the state of a certain processing stage or a derived feature. RawData, ProcessedData, and TrainableData correspond to the original data, intermediate state during processing, and trainable form, respectively, and can be further supplemented with feature columns such as quality, domain, security, and deduplication.
Two categories of system fields are responsible for aligning these columns. The _ai_unique_id_ serves as a global primary key, using the same identifier for the same data across different sources, processing stages, and feature columns; the _ai_append_name_ records the access batch, source, and version. Data backfilling, spot checking, and lineage tracing now have a stable anchor point.
Here, "one table" refers to a logical contract. The production environment is divided into four domain-wide logical wide tables, namely Web, code, PDF, and post-SFT, which collectively manage over 35 PB and more than 30.5 billion records. Each logical table's columns are supported by multiple physical tables in its Table Family and can be further split by row or column. The largest Web wide table manages approximately 25 PB and over 30 billion records, containing more than 800 logical columns and over 200 registered features. The paper also extends the logical columns to 2,500 in a controlled experiment with a fixed amount of approximately 2 PB of data.
The correspondence between logical columns and physical locations is maintained by the Catalog. At the underlying level, rows can be split, columns can be split, small files can be merged, partitions can be adjusted, or materialized views can be established for high-frequency column groups, while the upper-layer schema and column semantics remain unchanged. Downstream queries do not need to be modified with each physical adjustment. The paper also gives the cost of this design: establishing materialized results for hot column groups may increase storage overhead by about 8%–15%.
Figure 2: Catalog connects data access, feature execution, query export, and background governance. Source: Figure 2 from the paper.
Feature calculation has been changed from "drawing tasks" to "reporting target columns"
A unified logical view solves the problem of "where the data is", while Catalog continues to manage how features are generated.
When a feature is registered, it is necessary to specify the input columns, output columns, UDF/SQL/model inference logic, version, as well as the preferred execution mode, whether CPU or GPU. When an engineer submits a re-computation task, they only need to specify the target batch and target feature. OmniTable will query the current computation status, find the smallest dependency closure that has not been completed along the column-level dependency DAG, and then generate a physical execution plan in topological order.
For instance, a quality score depends on washed text, which in turn depends on parsing results. The old process required engineers to confirm the completeness of three tasks and locate input and output tables separately. OmniTable directly checks the status of these columns in the target batch: reused completed results, and missing ancestral columns are added to the plan. Multiple operators with shared inputs and the same execution engine can also be merged into a single scan.
After a task is successfully completed, Catalog atomically registers the status, version, physical location, and column-level lineage of the "batch-feature column"; results that have not been committed will not enter the stable logical view. Subsequently, when querying a column of data, the system can answer which input batch it used, which parent columns it depends on, which feature version it adopted, which engine calculated it, and where the results are stored.
Information that was previously scattered across scripts, scheduling platforms, and manual records is now integrated into the same metadata interface. Engineers are still responsible for defining feature semantics, while the system takes over dependency expansion, execution routing, state management, and result submission.
Figure 3: The system parses dependencies, generates plans, and schedules feature calculations. Source: Figure 3 of the paper.
A small number of bad samples will no longer require re-running the entire batch of data.
Unstructured data inevitably contains abnormal encoding, excessively long text, or damaged content. When the data reaches hundreds of millions or tens of billions of entries, even an extremely low proportion of abnormalities can result in a large number of bad samples. Traditional batch tasks often take the task as the unit of failure, where a single UDF OOM or timeout can cause the entire multi-terabyte calculation to exit.
OmniTable isolates common UDF faults to the record level. Each UDF call comes with a timeout and memory check; when encountering Python OOM, timeouts, or uncaught exceptions, the system records the sample ID, exception type, and error summary, writing that result as NULL, while the remaining records continue to be processed. Error records are unified into an error table, facilitating subsequent repairs and recalculations.
The paper conducted a comparison on a feature task with approximately 600 million records, totaling 500 GB. The data contained 31,247 abnormal records, accounting for 0.005%. After enabling failover, except for the 31,247 abnormal records, the remaining approximately 99.995% of records were processed in one pass, taking around 6.2 hours without requiring manual intervention. With the capability disabled, the task failed directly. The old process required three rounds of troubleshooting, deletion, and re-extraction, with a total duration of approximately 52 hours, of which around 18 hours involved manual handling.
Record-level packaging will increase execution overhead by about 3%–5% and cannot eliminate all failures, such as machine faults and network interruptions. It addresses the most common and time-consuming issues for engineers in production: single abnormal data points causing entire batches of tasks to fail.
Scan multiple columns at once, with CPU and GPU each doing their suitable tasks
The computational forms of large model data features vary greatly. Text length, character ratio, and rule filtering are typically suitable for CPU or SQL, while model inference can run on either CPU or GPU, depending on the model scale, operator image, and resource conditions. OmniTable selects the execution backend among Spark, MaxCompute SQL, and GPU inference platforms based on user declarations, operator images, engine capabilities, and cluster loads, and adjusts resource parameters in combination with historical running information.
Apart from routing, repeated scanning is also a significant overhead. When multiple features read from the same column and run on the same engine, OmniTable merges them into a single task, reading once to produce multiple column results.
In the operator fusion experiment in the paper, 8 CPU/Spark features all read parsed_text. The test data is approximately 2.5 PB, with over 300 billion entries. After fusion, the number of scans decreased from 8 to 1, CPU Hours decreased from 42,000 to 18,500, a reduction of 55.9%; end-to-end time decreased from 38 hours to 14 hours, accelerating 2.7 times.
Adaptive tuning is used to reduce parameter trial and error. In controlled experiments with batch sizes of 50 GB, 500 GB, and 2 TB, OmniTable's adaptive configuration achieved a first-submission success rate of 100%, with task costs not exceeding 5% of those achieved by manual expert tuning. These results correspond to specific BERT feature tasks, indicating that the system can provide usable parameters close to expert configurations, but do not represent that any task can automatically reach optimal results.
Figure 4: Operator fusion reduces repeated scans, and adaptive tuning approaches expert configuration under different batch sizes. Source: Paper performance evaluation.
Logical tables continue to widen, with the backend continuously optimizing physical layouts.
After the wide table is launched, batches and features will continue to be added. The accumulation of small files, partition skew, growth in the number of columns, and changes in query hotspots will all slow down access. OmniTable's background governance service continuously monitors these metrics and automatically performs small file merging, row splitting, column splitting, and materialized view construction.
The governance process uses Prepare-Execute-Commit: first, prepare the new layout and complete the physical rewrite, and after verification, atomically switch the Catalog mapping. The old layout continues to serve queries before switching, and failed governance tasks can be rolled back. Users continue to query the same set of logical columns, without needing to be aware of how files and table families change.
Figure 5: Background governance adjusts physical layout according to scale and access mode. Source: Paper Figure 4.
Column splitting allows the logical schema to exceed the physical column limit of a single engine. In a test with a fixed dataset of approximately 2 PB, where the queried column set remained unchanged, the number of logical columns increased from 200 to 2500, and the P95 latency rose from approximately 25 seconds to 38 seconds, crossing the underlying engine's physical limit of around 1200 columns. Another set of scalability tests covering 1 TB to 25 PB showed that the filtering and export throughput remained steady at 18-23 TB per hour.
Single-sample filtering takes a different path. The global ID index directly locates _ai_unique_id_ to the physical table, partition, and row group. On over 25 PB, 30 billion records, and more than 800 logical columns of web data, the P50 for querying a complete logical row is 8.3 seconds, and P99 is 14.7 seconds; full scans require over 184 seconds and 612 seconds, respectively.
When exporting multiple columns at once, the background materialized view pre-eliminates JOIN operations between hot column groups. In a representative scenario involving 15 columns and 4 physical tables, the filtered export throughput increased from 4.8 TB/hour to 20.1 TB/hour. Point queries, batch filtering, and large-scale exports use different paths, and Catalog provides a unified entry point for them.
5.6 times faster, mainly saving coordination and rework time
The paper conducted an end-to-end comparison using a real SFT data preparation task. The task included 8 data sources and 12 features, of which 9 were CPU UDFs and 3 were GPU inference.
The old process required about 2 days to locate and access the data, about 9.5 days to complete feature backfilling, and an additional 2.5 days to write multi-table JOINs and export the results, with a total cycle of about 14 days. The process involved about 45 manual steps, 24 independent pipelines or scripts, and 35 physical tables.
OmniTable shortened the access phase to approximately 0.5 days, feature backfill to approximately 1.7 days, and filtered export to approximately 0.3 days, totaling approximately 2.5 days. Manual steps were reduced to 12, independent commands to 10, and the entry point is a single SFT domain logic wide table. End-to-end speed increased 5.6 times, manual operation steps decreased 73.3%, and independent pipelines and scripts decreased 58.3%.
Figure 6: SFT data preparation tasks were shortened from approximately 14 days to approximately 2.5 days. Source: Paper on End-to-End Experiments.
This set of data corresponds to the same production task in the paper evaluation. It clearly shows where time is saved: fewer table compilations, shared inputs are no longer repeatedly scanned, a small number of exceptions do not frequently trigger a re-run of the entire batch, and physical layouts do not need to be manually adjusted until performance degradation occurs.
Wide tables are just the entry point, the complete lifecycle is the key
OmniTable consolidates four types of information that were previously scattered across multiple tools, including data batches, feature definitions, execution status, and column-level lineage. The logical wide table provides users with a stable entry point, while the Catalog maintains the relationship between data and features, and the execution and governance services continue to choose the appropriate physical organization underneath.
This approach also has clear costs. Materializing hotspots requires additional storage, and record-level fault tolerance incurs a small amount of execution overhead, while background governance occupies cluster resources. Different companies have different data domains, computing engines, and team habits, so OmniTable is more suitable as a system design reference that has been tested in 35+ PB production deployments. The idea it provides is to first stabilize logical semantics and then allow physical layouts to evolve continuously.
As large model training enters the petabyte era, the challenge for data engineering is no longer just about completing a single task, but about keeping the continuously growing data, features, and computing power manageable in the long term. What OmniTable aims to save is not just the machine running time, but also the time spent by engineers repeatedly searching for tables, supplementing tasks, and troubleshooting exceptions.
": " " " " " " " " " " " " " ' " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " not " " " " " " " ". " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " a " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " a " " " " " " ". " " " " " " " "." " " " " " " " " " " " " " " " " " " " " " " " " " a " " " " " " " " " " " " " " " " " " " " " " " " " " a " that " " " " " " " " " " " " " " " " " " " "uality " " " " " " " " " " " " " " " " "
This article is provided by Ant and reproduced by QbitAI with permission, and the views belong to the original author.