← Back to list
Big Data Engineer
Dongxu Technology
Responsibilities
- Design and implement video data collection pipelines to support automated ingestion from multiple sources, format standardization, deduplication, and the construction of data cleaning and preprocessing workflows. Establish quality filtering rules tailored to AI-generated video characteristics to eliminate invalid samples.
- Build an annotation task management system integrated with outsourced annotation platforms (e.g., DataTang, JD Crowdsourcing) to automate task distribution, progress tracking, and quality spot-checking.
- Establish data warehousing and version control mechanisms to ensure every dataset batch is traceable, rollable-back, and comparable, supporting algorithm teams' needs for experiment reproducibility.
- Design a data quality validation framework to perform consistency checks, anomaly detection, and statistical reporting on annotation results.
- Maintain and iterate the engineering infrastructure for Golden Set datasets, collaborating with AI Video Cognition Engineers to complete data updates and version releases.
- Monitor pipeline operational status, troubleshoot data anomalies and pipeline bottlenecks, and ensure the stability and timeliness of data supply.
Requirements
- Role Overview: Build the data infrastructure for the TianCe WanYing AI video production system from the ground up. Responsible for establishing the end-to-end engineering system for video generation data, covering data collection, cleaning, annotation pipelines, warehousing, and version management, providing high-quality, traceable data for model training and evaluation.
- Experience in data engineering or backend development; proficient in Python for pipeline development with a track record of production-grade ETL projects.
- Familiarity with data storage technology stacks, including relational databases (PostgreSQL / MySQL), object storage (OSS / S3), and basic concepts of vector databases.
- Practical experience with data version management tools such as DVC or LakeFS, or equivalent engineering capabilities.
- Proficiency in task queue and asynchronous processing development (e.g., Celery, RQ, or similar tools) capable of handling concurrent scheduling for large-scale data.
- Understanding of engineering requirements for data annotation workflows; experience in integrating with annotation platforms or designing annotation pipelines is a plus.
- Basic knowledge of video file processing (FFmpeg, frame extraction, metadata parsing); ability to write video data preprocessing scripts is a plus.
- Experience in MLOps or AI engineering projects; understanding of the role of data in the model training and evaluation loop is a plus.
- Not Required: Algorithm research or model training capabilities; frontend development skills.
About the company
Dongxu Technology is an AI-driven commercial video generation and omnichannel marketing operating system company. Its flagship product, Chorify AI (TianCe WanYing), is positioned as an "Enterprise-level AI Digital CGO (Chief Growth Officer), " replacing traditional marketing planning and video delivery teams through conversational interfaces.
- Industry
- AI Applications & LLMs
- Founded
- 2024-01-29