Configuration-Driven Distributed Data Center

A containerised, geo-distributed back end for a news-media platform, built for the Distributed Database Systems course at Tsinghua University. It simulates a real data center: a sharded MongoDB cluster, HDFS for image and video storage, Redis caching and a React admin dashboard, all orchestrated with Docker.

The core idea

The central contribution is a configuration-driven architecture. Sharding, geographic zones (Beijing / Hong Kong), fragmentation and replication rules, and data migration are all expressed as configuration. The cluster can grow from 2 shards to 3, rebalance, and migrate roughly 1,000,000 records across regions without changing a line of application code.

The React dashboard for monitoring shards, regions and data distribution.

Reading an article served from the distributed store, with media from HDFS.

How it fits together

• MongoDB (sharded): primary record store, partitioned and replicated per the configured rules.
• HDFS: bulk storage for images and videos referenced by articles.
• Redis: caching layer in front of hot reads.
• Flask API + React dashboard: application and operations front end.
• Docker: the whole topology is reproducible from containers.

My role

Infrastructure, dynamic sharding and data loading (with Louis Rollet). I focused on the configuration-driven expansion and migration path, adding a shard and rebalancing live data, and on the bulk data-loading pipeline.

The full write-up and deployment manual are embedded as reports below, along with the project presentation.

Tech

Python · Flask · React · MongoDB · HDFS (Hadoop) · Redis · Docker