{
    "componentChunkName": "component---src-templates-case-study-detail-ts",
    "path": "/case-studies/35",
    "result": {"pageContext":{"next":{"id":36,"attributes":{"feature":true,"title":"数据为王，携程国际火车票的 ShardingSphere 之路","content":"## 业务背景\n\n随着国际火车票业务的高速发展，订单量快速增长，单数据库瓶颈层面的问题逐渐显露，常规的数据库优化已无法达到期望的效果。同时，原先的底层数据库设计，也存在一些历史遗留问题，比如存在部分无用字段、表通过自增主键关联和各个应用直连数据库等问题。\n\n为此，经过讨论后，我们决定对订单库进行分库分表，同时对订单表进行重构，进而从根本上解决这些问题。\n\n## 业务挑战\n\n目标确定后，实践起来可不轻松，出现了很多的问题和挑战。这里列举一些典型问题，大致可以分为两大类：分库分表通用问题、具体业务关联问题。\n\n**分库分表通用问题**\n\n- 如何切分，垂直分还是水平分？分片的键，如何选取？\n- 如何根据键值路由到对应库、对应表？\n- 采用什么中间件，代理方式还是中间件的方式？\n- 跨库操作等问题，如跨库事务和跨库关联？\n- 数据扩容问题，后续如何进行扩容？\n\n**具体业务关联问题**\n\n- 各个应用直连数据如何解决？\n- 如何进行平滑过渡？\n- 历史数据如何恰当迁移？\n\n## 方案选型\n\n### **如何切分**\n\n切分方式，一般分为垂直分库、垂直分表、水平分库和水平分表四种，如何选择，一般是根据自己的业务需求决定。\n\n我们的目标是要从根本上解决数据量大、单机性能问题等问题，垂直方式并不能满足需求，所以我们选取了水平分库+水平分表的切分方式。\n\n### **分片键选取**\n\n一般是根据自己的实际业务，来选择字段来作为分片的键，同时可以结合考虑数据的热点问题 、分布问题。比如订单系统，不能根据国家字段进行分片，否则可能会出现某些国家很多的订单记录，某些国家几乎没有订单记录，进而数据分布不均。相对正确的方式，比如订单类系统，可以选择订单 ID；会员系统，可以选择会员 ID。\n\n### **如何路由**\n\n选定了分片的键之后，接下来需要探讨的问题，就是如何路由到具体的数据库和具体的表。\n\n- 映射路由\n- 分组路由\n- 哈希路由\n- 分组哈希路由\n\n在实践中，我们结合了前面的几种方式，借鉴了他们的优点不足，而采用了此种方式。因为分组方式，能很方便的进行扩容，解决了数据扩容问题；哈希方式，能解决分布相对均匀，无单点数据库热点问题。\n\n## 选择 Apache ShardingSphere 的理由\n\n在产品选型过程中，考虑了多款产品，也包括公司自己实现的产品，最终选择了 ShardingSphere，理由如下。\n\n### **技术环境**\n\n- 我们团队是 Java 体系下的，对 Java 中间件有一些偏爱\n- 更偏向于轻量级组件，可以深入研究的组件\n- 可能会需要一些个性定制化\n\n### **专业程度**\n\n- 取决于中间件由哪个团队进行维护，是否是名师打造，是否是行业标杆\n- 更新迭代频率，最好是更新相对频繁，维护较积极的\n- 流行度问题，偏向于流行度广、社区活跃的中间件\n- 性能问题，性能能满足我们的要求\n\n### **使用成本**\n\n- 学习成本、入门成本和定制改造成本\n- 弱浸入性，对业务能较少浸入\n- 现有技术栈下的迁移成本，我们当前技术栈是 SSM 体系下\n\n### **运维成本**\n\n- 高可用、高稳定性\n- 减少硬件资源，不希望再单独引入一个代理中间件，还要考虑运维成本\n- 丰富的埋点、完善的监控","date":"2021-08-24","author":"瑞华，携程高级后端开发工程师","excerpt":"国际火车票业务分库分表重构实践","metaDescription":"我们是携程火车票研发团队，负责火车票业务的开发以及创新。火车票研发在多种交通线路联程联运算法、多种交通工具一站式预定、高并发方向不断地深入探索和创新，持续优化用户体验，提高效率，致力于为全球人民买全球火车票。","createdAt":"2023-03-28T07:18:10.674Z","updatedAt":"2023-03-28T07:18:12.210Z","publishedAt":"2023-03-28T07:18:12.206Z","locale":"zh","caseStudiesType":{"data":null},"cover":{"data":null},"metaLogo":{"data":null},"localizations":{"data":[]}}},"prev":{"id":2,"attributes":{"feature":false,"title":"葫芦 App 后台数据处理的逻辑捷径","content":"> ShardingSphere 大大简化了分库分表的开发和维护工作，对于业务的快速上线起到了非常大的支撑作用，保守估计 ShardingSphere 至少为我们节省了 4 个月的研发成本。\n> \n> –– 史墨轩，易华录·技术总监\n\n### 项目背景\n\n伴随着易华录旗下面向个人用户的云服务产品【葫芦 App】正式上线，后台架构所承受的业务压力也与日俱增。\n\n为此，葫芦 App 研发团队选择采用 ShardingSphere 分库分表的功能对数据进行了横向的拆分，围绕 ShardingSphere 灵活敏捷的特性，满足了葫芦 App 业务对数据层扩展性的要求，避免团队重复“造轮子”，最大程度简化了随着业务增长而带来的愈发复杂化的分库分表的开发与维护工作。\n\n### 业务挑战\n\n- 扩展能力\n- 效率提升\n- 上线提前\n- 成熟技术\n\n### 解决方案\n\n- 可插拔架构的『高扩展性』\n\n由于业务特性，葫芦 App 原本有限的存储空间被消耗得非常快，并逐渐开始影响用户在前端的响应效率。通过采用 ShardingSphere 的分片策略，葫芦研发团队在应对海量计算+存储所带来的业务问题同时，能够确保分片扩展策略的灵活性。基于此，葫芦团队可以在ShardingSphere 上快速做出相应的功能扩展，为后续架构调整提供优化方案，进一步强化突出了 ShardingSphere 分库分表的优势。\n\n- 距离业务更紧密的『贴合性』\n\n后台架构的变化越小，对于业务而言就越可控。ShardingSphere 这种位于数据库之上的生态，距离业务更近，部署起来也更加轻量，无疑是解决葫芦 App 前台业务变更与后台架构调整之间矛盾的最优解。此外在灵活性层面，ShardingSphere 通过相关配置即可实现的方式，极大简化了葫芦研发团队在分库分表层面的研发与维护工作，对于业务的快速上线起到了非常大的支撑作用。\n\n- 对于业务架构的『零侵入性』\n\n葫芦 App 选择了 ShardingSphere-Proxy 部署模式，在不更换底层数据库的前提下通过 Proxy 来管理真实的数据库集群，基本无需对业务进行改造就已经完成了业务与数据在架构层面的分离，并避免了因更换数据库导致的业务不可用、漫长稳定周期等风险。另外，ShardingSphere 的无状态模式，几乎不会对前端用户产生任何可感知的影响，业务层也无需关注数据的存储方式。\n\n![ehualu_zh_1.png](https://official-media.sphere-ex.com/ehualu_zh_1_3fad8e2ccd.png)\n\n### 用户收益\n\n- 节省 4 人月成本\n- 明显改善业务体验\n- 系统灵活扩容\n\n**SphereEx，让数据与应用连接更简单。**","date":"2021-11-29","author":"SphereEx","excerpt":"ShardingSphere 大大简化了分库分表的开发和维护工作，对于业务的快速上线起到了非常大的支撑作用，保守估计 ShardingSphere 至少为我们节省了 4 个月的研发成本。","metaDescription":"北京易华录信息技术股份有限公司，成立于 2001 年，是国务院国资委直接监管的中央企业中国华录集团旗下控股的上市公司，致力于建设城市数字经济基础设施。","createdAt":"2021-12-11T23:11:13.132Z","updatedAt":"2022-11-30T08:25:32.894Z","publishedAt":"2021-12-11T23:11:16.643Z","locale":"zh","caseStudiesType":{"data":{"id":3,"attributes":{"name":"金融","createdAt":"2021-12-11T23:02:27.233Z","updatedAt":"2021-12-11T23:02:50.766Z","publishedAt":"2021-12-11T23:02:50.699Z","locale":"zh"}}},"cover":{"data":{"id":7,"attributes":{"name":"ehualu.png","alternativeText":"ehualu.png","caption":"ehualu.png","width":1030,"height":720,"formats":{"thumbnail":{"name":"thumbnail_ehualu.png","hash":"thumbnail_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":223,"height":156,"size":6.16,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_ehualu_6d29db12b5.png"},"large":{"name":"large_ehualu.png","hash":"large_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":1000,"height":699,"size":53.63,"path":null,"url":"https://official-media.sphere-ex.com/large_ehualu_6d29db12b5.png"},"medium":{"name":"medium_ehualu.png","hash":"medium_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":750,"height":524,"size":33.7,"path":null,"url":"https://official-media.sphere-ex.com/medium_ehualu_6d29db12b5.png"},"small":{"name":"small_ehualu.png","hash":"small_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":500,"height":350,"size":18.22,"path":null,"url":"https://official-media.sphere-ex.com/small_ehualu_6d29db12b5.png"}},"hash":"ehualu_6d29db12b5","ext":".png","mime":"image/png","size":24.54,"url":"https://official-media.sphere-ex.com/ehualu_6d29db12b5.png","previewUrl":null,"provider":"dine-strapi-provider-upload-aws-s3","provider_metadata":null,"createdAt":"2021-12-12T12:04:05.768Z","updatedAt":"2021-12-14T07:19:57.408Z"}}},"metaLogo":{"data":{"id":3,"attributes":{"name":"ehualu_logo.png","alternativeText":"ehualu_logo.png","caption":"ehualu_logo.png","width":303,"height":153,"formats":{"thumbnail":{"name":"thumbnail_ehualu_logo.png","hash":"thumbnail_ehualu_logo_26122a7add","ext":".png","mime":"image/png","width":245,"height":124,"size":22.6,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_ehualu_logo_26122a7add.png"}},"hash":"ehualu_logo_26122a7add","ext":".png","mime":"image/png","size":22.6,"url":"https://official-media.sphere-ex.com/ehualu_logo_26122a7add.png","previewUrl":null,"provider":"dine-strapi-provider-upload-aws-s3","provider_metadata":null,"createdAt":"2021-12-12T12:04:05.036Z","updatedAt":"2021-12-14T07:19:30.734Z"}}},"localizations":{"data":[{"id":7,"attributes":{"feature":false,"title":"eHualu Story’s Data Processing Shortcut","content":"> ShardingSphere has greatly simplified sharding development and maintenance, which has played a very important role in our rapid product release. We estimate that ShardingSphere has saved us at least 4 months of R&D costs.\n> \n> –– Shi Moxuan, Technical Director，E-Hualu\n\nAt the beginning of this year, E-hualu officially launched its cloud service product “Hulu App” that targets non-business users. Since then, its architecture’s data stress has increased on a daily basis.\n\nThe R&D team of Hulu App chose to use ShardingSphere’s sharding features and scale their data out. The flexible and agile features of ShardingSphere met the requirements of Hulu App. It requires data extensibility, and needs to prevent the team from “reinventing the wheel”.\n\nShardingSphere can simplify sharding development and maintenance to the greatest extent. As the service is growing, the work will be increasingly complicated.\n\n### Hulu App’s growth pressure: feature extension and new services\n\nThe high requirement for data storage and computing is in the genes of the Hulu App technical team.\n\nBecause Hulu App is rapidly growing, its service and feature changes are relatively frequent. Therefore its technical team needs to adapt quickly when a front-end service changes. As its users and service data volume are growing rapidly, its databases are faced with greater pressure.\n\nThe official Hulu App was released on May 17, 2020, and then its user data and service volume also showed a rapid growth trend. Therefore it was inevitable to scale out its databases for many times. At the same time, with business requirements changing very quickly, new challenges have emerged:\n\n- How to ensure feature extensibility\n\nBecause of increasing users and developing product versions, its user data has exploded. The storage and computing capability of its old architecture has been severely challenged. Therefore, Hulu App requires that its back-end data processing platform shall have both extensible features, and flexibility.\n\n- How to improve efficiency\n\nFaced with its changeable services, the R&D team of Hulu App needs to make rapid adjustments when necessary, so as to make its architecture adapt to its services. A highly-flexible, extensible data architecture can greatly improve development efficiency of the team. On the other hand, because of such large amounts of data, database retrieval efficiency will inevitably have problems such as delays, slow read and write. These problems will ultimately affect user experience.\n\n- How to facilitate feature release\n\nAll new products will face problems like frequent new feature release, bringing the scheduled release date forward, when they are just released. These problems pose a great challenge to a development team. At first, Hulu’s development team planned to base on their service demands and create their own sharding plan all by themselves. However, they don’t have enough time to do that, so it’s urgent for them to choose a new sharding plan.\n\n- How to guarantee system stability\n\nA newly-added technology will greatly challenge system stability. It’s especially true for bottom-level technologies, most of which can influence platform services and probably challenge system stability. So what Hulu App needs is a data application product that does not greatly stress databases, but quickly adapts to them and that is also highly stable.\n\n### Leverage ShardingSphere’s features to Create a Flexible, Highly-Available Data Architecture Solution\n\nHulu App’s team studied their needs and spent two weeks evaluating ShardingSphere and other similar solutions. They had evaluative indicators like product feature, maturity, stability and performance. At last, they chose ShardingSphere because of its powerful features, its support and its high maturity can fully meet their needs.\n\nThe Hulu team deployed ShardingSphere above Alibaba Cloud’s RDS. Instead of adjusting ontology databases, the Hulu team prefers to conduct data governance above its databases. ShardingSphere can effectively improve Hulu’s data architecture, because it can achieve three things: high scalability brought by its extensible architecture, closely adaptive to Hulu’s service. ShardingSphere brings Hulu with obvious improvement:\n\n#### “High scalability” of the pluggable architecture\n\nDue to its service characteristics, Hulu App’s limited storage space is consumed very quickly, and the problem gradually begins to affect response efficiency at the front end. Therefore, the team can quickly develop enhanced features by using ShardingSphere, which can provide optimized solutions for its subsequent architectural adjustments. The example strengthens the ShardingSphere’s advantage in sharding.\n\n#### Closely adaptive to service\n\nThe smaller the architecture change is, the more controllable it is for developers. ShardingSphere, an ecosystem located above databases, is closer to service and it deployment is even more lightweight. So it is undoubtedly the best solution to solve the contradiction between the front-end service changes and back-end architecture adjustment of Hulu App. In addition, in terms of flexibility, ShardingSphere supports instant achievement of relevant configurations, which greatly helps the team save a lot of sharding-related work. So it plays a very important role in Hulu App’s quick releases.\n\n#### “Zero attack” to the service architecture\n\nHulu App chose ShardingSphere’s Proxy deployment mode. It can manage the real database cluster through Proxy without. replacing underlying databases. It basically completed the separation of service and data at the architectural level without any changes. So it can avoid the risks caused by database changes, such as unavailable service and long stable cycle. In addition, the non-status mode of ShardingSphere hardly has any perceptible impact on users at the front end and the service layer does not need to pay attention to the data storage method .\n\nSo, Hulu App is a product with tight go-live schedule and fast feature iteration. ShardingSphere-Proxy can reuse original databases and help the Hulu development team to develop incremental features above databases such as sharding and data encryption. In addition, ShardingSphere allows developers not to worry about database configuration and shields user perception, so it quickly builds service-oriented database direct connection, which is well separated from the system architecture level, and ensures that daily maintenance work at the database agent, such as bug repair and version update, will not affect the services.\n\nThis time, ShardingSphere collaborated with the R&D team of E-hualu’s Hulu App, and helped them overcome data volume explosion for several times. Actually, this is just a tiny example of ShardingSphere’s various application scenarios around the world.\n\n### About E-hualu & Hulu App\n\nFounded in 2001, Beijing E-hualu Information Technology Co., Ltd. is a listed company under the state-owned enterprise China Hualu Group Co., LTD. directly administrated by the State-owned Assets Supervision and Administration Commission of the State Council (SASAC). Its mission is to reduce energy consumption and the cost of long-term data management.\n\nHulu App is the first to-customer product of E-hualu.\n\n### ShardingSphere Community:\n\nShardingSphere Github: https://github.com/apache/shardingsphere\n\nShardingSphere Twitter: https://twitter.com/ShardingSphere\n\nShardingSphere Slack Channel: apacheshardingsphere.slack.com","date":"2021-11-29","author":"SphereEx","excerpt":"ShardingSphere has greatly simplified sharding development and maintenance, which has played a very important role in our rapid product release. We estimate that ShardingSphere has saved us at least 4 months of R&D costs.","metaDescription":"Founded in 2001, Beijing E-hualu Information Technology Co., Ltd. is a listed company under the state-owned enterprise China Hualu Group Co., LTD. directly administrated by the State-owned Assets Supervision and Administration Commission of the State Council (SASAC). Its mission is to reduce energy consumption and the cost of long-term data management.","createdAt":"2021-12-12T12:33:14.398Z","updatedAt":"2022-09-26T10:27:50.013Z","publishedAt":"2021-12-12T12:35:01.820Z","locale":"en"}}]}}},"article":{"id":35,"attributes":{"feature":true,"title":"openGauss X ShardingSphere，分布式方案的另一种最佳实践","content":"Apache ShardingSphere 持续助力于 openGauss 分布式数据库能力的构建。openGauss 数据库自 2020 年 6 月开源以来，受到了业界的广泛关注，现已吸引众多伙伴、开发者参与其中，共建繁荣的数据库生态。面对如今海量数据，超高并发等诸多场景，openGauss 将目光转向于分布式解决方案，专注于解决海量数据存储、超高并发吞吐、大表瓶颈等众多难题，与 ShardingSphere 一起构建全栈开源分布式解决方案，实现 openGauss 的又一大突破。\n\n## 分布式解决方案\n![640.png](https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/640_f6ba503847.png)\n<center>图1 分布式解决方案整体框图</center>\n\nopenGauss 融合了众多开源组件，用以构建集数据水平扩展、分布式事务及治理一体化的全栈开源分布式解决方案，整体框图如图 1 所示。\n\n- 其中 ShardingSphere-Proxy 为开源分布式解决方案，具有分库、分表、分布式事务、弹性伸缩、读写分离等众多能力；\n- HAProxy 结合 Patroni 的 REST API，可以始终识别数据库的主节点，保证高可用场景，同时可实现负载均衡；\n- 每个 Patroni 高可用节点支持一主多备，每个节点使用 Paxos 协议保证数据的一致性，各个节点可以部署在相同或不同的区域，用以保证多地多中心的数据安全。\n本分布式方案运用 ShardingSphere-Proxy 强劲的分布式能力，通过 kubernetes 管理集群，prometheus 监控集群状态，从而构建全栈开源的分布式解决方案。\n\n## 产品优势\n\n1. **极致扩展能力，灵活扩缩容**\n计算与存储能力可通过水平拆分实现线性扩展，最高可达数据 6400 分片，性能随扩展准线性增长，可有效解决单表数据量膨胀问题；结合业务流量，灵活平滑进行数据节点的扩缩容，智能读写分离，实现分布式数据库的自动负载均衡。\n\n2. **丰富企业级特性**\n支持分布式存储过程、触发器，分布式事务，全密态数据加密，WDR 诊断报告，提供丰富的企业级特性。\n\n3. **一键部署，屏蔽底层依赖**\n标准化镜像确保多环境一致性交付，容器化部署，实现物理资源池化，降低对平台的依赖性，简洁高效，实现应用秒级部署。\n\n4. **超高可用，实现异地容灾**\n强有力的集群管理、运维能力，支持同城、异地、多地多中心灵活部署，基于 Paxos 协议保证数据的安全及强一致性，提供 RPO=0 的多种容灾能力。\n\n5. **开源开放，构建全栈生态**\n开源 openGauss 单机及分布式解决方案，鼓励更多伙伴、开发者共同参与其中，共建数据库的繁荣生态，打造全栈开源生态链。\n\n## 应用场景\n\n- **金融系统**\n基于多数派共识协议 Paxos，提供分布式强一致事务能力，确保分布式环境下数据的强一致能力；多地多中心，提供 RPO=0 的能力，保障金融级可靠。\n\n- **政企办公**\n提供全密态数据加密，安全可信，支持软硬件生态，保障高可用场景，为政企办公保驾护航。\n\n- **智慧电网**\n支持同城、异地、多地多中心灵活部署，根据业务的潮汐特点灵活扩缩容，助力智慧电网。","date":"2021-09-15","author":"SphereEx","excerpt":"Apache ShardingSphere 持续助力于 openGauss 分布式数据库能力的构建","metaDescription":null,"createdAt":"2023-03-28T07:02:39.490Z","updatedAt":"2023-03-28T07:10:25.644Z","publishedAt":"2023-03-28T07:10:25.641Z","locale":"zh","caseStudiesType":{"data":null},"cover":{"data":null},"metaLogo":{"data":null},"localizations":{"data":[]}}}}},
    "staticQueryHashes": []}