{
    "componentChunkName": "component---src-templates-case-study-detail-ts",
    "path": "/case-studies/22",
    "result": {"pageContext":{"next":{"id":23,"attributes":{"feature":false,"title":"Apache ShardingSphere & Keep: Asia’s Leading Workout & Fitness Trainer App Maker","content":"For our community’s first visit, we went to Asia’s leading workout & fitness trainer app maker “Keep” headquarters at Vanke Times Center and shared our technologies with developers at Keep Co.\n\nWay back in 2018, Keep had already deployed ShardingSphere capabilities such as sharding and read/write splitting in multiple application scenarios for its diversified lines of business.\n\nWith the release of Apache ShardingSphere 5.0, the concept of Database Plus and pluggable architecture has to some extent, reshaped the ShardingSphere ecosystem. During our visit, we conducted in-depth exchanges and discussions with our counterparts at Keep.\n\nKeep engineers expressed great interest in Apache ShardingSphere 5.0. At the event, Juan Pan, Apache ShardingSphere PMC and SphereEx CTO, presented a full overview of the initial ShardingSphere architecture, user-end access, community building and [the Database Plus concept.](https://www.infoq.com/articles/next-evolution-of-database-sharding-architecture/)\n\n## Database Plus: Freeing DBAs and Developers\n\nDatabase Plus is a design concept of the distributed database system. By building a standard layer and ecosystem for use and interaction above fragmented heterogeneous databases, and by multiplying and expanding computing capabilities (e.g. data sharding, data encryption and decryption), all the interaction between applications and databases is oriented to the standard layer built by Database Plus. This results in shielding the differentiated impact of database fragmentation on upper layer transactions.\n\nPan believes that the global database industry has boomed thanks in large to the following two reasons:\n\n**Business side demand:**\n\nIn terms of business, to ensure sustained visits and transactions growth, underlying databases must respond to requests as soon as possible. In addition, the breakup of microservices and subsequent modification of corresponding databases also generate demands from the business side.\n\n**Operations & maintenance side demand:**\n\nDBAs are responsible for running the whole business and data system, including data security, backup, distributed governance, and API smart monitoring of data clusters.\nIf the middle layer can meet transaction traffic and understand the request, DBAs can modify the request accordingly and then more operations can be performed. Therefore, we need to strike a balance between the two sides of demand to carry transaction traffic and support database capacity building, thus creating an efficient, collaborative ecosystem between the two sides.\n\nFollowing the version 5.0 release, the sharding function is no longer at the core of Apache ShardingSphere. Actually, sharding has been “downgraded” to a secondary function in the ShardingSphere ecosystem. Following the Database Plus concept, Apache ShardingSphere has built a pluggable architecture ecosystem, enabling the middle layer to achieve more value-added capabilities.\n\n![ShardingSphere’s Database Plus Architecture\n](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c8kc9fdyp0n1kd3eo341.png)\n\nTaking data encryption and decryption as an example, we can see how they are generally done at the business layer because MySQL and other databases do not support encrypted algorithms themselves. This means encryption & decryption is only be achievable at application and business layers. However, this poses a problem if we consider complex online businesses that face considerable tasks when they upgrade their encryption algorithms.\n\nThe best solution to this problem is to encrypt and decrypt data at the middle layer. ShardingSphere-Proxy can be directly bound to databases, be placed at the middle layer between applications and underlying databases. ShardingSphere-proxy will feel “DB-like” by taking advantage of protocols that are compatible with different databases, thus deciding which node the SQL quest will fall on, letting upper layer applications be programmed for ShardingSphere, and shielding the impact of underlying databases. Data is in plaintext status when the transaction sends a request, but it will be encrypted after going through Proxy and decrypted when being retrieved.\n\n![ShardingSphere Encryption & Decryption Capability](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hkesryh371m8tzlqct3c.png)\n\nAs such, the ShardingSphere encryption and decryption process can be separated from the existing system of applications and databases and can be linked with special encryption algorithms, especially in the cases where a cipher machine is involved. ShardingSphere capabilities such as encryption can free significant amounts of DBAs’ and developers’ time, allowing them to focus more on businesses.\n\n\n## Apache ShardingSphere Open Source Project Links:\n\n[ShardingSphere Github](https://github.com/apache/shardingsphere)\n[ShardingSphere Twitter](https://twitter.com/ShardingSphere)\n[ShardingSphere Slack Channel](https://apacheshardingsphere.slack.com/ssb/redirect)\n\n","date":"2022-01-10","author":"Yacine Si Tayeb","excerpt":null,"metaDescription":"Keep is a mobile fitness community that offers a variety of training programs for different groups of people.","createdAt":"2022-09-26T07:48:54.261Z","updatedAt":"2022-09-27T07:00:22.104Z","publishedAt":"2022-09-27T07:00:22.099Z","locale":"en","caseStudiesType":{"data":{"id":9,"attributes":{"name":"Internet","createdAt":"2022-09-26T10:34:03.086Z","updatedAt":"2022-09-27T06:59:34.838Z","publishedAt":"2022-09-27T06:59:34.836Z","locale":"en"}}},"cover":{"data":{"id":344,"attributes":{"name":"keep.png","alternativeText":"keep.png","caption":"keep.png","width":880,"height":347,"formats":{"thumbnail":{"name":"thumbnail_keep.png","hash":"thumbnail_keep_d033ce5311","ext":".png","mime":"image/png","width":245,"height":97,"size":18.27,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_keep_d033ce5311.png"},"medium":{"name":"medium_keep.png","hash":"medium_keep_d033ce5311","ext":".png","mime":"image/png","width":750,"height":296,"size":83.73,"path":null,"url":"https://official-media.sphere-ex.com/medium_keep_d033ce5311.png"},"small":{"name":"small_keep.png","hash":"small_keep_d033ce5311","ext":".png","mime":"image/png","width":500,"height":197,"size":47.39,"path":null,"url":"https://official-media.sphere-ex.com/small_keep_d033ce5311.png"}},"hash":"keep_d033ce5311","ext":".png","mime":"image/png","size":24.11,"url":"https://official-media.sphere-ex.com/keep_d033ce5311.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T07:53:26.070Z","updatedAt":"2022-09-26T07:53:26.070Z"}}},"metaLogo":{"data":{"id":357,"attributes":{"name":"keep logo.png","alternativeText":"keep logo.png","caption":"keep logo.png","width":205,"height":204,"formats":{"thumbnail":{"name":"thumbnail_keep logo.png","hash":"thumbnail_keep_logo_ecc24449b4","ext":".png","mime":"image/png","width":157,"height":156,"size":13.7,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_keep_logo_ecc24449b4.png"}},"hash":"keep_logo_ecc24449b4","ext":".png","mime":"image/png","size":18,"url":"https://official-media.sphere-ex.com/keep_logo_ecc24449b4.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T09:00:55.470Z","updatedAt":"2022-09-26T09:00:55.470Z"}}},"localizations":{"data":[]}}},"prev":{"id":11,"attributes":{"feature":false,"title":"The SphereEx-Console Enterprise Grade Data Governance Platform is Officially Released","content":">On January 27th, SphereEx officially launched SphereEx-Console, a data monitoring and management platform. It provides a one-stop solution incorporating Apache ShardingSphere and SphereEx as the core, a powerful user interface to manage data clusters, and a platform to simplify user configuration.\n>\n>By simplifying operation & deployment processes, reducing entry barriers and data management challenges, SphereEx-Console improves enterprise operations and maintenance efficiency. \n\n\nExpanding data volumes and increasingly demanding agility requirements from front-end businesses have created significant challenges for enterprises in the finance, government and traditional manufacturing sectors. \n\nImproving underlying data architectures' adaptability, and upgrading it to a distributed ecosystem represent some of the new requirements - however, traditional centralized databases are inherently inadequate in terms of massive data carrying capacity. \n\nAs a result, distributed databases have emerged in recent years to meet this demand. Nevertheless, the complexity of distributed architectures poses great challenges to users. Simplifying users' experience has become a key database industry driver.\n\nAs a popular open source project, Apache ShardingSphere has successfully been implemented in multiple production scenarios, with an increasing number of enterprises adopting the Database Plus approach to transition from traditional relational databases to distributed databases. \n\nBased on Apache ShardingSphere a data platform on top of the existing databases is built, enabling distributed data capability to be deployed in the middle layer without requiring adjustments to the underlying data architecture. This approach allows enterprises to smoothly transition to distributed and cloud based platforms.\n\nThe SphereEx Enterprise Data Service Platform was created by leveraging ShardingSphere's open source kernel, and enhancing it with enterprise grade features. It provides improved data service capabilities, such as data sharding, data security etc.\n\n**SphereEx-Console is a powerful dashboard applied to the management and control of the SphereEx Enterprise Data Service Platform. It provides a one-stop solution with a user-friendly experience packaging resources, instances and plugins.**\n\nIt provides a visual approach to manage data clusters with Apache ShardingSphere and SphereEx as the core, while offering a platform to simplify user configuration, and ultimately improve enterprise operations & maintenance efficiency.\n\n## What is SphereEx-Console?\n\nThe SphereEx Enterprise Data Service Platform was created by leveraging ShardnginSphere's open source kernel, and enhancing it with enterprise grade features. It provides improved data service capabilities, such as data sharding, data security etc.\n\nSphereEx-Console is a powerful dashboard applied to the management and control of the SphereEx Enterprise Data Service Platform. It provides a one-stop solution with a user-friendly experience packaging multiple functions such as resource layer, instance layer and application layer.\n\nThe SphereEx-Console is used to manage the operation and maintenance of SphereEx Enterprise Data Service Platform. It supports operation, maintenance and development scenarios in Internet, finance and government industries.\n\nAs a SphereEx business data services cluster platform with ShardingSphere at its core, the SphereEx-Console visual data control platform provides a one-stop solution with a user friendly interface instead of complex code input, packaged with various features for the resource, instance and application layers. \n\n![Diagram Console EN.png](https://official-media.sphere-ex.com/Diagram_Console_EN_5f98720b3f.png)\n\n\n### SphereEx-Console Advantages\n\n**Easy to Use**\n\nIt improves the user experience and eliminates configuration errors. Users are not required to operate the SphereEx Enterprise Data Service Platform through configuration commands, greatly improving user experience. \n\n**Comprehensive**\n\nIt provides a unified solution for the management and control of the SphereEx Enterprise Data Service Platform - from basic resources to plugin capabilities.\n\n**Visual Monitoring**\n\nWith its data monitoring & management dashboard, SphereEx Enterprise Data Service Platform's clusters, instances and hosts monitoring data are visible online and in real time.\n\n### Industry Leading\n\nThe release of SphereEx-Console is an essential milestone for SphereEx's data management ecosystem. It improves and evolves Apache ShardingSphere in areas such as management difficulty and user interface. \n\nThe user interface greatly reduces enterprise operations & maintenance teams' learning costs, while the unified data management platform greatly improves operations and maintenance efficiency.\n\nAs a business founded the open source project Apache ShardingSphere's core team, SphereEx continues to enhance data governance as well as management & control capabilities. We are committed to providing users with a comprehensive Apache ShardingSphere application ecosystem and SphereEx enterprise grade capabilities.\n\n[Click here for more details!](https://sphere-ex.com/en/products/console/)","date":"2022-01-26","author":"SphereEx","excerpt":"Enhancing Enterprise Data Governance Capabilities","metaDescription":null,"createdAt":"2022-03-11T09:38:54.812Z","updatedAt":"2022-09-27T08:42:51.504Z","publishedAt":"2022-03-11T09:59:17.395Z","locale":"en","caseStudiesType":{"data":null},"cover":{"data":{"id":178,"attributes":{"name":"SphereEx-Console.png","alternativeText":"SphereEx-Console.png","caption":"SphereEx-Console.png","width":3751,"height":1596,"formats":{"thumbnail":{"name":"thumbnail_SphereEx-Console.png","hash":"thumbnail_Sphere_Ex_Console_46205ff4d3","ext":".png","mime":"image/png","width":245,"height":104,"size":26.2,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_Sphere_Ex_Console_46205ff4d3.png"},"large":{"name":"large_SphereEx-Console.png","hash":"large_Sphere_Ex_Console_46205ff4d3","ext":".png","mime":"image/png","width":1000,"height":425,"size":183.98,"path":null,"url":"https://official-media.sphere-ex.com/large_Sphere_Ex_Console_46205ff4d3.png"},"medium":{"name":"medium_SphereEx-Console.png","hash":"medium_Sphere_Ex_Console_46205ff4d3","ext":".png","mime":"image/png","width":750,"height":319,"size":129.65,"path":null,"url":"https://official-media.sphere-ex.com/medium_Sphere_Ex_Console_46205ff4d3.png"},"small":{"name":"small_SphereEx-Console.png","hash":"small_Sphere_Ex_Console_46205ff4d3","ext":".png","mime":"image/png","width":500,"height":213,"size":72.9,"path":null,"url":"https://official-media.sphere-ex.com/small_Sphere_Ex_Console_46205ff4d3.png"}},"hash":"Sphere_Ex_Console_46205ff4d3","ext":".png","mime":"image/png","size":345.31,"url":"https://official-media.sphere-ex.com/Sphere_Ex_Console_46205ff4d3.png","previewUrl":null,"provider":"dine-strapi-provider-upload-aws-s3","provider_metadata":null,"createdAt":"2022-03-11T09:32:09.548Z","updatedAt":"2022-03-11T09:32:09.548Z"}}},"metaLogo":{"data":null},"localizations":{"data":[{"id":12,"attributes":{"feature":true,"title":"增强企业数据服务及管控能力｜SphereEx-Console 可视化数据管控平台正式发布","content":">**1 月 27 日，SphereEx 正式推出可视化数据服务管控平台：SphereEx-Console。**\n>\n>作为一款以管理 ShardingSphere 为核心的 SphereEx 商业数据服务集群平台，SphereEx-Console 以平台的形式简化用户配置，极大简化用户操作和部署流程，进而降低新手入门门槛和企业管控难度，进一步提升企业运维效率。\n\n随着数据体量增长以及前端业务敏捷性的要求，金融、政府、传统制造等领域均面临着提升底层数据架构适配性、面向分布式生态进行架构升级。但传统集中式数据库在海量数据承载能力方面存在先天性不足，于是近些年来分布式数据库不断涌现，很好地满足这一诉求。但我们也看到，分布式架构天然的复杂性，对使用者提出了更高的要求，如何在新架构下简化用户体验，便捷普惠地享受到新架构带来的优势，成为各家重点考虑的因素。\n\n作为近些年火热的开源项目，Apache ShardingSphere 的应用范围越来越广，越来越多的企业选择采用 “数据库 + ShardingSphere” 的方式来实现传统关系型数据库到分布式数据库的转型过渡。以 Apache ShardingSphere 为基础，在原有数据库之上构建一套数据中间平台，将分布式数据服务增强能力部署在中间层，而不需对底层数据架构做出调整，进而帮助企业以更加平滑的方式获得分布式以及云上迁移能力。\n\n因此，为进一步降低用户应用门槛，拓宽 Apache ShardingSphere 的使用范围，**SphereEx 在开源内核 Apache ShardingSphere 的基础上推出 SphereEx-Console 可视化数据管控平台。** 作为一款以管理 ShardingSphere 为核心的 SphereEx 商业数据服务集群平台，SphereEx-Console 以平台的形式简化用户配置，极大简化用户操作和部署流程，进而降低新手入门门槛和企业管控难度，进一步提升企业运维效率。\n\n## 什么是 SphereEx-Console\n\n**SphereEx 控制台主要用于对 SphereEx 数据平台的运维管控，在互联网、金融、政务行业均能够支持运维和开发人员常见的操作场景。**\n\n作为一款以管理 ShardingSphere 为核心的 SphereEx 商业数据服务集群平台，SphereEx-Console 可视化数据管控平台以简便易用的操作界面代替相对复杂代码输入方式，结合内部针对资源、实例、插件等多种功能封装，为用户带来一站式的使用体验。\n\n![1.png](https://official-media.sphere-ex.com/1_5bd2d7ea7f.png)\n\n### 产品优势\n\n**功能易用**\n\n提高了用户体验，避免了配置错误，用户不需要通过配置和命令操作 SphereEx 企业数据服务平台，易用性得到极大提升，实现了平台使用的“零”瓶颈。\n\n**功能全面**\n\n完善开源版本的管理能力，提供了对 SphereEx 企业数据服务平台从基础资源到插件能力的管控，形成了对周边插件实现统一管控的解决方案。\n\n**功能可视**\n\n原生提供可视化的监控数据面板，可以实时在线查看 SphereEx 企业数据服务平台的集群、实例、主机的监控数据。\n\n\n### 行业价值\n\n**本次 SphereEx-Console 的发布，是 SphereEx 在数据管理生态下一次关键尝试，弥补了 Apache ShardingSphere 过去在管理易用性、可视化界面等维度的不足。** SphereEx-Console 以可视的操作界面，极大降低了企业运维人员的学习成本；统一的数据管控平台，也极大提升了企业的运维管理效率。\n\n作为一家以开源项目 Apache ShardingSphere 为基本的商业化公司，SphereEx 将持续增强自身在数据治理、管控等方面的能力，配以周边完善的生态基础，为用户提供更加全面和完善的 Apache ShardingSphere 应用生态以及 SphereEx 产品矩阵能力。\n\n**[点击了解详情](https://sphere-ex.com/zh/products/console/)**","date":"2022-01-26","author":"SphereEx","excerpt":"提供企业级的平台管控能力","metaDescription":null,"createdAt":"2022-03-11T09:39:49.579Z","updatedAt":"2022-03-11T09:55:15.098Z","publishedAt":"2022-03-11T09:40:24.284Z","locale":"zh"}}]}}},"article":{"id":22,"attributes":{"feature":false,"title":"F6 Automobile Technology’s Multimillion Rows of Data Sharding Strategy Based on  Apache ShardingSphere","content":"[F6 Automobile Technology](https://www.f6car.com/) is an Internet platform company focusing on the informatization of the automotive aftermarket. It helps automotive repair companies (clients) build their smart management systems to digitally transform the auto aftermarket.\n\nThe data of different auto repair companies will certainly be isolated from each other, so theoretically the data can be stored in different tables of different databases. However, fast-growing enterprises face increasing data volume challenges: sometimes total data volume in a single table may approach 10 million or even 100 million entries.\n\nThis issue definitely challenges business growth. Moreover, growing enterprises are now also planning to split their systems into many microservices based on domains or business types, and accordingly, different databases are vertically required for different business cases.\n\n## Why Did We Need Data Sharding?\n\nRelational databases are bottlenecks when it comes to storage capacity, connection count, and processing capabilities.\n\nFirst, we always prioritize database performance. When the data volume of a single table reaches tens of millions, and there are a relatively large number of query dimensions, system performance would still prove unsatisfactory even if we added more slave databases and optimize indexes. This meant it was time for us to consider data sharding.\n\nThe purpose of data sharding is to reduce database load stress and query time. Additionally, since a single database often has a limited number of connections, when Queries Per Second (QPS) indicator of the database is too high, database sharding is certainly needed to share connection stress.\n\nSecond, to ensure availability was another important reason. If unfortunately, an accident occurs in a single database, we’d likely lose all data and further affect all services. Database sharding can minimize risk and the negative impact on business services. Generally, when the data volume of a table is greater than 2GB or the number of data rows is greater than 10 million, not to mention that the data is also growing rapidly, we’d better use data sharding.\n\n## What’s Data Sharding?\n\nThere are four common types of data sharding in the industry:\n\nVertical table sharding: split big tables into small ones, field-based, which means less frequently used or relatively long fields are split into extended tables.\n\nVertical database sharding: business-based database sharding is used to solve performance bottlenecks of a single database.\n\nHorizontal table sharding: distribute tables’ data rows into different tables according to some rules in order to decrease the data volume of single tables and optimize query performance. In terms of the database layer, it still faces bottlenecks.\n\nHorizontal data sharding: based on horizontal table sharding, distribute data into different databases to effectively improve performance, lower stress of stand-alone machine and single databases, and break the shackles of I/O, connections, and hardware resources.\n\n## Excellent Data Sharding Solutions\n\n**1. [Apache ShardingSphere-JDBC](https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-jdbc-quick-start/)**\n\n**Pros：**\n\n- Supported by an active open source community. Now, Apache ShardingSphere 5.0 version has been released and the development iteration speed is fast.\n- Proven efficacy by many successful enterprise application cases: big companies such as JD Technology and Dangdang.com have applied ShardingSphere.\n- Easy deployment: Sharding-JDBC can be quickly integrated into any project without extra services deployed.\n- Excellent compatibility: it can route to a single data node and perfectly support SQL.\n- Excellent performance and low loss: test results can be found on the Apache ShardingSphere website.\n\n**Cons:**\n\nPotential increase in operations and maintenance costs, and troublesome field changes and index creation after data sharding. To fix the issue, users need to deploy Sharding-Proxy that supports heterogeneous languages and is more friendly to DBAs.\n\nSo far, the project doesn’t support data shards dynamic migration yet. Therefore, feature implementation is required.\n\n**2. MyCat**\n\n**Pros:**\n\n- MyCat is a middleware placed between applications and databases to handle data processing and interactions. It cannot be perceived during development, and integrating MyCat does not cost much.\n- Use JDBC to connect databases such as Oracle, DB2, SQL Server, and MySQL.\n- Support multiple languages plus easy deployment and implementation across different platforms.\n- High availability and auto-switch triggered by a crash.\n\n**Cons:**\n\n- High operations and maintenance costs: to use MyCat, it’s required to configure a series of parameters plus HA load balancer.\n- Users have to independently deploy the service, which may increase system risks.\n\nOf course, there are similar solutions such as Cobar, Zebra, MTDDL, and TiDB but honestly, we didn’t spend much time researching other solutions, because we decided to use ShardingSphere as we felt it meets the company’s needs.\n\n## F6 Automobile Technology’s Overall Plan\n\nBased on our company’s business model, we chose Client ID as Sharding Key to ensure that work order data of one client is stored in the same single table of the same client-specific database. Therefore, performance loss caused by multi-table correlated queries is avoided; plus later, even if multi-databases sharding is required, cross-database transactions and cross-database JOIN can be avoided.\n\nAmong client ID databases, the type BIGINT(20) applies UID (Unique Identification Number, or we call it “gene” ) to ensure potential database scaling in the future; the last two digits of a client ID are its UID, so according to the double scaling rule, the maximum reaches 64 databases. The values of left bits can be used for table sharding, which can be split into 32 sharding tables.\n\nTake 10545055917999668983 as the client ID example and the rules are shown as follows:\n\n```\n105450559179996689 83\nTable sharding uid value % 32 database sharding uid value % 1\n```\nThe last two digits (i.e. 83) are used for database sharding, of which temporary data is only sharded into the library f6xxx, so the remainder is 0. Later, increasing data volume can be expanded to multiple libraries. The remaining value 105450559179996689 is used for table sharding. At first time, it is divided into 32 single tables so the modulo remainders correspond to the specific sharding table subscripts are 0~31.\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rdc5apgqko2381hhrvxw.png)\n \nGiven that the business system is growing and we adopt rapid iteration method to develop features step by step, we plan to shard tables first and then do database sharding.\n\nData sharding has a great impact on the system, so we need greyscale release— if unfortunately, an issue occurs, the system can quickly start roll-back, to ensure a functioning business system. The implementation details are given below:\n\n**Table Sharding**\n\n1. Switch from JDBC to Sharding-JDBC to connect data sources\n2. Decouple the write databases, and then migrate codes\n3. Synchronize historical data and incremental data\n4. Switch sharding tables\n\n**Database Sharding**\n\n1. Migrate the read-only databases\n2. Data migration\n3. Switch read-only databases\n4. Switch write-only databases\n\n## Table Sharding Details\n\n**_Number of Sharding Tables_**\n\nIn the industry, the data of a single table should usually be limited to 5 million rows, and the number of sharding tables should be a power of two to make them scalable. The exact number of sharding tables is calculated based on business development speed and future data increase as well as the future data archiving plan. After sharding table count and sharding algorithms are defined, it’s OK to assess the current data volume in each sharding table.\n\n**_Preparation_**\n\n**- Replace the database & auto table ID generator**\n\nAfter table sharding, we could no longer use the auto database ID generator anymore, so we had to find a feasible solution. We had two plans:\n\nPlan 1: Use other keys such as snowflake\n\nPlan 2: Implement an incremental component (database or Redis) all by ourselves\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qc0qn1rn3d2261fn12ys.png)\n \nAfter comparing the two solutions and the business condition, we decided to choose Plan 2 and concurrently, provided a new comprehensive table-level ID generator solution.\n\n**- Check whether all requests are carried with shard keys**\n\nNow, the microservice traffic entrances include:\n\n- HTTP\n- Dubbo\n- XXLJOB scheduled job\n- Message Queue (MQ)\n\nAfter table sharding, to quickly locate data shards, all requests must carry their shard keys.\n\n**- Decoupling**\n\n1. Decouple business systems of each domain and use interfaces to interact with read and write data.\n\n2. Remove Direct Table JOIN and use interfaces instead.\n\nThe biggest problem brought by the decoupling is the distributed transaction problem: how to ensure data consistency. Usually, developers introduce distributed transaction components to ensure transaction consistency or they use compensation or other mechanisms to ensure final data consistency.\n\n_**Grayscale Release Plan**_\n\nIn order to ensure quick roll-back when problems caused by new feature releases occur, all online modifications are released step by step based on clients. Our grayscale release plan is shown as follows:\n\n**Plan 1:** Maintain two sets of Mapper interfaces: one uses Sharding-JDBC data sources to connect to databases while the other uses JDBC data sources to connect to databases. At the service layer, it’s necessary to select one of the two interfaces based on the decision workflow diagram below:\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eivsoq9mms92ess7g0m1.png)\n \nHowever, the solution causes another problem: all codes visiting the Mapper layer have an `if else` branch, resulting in major business code changes, potential code intrusion, and harder code maintenance. Therefore, we found another solution and we call it Plan 2.\n\n**Plan 2 — Adaptive Mapper Selection Plan:** one set of Mapper interface is with two data sources and two sets of implementations. Based on the grayscale configuration, different client requests will go through different Mapper implementations, and one service corresponds to two data sources and two sets of transaction managers, and based on the grayscale configuration, different clients’ requests go to different transaction managers. Accordingly, we leverage multiple Mapper scanners of MyBatis to generate multiple mapperInterfaces, and concurrently generate a mapperInterface for wrapping. The wrapper class supports hintManagerto automatically select mappers; the transaction manager is similar to wrapper class generation. The wrapper class supports hintManager to automatically select various transaction managers to manage transactions. This solution actually avoids intrusion because for codes of the service layer, there is only one Mapper interface.\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/obzvh9hgdwke71zbnpur.png)\n \n**_Data Source Connection Switch_**\n\nApache ShardingSphere already lists some grammars currently not supported by Sharding-JDBC on its website, but we still found the following problematic SQL statements that Sharding-JDBC parser cannot handle:\n\n- A subquery without a shard key.\n- Not support` Insert` statements whose values include cast ifnull now and other functions.\n- Not support` ON DUPLICATE KEY UPDATE`.\n- By default, select for update goes to the slave database (the issue has fixed since 4.0.0.RC3).\n- Sharding-JDBC does not support the statement ResultSet.first () of MySqlMapper with Optimistic Concurrency Control used to query vision.\n- No such statement for batch updates.\n- Even if `UNION ALL` does not support the grayscale release plan, we only need to copy a set of mapper.xml, and modify it based on the syntax of Sharding-JDBC before release.\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/33j5pacig5eg1afdq4vs.png)\n\n**_Historical Data Sync_**\n\n[DataX](https://www.alibabacloud.com/help/en/doc-detail/126635.html) is Alibaba Group’s offline data synchronization tool that can effectively sync heterogeneous data sources such as MySQL, Oracle, SqlServer, Postgre SQL, HDFS, Hive, ADS, HBase, TableStore(OTS), MaxCompute(ODPS) and DRDS.\n\nThe data synchronization framework DataX can abstract the synchronization of different data sources as a Reader Plugin that reads data from the data source, and then as a Writer Plugin that writes data to the target. In theory, the DataX framework can support data synchronization of all data source types. Additionally, the DataX plugin ecosystem can allow every newly-added data source to immediately interact with the old data sources.\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fzcjfuckkoo0ho77w1w4.png)\n\n**_Verify Data Synchronization_**\n\n- Use timed tasks to compare the number of data of the original table and of the sharding table.\n- Use timed tasks to compare the values of key fields.\n\n**_Read/write Splitting and Table Sharding_**\n\nBefore read/write splitting, we needed to configure incremental data synchronization first.\n\n**-Incremental Data Synchronization**\n\nWe used another open source distributed database sync project named Otter to synchronize incremental data. Based on database incremental log parsing, otter can synchronize data of MySQL/Oracle databases of the local computer room or the remote computer room. To use Otter, we needed to pay extra attention to the following tips:\n\n- For MySQL databases, users must have binlog enabled, and set its mode as ROW.\n- The user must have the query permission of binlog so they need to apply for that as an otter user.\n- Now, the binlog of DMS database is stored for only 3 days. In Otter, users can define the starting position of binlog synchronization and the starting point of incremental synchronization by themselves: first, select slave-testDb on the SQL platform, and use the SQL statement “show master status” to query.\n\n> Note: the execution results of show master statusof master and slave may be different, so if you set it, you need to get the execution result of the master database. We think this function is really useful because when Otter’s data synchronization fails, we can reset points and synchronize again from the beginning.\n\n- When Otter is disabled, it will automatically record the last point of synchronization, and continue to synchronize data from this point next time.\n- Otter allows developers to define their own processing process. For example, we can configure data routing rules and control the direction of client data synchronization data from subtable to parent table or vice versa.\n- Disabling Otter will not invalidate the cache defined in the user-defined Otter processing process. To fix it, the solution is to modify the code comment and save it.\n\n- **Read/write Splitting Plan**\n\nOur greyscale switch plan is shown below:\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r2ftt2x3pv1xwrbom1m6.png)\n\nWe chose the grayscale release solution, which means it was necessary to ensure real-time data updates in both subtables and parent tables. Therefore, all data was synchronized in two directions: for clients with grayscale release being on, reads and writes went to subtables and the data was synchronized to parent tables in real time via Otter, while for clients with grayscale release being off, reads and writes went to parent tables and the data is synchronized to subtables in real time via Otter.\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ic3nh61yipayjvsf259t.png)\n\n##  Database Sharding Details\n**_Preparations_**\n\n**- Primary Key**\nPrimary key should be auto increment type (sharding tables should be global auto increment) or access the only primary key numbering service (server_id independent of DB). Table auto increment primary key generation or uuid_short generated key require switching.\n\n**- Storage Procedure, Functions, Trigger and EVENT**\nTry to remove them first if present; if they cannot be removed, create them in advance in new databases.\n\n**- Data Synchronization**\nData synchronization uses DTS or sqldump (historical data) + otter (incremental data) for synchronization.\n\n**- Database Change Procedure**\nTo avoid potential performance and compatibility problems, database change plan must follow two criterion:\n\n- Greyscale switching: traffic is gradually switching to RDS (Alibaba Cloud Relational Database Service, a.k.a. RDS), allowing obervance of database performance at any time.\n- Quick Rollback: achieving quick reversion when problems occur with little impact on user experience.\n\n**Status Quo:** four application instances +one master db and two slave db\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8l2szt3kc9ygpcu1o65k.png)\n\n**Step 1:** add a new application instance and switch it to RDS, write into or pass dms master database, and the data in dms master database will be synced to rds in real time\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pv6r9qptfy7lsfpmsfzj.png)\n\n**Step 2:**add three more application instances, and cut 50% of the data to write into rds database \n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cybdga96h0wnj5byix4r.png)\n \n**Step 3:** remove the four original instance traffic, and read them into rds instances while writing still goes into dms master database\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e12d8o29ugwuz0l1xzcp.png)\n \n**Step 4:** switch the master database into rds, and rds data will be reversely synced to dms master database to make it easier for the quick rollback of the data\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9c4v451ofi535xwx28nl.png)\n\n**Step 5:** Completion\n\n ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/57jimac2lhn2gflcbdup.png)\n\nEach step mentioned above can be quickly rolled back through traffic switching so as to ensure the availability and stability of system.\n \n**Sharding & Scaling**\n\nWhen the performance of a single database reaches a plateau, we can scale out the database by modifying sharding database routing algorithms and migrating data.\n\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0rpthxwsficzwqfe2zn7.png)\n\nWhen the capacity of a single table reaches its maximum size, we can scale out the table by modifying sharding table routing algorithms and migrating data.\n\n ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/86tuypkympqtcez27m4w.png)\n \n\n## FAQ\n**_Q: Sometimes, Otter receives binlog data but the data cannot be found in the database?_**\n\n**A:**To make our MySQL compatible with replicas of other non-transactional engines, we added binlog at the server layer. Binlog can record all engine modification operations, so it can support replication function for all engines. The problem is a potential inconsistency between redo log and binlog but MySQL uses its internal XA mechanism to fix the issue.\n\n**Step 1:**Not perform operations on InnoDB prepare, write/sync redo log and binlog.\n\n**Step 2:** First, write/sync Binlog and then InnoDB commit (commit in memory).\n\nOf course, group commit has been added since version 5.6. The development improves I/O performance to some degree, but it doesn’t change the execution order.\n\nAfter write/sync Binlog is done, the binlog has been written, so MySQL considers that the transaction has been committed and persisted (now, the binlog is ready to be sent to subscribers). Even if a database crashes, the transaction can still be recovered correctly after MySQL reboot. However, before this step, any operation failure may cause transaction rollback.\n\nInnoDB commit is centered on memory commit such as killing locks, read views related to multiversion concurrency control release. MySQL believes that no errors occur in this step — once an error really occurs, the database will crash — MySQL itself cannot handle the crash. This step does not have any logic that causes transaction rollback. In terms of program operations, only after this step is completed, the changes caused by the transaction can be shown through the API or queries at the client-side.\n\nThe reason why the problem may occur is that the binlog is sent first, and then db commit is done. We use query retries to fix this issue.\n\n**_Q： When it comes to multi-table queries, sometimes, why some tables cannot get data?_**\n\n**A:**The master/slave routing strategy of Sharding-JDBC is shown below:\n\nMaster databases are chosen in the following scenarios:\n\n- SQL statements that include lock such as select for update ( of Version 4.0.0.RC3);\n- Not SELECT statements;\n- Threads that have already gone to master databases;\n- Codes specify the requests for master databases.\n\nAlgorithms used to choose from multiple slave databases:\n\n- Polling Strategy\n- Load Balancer Strategy\n\nThe default is the polling strategy. However, one query may go to different slave databases, or it may go to the master library and slave databases, which occurs when there is time inconsistency between master-slave database latency or multi-slave latency.\n\n**_Q: How can we remove network traffic?_**\n**A:**\n\n- http: use nginx to remove upstream;\n- dubbo: leverage its qos module to execute offline/online command;\n- xxljob: manually enter execution IP of the executor to specify instances;\n- MQ: use the API provided by Alibaba Cloud to enable or disable consumer bean.\n\n## Apache ShardingSphere Open Source Project Links:\n\n[ShardingSphere Github](https://github.com/apache/shardingsphere)\n[ShardingSphere Twitter](https://twitter.com/ShardingSphere)\n[ShardingSphere Slack Channel](https://apacheshardingsphere.slack.com/ssb/redirect)\n","date":"2022-01-24","author":"SphereEx","excerpt":"F6 Automobile Technology is an Internet platform for the automotive aftermarket. It switched to sharding and chose ShardingSphere for its multimillion rows of data. ","metaDescription":"F6 Automobile Technology is an Internet platform company focusing on the informatization of the automotive aftermarket.","createdAt":"2022-09-26T07:41:17.269Z","updatedAt":"2022-09-27T07:00:17.977Z","publishedAt":"2022-09-27T07:00:17.973Z","locale":"en","caseStudiesType":{"data":{"id":9,"attributes":{"name":"Internet","createdAt":"2022-09-26T10:34:03.086Z","updatedAt":"2022-09-27T06:59:34.838Z","publishedAt":"2022-09-27T06:59:34.836Z","locale":"en"}}},"cover":{"data":{"id":342,"attributes":{"name":"F6 automobile.png","alternativeText":"F6 automobile.png","caption":"F6 automobile.png","width":720,"height":443,"formats":{"thumbnail":{"name":"thumbnail_F6 automobile.png","hash":"thumbnail_F6_automobile_c02eaa72b3","ext":".png","mime":"image/png","width":245,"height":151,"size":17.64,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_F6_automobile_c02eaa72b3.png"},"small":{"name":"small_F6 automobile.png","hash":"small_F6_automobile_c02eaa72b3","ext":".png","mime":"image/png","width":500,"height":308,"size":42.94,"path":null,"url":"https://official-media.sphere-ex.com/small_F6_automobile_c02eaa72b3.png"}},"hash":"F6_automobile_c02eaa72b3","ext":".png","mime":"image/png","size":67.32,"url":"https://official-media.sphere-ex.com/F6_automobile_c02eaa72b3.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T07:41:48.589Z","updatedAt":"2022-09-26T07:41:48.589Z"}}},"metaLogo":{"data":{"id":356,"attributes":{"name":"F6 logo.png","alternativeText":"F6 logo.png","caption":"F6 logo.png","width":1200,"height":600,"formats":{"thumbnail":{"name":"thumbnail_F6 logo.png","hash":"thumbnail_F6_logo_6330d93329","ext":".png","mime":"image/png","width":245,"height":123,"size":4.96,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_F6_logo_6330d93329.png"},"large":{"name":"large_F6 logo.png","hash":"large_F6_logo_6330d93329","ext":".png","mime":"image/png","width":1000,"height":500,"size":35.19,"path":null,"url":"https://official-media.sphere-ex.com/large_F6_logo_6330d93329.png"},"medium":{"name":"medium_F6 logo.png","hash":"medium_F6_logo_6330d93329","ext":".png","mime":"image/png","width":750,"height":375,"size":22.18,"path":null,"url":"https://official-media.sphere-ex.com/medium_F6_logo_6330d93329.png"},"small":{"name":"small_F6 logo.png","hash":"small_F6_logo_6330d93329","ext":".png","mime":"image/png","width":500,"height":250,"size":12.46,"path":null,"url":"https://official-media.sphere-ex.com/small_F6_logo_6330d93329.png"}},"hash":"F6_logo_6330d93329","ext":".png","mime":"image/png","size":32.65,"url":"https://official-media.sphere-ex.com/F6_logo_6330d93329.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:51:46.576Z","updatedAt":"2022-09-26T08:51:46.576Z"}}},"localizations":{"data":[]}}}}},
    "staticQueryHashes": []}