{
    "componentChunkName": "component---src-templates-case-study-detail-ts",
    "path": "/case-studies/18",
    "result": {"pageContext":{"next":{"id":19,"attributes":{"feature":false,"title":"Apache ShardingSphere & Huimin: Huimin’s Transaction Middle Platform Architecture","content":"The FMCG (fast moving consumer-goods) industry can no longer meet current development requirements. The traditional way of only upgrading the end of the supply chain cannot achieve high-quality business development. [Huimin](https://huimin.cn/) focuses on order fulfillment and delivery service for over one million community supermarkets, transforming and upgrading from serving retail terminal stores to promoting the digitization of the whole FMCG industry chain.\n\nIt is committed to building a B2B2C data closed-loop operating system, providing brand owners, distributors, and community supermarkets with transformed and upgraded solutions for the whole chain covering purchasing, marketing, operation, and distribution. All these commitments cannot be fulfilled without its powerful “middle platform” system.\n\n## Introducing the “middle platform”\nHuimin began to transferred [PHP](https://www.php.net/) technology stack to the Java technology stack, and shifted to microservices from 2016 to 2017, and started to build the “middle platform” in 2018.\n\nThe phase-one project of its transaction middle platform went live in early 2021, and phase-two in March 2022.\n\nOwing to business growth and various changes, Huimin launched a strong “middle platform” development strategy to improve efficiency and reduce costs. Measures were taken to reduce the coupling degree between systems, increase scalability, and ensure low latency while extracting business commonalities. At the beginning of 2021, the company finihsed rebuilding the previous order management system (OMS) and launched a packaged business capability project for a transaction middle platform centered on the transaction process of orders, which is divided into the following four core modules: order status flow system, order management system, order fulfillment system and order cost calculation system. At the same time, based on business stratification, the project is also split into an order query system, report system, and order management backend.\n\nTo synergize technical and business requirements during the rebuilding process, the go-live process was divided into two phases: application system splitting and data splitting.\n\n## System splitting\nR&D teams having to frequently adjust order management system and adapt it to multiple business lines may cause inefficiencies. In response to these problems, the following principles were formulated to split and rebuild the system:\n\n1. Avoid the logic coupling of different lines of business. Different business lines should contain and minimize influence on each other to reduce logic complexity.\n2. Responsibilities are divided according to the order lifecycle to ensure that each system is independent with an exclusive responsibility.\n3. Read/write splitting. Ensure the stability of core functions and avoid the impact of frequent iterations on core functions.\n4. [ElasticSearch](https://www.elastic.co/) was introduced to solve the external index problem, and reduce database index pressure.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1223xl1pboe6emvrjian.png)\n \n\n## Data splitting\nThe purpose of data splitting is to reduce the maintenance pressure of a single table. When the amount of data reaches tens of millions, the index and field maintenance of database tables will have a great impact on the online environment. Before splitting a table, you have to consider the following questions:\n\n1. Splitting rules: what tables are data-intensive? What about the number of splits?\n2. Data read/write: how to solve the read/write problem after splitting tables?\n3. Go-live solution: how to go live smoothly?\n\n**Splitting rules**\n\nDuring a 3 years iteration period, the number of splits = ((amount of data added per month * 36) + Amount of historical data) / maximum amount of data in a single table.\n\nWhen we use the [InnoDB](https://en.wikipedia.org/wiki/InnoDB) engine for DDL operation to maintain [Alibaba Cloud](https://www.alibabacloud.com/?spm=a2796.143805.6791778070.2.b8fa46e3JLecKh) RDS MySQL, we come up with the following results:\n\nWhen the amount of data is less than 5 million, the execution time is about tens of seconds. Between 5 million and less than 10 million, the execution time is about 500 seconds; between 10 million and less than 50 million, the execution time is about 1000 seconds; and when more than 50 million, the execution time is more than 2000 seconds.\n\nThe upper limit for splitting a single table depends on the impact of a single table operation on business. Of course, some solutions can tackle the problem of table lock resulting from DDL operation. For example, dual-table synchronization and changes in table names can reduce the impact to within seconds.\n\nThe split number of the library table is preferably 2 to the power of N, which is conducive to later horizontal scaling.\n\n**Technology selection**\n\nAfter the data table is split, data hashing and query issues need to be resolved. As a SME (small & medium-sized enterprise), the cost of developing a set of sharding middleware is too high, and it will cause various risks in the early stage. A reasonable solution is to adopt a set of open source database sharding middleware.\n\n**[Apache ShardingSphere](https://shardingsphere.apache.org/)**\n\nApache ShardingSphere is an open source ecosystem that allows you to transform any database into a distributed database system. The project includes a JDBC, a Proxy and Sidecar (Planning), which can be deployed independently and support mixed deployment. They all provide functions of data scale-out, distributed transactions and distributed governance.\n\nApache ShardingSphere 5.x introduces the concept of [Database Plus](https://faun.pub/whats-the-database-plus-concepand-what-challenges-can-it-solve-715920ba65aa), and the new version focuses on a plugin oriented architecture. Currently, functions including data sharding, read/write splitting, data encryption, shadow library pressure testing, as well as SQL and protocol support such as [MySQL](https://www.mysql.com/), [PostgreSQL](https://www.postgresql.org/), [SQL Server](https://www.microsoft.com/en-us/sql-server/sql-server-downloads), [Oracle](https://www.oracle.com/index.html), etc. are woven into the project through plugins, allowing developers to customize their own unique systems.\n\nWhy Huimin chose Apache ShardingSphere:\n\n1. Apache ShardingSphere has a variety of functions and can solve current challenges with strong scalability.\n\n2. The Apache ShardingSphere community is very active, which means the possibility to find support quickly.\n\n3. The company adopts the SpringCloud technology stack which is convenient for integration and can lower costs.\n\n4. Its performance can fully support Huimin’s existing business.\n\nShardingSphere supports the following 3 modes:\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/19343yqtstyzsmvtu69j.png)\n \n\nCurrently, Huimin’s server-side technology stack only involves Java language, so there is no need to consider heterogeneous scenarios. Taking flexibility, code intrusion and deployment cost into consideration, the implementation of [ShardingSphere-JDBC](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-jdbc) wasfinally selected.\n\n**Technology implementation**\n\nThe most complicated part of the process is actually the go-live part, not the plugin integration part as would someone expect.\n\nTo achieve non-perception and rollback, we divided the go-live process into the following steps, as shown in the figure:\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dyrbcvaql5nhnw4m0xns.png)\n \n\n**Step 1:** Processing all the data needs to ensure the data consistency between the old and new libraries. Since there currently is no synchronization tool supporting data sharding policy, we developed a data synchronization tool to support the configuration of the sharding policy.\n\n**Step 2:** Process incremental data to keep data consistency between the old and new libraries. Our implementation scheme here was to use the open source component, to listen to the binlog of the database and synchronize the database changes to the message queue, and then the data synchronization tool listens to the messages and writes them into the new library.\n\nThe architecture scheme is not complicated, but great importance was attached to the data consistency in this process. Incremental data cannot be lost, and single-line data should be written in sequential order to prevent ABA problems.\n\n**Step 3:** Process the gray-release of read-traffic. At this point, we were writing an old library and reading a new library in the database. In this process, we should paid attention to the sensitivity of business to data consistency, because the delay from an old library to a new library is within seconds. Reading an old library is needed when we come across some sensitive scenarios requiring high consistency.\n\n**Step 4:** Switch all read-traffic of the application to the new library, maintaining the status of writing the old library and reading the new library.\n\n**Step 5:** Processing write-traffic didn’t take gray-release scenarios into consideration in order to reduce the complexity of the program. Of course, the grayscale scheme can also achieve the same effect. We supported the rollback strategy by writing data from the new library back to the old library, and then uniformly transferring all traffic to the new library within one release.\n\nSo far, the whole process of release and going live has been completed. Due to the low cost of completing the transformation with the help of ShardingSphere-JDBC middleware to rewrite SQL and merge results, there was almost no code intrusion and going live went smoothly.\n\nThere are also some issues that you may need to be aware of when accessing the ShardingSphere-JDBC scheme:\n\n1. ShardingSphere-JDBC supports a default processing for data insertion logic. If there is no shard column in SQL, the same data will be inserted into each shard table.\n\n2. ShardingSphere-JDBC parses only the logil table of the first SQL for batches of SQL semantic analysis, which results in an error. This problem has been reported and should be fixed in the near future. Additionally, a detailed SQL sample is provided, which lists in detail the scope of support.\n\n3. ShardingSphere connection mode selection. The number of database connections visited by businesses should be limited for the sake of resource control. It can effectively prevent a service operation from occupying too many resources, which depletes the database connection resources and affects the normal access of other services. Especially in the case of many sub-tables in a database instance, a logical SQL without shard keys will produce a large number of actual SQL that falls on different tables in the same library. If each actual SQL occupies a separate connection, a query will undoubtedly occupy too many resources.\n\nFrom the perspective of execution efficiency, maintaining a separate database connection for each shard query can make more effective use of multi-threading to improve execution efficiency. Building a separate thread for each database connection allows I/O consumption to be processed concurrently, maintains a separate database connection for each shard, and avoids premature loading of query result data into memory. A separate database connection can hold a reference to the cursor location of the query result set and move the cursor when it needs to fetch the corresponding data.\n\nThe result merging method by moving the result set cursor down is called streaming merge. It does not have to load all the result data into memory, which can effectively save memory resources and reduce the frequency of garbage collection. If each shard query cannot be guaranteed to have a separate database connection, the current query result set needs to be loaded into memory before reusing the database connection to obtain the query result set of the next shard table. Therefore, even if streaming merge could be used, it would degenerate into a memory merge in this scenario.\n\nIt is a problem for the ShardingSphere execution engine to balance the relationship between the control and protection of database connection resources and saving middleware memory resources by adopting a better merging mode. Specifically, if a SQL needs to operate on 200 tables under a database instance after its sharding in ShardingSphere, whether to create 200 connections and execute them concurrently or create one connection and execute it in sequence? How to choose between efficiency and resource control?\n\nIn response to the above scenario, ShardingSphere provides a solution. It introduces the concept of Connection Mode and divides it into `MEMORY_STRICTLY` mode and `CONNECTION_STRICTLY` mode.\n\n`MEMORY_STRICTLY`\n\nThe premise of using this pattern is that ShardingSphere does not limit the number of database connections required by an operation. If the actual SQL executed needs to operate on 200 tables in a database instance, a new database connection is created for each table and processed concurrently through multi-threading to maximize the execution efficiency. When SQL meets the conditions, streaming merge is preferred to prevent memory overflow or frequent garbage collection.\n\n`CONNECTION_STRICTLY`\n\nThe premise for using this pattern is that ShardingSphere strictly limits the number of database connections required for an operation. If the actual SQL executed needs to operate on 200 tables in a database instance, only one single database connection is created and its 200 tables are processed sequentially. If shards in an operation are scattered across different databases, multi-threading is still used to handle operations on different libraries, but only one database connection is created for each operation of each library. The practice prevents the problem of consuming too many database connections for one request. This pattern adopts memory merge all the time.\n\nThe `MEMORY_STRICTLY` mode is applicable for OLAP operations and can improve system throughput by relaxing limits on database connections. The `CONNECTION_STRICTLY` mode applies to OLTP operations, which usually have shard keys and are routed to a single shard. Therefore, it is a wise choice to strictly control database connections to ensure that database resources in an online system can be used by more applications.\n\nWe found that in the `MEMORY_STRICTLY` mode, the operation would become I/O-intensive due to the cache buffer loading strategy of MySQL InnoDB engine, resulting in SQL timeouts. Its solution is to add another layer of processing without changing database resources. If there is no shard key, verify the shard key in the external index, and then use the shard key to perform database retrieval.\n\n## ShardingSphere Benefits\n**1. Performance improvement**\nThrough architecture rebuilding, it can effectively control the amount of single table data, and greatly reduce the slow SQL, down nearly 50%.\n\n**2. Save R&D resources and lower cost**\n\nThe introduction of Apache ShardingSphere does not require the re-development of sharding components, which reduces the R&D cost and lowers the risks.\n\n**3. Strong scalability**\n\nApache ShardingSphere has good scalability in terms of data encryption, distributed transactions, shadow library and other aspects.\n\n**Conclusion**\nIn his book [Building Microservices, Sam Newman](https://samnewman.io/books/building_microservices/) writes, “ Our requirements shift more rapidly than they do for people who design and build buildings — as do the tools and techniques at our disposal. The things we create are not fixed points in time. Once launched into production, our software will continue to evolve as the way it is used changes. For most things we create, we have to accept that once the software gets into the hands of our customers we will have to react and adapt, rather than it being a never-changing artifact. Thus, our architects need to shift their thinking away from creating the perfect end product, and instead focus on helping create a framework in which the right systems can emerge, and continue to grow as we learn more (Sam Newman, 2015: 15).” Apache ShardingSphere is doing exactly what the book says, and is definitely on a promising path.\n\n## Author\n**Zhai Yang**\n\nSoftware architect at Zhongshang Huimin E-Commerce platform team.\n\nParticipated in the 0–1 process of middle platform construction, mainly responsible for the R&D and building the transaction and commodity middle platform and the search platform.","date":"2022-05-17","author":"SphereEx","excerpt":"HuiMin does order fulfillment for over 1 million supermarkets. With ShardingSphere HuiMin shifted to microservices & built a new \"middle platform\" architecture with packaged business capability for order transaction processing.","metaDescription":"Huimin is a B2B e-commerce platform that focuses on small-scale supermarkets that provides community supermarket websites for communities.","createdAt":"2022-09-26T07:01:29.374Z","updatedAt":"2022-09-27T06:59:59.417Z","publishedAt":"2022-09-27T06:59:59.413Z","locale":"en","caseStudiesType":{"data":{"id":2,"attributes":{"name":"E-commerce","createdAt":"2021-12-11T23:02:08.395Z","updatedAt":"2021-12-11T23:02:11.020Z","publishedAt":"2021-12-11T23:02:10.947Z","locale":"en"}}},"cover":{"data":{"id":338,"attributes":{"name":"Zhongshanghuimin.png","alternativeText":"Zhongshanghuimin.png","caption":"Zhongshanghuimin.png","width":720,"height":339,"formats":{"thumbnail":{"name":"thumbnail_Zhongshanghuimin.png","hash":"thumbnail_Zhongshanghuimin_6ceb4a8b12","ext":".png","mime":"image/png","width":245,"height":115,"size":19.04,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_Zhongshanghuimin_6ceb4a8b12.png"},"small":{"name":"small_Zhongshanghuimin.png","hash":"small_Zhongshanghuimin_6ceb4a8b12","ext":".png","mime":"image/png","width":500,"height":235,"size":60.18,"path":null,"url":"https://official-media.sphere-ex.com/small_Zhongshanghuimin_6ceb4a8b12.png"}},"hash":"Zhongshanghuimin_6ceb4a8b12","ext":".png","mime":"image/png","size":73.86,"url":"https://official-media.sphere-ex.com/Zhongshanghuimin_6ceb4a8b12.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T07:01:49.173Z","updatedAt":"2022-09-26T07:01:49.173Z"}}},"metaLogo":{"data":{"id":353,"attributes":{"name":"zhongshanghuimin logo.png","alternativeText":"zhongshanghuimin logo.png","caption":"zhongshanghuimin logo.png","width":608,"height":430,"formats":{"thumbnail":{"name":"thumbnail_zhongshanghuimin logo.png","hash":"thumbnail_zhongshanghuimin_logo_978897c9d6","ext":".png","mime":"image/png","width":221,"height":156,"size":9.77,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/thumbnail_zhongshanghuimin_logo_978897c9d6.png"},"small":{"name":"small_zhongshanghuimin logo.png","hash":"small_zhongshanghuimin_logo_978897c9d6","ext":".png","mime":"image/png","width":500,"height":354,"size":31.4,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/small_zhongshanghuimin_logo_978897c9d6.png"}},"hash":"zhongshanghuimin_logo_978897c9d6","ext":".png","mime":"image/png","size":12.04,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/zhongshanghuimin_logo_978897c9d6.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:42:11.374Z","updatedAt":"2022-10-12T05:56:46.710Z"}}},"localizations":{"data":[]}}},"prev":{"id":17,"attributes":{"feature":false,"title":"Heterogeneous migration: reducing Dangdang’s customer system RTO 60x and increasing speed by 20%","content":"> [Apache ShardingSphere](https://shardingsphere.apache.org/) helps [Dangdang](https://www.crunchbase.com/organization/dangdang-com) rebuild its customer system with 350 million users, and seamlessly transition from a [PHP](https://www.php.net/)+[SQL Server](https://www.microsoft.com/en-us/sql-server/sql-server-downloads) technology stack to a Java+ShardingSphere+[MySQL](https://www.mysql.com/) stack. The performance, availability, and maintainability of its customer system have been significantly improved, which is the best practice of ShardingSphere’s heterogeneous migration.\n\n## Dangdang’s customer system\nDangdang’s customer system is mainly responsible for account registration, login, and privacy data maintenance. Its previous technology stack was based on PHP and SQL Server, which means a standard centralized architecture, as shown in the figure below.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lorqkp14im9bxbco4yuw.png)\n \n\nBefore the rebuild project started, several business modules of the customer system had encountered multiple problems and technical challenges, such as logical decentralization, low throughput, and high operation & maintenance costs.\n\nTo improve customers’ shopping experience, Dangdang’s technical team decided to optimize the business logic and underlying data architecture to achieve the availability, scalability, and comprehensive improvement of the customer system in multiple scenarios. The rebuild also introduced many technological innovations such as cross-data source double write, read/write splitting, intelligent gateway, and gray release.\n\nDangdang’s technical team completed the system rebuild within half a year, from demand design, sharding planning, logic optimization, and stress testing to its official launch.\n\nThe project used Java to reconstruct more than ten modules, build distributed database solutions through ShardingSphere & MySQL, and finally complete the online migration of heterogeneous databases. The project boasts the following highlights:\n\n- Reconstruct PHP business code using Java language.\n- Replace SQL Server with ShardingSphere & MySQL.\n- Complete online data migration of 350 million users.\n- Complete a seamless launch through the data double-write scheme.\n\n## Pain points & challenges\n\n**Business pain points**\nAt the business level, the registration and login logic of some modules of the customer system was scattered at different ends. This resulted in high maintenance costs, and the old technical architecture was limited in terms of performance improvement and high availability.\n\n- **Maintenance difficulty:** the registration and login logic of multiple platforms is scattered, so business maintenance is complicated.\n- **Limited performance:** the PHP & SQL Server, a centralized technical architecture, had insufficient throughput.\n- **Poor availability and security:** If the active/standby status of SQL Server changes, the subscription database becomes invalid and the reconfiguration takes a window of time. The security of SQL Server running on Windows Server is poor due to viruses, and the upgrading takes a long time (>30min) after the patch is installed.\n\n**Challenges**\n\n- **Data integrity:** the customer system involves data of more than 350 million users. It is necessary to ensure data consistency and integrity after migrating from SQL Server to MySQL.\n- **API transparency:** the API is transparent to the caller to ensure that the caller does not change and to minimize the change of interface.\n- **Seamless switch:** the business system must be seamlessly switched over without impact on business.\n- **Time is short:** the system will be blocked before and after “[618 (aka JD.com Day)](https://edition.cnn.com/2020/06/18/tech/jd-618-china-coronavirus-intl-hnk/index.html) and [11.11 (aka Singles Day)](https://en.wikipedia.org/wiki/Singles%27_Day)” (two online shopping festivals in China), so we need to switch it between the two shopping promotions in a limited window of time, and then undergo the tests to prepare for the 11.11 shopping festival.\n\n## Solutions\n**Overall planning**\nTo improve the maintainability, availability, and performance of the customer system, the R&D team reorganized the customer system architecture.\n\nAt the application layer, the goal was to unify the function logic of all terminals and improve business maintainability.\n\nAt the database layer, the centralized architecture was transformed into a distributed database architecture to improve performance and availability, which is exactly the open-source distributed solution built by ShardingSphere & MySQL.\n\n- **Application layer:** As Dangdang’s overall technology stack changed, its business development language changed from PHP to Java.\n- **Middleware:** As a mature open-source database middleware, ShardingSphere, was used to achieve data sharding.\n- **Database:** Multiple MySQL clusters were used to replace SQL Server databases.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bhrntsoik9fb41raqqbe.png)\n \nThe overall architecture design introduced multiple schemes, such as distributed primary-key generation strategy, shard management, data migration verification, and gray release.\n\n## Distributed primary-key generation strategy\n\nDistributed primary-key generation strategy is the first problem to be solved if database architecture is to be transformed from a centralized architecture to a distributed one based on middleware.\n\nDuring the system rebuild, we chose to build two or more database ID-generating servers. Each server had a `Sequence` table that records the current `ID` of each table. The step size of `ID` that increases in the `Sequence` table is the number of servers. The starting values are staggered so that the ID generation is hashed to each server node.\n\n## Implementing sharding (Apache ShardingSphere）\nDuring the customer system rebuild, database sharding was completed through Apache ShardingSphere, and the read/write splitting function was also enabled.\n\nDue to the requirements of the customer system for high concurrency and low latency, the access end chose [ShardingSphere-JDBC](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-jdbc), which is positioned as a lightweight Java framework and provides additional services in Java’s JDBC layer.\n\nIt connects directly to the database via the client and provides services in the form of a `jar` package without additional deployment and dependence. It can be viewed as an enhanced version of the JDBC driver, fully compatible with JDBC and various [ORM](https://www.techopedia.com/definition/24200/object-relational-mapping--orm) frameworks.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6yyoi7xsp7qpayu1it5v.png)\n \n\n**Sharding:** ShardingSphere supports a complete set of sharding algorithms, including `modulo` operation, `hash`, `range`, `time`, and customized algorithms. Customers use the `modulo` sharding algorithm to split large tables.\n**Read-write splitting:** in addition to Sharding, ShardingSphere’s read/write splitting function is also enabled to make full use of [MHA](https://myheroacademia.fandom.com/wiki/Cluster) cluster resources and improve system throughput capacity.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/chx6mgi5yfr2tqv86ho3.png)\n \n\n## Double-write & data synchronization\nData synchronization runs through the whole rebuild project, and the integrity and consistency of data migration are vital to the rebuild.\n\nThis example periodically synchronizes SQL Server’s historical data to MySQL based on Elastic-Job synchronization. During the database switchover, a backup scheme is used to double-write the database to ensure data consistency. The process consists of:\n\n**Step 1:** implement the double-write mechanism\n\nDisconnect link 1, get through links 2, 3, 4, and then 9, 10.\n\n**Step 2:** switch the login service\n\nDisconnect links 9,10, get through link 7 and disconnect link 5.\n\n**Step 3:** switch read service\n\nGet through link 8 and disconnect link 6.\n\n**Step 4:** cancel the double-write mechanism\n\nDisconnect link 2 and complete the switchover.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c2f4q20g116681261m9m.png)\n \n\nData verification is performed periodically on both the service side and the database side. Different frequencies are used in different time periods to sample or fully check data integrity. `COUNT/SUM` is also verified on the database side.\n\nCustomer system reconstruction adopts an apollo-based gray release. In the process of new login processing, configuration items are gradually released and sequential cutover within a small range is implemented to ensure the launch success rate. The rebuilt system architecture is shown in the following figure.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iru06ju7apswukev0wxc.png)\n \n\n## Advantages\nAfter the rebuild, the response speed of Dangdang’s customer system is significantly improved, and the daily operation & maintenance costs are also reduced.\n\nThe distributed solution provided by ShardingSphere plays a big part in this. The solution is suitable for various high-traffic Internet platform services, as well as e-commerce platforms and other data-processing systems.\n\n- **Performance improvement:** response speed increased by more than 20%.\n- **High availability:** RTO is reduced to less than 30s owing to the ShardingSphere & MySQL design.\n- **Easy to maintain:** business logic and database maintainability are significantly improved.\n- **Seamless migration:** complete online cutover of each module within 6 months, and window time is zero.\n\n## Conclusion\nThis is ShardingSphere’s second implementation by Dangdang, following the previous one we shared in the post “[Asia’s E-Commerce Giant Dangdang Increases Order Processing Speed by 30% — Saves Over Ten Million in Technology Budget with Apache ShardingSphere](https://shardingsphere.medium.com/asias-e-commerce-giant-dangdang-increases-order-processing-speed-by-30-saves-over-ten-million-113a976e0165)”.\n\nApache ShardingSphere provides strong support for enterprise systems, as the project strives for simplicity and perfection, to achieve simpler business logic and maximum performance.\n\n**Apache ShardingSphere Project Links:**\n\n[ShardingSphere Github](https://github.com/apache/shardingsphere/issues?page=1&q=is%3Aopen+is%3Aissue+label%3A%22project%3A+OpenForce+2022%22)\n\n[ShardingSphere Twitter](https://twitter.com/ShardingSphere)\n\n[ShardingSphere Slack](https://join.slack.com/t/apacheshardingsphere/shared_invite/zt-sbdde7ie-SjDqo9~I4rYcR18bq0SYTg)\n\n[Contributor Guide](https://shardingsphere.apache.org/community/cn/contribute/)","date":"2022-06-21","author":"SphereEx","excerpt":"E-commerce giant DangDang rebuilt its architecture & migrated 350M users with ShardingSphere & MySQL. RTO improved 60x, response speed by 20%, and introduced cross-data source double write, R/W split, and intelligent gateway innovations.","metaDescription":"Dangdang is a B2C-based e-commerce company in China offering a large selection of products both online and offline.","createdAt":"2022-09-26T06:47:46.510Z","updatedAt":"2022-09-27T06:55:21.087Z","publishedAt":"2022-09-27T06:55:21.084Z","locale":"en","caseStudiesType":{"data":{"id":9,"attributes":{"name":"Internet","createdAt":"2022-09-26T10:34:03.086Z","updatedAt":"2022-09-27T06:59:34.838Z","publishedAt":"2022-09-27T06:59:34.836Z","locale":"en"}}},"cover":{"data":{"id":336,"attributes":{"name":"Dangdang.png","alternativeText":"Dangdang.png","caption":"Dangdang.png","width":720,"height":432,"formats":{"thumbnail":{"name":"thumbnail_Dangdang.png","hash":"thumbnail_Dangdang_f80baa2e02","ext":".png","mime":"image/png","width":245,"height":147,"size":11.34,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_Dangdang_f80baa2e02.png"},"small":{"name":"small_Dangdang.png","hash":"small_Dangdang_f80baa2e02","ext":".png","mime":"image/png","width":500,"height":300,"size":28.45,"path":null,"url":"https://official-media.sphere-ex.com/small_Dangdang_f80baa2e02.png"}},"hash":"Dangdang_f80baa2e02","ext":".png","mime":"image/png","size":37.5,"url":"https://official-media.sphere-ex.com/Dangdang_f80baa2e02.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T06:48:03.795Z","updatedAt":"2022-09-26T06:48:03.795Z"}}},"metaLogo":{"data":{"id":351,"attributes":{"name":"dangdang logo.jpg","alternativeText":"dangdang logo.jpg","caption":"dangdang logo.jpg","width":500,"height":256,"formats":{"thumbnail":{"name":"thumbnail_dangdang logo.jpg","hash":"thumbnail_dangdang_logo_539787f05a","ext":".jpg","mime":"image/jpeg","width":245,"height":125,"size":5.42,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/thumbnail_dangdang_logo_539787f05a.jpg"}},"hash":"dangdang_logo_539787f05a","ext":".jpg","mime":"image/jpeg","size":12.11,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/dangdang_logo_539787f05a.jpg","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:37:49.195Z","updatedAt":"2022-10-12T05:57:41.631Z"}}},"localizations":{"data":[]}}},"article":{"id":18,"attributes":{"feature":false,"title":"Apache ShardingSphere & Bilibili","content":"> To further understand application scenarios, and enterprises’ needs, and improve dev teams’ understanding of Apache ShardingSphere, our community launched the “Enterprise Visits”\n> series.\n\n\n[Apache ShardingSphere](https://shardingsphere.apache.org/)’s core contributor team was invited to the headquarters of [Bilibili](https://www.bilibili.com/) in Shanghai. Our PMC Chair, Zhang Liang, discussed with Bilibili’s tech team the e-commerce and digital entertainment verticals application scenarios, and the capabilities of different versions of ShardingSphere.\n\nWith the unprecedented growth of data volume and increasingly diversified data application scenarios, different platforms, businesses, and scenarios have resulted in database applications’ fragmentation. The challenges facing databases today are totally different from the ones they were conceived to solve. This is also reflected in the growth of Bilibili’s e-commerce business.\n\nFor over a decade, Bilibili has built an ecosystem community centered on users, creators, and content, while producing high-quality videos as well. With an increasing number of users, Bilibili has also gradually developed peripheral business ecosystems such as its subscription revenue model. The expansion of business lines and application scenarios brought great challenges for Bilibili’s tech team, especially in the management and application of backend data.\n\nDuring the visit, our community and Bilibili mainly discussed these three points:\n\n## The SQL warm-up feature jointly built with Bilibili:\nWhen Bilibili queries goods and orders with batch priority in its business, it is often necessary to manually warm-up SQL during the initial linking process to improve overall performance. However, in the process of manual warm-up, parameters of different lengths cannot be used as a SQL to warm-up due to the use of `foreach` syntax of the [Mybatis](https://mybatis.org/mybatis-3/index.html) framework.\n\nApache ShardingSphere can see the SQL execution plan through the `preview`, thus warming up the SQL. In the future, we plan to provide a separate SQL warm-up interface and merge the SQL warm-up time into the startup time. ShardingSphere will start on its own after SQL is warmed up.\n\n[Apache ShardingSphere](https://shardingsphere.apache.org/) is an open-source project driven by the needs of the community. Currently, many capabilities’ development is driven by specific user requirements, which are fed back to the community after development and gradually integrated into Apache ShardingSphere.\n\nTherefore, the ShardingSphere community invited Bilibili’s tech team to get involved in terms of the SQL warm-up feature.\n\nWith the expansion of Apache ShardingSphere’s application scenarios, there is a higher expectation for ShardingSphere’s capabilities to adapt to a variety of special business scenarios. In the previous “Enterprise Visits” series, the ShardingSphere community recognized that even though it has more than 100 feature modules, enterprises from various verticals have different expectations from ShardingSphere.\n\n## Bilibili’s circuit breaker automation in traffic spike scenarios:\n\nWith the growth of Bilibili’s e-commerce scale, as is often the case with many e-commerce businesses, started to adopt front-end strategies such as cash rebates and flash deals in order to improve user retention.\n\nOn the backend, however, when the rebate performance of the warehouse is equal to the number of concurrent transactions while exceeding its own connection number limit, it can only be stopped through manual disconnection via DBA based on the number of SKU at the business level. Manual intervention is inefficient and consumes DBAs’ energy, so [ShardingSphere-Proxy](https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/) is expected to provide an automatic circuit breaker capability.\n\nHowever, the flash deal scenario is too demanding, so [Redis](https://redis.io/) still represents a better choice. Achieving automation simply requires setting rules and thresholds.\n\nSinking data to Proxy can also achieve the capabilities of Redis, but no matter how the upper layer changes, the bottom of the database will not change. Therefore, it is better for DBAs to take the initiative to operate the circuit breaker mechanism. Otherwise, after setting a threshold, if the connection between the application and the database has a slight timeout, a large number of transactions will be cut off instantly, which can easily cause a business outage.\n\nCurrently, a better way is to dig out all kinds of key information and display it on the visual interface based on Proxy, to facilitate the real-time comparison and operation of DBA instead of achieving automation.\n\n## The Apache ShardingSphere registry:\nIn the Apache ShardingSphere architecture, the registry provides distributed governance capabilities and is fully open to users since its computing node ([Shardingsphere-proxy](https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/)) is stateless without data storage capability. Therefore, user accounts and authorization information need to be stored in the registry.\n\nConcurrently, with the help of the registry, Apache ShardingSphere can distribute information to multiple computing nodes in the cluster in real-time, greatly reducing maintenance costs for users when using the cluster and improving management efficiency.\n\nIn cluster mode, Apache ShardingSphere integrates third-party registry components [ZooKeeper](https://zookeeper.apache.org/) and [Etcd](https://etcd.io/) to achieve metadata and configuration sharing in the cluster environment. At the same time, with the help of the notification and coordination ability of the registry, it ensures the real-time synchronization of the cluster when the shared data changes. And the business will not be aware of changes from the registry.\n\n## Q&A with Bilibili\n**Q: Is there a performance loss when using JDBC to connect to the governance center?**\n\nA: No. It only connects to the governance center during initialization, and the governance center will send a push when there is a change.\n\n**Q: How does [Sysbench](https://wiki.gentoo.org/wiki/Sysbench) conduct a stress test on Apache ShardingSphere?**\n\nA: The two deployment types of Apache ShardingSphere, JDBC and Proxy, are both tested by Sysbench. ShardingSphere has a newly designed Sysbench-like Java program on the JDBC end that can be used to conduct a pressure test to JDBC and Proxy. It can also ensure that the official Sysbench and our Java program share the same standard.\n\n**Q: Can ShardingSphere converge the connection number?**\n\nA: ShardingSphere-Proxy can converge the connection number, but it will definitely lead to performance loss.\n\n**Q: Can Proxy identify slow SQL?**\n\nA: Currently, the open-source version doesn’t support this function, because most users of the open-source version are Internet enterprises with a low tolerance for performance loss. Thus, the number of probes is relatively small.\n\n**Q: Does [ElasticJob](https://shardingsphere.apache.org/elasticjob/) belong to Apache ShardingSphere?**\n\nA: ElasticJob is currently used as the migration tool by Apache ShardingSphere. Additionally, ElasticJob can also be used for liveness probes.\n\n**Q: Are Internet enterprises using Proxy on a large scale?**\n\nA: Most Internet users choose the mixed deployment model, using JDBC for development and better performance, and Proxy for management. Financial customers prefer to use Proxy because they can take Proxy as a database for unified management without additional learning costs.\n\n**Q: We are currently using ShardingSphere version 4.1.1, what does it support for transactions?**\n\nA: Both versions 4.11 and 5.1.0 support [XA](https://docs.oracle.com/database/121/TTCDV/xa_dtp.htm#TTCDV327) distributed transaction management. We plan to develop the global transaction manager ([GTM](https://docs.oracle.com/cd/E17276_01/html/programmer_reference/xa_build.html)) which is scheduled to start in the second half of this year.\n\n## Get in touch with us\nIf you have applied Apache ShardingSphere solutions in your business or if you want to quickly understand and introduce the Apache ShardingSphere 5.X ecosystem to your business, you’ll probably like for someone from our community to help you out and share the Apache ShardingSphere technology with your team.\n\nFeel free to reach out to us on one of our official community channels, such as Twitter or Slack.\n\nIf we both agree that ShardingSphere is suitable for your business scenarios, our community team will be happy to connect with you and your engineers to take their questions.\n\n**Apache ShardingSphere Project Links:**\n\n[ShardingSphere Github](https://github.com/apache/shardingsphere/issues?page=1&q=is%3Aopen+is%3Aissue+label%3A%22project%3A+OpenForce+2022%22)\n\n[ShardingSphere Twitter](https://twitter.com/ShardingSphere)\n\n[ShardingSphere Slack](https://join.slack.com/t/apacheshardingsphere/shared_invite/zt-sbdde7ie-SjDqo9~I4rYcR18bq0SYTg)\n\n[Contributor Guide](https://shardingsphere.apache.org/community/cn/contribute/)","date":"2022-06-07","author":"SphereEx","excerpt":"The ShardingSphere community visited one of its enterprise users: the major Chinese video-on-demand streaming platform - Bilibili. Originally known for anime, comics & games, Bilibili has since expanded - as did its total users.\n","metaDescription":"Bilibili is an online entertainment platform featuring videos, live broadcasting, and mobile games.","createdAt":"2022-09-26T06:51:57.325Z","updatedAt":"2022-09-27T06:59:09.655Z","publishedAt":"2022-09-27T06:59:09.651Z","locale":"en","caseStudiesType":{"data":{"id":9,"attributes":{"name":"Internet","createdAt":"2022-09-26T10:34:03.086Z","updatedAt":"2022-09-27T06:59:34.838Z","publishedAt":"2022-09-27T06:59:34.836Z","locale":"en"}}},"cover":{"data":{"id":337,"attributes":{"name":"bilibili.jpeg","alternativeText":"bilibili.jpeg","caption":"bilibili.jpeg","width":1024,"height":450,"formats":{"thumbnail":{"name":"thumbnail_bilibili.jpeg","hash":"thumbnail_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":245,"height":108,"size":10.11,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_bilibili_2865813b49.jpeg"},"large":{"name":"large_bilibili.jpeg","hash":"large_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":1000,"height":439,"size":86.56,"path":null,"url":"https://official-media.sphere-ex.com/large_bilibili_2865813b49.jpeg"},"medium":{"name":"medium_bilibili.jpeg","hash":"medium_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":750,"height":330,"size":56.83,"path":null,"url":"https://official-media.sphere-ex.com/medium_bilibili_2865813b49.jpeg"},"small":{"name":"small_bilibili.jpeg","hash":"small_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":500,"height":220,"size":30.65,"path":null,"url":"https://official-media.sphere-ex.com/small_bilibili_2865813b49.jpeg"}},"hash":"bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","size":90.83,"url":"https://official-media.sphere-ex.com/bilibili_2865813b49.jpeg","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T06:52:45.832Z","updatedAt":"2022-09-26T06:52:45.832Z"}}},"metaLogo":{"data":{"id":352,"attributes":{"name":"bilibili logo.jpg","alternativeText":"bilibili logo.jpg","caption":"bilibili logo.jpg","width":1050,"height":758,"formats":{"thumbnail":{"name":"thumbnail_bilibili logo.jpg","hash":"thumbnail_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":216,"height":156,"size":4.21,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_bilibili_logo_560e3483f4.jpg"},"large":{"name":"large_bilibili logo.jpg","hash":"large_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":1000,"height":722,"size":27.06,"path":null,"url":"https://official-media.sphere-ex.com/large_bilibili_logo_560e3483f4.jpg"},"medium":{"name":"medium_bilibili logo.jpg","hash":"medium_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":750,"height":541,"size":18.89,"path":null,"url":"https://official-media.sphere-ex.com/medium_bilibili_logo_560e3483f4.jpg"},"small":{"name":"small_bilibili logo.jpg","hash":"small_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":500,"height":361,"size":11.77,"path":null,"url":"https://official-media.sphere-ex.com/small_bilibili_logo_560e3483f4.jpg"}},"hash":"bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","size":27.87,"url":"https://official-media.sphere-ex.com/bilibili_logo_560e3483f4.jpg","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:40:54.591Z","updatedAt":"2022-09-26T08:40:54.591Z"}}},"localizations":{"data":[]}}}}},
    "staticQueryHashes": []}