{
    "componentChunkName": "component---src-templates-case-study-detail-ts",
    "path": "/case-studies/19",
    "result": {"pageContext":{"next":{"id":20,"attributes":{"feature":false,"title":"Apache ShardingSphere & Zhuanzhuan: Zhuanzhuan’s Transaction System with 100s of Millions of Records","content":"## Background and Challenges\n[Zhuanzhuan](https://www.zhuanzhuan.com/index.html) is an internet platform that allows it users to sell their second-hand stuff — sort of an eBay of the East. Its business had been booming, and with it the ordering system started to face increasing performances challenges. The order database is the cornerstone of the system, and its performance should not be underestimated.\n\n**Challenges:**\n\n- During promotions and special discount periods the burden on databases is heaby with tens of thousands of single database queries per second (qps) taking up huge database resources, and causing a significant reduction in write performance.\n- Increased data pressure, with a single database containing several large tables with hundreds of millions of data, which challenges the capacity limit of the server.\n- Overwhelming data volume, and data backup and recovery take a long time, posing high risks of data loss in extreme cases.\n\n## Why ShardingSphere?\nIn the beginning, ZhuanZhuan’s team took adjustment measures to ease the database pressure. Exmaples include:\n\n**- Optimized major transactions, reduced transactions, and even eliminated transactions**\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jgvh0xy3l3xvjkbn7c4j.png)\n \n\nWe adjusted the original transaction order by putting table generation, the core step at the end, and keeping the transaction only in the order primary database. When the operation of the main table was abnormal, dirty reads were allowed on other order-related tables.\n\n**- Order data cache**\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ueib5czc3uni28w0yk5.png)\n \n\nData consistency was the trickiest part of the cache. As order data involved account settlements and commission, non-real-time and inconsistent data would cause serious accidents.\n\nStrictly keeping cache data consistency would complex coding and reduce system concurrency. Therefore, we made some compromises on cache plans:\n\n1. Allowing direct query when cache failed.\n2. Adding version serial number, and querying the latest version’s data to ensure real-time data.\n3. Complex queries were conducted by [Elasticsearch (ES)](https://www.elastic.co/) and primary and secondary separation, and for some large tables, we adopted hot and cold data separation.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k92mmvjo5y2ldbkskqc6.png)\n \n\nThrough these optimizations, database pressure was eased. However, it still seemed overwhelming under high concurrency scenarios, such as discount season.\n\nTo fundamentally solve the performance problem of order database, ZhuanZhuan decided to adopt data sharding (database and table splitting) on the `order` database so that we wouldn’t have to worry about order capacity in the future 3–5 years.\n\nZhuangzhuang chose [ShardingSphere](https://shardingsphere.apache.org/) after comparing the efficiency, stability, learning cost and etc. of different data sharding components.\n\nAdvantages of ShardingSphere:\n\n- It provides standardized data sharding, distributed transactions and database governance, and it’s applicable in a variety of situations such as Java isomorphism, heterogeneous language and cloud native.\n- It has flexible sharding strategies, supporting multiple sharding methods.\n- It’s easy to integrate with other components and has a low level of transaction intrusions.\n- It has extensive documentation and an active community.\n\nShardingSphere initiated the Database Plus concept and adopts a plugin oriented architecture where all modules are independent of each other, allowing each to be used individually or flexibly combined.\n\nIt consists of three products, namely [JDBC](https://shardingsphere.apache.org/), [Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy) and [Sidecar ](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-sidecartodo)(Planning), which supports both independent and hybrid deployment.\n\nBelow is a feature comparison of the three products:\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7517wpdgk6gmgb0s8202.png)\n \n\nBy comparison, and considering the high order concurrency, we chose ShardingSphere-JDBC as our data sharding middleware.\n\nShardingSphere-JDBC is a lightweight Java framework, proving extra service at the JDBC layer. It directly connects to the database by the client-side, provides services by Jar package, and requires no extra deployment and reliance. It can be seen as an enhanced JDBC driver, fully compatible with JDBC and other Object-Relational Mapping(ORM) frameworks.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/luou74i1f1gpgowequk9.png)\n \n\n## Key Points in Project Implementation\n\n**- Sharding Key**\nThe current order ID is generated by `timestamp+user identification code+machine code+incremental sequence`. The user identification code is taken from bits 9 to 16 of the buyer ID, a true random number when the user ID is generated, and is thus suitable as a sharding key.\n\nChoosing user identification code as the sharding key has some advantages:\n\n- The data can be distributed as evenly as possible to each database and table.\n- Specific sharding locations can be quickly located either by order ID or user ID.\n- Data of the same buyer can be distributed to the same databases and tables, facilitating the integrated query of the buyer information.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jkilhtbnnwk9gcdn7l5x.png)\n \nThe sharding strategy: we adopt 16 databases and 16 tables. User identification codes are used to split databases, and higher 4 bits are used to split tables.\n\n**- Data Migration between Old and New Databases**\nThe migration must be online, and downtime migration cannot be accepted, as there will be new data writes during the migration process.\n\nThe data should be intact, and the migration process should be insensible to the client-side. After the migration, data in the new database should be consistent with the ones in the old databases.\n\nThe migration should allow rollback, so that when a problem occurs during the migration process, it should be able to roll back to the source database without impacting system availability.\n\nData migration steps are as follows: dual writes-> migrate historical data-> verify-> old database offline.\n\n## Effects and Benefits\n\n- It solves the problem of single database capacity limit.\n- The data volume of a single database and table is greatly reduced after sharding. The data volume of a single table is reduced from nearly a hundred million level to several millions level, which greatly improves the overall performance.\n- It reduces the risk of data losses due to oversized single databases and tables in extreme cases and eases the pressure of operation and maintenance.\nThe following is a comparison of the number of interface calls of the order placement service and the time consumed by the interface during two promotion and discount periods:\n\n**Promotion before adopting ShardingSphere**\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l8d6f5tchgq7uipwbip0.png)\n \n\n**Promotion after adopting ShardingSphere**\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3s9mtqhr0kpkecyz94kl.png)\n \n\n## Summary\nShardingSphere simplifies the development of data sharding with its well-designed architecture, highly flexible, pluggable and scalable capabilities, allowing R&D teams to focus only on the business itself, thus enabling flexible scaling of the data architecture.\n\n**Apache ShardingSphere Project Links:**\n\n[ShardingSphere Github](https://github.com/apache/shardingsphere/issues?page=1&q=is%3Aopen+is%3Aissue+label%3A%22project%3A+OpenForce+2022%22)\n\n[ShardingSphere Twitter](https://twitter.com/ShardingSphere)\n\n[ShardingSphere Slack](https://join.slack.com/t/apacheshardingsphere/shared_invite/zt-sbdde7ie-SjDqo9~I4rYcR18bq0SYTg)\n\n[Contributor Guide](https://shardingsphere.apache.org/community/cn/contribute/)","date":"2022-04-29","author":"SphereEx","excerpt":"Sales promotions drive traffic, and put great pressure on databases. Here's how \n ShardingSphere solves an e-commerce platform's data challenges created by sales promotion days.","metaDescription":"Zhuanzhuan is an online marketplace on which users can buy and sell secondhand goods.","createdAt":"2022-09-26T07:06:47.526Z","updatedAt":"2022-09-27T07:00:05.205Z","publishedAt":"2022-09-27T07:00:05.202Z","locale":"en","caseStudiesType":{"data":{"id":9,"attributes":{"name":"Internet","createdAt":"2022-09-26T10:34:03.086Z","updatedAt":"2022-09-27T06:59:34.838Z","publishedAt":"2022-09-27T06:59:34.836Z","locale":"en"}}},"cover":{"data":{"id":339,"attributes":{"name":"zhuanzhuan.png","alternativeText":"zhuanzhuan.png","caption":"zhuanzhuan.png","width":720,"height":534,"formats":{"thumbnail":{"name":"thumbnail_zhuanzhuan.png","hash":"thumbnail_zhuanzhuan_a2afd90ff3","ext":".png","mime":"image/png","width":210,"height":156,"size":11,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_zhuanzhuan_a2afd90ff3.png"},"small":{"name":"small_zhuanzhuan.png","hash":"small_zhuanzhuan_a2afd90ff3","ext":".png","mime":"image/png","width":500,"height":371,"size":36.3,"path":null,"url":"https://official-media.sphere-ex.com/small_zhuanzhuan_a2afd90ff3.png"}},"hash":"zhuanzhuan_a2afd90ff3","ext":".png","mime":"image/png","size":47.13,"url":"https://official-media.sphere-ex.com/zhuanzhuan_a2afd90ff3.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T07:07:25.422Z","updatedAt":"2022-09-26T07:07:25.422Z"}}},"metaLogo":{"data":{"id":354,"attributes":{"name":"zhuanzhuan logo.png","alternativeText":"zhuanzhuan logo.png","caption":"zhuanzhuan logo.png","width":130,"height":130,"formats":null,"hash":"zhuanzhuan_logo_fd59d35610","ext":".png","mime":"image/png","size":1.85,"url":"https://official-media.sphere-ex.com/zhuanzhuan_logo_fd59d35610.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:45:59.332Z","updatedAt":"2022-09-26T08:45:59.332Z"}}},"localizations":{"data":[]}}},"prev":{"id":18,"attributes":{"feature":false,"title":"Apache ShardingSphere & Bilibili","content":"> To further understand application scenarios, and enterprises’ needs, and improve dev teams’ understanding of Apache ShardingSphere, our community launched the “Enterprise Visits”\n> series.\n\n\n[Apache ShardingSphere](https://shardingsphere.apache.org/)’s core contributor team was invited to the headquarters of [Bilibili](https://www.bilibili.com/) in Shanghai. Our PMC Chair, Zhang Liang, discussed with Bilibili’s tech team the e-commerce and digital entertainment verticals application scenarios, and the capabilities of different versions of ShardingSphere.\n\nWith the unprecedented growth of data volume and increasingly diversified data application scenarios, different platforms, businesses, and scenarios have resulted in database applications’ fragmentation. The challenges facing databases today are totally different from the ones they were conceived to solve. This is also reflected in the growth of Bilibili’s e-commerce business.\n\nFor over a decade, Bilibili has built an ecosystem community centered on users, creators, and content, while producing high-quality videos as well. With an increasing number of users, Bilibili has also gradually developed peripheral business ecosystems such as its subscription revenue model. The expansion of business lines and application scenarios brought great challenges for Bilibili’s tech team, especially in the management and application of backend data.\n\nDuring the visit, our community and Bilibili mainly discussed these three points:\n\n## The SQL warm-up feature jointly built with Bilibili:\nWhen Bilibili queries goods and orders with batch priority in its business, it is often necessary to manually warm-up SQL during the initial linking process to improve overall performance. However, in the process of manual warm-up, parameters of different lengths cannot be used as a SQL to warm-up due to the use of `foreach` syntax of the [Mybatis](https://mybatis.org/mybatis-3/index.html) framework.\n\nApache ShardingSphere can see the SQL execution plan through the `preview`, thus warming up the SQL. In the future, we plan to provide a separate SQL warm-up interface and merge the SQL warm-up time into the startup time. ShardingSphere will start on its own after SQL is warmed up.\n\n[Apache ShardingSphere](https://shardingsphere.apache.org/) is an open-source project driven by the needs of the community. Currently, many capabilities’ development is driven by specific user requirements, which are fed back to the community after development and gradually integrated into Apache ShardingSphere.\n\nTherefore, the ShardingSphere community invited Bilibili’s tech team to get involved in terms of the SQL warm-up feature.\n\nWith the expansion of Apache ShardingSphere’s application scenarios, there is a higher expectation for ShardingSphere’s capabilities to adapt to a variety of special business scenarios. In the previous “Enterprise Visits” series, the ShardingSphere community recognized that even though it has more than 100 feature modules, enterprises from various verticals have different expectations from ShardingSphere.\n\n## Bilibili’s circuit breaker automation in traffic spike scenarios:\n\nWith the growth of Bilibili’s e-commerce scale, as is often the case with many e-commerce businesses, started to adopt front-end strategies such as cash rebates and flash deals in order to improve user retention.\n\nOn the backend, however, when the rebate performance of the warehouse is equal to the number of concurrent transactions while exceeding its own connection number limit, it can only be stopped through manual disconnection via DBA based on the number of SKU at the business level. Manual intervention is inefficient and consumes DBAs’ energy, so [ShardingSphere-Proxy](https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/) is expected to provide an automatic circuit breaker capability.\n\nHowever, the flash deal scenario is too demanding, so [Redis](https://redis.io/) still represents a better choice. Achieving automation simply requires setting rules and thresholds.\n\nSinking data to Proxy can also achieve the capabilities of Redis, but no matter how the upper layer changes, the bottom of the database will not change. Therefore, it is better for DBAs to take the initiative to operate the circuit breaker mechanism. Otherwise, after setting a threshold, if the connection between the application and the database has a slight timeout, a large number of transactions will be cut off instantly, which can easily cause a business outage.\n\nCurrently, a better way is to dig out all kinds of key information and display it on the visual interface based on Proxy, to facilitate the real-time comparison and operation of DBA instead of achieving automation.\n\n## The Apache ShardingSphere registry:\nIn the Apache ShardingSphere architecture, the registry provides distributed governance capabilities and is fully open to users since its computing node ([Shardingsphere-proxy](https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/)) is stateless without data storage capability. Therefore, user accounts and authorization information need to be stored in the registry.\n\nConcurrently, with the help of the registry, Apache ShardingSphere can distribute information to multiple computing nodes in the cluster in real-time, greatly reducing maintenance costs for users when using the cluster and improving management efficiency.\n\nIn cluster mode, Apache ShardingSphere integrates third-party registry components [ZooKeeper](https://zookeeper.apache.org/) and [Etcd](https://etcd.io/) to achieve metadata and configuration sharing in the cluster environment. At the same time, with the help of the notification and coordination ability of the registry, it ensures the real-time synchronization of the cluster when the shared data changes. And the business will not be aware of changes from the registry.\n\n## Q&A with Bilibili\n**Q: Is there a performance loss when using JDBC to connect to the governance center?**\n\nA: No. It only connects to the governance center during initialization, and the governance center will send a push when there is a change.\n\n**Q: How does [Sysbench](https://wiki.gentoo.org/wiki/Sysbench) conduct a stress test on Apache ShardingSphere?**\n\nA: The two deployment types of Apache ShardingSphere, JDBC and Proxy, are both tested by Sysbench. ShardingSphere has a newly designed Sysbench-like Java program on the JDBC end that can be used to conduct a pressure test to JDBC and Proxy. It can also ensure that the official Sysbench and our Java program share the same standard.\n\n**Q: Can ShardingSphere converge the connection number?**\n\nA: ShardingSphere-Proxy can converge the connection number, but it will definitely lead to performance loss.\n\n**Q: Can Proxy identify slow SQL?**\n\nA: Currently, the open-source version doesn’t support this function, because most users of the open-source version are Internet enterprises with a low tolerance for performance loss. Thus, the number of probes is relatively small.\n\n**Q: Does [ElasticJob](https://shardingsphere.apache.org/elasticjob/) belong to Apache ShardingSphere?**\n\nA: ElasticJob is currently used as the migration tool by Apache ShardingSphere. Additionally, ElasticJob can also be used for liveness probes.\n\n**Q: Are Internet enterprises using Proxy on a large scale?**\n\nA: Most Internet users choose the mixed deployment model, using JDBC for development and better performance, and Proxy for management. Financial customers prefer to use Proxy because they can take Proxy as a database for unified management without additional learning costs.\n\n**Q: We are currently using ShardingSphere version 4.1.1, what does it support for transactions?**\n\nA: Both versions 4.11 and 5.1.0 support [XA](https://docs.oracle.com/database/121/TTCDV/xa_dtp.htm#TTCDV327) distributed transaction management. We plan to develop the global transaction manager ([GTM](https://docs.oracle.com/cd/E17276_01/html/programmer_reference/xa_build.html)) which is scheduled to start in the second half of this year.\n\n## Get in touch with us\nIf you have applied Apache ShardingSphere solutions in your business or if you want to quickly understand and introduce the Apache ShardingSphere 5.X ecosystem to your business, you’ll probably like for someone from our community to help you out and share the Apache ShardingSphere technology with your team.\n\nFeel free to reach out to us on one of our official community channels, such as Twitter or Slack.\n\nIf we both agree that ShardingSphere is suitable for your business scenarios, our community team will be happy to connect with you and your engineers to take their questions.\n\n**Apache ShardingSphere Project Links:**\n\n[ShardingSphere Github](https://github.com/apache/shardingsphere/issues?page=1&q=is%3Aopen+is%3Aissue+label%3A%22project%3A+OpenForce+2022%22)\n\n[ShardingSphere Twitter](https://twitter.com/ShardingSphere)\n\n[ShardingSphere Slack](https://join.slack.com/t/apacheshardingsphere/shared_invite/zt-sbdde7ie-SjDqo9~I4rYcR18bq0SYTg)\n\n[Contributor Guide](https://shardingsphere.apache.org/community/cn/contribute/)","date":"2022-06-07","author":"SphereEx","excerpt":"The ShardingSphere community visited one of its enterprise users: the major Chinese video-on-demand streaming platform - Bilibili. Originally known for anime, comics & games, Bilibili has since expanded - as did its total users.\n","metaDescription":"Bilibili is an online entertainment platform featuring videos, live broadcasting, and mobile games.","createdAt":"2022-09-26T06:51:57.325Z","updatedAt":"2022-09-27T06:59:09.655Z","publishedAt":"2022-09-27T06:59:09.651Z","locale":"en","caseStudiesType":{"data":{"id":9,"attributes":{"name":"Internet","createdAt":"2022-09-26T10:34:03.086Z","updatedAt":"2022-09-27T06:59:34.838Z","publishedAt":"2022-09-27T06:59:34.836Z","locale":"en"}}},"cover":{"data":{"id":337,"attributes":{"name":"bilibili.jpeg","alternativeText":"bilibili.jpeg","caption":"bilibili.jpeg","width":1024,"height":450,"formats":{"thumbnail":{"name":"thumbnail_bilibili.jpeg","hash":"thumbnail_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":245,"height":108,"size":10.11,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_bilibili_2865813b49.jpeg"},"large":{"name":"large_bilibili.jpeg","hash":"large_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":1000,"height":439,"size":86.56,"path":null,"url":"https://official-media.sphere-ex.com/large_bilibili_2865813b49.jpeg"},"medium":{"name":"medium_bilibili.jpeg","hash":"medium_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":750,"height":330,"size":56.83,"path":null,"url":"https://official-media.sphere-ex.com/medium_bilibili_2865813b49.jpeg"},"small":{"name":"small_bilibili.jpeg","hash":"small_bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","width":500,"height":220,"size":30.65,"path":null,"url":"https://official-media.sphere-ex.com/small_bilibili_2865813b49.jpeg"}},"hash":"bilibili_2865813b49","ext":".jpeg","mime":"image/jpeg","size":90.83,"url":"https://official-media.sphere-ex.com/bilibili_2865813b49.jpeg","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T06:52:45.832Z","updatedAt":"2022-09-26T06:52:45.832Z"}}},"metaLogo":{"data":{"id":352,"attributes":{"name":"bilibili logo.jpg","alternativeText":"bilibili logo.jpg","caption":"bilibili logo.jpg","width":1050,"height":758,"formats":{"thumbnail":{"name":"thumbnail_bilibili logo.jpg","hash":"thumbnail_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":216,"height":156,"size":4.21,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_bilibili_logo_560e3483f4.jpg"},"large":{"name":"large_bilibili logo.jpg","hash":"large_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":1000,"height":722,"size":27.06,"path":null,"url":"https://official-media.sphere-ex.com/large_bilibili_logo_560e3483f4.jpg"},"medium":{"name":"medium_bilibili logo.jpg","hash":"medium_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":750,"height":541,"size":18.89,"path":null,"url":"https://official-media.sphere-ex.com/medium_bilibili_logo_560e3483f4.jpg"},"small":{"name":"small_bilibili logo.jpg","hash":"small_bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","width":500,"height":361,"size":11.77,"path":null,"url":"https://official-media.sphere-ex.com/small_bilibili_logo_560e3483f4.jpg"}},"hash":"bilibili_logo_560e3483f4","ext":".jpg","mime":"image/jpeg","size":27.87,"url":"https://official-media.sphere-ex.com/bilibili_logo_560e3483f4.jpg","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:40:54.591Z","updatedAt":"2022-09-26T08:40:54.591Z"}}},"localizations":{"data":[]}}},"article":{"id":19,"attributes":{"feature":false,"title":"Apache ShardingSphere & Huimin: Huimin’s Transaction Middle Platform Architecture","content":"The FMCG (fast moving consumer-goods) industry can no longer meet current development requirements. The traditional way of only upgrading the end of the supply chain cannot achieve high-quality business development. [Huimin](https://huimin.cn/) focuses on order fulfillment and delivery service for over one million community supermarkets, transforming and upgrading from serving retail terminal stores to promoting the digitization of the whole FMCG industry chain.\n\nIt is committed to building a B2B2C data closed-loop operating system, providing brand owners, distributors, and community supermarkets with transformed and upgraded solutions for the whole chain covering purchasing, marketing, operation, and distribution. All these commitments cannot be fulfilled without its powerful “middle platform” system.\n\n## Introducing the “middle platform”\nHuimin began to transferred [PHP](https://www.php.net/) technology stack to the Java technology stack, and shifted to microservices from 2016 to 2017, and started to build the “middle platform” in 2018.\n\nThe phase-one project of its transaction middle platform went live in early 2021, and phase-two in March 2022.\n\nOwing to business growth and various changes, Huimin launched a strong “middle platform” development strategy to improve efficiency and reduce costs. Measures were taken to reduce the coupling degree between systems, increase scalability, and ensure low latency while extracting business commonalities. At the beginning of 2021, the company finihsed rebuilding the previous order management system (OMS) and launched a packaged business capability project for a transaction middle platform centered on the transaction process of orders, which is divided into the following four core modules: order status flow system, order management system, order fulfillment system and order cost calculation system. At the same time, based on business stratification, the project is also split into an order query system, report system, and order management backend.\n\nTo synergize technical and business requirements during the rebuilding process, the go-live process was divided into two phases: application system splitting and data splitting.\n\n## System splitting\nR&D teams having to frequently adjust order management system and adapt it to multiple business lines may cause inefficiencies. In response to these problems, the following principles were formulated to split and rebuild the system:\n\n1. Avoid the logic coupling of different lines of business. Different business lines should contain and minimize influence on each other to reduce logic complexity.\n2. Responsibilities are divided according to the order lifecycle to ensure that each system is independent with an exclusive responsibility.\n3. Read/write splitting. Ensure the stability of core functions and avoid the impact of frequent iterations on core functions.\n4. [ElasticSearch](https://www.elastic.co/) was introduced to solve the external index problem, and reduce database index pressure.\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1223xl1pboe6emvrjian.png)\n \n\n## Data splitting\nThe purpose of data splitting is to reduce the maintenance pressure of a single table. When the amount of data reaches tens of millions, the index and field maintenance of database tables will have a great impact on the online environment. Before splitting a table, you have to consider the following questions:\n\n1. Splitting rules: what tables are data-intensive? What about the number of splits?\n2. Data read/write: how to solve the read/write problem after splitting tables?\n3. Go-live solution: how to go live smoothly?\n\n**Splitting rules**\n\nDuring a 3 years iteration period, the number of splits = ((amount of data added per month * 36) + Amount of historical data) / maximum amount of data in a single table.\n\nWhen we use the [InnoDB](https://en.wikipedia.org/wiki/InnoDB) engine for DDL operation to maintain [Alibaba Cloud](https://www.alibabacloud.com/?spm=a2796.143805.6791778070.2.b8fa46e3JLecKh) RDS MySQL, we come up with the following results:\n\nWhen the amount of data is less than 5 million, the execution time is about tens of seconds. Between 5 million and less than 10 million, the execution time is about 500 seconds; between 10 million and less than 50 million, the execution time is about 1000 seconds; and when more than 50 million, the execution time is more than 2000 seconds.\n\nThe upper limit for splitting a single table depends on the impact of a single table operation on business. Of course, some solutions can tackle the problem of table lock resulting from DDL operation. For example, dual-table synchronization and changes in table names can reduce the impact to within seconds.\n\nThe split number of the library table is preferably 2 to the power of N, which is conducive to later horizontal scaling.\n\n**Technology selection**\n\nAfter the data table is split, data hashing and query issues need to be resolved. As a SME (small & medium-sized enterprise), the cost of developing a set of sharding middleware is too high, and it will cause various risks in the early stage. A reasonable solution is to adopt a set of open source database sharding middleware.\n\n**[Apache ShardingSphere](https://shardingsphere.apache.org/)**\n\nApache ShardingSphere is an open source ecosystem that allows you to transform any database into a distributed database system. The project includes a JDBC, a Proxy and Sidecar (Planning), which can be deployed independently and support mixed deployment. They all provide functions of data scale-out, distributed transactions and distributed governance.\n\nApache ShardingSphere 5.x introduces the concept of [Database Plus](https://faun.pub/whats-the-database-plus-concepand-what-challenges-can-it-solve-715920ba65aa), and the new version focuses on a plugin oriented architecture. Currently, functions including data sharding, read/write splitting, data encryption, shadow library pressure testing, as well as SQL and protocol support such as [MySQL](https://www.mysql.com/), [PostgreSQL](https://www.postgresql.org/), [SQL Server](https://www.microsoft.com/en-us/sql-server/sql-server-downloads), [Oracle](https://www.oracle.com/index.html), etc. are woven into the project through plugins, allowing developers to customize their own unique systems.\n\nWhy Huimin chose Apache ShardingSphere:\n\n1. Apache ShardingSphere has a variety of functions and can solve current challenges with strong scalability.\n\n2. The Apache ShardingSphere community is very active, which means the possibility to find support quickly.\n\n3. The company adopts the SpringCloud technology stack which is convenient for integration and can lower costs.\n\n4. Its performance can fully support Huimin’s existing business.\n\nShardingSphere supports the following 3 modes:\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/19343yqtstyzsmvtu69j.png)\n \n\nCurrently, Huimin’s server-side technology stack only involves Java language, so there is no need to consider heterogeneous scenarios. Taking flexibility, code intrusion and deployment cost into consideration, the implementation of [ShardingSphere-JDBC](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-jdbc) wasfinally selected.\n\n**Technology implementation**\n\nThe most complicated part of the process is actually the go-live part, not the plugin integration part as would someone expect.\n\nTo achieve non-perception and rollback, we divided the go-live process into the following steps, as shown in the figure:\n![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dyrbcvaql5nhnw4m0xns.png)\n \n\n**Step 1:** Processing all the data needs to ensure the data consistency between the old and new libraries. Since there currently is no synchronization tool supporting data sharding policy, we developed a data synchronization tool to support the configuration of the sharding policy.\n\n**Step 2:** Process incremental data to keep data consistency between the old and new libraries. Our implementation scheme here was to use the open source component, to listen to the binlog of the database and synchronize the database changes to the message queue, and then the data synchronization tool listens to the messages and writes them into the new library.\n\nThe architecture scheme is not complicated, but great importance was attached to the data consistency in this process. Incremental data cannot be lost, and single-line data should be written in sequential order to prevent ABA problems.\n\n**Step 3:** Process the gray-release of read-traffic. At this point, we were writing an old library and reading a new library in the database. In this process, we should paid attention to the sensitivity of business to data consistency, because the delay from an old library to a new library is within seconds. Reading an old library is needed when we come across some sensitive scenarios requiring high consistency.\n\n**Step 4:** Switch all read-traffic of the application to the new library, maintaining the status of writing the old library and reading the new library.\n\n**Step 5:** Processing write-traffic didn’t take gray-release scenarios into consideration in order to reduce the complexity of the program. Of course, the grayscale scheme can also achieve the same effect. We supported the rollback strategy by writing data from the new library back to the old library, and then uniformly transferring all traffic to the new library within one release.\n\nSo far, the whole process of release and going live has been completed. Due to the low cost of completing the transformation with the help of ShardingSphere-JDBC middleware to rewrite SQL and merge results, there was almost no code intrusion and going live went smoothly.\n\nThere are also some issues that you may need to be aware of when accessing the ShardingSphere-JDBC scheme:\n\n1. ShardingSphere-JDBC supports a default processing for data insertion logic. If there is no shard column in SQL, the same data will be inserted into each shard table.\n\n2. ShardingSphere-JDBC parses only the logil table of the first SQL for batches of SQL semantic analysis, which results in an error. This problem has been reported and should be fixed in the near future. Additionally, a detailed SQL sample is provided, which lists in detail the scope of support.\n\n3. ShardingSphere connection mode selection. The number of database connections visited by businesses should be limited for the sake of resource control. It can effectively prevent a service operation from occupying too many resources, which depletes the database connection resources and affects the normal access of other services. Especially in the case of many sub-tables in a database instance, a logical SQL without shard keys will produce a large number of actual SQL that falls on different tables in the same library. If each actual SQL occupies a separate connection, a query will undoubtedly occupy too many resources.\n\nFrom the perspective of execution efficiency, maintaining a separate database connection for each shard query can make more effective use of multi-threading to improve execution efficiency. Building a separate thread for each database connection allows I/O consumption to be processed concurrently, maintains a separate database connection for each shard, and avoids premature loading of query result data into memory. A separate database connection can hold a reference to the cursor location of the query result set and move the cursor when it needs to fetch the corresponding data.\n\nThe result merging method by moving the result set cursor down is called streaming merge. It does not have to load all the result data into memory, which can effectively save memory resources and reduce the frequency of garbage collection. If each shard query cannot be guaranteed to have a separate database connection, the current query result set needs to be loaded into memory before reusing the database connection to obtain the query result set of the next shard table. Therefore, even if streaming merge could be used, it would degenerate into a memory merge in this scenario.\n\nIt is a problem for the ShardingSphere execution engine to balance the relationship between the control and protection of database connection resources and saving middleware memory resources by adopting a better merging mode. Specifically, if a SQL needs to operate on 200 tables under a database instance after its sharding in ShardingSphere, whether to create 200 connections and execute them concurrently or create one connection and execute it in sequence? How to choose between efficiency and resource control?\n\nIn response to the above scenario, ShardingSphere provides a solution. It introduces the concept of Connection Mode and divides it into `MEMORY_STRICTLY` mode and `CONNECTION_STRICTLY` mode.\n\n`MEMORY_STRICTLY`\n\nThe premise of using this pattern is that ShardingSphere does not limit the number of database connections required by an operation. If the actual SQL executed needs to operate on 200 tables in a database instance, a new database connection is created for each table and processed concurrently through multi-threading to maximize the execution efficiency. When SQL meets the conditions, streaming merge is preferred to prevent memory overflow or frequent garbage collection.\n\n`CONNECTION_STRICTLY`\n\nThe premise for using this pattern is that ShardingSphere strictly limits the number of database connections required for an operation. If the actual SQL executed needs to operate on 200 tables in a database instance, only one single database connection is created and its 200 tables are processed sequentially. If shards in an operation are scattered across different databases, multi-threading is still used to handle operations on different libraries, but only one database connection is created for each operation of each library. The practice prevents the problem of consuming too many database connections for one request. This pattern adopts memory merge all the time.\n\nThe `MEMORY_STRICTLY` mode is applicable for OLAP operations and can improve system throughput by relaxing limits on database connections. The `CONNECTION_STRICTLY` mode applies to OLTP operations, which usually have shard keys and are routed to a single shard. Therefore, it is a wise choice to strictly control database connections to ensure that database resources in an online system can be used by more applications.\n\nWe found that in the `MEMORY_STRICTLY` mode, the operation would become I/O-intensive due to the cache buffer loading strategy of MySQL InnoDB engine, resulting in SQL timeouts. Its solution is to add another layer of processing without changing database resources. If there is no shard key, verify the shard key in the external index, and then use the shard key to perform database retrieval.\n\n## ShardingSphere Benefits\n**1. Performance improvement**\nThrough architecture rebuilding, it can effectively control the amount of single table data, and greatly reduce the slow SQL, down nearly 50%.\n\n**2. Save R&D resources and lower cost**\n\nThe introduction of Apache ShardingSphere does not require the re-development of sharding components, which reduces the R&D cost and lowers the risks.\n\n**3. Strong scalability**\n\nApache ShardingSphere has good scalability in terms of data encryption, distributed transactions, shadow library and other aspects.\n\n**Conclusion**\nIn his book [Building Microservices, Sam Newman](https://samnewman.io/books/building_microservices/) writes, “ Our requirements shift more rapidly than they do for people who design and build buildings — as do the tools and techniques at our disposal. The things we create are not fixed points in time. Once launched into production, our software will continue to evolve as the way it is used changes. For most things we create, we have to accept that once the software gets into the hands of our customers we will have to react and adapt, rather than it being a never-changing artifact. Thus, our architects need to shift their thinking away from creating the perfect end product, and instead focus on helping create a framework in which the right systems can emerge, and continue to grow as we learn more (Sam Newman, 2015: 15).” Apache ShardingSphere is doing exactly what the book says, and is definitely on a promising path.\n\n## Author\n**Zhai Yang**\n\nSoftware architect at Zhongshang Huimin E-Commerce platform team.\n\nParticipated in the 0–1 process of middle platform construction, mainly responsible for the R&D and building the transaction and commodity middle platform and the search platform.","date":"2022-05-17","author":"SphereEx","excerpt":"HuiMin does order fulfillment for over 1 million supermarkets. With ShardingSphere HuiMin shifted to microservices & built a new \"middle platform\" architecture with packaged business capability for order transaction processing.","metaDescription":"Huimin is a B2B e-commerce platform that focuses on small-scale supermarkets that provides community supermarket websites for communities.","createdAt":"2022-09-26T07:01:29.374Z","updatedAt":"2022-09-27T06:59:59.417Z","publishedAt":"2022-09-27T06:59:59.413Z","locale":"en","caseStudiesType":{"data":{"id":2,"attributes":{"name":"E-commerce","createdAt":"2021-12-11T23:02:08.395Z","updatedAt":"2021-12-11T23:02:11.020Z","publishedAt":"2021-12-11T23:02:10.947Z","locale":"en"}}},"cover":{"data":{"id":338,"attributes":{"name":"Zhongshanghuimin.png","alternativeText":"Zhongshanghuimin.png","caption":"Zhongshanghuimin.png","width":720,"height":339,"formats":{"thumbnail":{"name":"thumbnail_Zhongshanghuimin.png","hash":"thumbnail_Zhongshanghuimin_6ceb4a8b12","ext":".png","mime":"image/png","width":245,"height":115,"size":19.04,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_Zhongshanghuimin_6ceb4a8b12.png"},"small":{"name":"small_Zhongshanghuimin.png","hash":"small_Zhongshanghuimin_6ceb4a8b12","ext":".png","mime":"image/png","width":500,"height":235,"size":60.18,"path":null,"url":"https://official-media.sphere-ex.com/small_Zhongshanghuimin_6ceb4a8b12.png"}},"hash":"Zhongshanghuimin_6ceb4a8b12","ext":".png","mime":"image/png","size":73.86,"url":"https://official-media.sphere-ex.com/Zhongshanghuimin_6ceb4a8b12.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T07:01:49.173Z","updatedAt":"2022-09-26T07:01:49.173Z"}}},"metaLogo":{"data":{"id":353,"attributes":{"name":"zhongshanghuimin logo.png","alternativeText":"zhongshanghuimin logo.png","caption":"zhongshanghuimin logo.png","width":608,"height":430,"formats":{"thumbnail":{"name":"thumbnail_zhongshanghuimin logo.png","hash":"thumbnail_zhongshanghuimin_logo_978897c9d6","ext":".png","mime":"image/png","width":221,"height":156,"size":9.77,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/thumbnail_zhongshanghuimin_logo_978897c9d6.png"},"small":{"name":"small_zhongshanghuimin logo.png","hash":"small_zhongshanghuimin_logo_978897c9d6","ext":".png","mime":"image/png","width":500,"height":354,"size":31.4,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/small_zhongshanghuimin_logo_978897c9d6.png"}},"hash":"zhongshanghuimin_logo_978897c9d6","ext":".png","mime":"image/png","size":12.04,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/zhongshanghuimin_logo_978897c9d6.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:42:11.374Z","updatedAt":"2022-10-12T05:56:46.710Z"}}},"localizations":{"data":[]}}}}},
    "staticQueryHashes": []}