{
    "componentChunkName": "component---src-templates-case-study-detail-ts",
    "path": "/case-studies/6",
    "result": {"pageContext":{"next":{"id":7,"attributes":{"feature":false,"title":"eHualu Story’s Data Processing Shortcut","content":"> ShardingSphere has greatly simplified sharding development and maintenance, which has played a very important role in our rapid product release. We estimate that ShardingSphere has saved us at least 4 months of R&D costs.\n> \n> –– Shi Moxuan, Technical Director，E-Hualu\n\nAt the beginning of this year, E-hualu officially launched its cloud service product “Hulu App” that targets non-business users. Since then, its architecture’s data stress has increased on a daily basis.\n\nThe R&D team of Hulu App chose to use ShardingSphere’s sharding features and scale their data out. The flexible and agile features of ShardingSphere met the requirements of Hulu App. It requires data extensibility, and needs to prevent the team from “reinventing the wheel”.\n\nShardingSphere can simplify sharding development and maintenance to the greatest extent. As the service is growing, the work will be increasingly complicated.\n\n### Hulu App’s growth pressure: feature extension and new services\n\nThe high requirement for data storage and computing is in the genes of the Hulu App technical team.\n\nBecause Hulu App is rapidly growing, its service and feature changes are relatively frequent. Therefore its technical team needs to adapt quickly when a front-end service changes. As its users and service data volume are growing rapidly, its databases are faced with greater pressure.\n\nThe official Hulu App was released on May 17, 2020, and then its user data and service volume also showed a rapid growth trend. Therefore it was inevitable to scale out its databases for many times. At the same time, with business requirements changing very quickly, new challenges have emerged:\n\n- How to ensure feature extensibility\n\nBecause of increasing users and developing product versions, its user data has exploded. The storage and computing capability of its old architecture has been severely challenged. Therefore, Hulu App requires that its back-end data processing platform shall have both extensible features, and flexibility.\n\n- How to improve efficiency\n\nFaced with its changeable services, the R&D team of Hulu App needs to make rapid adjustments when necessary, so as to make its architecture adapt to its services. A highly-flexible, extensible data architecture can greatly improve development efficiency of the team. On the other hand, because of such large amounts of data, database retrieval efficiency will inevitably have problems such as delays, slow read and write. These problems will ultimately affect user experience.\n\n- How to facilitate feature release\n\nAll new products will face problems like frequent new feature release, bringing the scheduled release date forward, when they are just released. These problems pose a great challenge to a development team. At first, Hulu’s development team planned to base on their service demands and create their own sharding plan all by themselves. However, they don’t have enough time to do that, so it’s urgent for them to choose a new sharding plan.\n\n- How to guarantee system stability\n\nA newly-added technology will greatly challenge system stability. It’s especially true for bottom-level technologies, most of which can influence platform services and probably challenge system stability. So what Hulu App needs is a data application product that does not greatly stress databases, but quickly adapts to them and that is also highly stable.\n\n### Leverage ShardingSphere’s features to Create a Flexible, Highly-Available Data Architecture Solution\n\nHulu App’s team studied their needs and spent two weeks evaluating ShardingSphere and other similar solutions. They had evaluative indicators like product feature, maturity, stability and performance. At last, they chose ShardingSphere because of its powerful features, its support and its high maturity can fully meet their needs.\n\nThe Hulu team deployed ShardingSphere above Alibaba Cloud’s RDS. Instead of adjusting ontology databases, the Hulu team prefers to conduct data governance above its databases. ShardingSphere can effectively improve Hulu’s data architecture, because it can achieve three things: high scalability brought by its extensible architecture, closely adaptive to Hulu’s service. ShardingSphere brings Hulu with obvious improvement:\n\n#### “High scalability” of the pluggable architecture\n\nDue to its service characteristics, Hulu App’s limited storage space is consumed very quickly, and the problem gradually begins to affect response efficiency at the front end. Therefore, the team can quickly develop enhanced features by using ShardingSphere, which can provide optimized solutions for its subsequent architectural adjustments. The example strengthens the ShardingSphere’s advantage in sharding.\n\n#### Closely adaptive to service\n\nThe smaller the architecture change is, the more controllable it is for developers. ShardingSphere, an ecosystem located above databases, is closer to service and it deployment is even more lightweight. So it is undoubtedly the best solution to solve the contradiction between the front-end service changes and back-end architecture adjustment of Hulu App. In addition, in terms of flexibility, ShardingSphere supports instant achievement of relevant configurations, which greatly helps the team save a lot of sharding-related work. So it plays a very important role in Hulu App’s quick releases.\n\n#### “Zero attack” to the service architecture\n\nHulu App chose ShardingSphere’s Proxy deployment mode. It can manage the real database cluster through Proxy without. replacing underlying databases. It basically completed the separation of service and data at the architectural level without any changes. So it can avoid the risks caused by database changes, such as unavailable service and long stable cycle. In addition, the non-status mode of ShardingSphere hardly has any perceptible impact on users at the front end and the service layer does not need to pay attention to the data storage method .\n\nSo, Hulu App is a product with tight go-live schedule and fast feature iteration. ShardingSphere-Proxy can reuse original databases and help the Hulu development team to develop incremental features above databases such as sharding and data encryption. In addition, ShardingSphere allows developers not to worry about database configuration and shields user perception, so it quickly builds service-oriented database direct connection, which is well separated from the system architecture level, and ensures that daily maintenance work at the database agent, such as bug repair and version update, will not affect the services.\n\nThis time, ShardingSphere collaborated with the R&D team of E-hualu’s Hulu App, and helped them overcome data volume explosion for several times. Actually, this is just a tiny example of ShardingSphere’s various application scenarios around the world.\n\n### About E-hualu & Hulu App\n\nFounded in 2001, Beijing E-hualu Information Technology Co., Ltd. is a listed company under the state-owned enterprise China Hualu Group Co., LTD. directly administrated by the State-owned Assets Supervision and Administration Commission of the State Council (SASAC). Its mission is to reduce energy consumption and the cost of long-term data management.\n\nHulu App is the first to-customer product of E-hualu.\n\n### ShardingSphere Community:\n\nShardingSphere Github: https://github.com/apache/shardingsphere\n\nShardingSphere Twitter: https://twitter.com/ShardingSphere\n\nShardingSphere Slack Channel: apacheshardingsphere.slack.com","date":"2021-11-29","author":"SphereEx","excerpt":"ShardingSphere has greatly simplified sharding development and maintenance, which has played a very important role in our rapid product release. We estimate that ShardingSphere has saved us at least 4 months of R&D costs.","metaDescription":"Founded in 2001, Beijing E-hualu Information Technology Co., Ltd. is a listed company under the state-owned enterprise China Hualu Group Co., LTD. directly administrated by the State-owned Assets Supervision and Administration Commission of the State Council (SASAC). Its mission is to reduce energy consumption and the cost of long-term data management.","createdAt":"2021-12-12T12:33:14.398Z","updatedAt":"2022-09-26T10:27:50.013Z","publishedAt":"2021-12-12T12:35:01.820Z","locale":"en","caseStudiesType":{"data":{"id":1,"attributes":{"name":"Finance","createdAt":"2021-12-11T23:01:56.177Z","updatedAt":"2021-12-11T23:01:58.635Z","publishedAt":"2021-12-11T23:01:58.567Z","locale":"en"}}},"cover":{"data":{"id":7,"attributes":{"name":"ehualu.png","alternativeText":"ehualu.png","caption":"ehualu.png","width":1030,"height":720,"formats":{"thumbnail":{"name":"thumbnail_ehualu.png","hash":"thumbnail_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":223,"height":156,"size":6.16,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_ehualu_6d29db12b5.png"},"large":{"name":"large_ehualu.png","hash":"large_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":1000,"height":699,"size":53.63,"path":null,"url":"https://official-media.sphere-ex.com/large_ehualu_6d29db12b5.png"},"medium":{"name":"medium_ehualu.png","hash":"medium_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":750,"height":524,"size":33.7,"path":null,"url":"https://official-media.sphere-ex.com/medium_ehualu_6d29db12b5.png"},"small":{"name":"small_ehualu.png","hash":"small_ehualu_6d29db12b5","ext":".png","mime":"image/png","width":500,"height":350,"size":18.22,"path":null,"url":"https://official-media.sphere-ex.com/small_ehualu_6d29db12b5.png"}},"hash":"ehualu_6d29db12b5","ext":".png","mime":"image/png","size":24.54,"url":"https://official-media.sphere-ex.com/ehualu_6d29db12b5.png","previewUrl":null,"provider":"dine-strapi-provider-upload-aws-s3","provider_metadata":null,"createdAt":"2021-12-12T12:04:05.768Z","updatedAt":"2021-12-14T07:19:57.408Z"}}},"metaLogo":{"data":{"id":3,"attributes":{"name":"ehualu_logo.png","alternativeText":"ehualu_logo.png","caption":"ehualu_logo.png","width":303,"height":153,"formats":{"thumbnail":{"name":"thumbnail_ehualu_logo.png","hash":"thumbnail_ehualu_logo_26122a7add","ext":".png","mime":"image/png","width":245,"height":124,"size":22.6,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_ehualu_logo_26122a7add.png"}},"hash":"ehualu_logo_26122a7add","ext":".png","mime":"image/png","size":22.6,"url":"https://official-media.sphere-ex.com/ehualu_logo_26122a7add.png","previewUrl":null,"provider":"dine-strapi-provider-upload-aws-s3","provider_metadata":null,"createdAt":"2021-12-12T12:04:05.036Z","updatedAt":"2021-12-14T07:19:30.734Z"}}},"localizations":{"data":[{"id":2,"attributes":{"feature":false,"title":"葫芦 App 后台数据处理的逻辑捷径","content":"> ShardingSphere 大大简化了分库分表的开发和维护工作，对于业务的快速上线起到了非常大的支撑作用，保守估计 ShardingSphere 至少为我们节省了 4 个月的研发成本。\n> \n> –– 史墨轩，易华录·技术总监\n\n### 项目背景\n\n伴随着易华录旗下面向个人用户的云服务产品【葫芦 App】正式上线，后台架构所承受的业务压力也与日俱增。\n\n为此，葫芦 App 研发团队选择采用 ShardingSphere 分库分表的功能对数据进行了横向的拆分，围绕 ShardingSphere 灵活敏捷的特性，满足了葫芦 App 业务对数据层扩展性的要求，避免团队重复“造轮子”，最大程度简化了随着业务增长而带来的愈发复杂化的分库分表的开发与维护工作。\n\n### 业务挑战\n\n- 扩展能力\n- 效率提升\n- 上线提前\n- 成熟技术\n\n### 解决方案\n\n- 可插拔架构的『高扩展性』\n\n由于业务特性，葫芦 App 原本有限的存储空间被消耗得非常快，并逐渐开始影响用户在前端的响应效率。通过采用 ShardingSphere 的分片策略，葫芦研发团队在应对海量计算+存储所带来的业务问题同时，能够确保分片扩展策略的灵活性。基于此，葫芦团队可以在ShardingSphere 上快速做出相应的功能扩展，为后续架构调整提供优化方案，进一步强化突出了 ShardingSphere 分库分表的优势。\n\n- 距离业务更紧密的『贴合性』\n\n后台架构的变化越小，对于业务而言就越可控。ShardingSphere 这种位于数据库之上的生态，距离业务更近，部署起来也更加轻量，无疑是解决葫芦 App 前台业务变更与后台架构调整之间矛盾的最优解。此外在灵活性层面，ShardingSphere 通过相关配置即可实现的方式，极大简化了葫芦研发团队在分库分表层面的研发与维护工作，对于业务的快速上线起到了非常大的支撑作用。\n\n- 对于业务架构的『零侵入性』\n\n葫芦 App 选择了 ShardingSphere-Proxy 部署模式，在不更换底层数据库的前提下通过 Proxy 来管理真实的数据库集群，基本无需对业务进行改造就已经完成了业务与数据在架构层面的分离，并避免了因更换数据库导致的业务不可用、漫长稳定周期等风险。另外，ShardingSphere 的无状态模式，几乎不会对前端用户产生任何可感知的影响，业务层也无需关注数据的存储方式。\n\n![ehualu_zh_1.png](https://official-media.sphere-ex.com/ehualu_zh_1_3fad8e2ccd.png)\n\n### 用户收益\n\n- 节省 4 人月成本\n- 明显改善业务体验\n- 系统灵活扩容\n\n**SphereEx，让数据与应用连接更简单。**","date":"2021-11-29","author":"SphereEx","excerpt":"ShardingSphere 大大简化了分库分表的开发和维护工作，对于业务的快速上线起到了非常大的支撑作用，保守估计 ShardingSphere 至少为我们节省了 4 个月的研发成本。","metaDescription":"北京易华录信息技术股份有限公司，成立于 2001 年，是国务院国资委直接监管的中央企业中国华录集团旗下控股的上市公司，致力于建设城市数字经济基础设施。","createdAt":"2021-12-11T23:11:13.132Z","updatedAt":"2022-11-30T08:25:32.894Z","publishedAt":"2021-12-11T23:11:16.643Z","locale":"zh"}}]}}},"prev":{"id":25,"attributes":{"feature":true,"title":"Apache ShardingSphere & iQiyi: What's Database Mesh?","content":"> _To further understand application scenarios, enterprises’ needs, and improve dev teams’ understanding of Apache ShardingSphere, SphereEx launched the “Enterprise Visits” series. The second stop took us to the leading online streaming platform in Asia, iQiyi._\n\nIn November 2021, a team from the ShardingSphere community visited the innovation center of iQiyi for in-depth interactions and discussions with their counterparts from Beijing and Shanghai. During the meetup, Zhang Liang, Apache ShardingSphere PMC Chair and SphereEx Founder provided details on the latest Apache ShardingSphere community initiatives, its future development, and Database Mesh.\n\nDuring the meetup, iQiyi was especially interested in the capabilities and future plans of ShardingSphere and Database Mesh. here are some key takeaway questions from the visit:\n\n**Q: How does ShardingSphere manage its encryption keys?**\n\n[Encryption](https://shardingsphere.apache.org/document/current/en/features/encrypt/) key management is a capability subordinate to the encryption algorithm. When ShardingSphere needs an encryption key in its decryption, the key could be passed through properties configured by the encryption algorithm. In principle, encryption key management is not a capability of ShardingSphere but the job of the encryption algorithm.\n\n**Q:Is there a built-in cryptographic algorithm in ShardingSphere? Or is it done through outside cooperation?**\n\nStrictly speaking, ShardingSphere is not equipped with built-in capability. That’s because the pluggable architecture of ShardingSphere is divided into multiple layers, and the encryption module itself is only a capability of L2 of the pluggable architecture.\n\nThe encryption module only defines the top-layer encryption connector and the detailed encryption algorithm achieves the pluggable part of the connector. Currently, ShardingSphere supports several of the most commonly used open source algorithms, but it does not support any national cryptographic algorithm in particular.\n\nIntegrating an encryption algorithm is easy for developers: instead of changing source codes, all they need is a ShardingSphere encryption algorithm connector. ShardingSphere’s data encryption process is completely transparent to users and very easy to use.\n\n**What’s the future plan for Database Mesh?**\n\nDatabase Mesh is another ShardingSphere initiative. We are now planning to integrate Database Mesh through Sidecar. Sidecar will be used to manage traffic, while ShardingSphere will be used to manage to compute, and the underlying databases’ nodes to manage storage.\n\nSidecar’s and ShardingSphere functions overlap in part, but not completely. ShardingSphere has more than 190 modules, therefore it’s not possible to completely replace ShardingSphere with Sidecar. That’s why ShardingSphere empowers some lightweight capabilities to Sidecar, and Sidecar is equipped with the capabilities to buffer some traffic and requests.\n\nThe database computing capability is still processed by ShardingSphere, as it would consume too many application resources to run heavyweight computation via Sidecar for the distributed transaction, query and optimization of heterogeneous databases.\n\nIn the future, other Sidecar capabilities, such as management, SQL auditing and authorities, will be developed. In this way, Sidecar and ShardingSphere will have their own focuses, providing a better solution. \n\n**Sidecar is equivalent to a data plane. Will you combine the control plane with [Istio](https://istio.io/) or will you build the control plane separately?**\n\nShardingSphere uses DistSQL (Distributed SQL) to create rules for sharding, encryption and so on. ShardingSphere is already equipped with the control plane capability of Sidecar. Any capability achieved on Sidecar can be controlled by ShardingSphere’s [DistSQL](https://opensource.com/article/21/9/distsql). In the future, we will consider integrating Sidecar into the Istio ecosystem.\n\nSo far, Apache ShardingSphere with the release of a version of 5.0 has officially repositioned itself in a new field with the Database Plus concept at its core. In addition to the powerful incremental capability at the upper layer of databases, the Database Plus architecture also opens a highly extensible database ecosystem to developers and users, charting a new future development course for Apache ShardingSphere.\n\n**Apache ShardingSphere Open Source Project Links:**\n\n[ShardingSphere Github]\n(https://github.com/apache/shardingsphere/issues?page=1&q=is%3Aopen+is%3Aissue+label%3A%22project%3A+OpenForce+2022%22)\n\n[ShardingSphere Twitter](https://twitter.com/ShardingSphere)\n\n[Contributor Guideline](https://shardingsphere.apache.org/community/cn/contribute/)","date":"2022-01-10","author":"Yacine Si Tayeb","excerpt":"Also known as the Netflix of Asia, iQiyi is an ad-supported television and movie portal providing fully-licensed, high-definition media content.","metaDescription":null,"createdAt":"2022-09-26T07:52:25.745Z","updatedAt":"2022-09-28T02:38:53.315Z","publishedAt":"2022-09-27T07:00:31.240Z","locale":"en","caseStudiesType":{"data":{"id":9,"attributes":{"name":"Internet","createdAt":"2022-09-26T10:34:03.086Z","updatedAt":"2022-09-27T06:59:34.838Z","publishedAt":"2022-09-27T06:59:34.836Z","locale":"en"}}},"cover":{"data":{"id":389,"attributes":{"name":"iqiyi2.jpg","alternativeText":"iqiyi2.jpg","caption":"iqiyi2.jpg","width":500,"height":360,"formats":{"thumbnail":{"name":"thumbnail_iqiyi2.jpg","hash":"thumbnail_iqiyi2_3b19419f07","ext":".jpg","mime":"image/jpeg","width":217,"height":156,"size":5.22,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_iqiyi2_3b19419f07.jpg"}},"hash":"iqiyi2_3b19419f07","ext":".jpg","mime":"image/jpeg","size":12.26,"url":"https://official-media.sphere-ex.com/iqiyi2_3b19419f07.jpg","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-28T02:38:45.900Z","updatedAt":"2022-09-28T02:38:45.900Z"}}},"metaLogo":{"data":null},"localizations":{"data":[]}}},"article":{"id":6,"attributes":{"feature":true,"title":"Apache ShardingSphere & JD Baitiao: Story of an Implementation Journey","content":">Apache ShardingSphere has become the best solution for JD Baitiao’s very large storage scenarios and scale-up. JD’s shopping festival weighed very much on our big data storage. I mean, like hundreds of millions of data sets.\n>\n>JD Baitiao, an internet credit products provider launched in 2014, has always been challenged by data explosion. Every shopping festival is like a technical test for us, but every time the technical staff professionals take proactive strategic initiatives in data architecture development to ensure everything goes smoothly.\n>\n>— Dongfang Zhang，JD Batiao R&D Director\n\n## The Baitiao Data Architecture Evolution\n\nLaunched in Feb. 2014, JD Baitiao’s data architecture has already evolved for several times because of rapid increase in data volume and rapid business growth.\n\n### From 2014 to 2015\n\nBaitiao first adopted the Solr + HBase solution to connect core business systems and other systems with key databases. Solr indexing is able to achieve fast search responses and HBase is used to store a massive amount of data.\n\n- A Solr cluster can help the core database reduce reads and writes stress.\n- However, Solr is not perfect: it has unsatisfying extensibility and intrusions.\n\n### From 2015 to 2016\n\nThe company chose NoSQL to store monthly data tables in Mongo DB clusters, temporarily meeting the demand for importing/exporting massive data in payment settlement scenarios.\n\n- The change made it quite efficient to query hot data. In addition, thanks to such unstructured data storage, staff can easily modify table structures.\n- Nevertheless, the plan was still problematic: bad extensibility, intrusions, and memory hog.\n\n### From 2016 to 2017\n\nGrowing business and growing data. Tens of billions of data volume imposed more performance and volume pressure on MongoDB. JD Baitiao’s Big Data Platform used DBRep and MySQL Slave to capture data changes and store the information data in its message center, to later be written to ES and HBase.\n\n- This plan was actually much better. It focused on real-time data and improved extensibility.\n- However, Baitiao faced high costs to maintain its code because of the data sharding problem in its architecture.\n\nThe architecture evolution of JD Baitiao is just an illustration of rapidly growing Internet consumer finance. All the methods Baitiao adopted in the past were not comprehensive, indicating that every solution would be outdated soon.\n\n## Time to Adopt a Decoupled Architecture\n\nTo ensure good system performance even with rapidly increasing data volumes, the technical team first chose the sharding architecture pattern that not only guaranteed excellent performance, but also kept code under control.\n\n### The data striping plan was actually based on application architecture:\n\nThe product continuously developed but the early solution had now become one of the biggest problems. The old data sharding plan makes code even more complex and increases maintenance expenditure. Developers have to spend much time adjusting sharding every time when the application upgrades, so they cannot concentrate on their own developments. In fact, tight coupling is to blame for the problem.\n\nTherefore, the team decided to use a mature sharding component that simplifies system upgrades and architecture changes. To compare Baitiao’s shard and ShardingSphere shard, their differences are shown in the following table.\n\n![jd_en_1.png](https://official-media.sphere-ex.com/jd_en_1_eb365b4a93.png)\n\nFor Baitiao, decoupling was the next task.\n\nApparently, JD Baitiao‘s data architecture will experience a brand-new journey to decoupling. The following three directions definitely fueled its transformation:\n\n**Concentration:** instead of database sharding built in its architecture, it applied a sharding component to place more energy on it own product development.\n\n**Easier Upgrade:** it used a decoupled architecture to simplify the R&D process of system upgrades.\n\n**Future-Oriented Plan:** it aims to improve system extensibility so that Baitiao will be more capable of confidently holding large online shopping festivals like the 618 Shopping Festival and Double 11 Shopping Festival (also known as Singles Day on November 11th).\n\nJD Baitiao is a huge business, so its business scenario is truly related to finance, high concurrency, and massive data volumes. Baitiao’s sharding component must have the following features:\n\n1. Be a mature product\n2. Have excellent performance\n3. Be able to handle big data\n4. Have an extensible architecture\n\n## The Apache ShardingSphere Solution\n\nThe lightweight Java framework ShardingSphere-JDBC is Apache ShardingSpheres’s first product providing services such as a Java Database Connectivity (JDBC) API. ShardingSphere JDBC uses jar archive file installation package and allows the client-side to directly connect to databases. So, it requires no extra deployment dependencies. It is like an enhanced JDBC driver fully compatible with JDBC and ORM frameworks.\n\nShardingSphere — JDBC provides the following features, making it the best solution in the Baitiao scenario.\n\nMature Product: ShardingSphere — JDBC is a mature product developed for years, and its open source community is very active.\n\nExcellent Performance: its micro-kernel and lightweight design hardly hinder performance.\n\nMinimum Changes: it supports native MySQL protocol and minimizes R&D workload.\n\nExtensibility: users can combine with the migration sync component to easily extend data.\n\n![jd_en_2.png](https://official-media.sphere-ex.com/jd_en_2_c92fb9e5ef.png)\n\nAfter being systematically tested for several times, Apache ShardingSphere became JD Baitiao’s first choice of data sharding middleware. The cooperation started at the end of 2018.\n\n## Product Adaptation\n\nTo provide Baitiao with better services and support the business, Apache ShardingSphere has made many improvements to its product features and performance during the implementation process. At the same time, the user case helps optimize the product in return.\n\n### Upgrade the SQL Engine\n\nBaitiao’s business logic is extremely complicated; its diversified scenario demands require the best SQL compatibility levels. Thus, Apache ShardingSphere restructures the SQL parser module to support more SQLs.\n\n- Single data node routing: 100% SQL compatibility\n- Multi data node routing：comprehensively support DML, DDL, DCL, TCL and partly DAL. Support features such as pagination, deduplication, ranking, grouping, aggregation and correlated query.\n\n### Distributed Key\n\nApache ShardingSphere provides built-in distributed key generators, such as UUID, and SNOWFLAKE. It also provides its distributed key generation API so that users can develop custom key generation algorithms to meet their special needs.\n\n### Shard Key Value Injection\n\nIf a SQL has no shard condition, Apache ShardingSphere uses ThreadLoad to manage shard key value, and users can program and add a shard condition to HintManager, making the condition only effective in the current thread. That’s so-called SQL zero intrusion.\n\nAdditionally, Apache ShardingSphere continues to optimize its other features in order to meet Baitiao’s need for high performance such as：\n\n- SQL parsing result cache\n- JDBC metadata cache\n- Bind table & Broadcast table\n- Automated execution engine & Stream merge\n\nJD Baitiao & ShardingSphere teams joint efforts amde all indicators of the product live up to their expectation. The resulting final performance is almost the same as that of a native JDBC.\n\n![jd_en_3.png](https://official-media.sphere-ex.com/jd_en_3_582e2afe89.png)\n\n## Cutover\n\nApache ShardingSphere uses its custom HASH strategy to shard data, effectively avoiding the hot data problem. The total of data nodes almost reaches ten thousand. The cutover process lasts for about 4 weeks.\n\n1. After DBRep reads data, Apache ShardingSphere synchronizes the data to the target database cluster.\n2. Two clusters run together. After a data migration, Baitiao uses its own tool to verify business and data.\n\nDBRep is fundamental to the product design of ShardingSphere-Scaling. The automation feature of Scaling definitely facilitates migration and scale-up.\n\n## Apache ShardingSphere Brings Many Benefits\n\n### Simplified Upgrade Path\n\nThe decoupled architecture effectively simplifies a technology stack required in a system upgrade, so the developer team no longer need to worry about table sharding. Instead, they can focus on the business itself. In brief, ShardingSphere greatly helps Baitiao optimize the upgrade path.\n\n### Save R&D Cost\n\nBaitiao directly uses the mature product Apache ShardingSphere, so they do not need to reinvent the wheel and save a lot of time and energy.\n\n### Improve Architecture Extensibility\n\nBaitiao also uses the sync migration component Scaling that makes system scale-up even more flexible. It’s the company’s secret for online shopping festival successes.\n\n## Summary\n\nThe growth of JD Baitiao’s business greatly stimulates its data architecture upgrades. This time, it chose Apache ShardingSphere to decouple its architecture and make future upgrades less complicated. Thank to this from now on, developers only have to concentrate on their own product optimization because its data architecture is extensible enough. The case is really a good example of ShardingSphere application in the consumer finance scenario.\n\nNow, there are more and more internet credit payment models. In the future, Apache ShardingSphere in collaboration with JD Technology will explore more business scenarios, and contribute finance-related technological innovations, further improving Internet finance.\n\n### About Apache ShardingSphere\n\nShardingSphere is one of the Apache Software Foundation Top-Level Open Source projects, used by over 170 enterprises worldwide, across various verticals such as finance, e-commerce, cloud services, tourism, logistics, education, and entertainment. Its GitHub community has so far cumulated over 14,000 stars.\n\nWe welcome more technical professionals to write articles and share their experiences and thoughts. If you are interested, feel free to contact us:\n\nShardingSphere Github: https://github.com/apache/shardingsphere\n\nShardingSphere Twitter: https://twitter.com/ShardingSphere\n\nShardingSphere Slack Channel: apacheshardingsphere.slack.com","date":"2021-12-01","author":"SphereEx","excerpt":"Apache ShardingSphere has become the best solution for JD Baitiao’s very large storage scenarios and scale-up. JD’s shopping festival weighed very much on our big data storage. I mean, like hundreds of millions of data sets.","metaDescription":"JD Technology Group is a business subgroup of JD Group that focuses on technology as an industry service, and is committed to providing technical products and solutions for enterprises, financial institutions, governments and other types of customers across the value chain.","createdAt":"2021-12-12T12:30:44.957Z","updatedAt":"2022-09-26T09:16:15.245Z","publishedAt":"2021-12-12T12:35:08.683Z","locale":"en","caseStudiesType":{"data":{"id":2,"attributes":{"name":"E-commerce","createdAt":"2021-12-11T23:02:08.395Z","updatedAt":"2021-12-11T23:02:11.020Z","publishedAt":"2021-12-11T23:02:10.947Z","locale":"en"}}},"cover":{"data":{"id":345,"attributes":{"name":"JDT.png","alternativeText":"JDT.png","caption":"JDT.png","width":720,"height":540,"formats":{"thumbnail":{"name":"thumbnail_JDT.png","hash":"thumbnail_JDT_2704eac855","ext":".png","mime":"image/png","width":208,"height":156,"size":8.85,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_JDT_2704eac855.png"},"small":{"name":"small_JDT.png","hash":"small_JDT_2704eac855","ext":".png","mime":"image/png","width":500,"height":375,"size":28.29,"path":null,"url":"https://official-media.sphere-ex.com/small_JDT_2704eac855.png"}},"hash":"JDT_2704eac855","ext":".png","mime":"image/png","size":14.08,"url":"https://official-media.sphere-ex.com/JDT_2704eac855.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-26T08:02:50.084Z","updatedAt":"2022-09-26T08:02:50.084Z"}}},"metaLogo":{"data":{"id":10,"attributes":{"name":"jd_logo.png","alternativeText":"jd_logo.png","caption":"jd_logo.png","width":300,"height":44,"formats":{"thumbnail":{"name":"thumbnail_jd_logo.png","hash":"thumbnail_jd_logo_591b2e5e14","ext":".png","mime":"image/png","width":245,"height":36,"size":6.43,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_jd_logo_591b2e5e14.png"}},"hash":"jd_logo_591b2e5e14","ext":".png","mime":"image/png","size":2.36,"url":"https://official-media.sphere-ex.com/jd_logo_591b2e5e14.png","previewUrl":null,"provider":"dine-strapi-provider-upload-aws-s3","provider_metadata":null,"createdAt":"2021-12-12T12:04:06.233Z","updatedAt":"2021-12-14T07:19:09.532Z"}}},"localizations":{"data":[{"id":1,"attributes":{"feature":true,"title":"Apache ShardingSphere 在京东白条场景的落地之旅","content":"\n>京东白条使用 Apache ShardingSphere 解决了千亿数据存储和扩容的问题，为大促活动奠定了基础。\n>\n>2014 年初，“京东白条”作为业内互联网信用支付产品，数据量爆发式的增长，每一次大促备战都是对技术人员的考验，每一次的战略转型驱动着数据架构的成长。\n>\n>— 张栋芳，京东白条研发负责人\n\n### 项目背景\n\n自 2014 年 2 月京东白条业务上线起，为满足快速发展的业务和激增的海量数据，白条的数据架构经历了数次演进。\n\n白条数据架构的演进间接地反应了互联网消费金融的飞速发展，也说明了每一种解决方案在不同背景下都有不同的保质期。\n\n### 业务挑战\n\n随着产品升级迭代，早期的解决方案演变成为了眼前的问题，通过业务框架实现的数据分片方案导致业务代码复杂度增加、维护成本不断攀升，紧耦合的弊端原形毕露，应用每次升级都需要投入较多的精力对分片做相应调整，研发人员难以专注于业务本身。\n\n技术团队经权衡后开始考虑使用成熟的分库分表组件来承担这部分工作，让业务系统升级和架构调整不再复杂。\n\n### 解决方案\n\nShardingSphere-JDBC 是 Apache ShardingSphere 的第一款产品，它定位为轻量级 Java 框架，在 Java 的 JDBC 层提供的额外服务。它使用客户端直连数据库，以 jar 包形式提供服务，无需额外部署和依赖，可理解为增强版的 JDBC 驱动，完全兼容 JDBC 和各种 ORM 框架。\n\nShardingSphere-JDBC 的以下特点能够很好地满足白条业务场景：\n\n- 产品成熟：经数年打磨产品成熟度高，且社区活跃；\n- 性能良好：微内核、轻量化的设计，性能损耗极小；\n- 改造量小：支持原生的 MySQL 协议，研发工作量小；\n- 扩展灵活：搭配使用迁移同步组件轻松实现数据扩展。\n\n经内部大量系统性验证之后，Apache ShardingSphere 成为了京东白条数据分片中间件的首选方案，2018 年底正式开始对接。\n\n![jd_zh_1.png](https://official-media.sphere-ex.com/jd_zh_1_fac28cd6ec.png)\n\n### 用户收益\n\n- 简化升级路径\n\n通过架构解耦，业务系统升级所涉及技术栈得到有效缩短，研发团队不再需要关注分表设计，精力全部聚焦于业务本身，升级路径得到大幅度优化；\n\n- 节省研发力量\n\n引入成熟的 Apache ShardingSphere 无需重新开发分表组件，在简化业务升级路径的基础上节省了大量研发力量；\n\n- 架构灵活扩展\n\n搭配使用 Scaling 同步迁移组件从容面对“618”和“11.11”等大型活动，系统灵活扩容。\n\n**SphereEx，让数据与应用连接更简单。**\n\n","date":"2021-12-01","author":"SphereEx","excerpt":"京东白条使用 Apache ShardingSphere 解决了千亿数据存储和扩容的问题，为大促活动奠定了基础。","metaDescription":"京东科技集团是京东集团旗下专注于以技术为产业服务的业务子集团，致力于为企业、金融机构、政府等各类客户提供全价值链的技术性产品与解决方案。","createdAt":"2021-12-11T23:10:03.172Z","updatedAt":"2021-12-14T07:22:15.304Z","publishedAt":"2021-12-11T23:10:07.307Z","locale":"zh"}}]}}}}},
    "staticQueryHashes": []}