{
    "componentChunkName": "component---src-templates-news-detail-ts",
    "path": "/news/63",
    "result": {"pageContext":{"next":{"id":57,"attributes":{"feature":true,"title":"iQiyi goes cloud-native with Apache ShardingSphere & Database Mesh","content":" \n \nIn May of this year, we at [SphereEx](https://www.sphere-ex.com/en/) proposed the [Database Mesh 2.0](https://medium.com/faun/database-mesh-2-0-database-governance-in-a-cloud-native-environment-ac24080349eb) concept. \n[Database Mesh](https://www.database-mesh.io/) is a dynamic concept that is constantly evolving, focusing on database traffic governance and providing sharding, load balancing, observability, and audit capabilities based on database protocol awareness. These capabilities address some of the traffic governance issues. \nFurthermore, Database Mesh emphasizes the development of database reliability engineering (DBRE), providing easier-to-use and superior database governance capability. \n \n[iQiyi](https://www.crunchbase.com/organization/iqiyi) shares our view for the concept and vision of Database Mesh, which is to **achieve high-performance database expansion while tackling data governance issues in the cloud**. \n\niQiyi expanded [ShardingSphere-JDBC](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-jdbc) based on its business requirements and conducted a series of tests combined with [Pisanix](https://github.com/database-mesh/pisanix), for the implementation of the Database Mesh concept. \n \n### How did iQiyi prepare to go cloud-native with Database Mesh & ShardingSphere-JDBC?\n\nWith the expansion in the number of features, products & service offerings and the surge in the number of users, enterprises have diversified the number of promotional activities to engage and retain users (flash sales, events, etc.). In successful cases, this led to a huge amount of traffic putting great pressure on their databases. \n\nAs a result, enterprises encounter database issues such as secondary delays and slow queries, with some operations failing to meet business requirements. \nMicroservices and cloud-native bring new possibilities for the business roll-out process and governance. But with more diversified business scenarios and stovepipe data application schemes, data control tends to be isolated. Tech teams face problems such as difficult technology selection, high costs, and complicated management and control.\n \nParticularly, cloud-native architecture is growing mature, and the relationship between business applications and database infrastructure is changing gradually. iQiyi hopes to grasp this new trend and adopt unified management to expand and update databases, thus supporting more businesses and applications migrating to the cloud.\n \nTo meet the requirements for database performance and availability in cloud environments, iQiyi needed to migrate [ShardingSphere](https://shardingsphere.apache.org/)'s local distributed capability to the cloud. To achieve this, iQiyi was looking for a tool that can unify cloud database traffic access in a cloud-native environment and achieve the unified and efficient management of cloud traffic and data.\n \nWhile investigating and testing Pisanix, a Database Mesh solution provided by SphereEx, iQiyi also redeveloped ShardingSphere-JDBC, to meet the requirements for sharding, load balancing, configuration and storage, and security when accessing businesses to database governance platforms.\n \n#### 1. Preparation: iQiyi transforms ShardingSphere-JDBC\n\nCurrently, iQiyi uses a unified config center to store database connection configuration. [KMS technology](https://kms-technology.com/) is used to encrypt database access configuration and ShardingSphere-JDBC is used to implement sharding and load balancing. The complete architecture is shown below.\n \n![iQiyi 1.png](https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/i_Qiyi_1_ca3a63a55b.png)\n\n \nWhen businesses are connected to the data governance platform, they apply for relevant connection configuration. After they are transformed and the access information is encrypted via KMS, they are stored in the unified configuration storage center. When the application starts, the transformed ShardingSphere-JDBC fetches the configuration and monitors configuration changes to support hot configuration updates.\n \nBefore the transformation, when there was a need to change the configuration, scale-out sharding clusters, upgrade cluster version, or migrate a database to the cloud, it usually required the release of a new version. Also, the DevOps teams had to design complicated procedures such as switchover, rollback, timing selection, grayscale traffic, and data verification, to account for various scenarios.\n \nAfter the transformation, the customized ShardingSphere-JDBC can support sharding cluster scaling or binding changes when adding or modifying table sharding configuration. In the configuration center, you can perform visual operations to modify configurations or bind clusters, and select the configuration of reload timing. When the SDK receives the latest configurations, it starts asynchronous tasks to close the old connection pool and replace the existing one. This facilitates the smooth migration of read/write traffic and greatly simplifies the migration of data governance capabilities to the cloud environment.\n \niQiyi plans to introduce Pisanix-Proxy by accessing Database Mesh, further sinking data governance capability from [SDK](https://en.wikipedia.org/wiki/Software_development_kit) to Sidecar. \n \n#### 2. Data governance capability with Sidecar and building a unified data governance based on Pisanix\n\nIn the traffic access layer, as cloud-native applications move closer to microservices and Serverless, users need to configure complex routing rules, support multiple application-layer protocols, and ensure service access security and the observability of traffic. In response to these requirements, iQiyi used middleware to manage [Redis](https://redis.io/) and [MySQL](https://www.mysql.com/) at the very beginning.\n\n![iQiyi 2.png](https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/i_Qiyi_2_fdfce31fdb.png)\n \nAdditionally, SmartJedis provided a unified configuration center to support iQiyi's hybrid cloud deployment. In the unified configuration center, configurations in different environments could be dynamically supported. In a non-mesh environment, a direct connection is adopted; while in a mesh environment, RedisProxy in Envoy is used to manage Redis protocol traffic and support hot updates of connection configuration, avoiding downtime after Redis is moved to the cloud. \n \nIn terms of MySQL, iQiyi's R&D team tested Pisanix, the specific implementation of Database Mesh. Written in [Go](https://go.dev/) and [Rust](lhttps://www.rust-lang.org/) for the [Kubernetes](https://kubernetes.io/) environment, Pisanix currently supports [MySQL](https://www.mysql.com/). It includes three components: Pisa-Controller, Pisa-Proxy, and Pisa-Daemon, which provide a local database for users and applications. It supports multi-protocol pluggable architecture, shields the status of real data sources, and provides unified database traffic control capabilities for data DevOps teams.\n\nCurrently, iQiyi still uses ShardingSphere-JDBC to support Java applications. Once Pisanix will be further implemented by iQiyi, the company will implement standardized automatic database maintenance via Pisanix, and achieve the cloud-native orchestration of multiple database governance behaviors by supporting multi-language applications. Based on Database Mesh's standard `CustomResourceDefinition`, such as unified database access declaration configuration and programmable database access resource limitations, iQiyi can rapidly achieve the governance and orchestration of cloud-native databases.\n \n#### 3. iQiyi's plan for Pisanix\n\n**1）Data sharding: achieve high performance on par with ShardingSphere-JDBC in the cloud**\n\nData sharding is an effective way to deal with massive data storage and computing, which is why iQiyi chose Pisanix for cloud-native and non-Java scenarios. Data sharding mainly includes four modules: SQL parser, SQL rewriting, SQL router, and result merger.\n \nTo facilitate the migration of ShardingSphere's powerful local sharding capability to the cloud, Pisanix provides data sharding governance capabilities in the cloud based on the underlying database, allowing users to achieve horizontal scaling computing through Pisanix. At the same time, more custom metrics are available to achieve intelligent, stable, and advanced auto-scaling for Pisa-Proxy.\n \nBased on the Pisa-Controller plane, iQiyi can achieve the management and control of data plane components. Pisa-Proxy can also be combined and deployed in the same Pod with business applications in Sidecar mode to monitor MySQL protocol and obtain the traffic of applications accessing the database. Pisanix also provides iQiyi with a variety of governance capabilities:\n- **SQL traffic governance:** achieve multiple load balancing strategies and current limiting by paring SQL.\n- **Access control:** achieve fine-grained permission control based on the relationship between users and data permission.\n- **SQL firewall:** prevent high-risk SQL from executing.\n- **Observability:** expose various database access metrics such as throughput and latency.\n \nFrom iQiyi's point of view, Pisanix enables the high-performance sharding of both Java and non-Java services in the cloud environment. This achievement lays the foundation for the smooth transition of more businesses.\n \n**2）Read/write splitting: increase database throughput.**\n\nTo improve throughput and availability, many systems adopt a primary-secondary database architecture configuration mode, which is a bit complicated. Therefore, when read requests outnumber write requests, read/write splitting should be used to overcome the performance bottleneck of the database in real-world application scenarios.\n \nRead/write splitting is a widely used technical solution to improve throughput in primary-secondary scenarios, and is capable of improving query performance and reducing server load. It also brings the same problem with data sharding, which makes it more complicated for DevOps teams to operate databases.\n \nCurrently, iQiyi evenly distributes query requests to multiple data copies through the configuration mode of one primary and multi-secondary, which improves the processing capability of the system. This method improves throughput and the availability of the system - even when a database breaks down or a disk is physically damaged, the system can still maintain normal functioning.\n \niQiyi plans to adopt Pisanix's dynamic read/write splitting feature to manage multi-primary and multi-secondary database clusters. After connecting to Pisanix, iQiyi will be able to transparently manage the primary/secondary database with read/write splitting so that users can use the database with the primary/secondary architecture just like a monolithic database.\n \n### Future plan\n\nCurrently, iQiyi has completed its internal transformation for ShardingSphere-JDBC. In the future, it plans to combine Pisanix and ShardingSphere to achieve the unified governance of MySQL. \n\nDriven by the ShardingSphere and Database Mesh communities, Pisanix will continue to develop cloud solutions to meet various usage scenarios, with SphereEx providing reliable technical support for iQiyi and accelerating the transition speed to the cloud. \n \nPisanix is a very young project, which means there are some shortcomings. iQiyi's test shows that Pisanix is limited in its expression support for database and table sharding, and its special configuration for SQL needs to be further improved. \n\nNext, the community will focus on improving Pisanix's online capabilities, including operating status visualization, metrics, circuit breaker degradation strategy, and tracing. \nAdditionally, SQL audit, Pisa-Controller's merge with [Istio](https://istio.io/), and other issues related to compatibility and performance have also been put on the agenda.\n \nIn the coming future, iQiyi will build a MySQL-based unified data access standard and solution based on ShardingSphere-JDBC and Pisanix that is still evolving under the Database Mesh concept. \n\nThrough a unified configuration center and customized Sidecar, iQiyi will gradually make the database access details fully transparent to developers. This way, it can simplify the operating process while enhancing the security of database access, simplifying moving applications moving to the cloud.\n \n----------------------------\nFor more information about Database Mesh and Pisanix, follow the links below:\n[Database Mesh 2.0: Database Governance in a Cloud Native Environment](https://medium.com/faun/database-mesh-2-0-database-governance-in-a-cloud-native-environment-ac24080349eb)\n[Pisanix is Available! An Open Source Database Mesh Solution Launched by SphereEx](https://www.sphere-ex.com/news/43/)\n[ Pisanix GitHub](https://github.com/database-mesh/pisanix)\n","date":"2022-12-16","author":"SphereEx","excerpt":"iQiyi migrates ShardingSphere’s local distributed capability to the cloud with Database Mesh’s Pisanix\n\n","createdAt":"2022-12-15T06:58:49.152Z","updatedAt":"2023-01-16T03:48:29.037Z","publishedAt":"2022-12-16T08:22:35.305Z","locale":"en","newsType":{"data":null},"cover":{"data":{"id":457,"attributes":{"name":"iQiyi & Database Mesh.png","alternativeText":"iQiyi & Database Mesh.png","caption":"iQiyi & Database Mesh.png","width":1800,"height":766,"formats":{"thumbnail":{"name":"thumbnail_iQiyi & Database Mesh.png","hash":"thumbnail_i_Qiyi_and_Database_Mesh_dc1577e484","ext":".png","mime":"image/png","width":245,"height":104,"size":23.87,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/thumbnail_i_Qiyi_and_Database_Mesh_dc1577e484.png"},"large":{"name":"large_iQiyi & Database Mesh.png","hash":"large_i_Qiyi_and_Database_Mesh_dc1577e484","ext":".png","mime":"image/png","width":1000,"height":426,"size":142.16,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/large_i_Qiyi_and_Database_Mesh_dc1577e484.png"},"medium":{"name":"medium_iQiyi & Database Mesh.png","hash":"medium_i_Qiyi_and_Database_Mesh_dc1577e484","ext":".png","mime":"image/png","width":750,"height":319,"size":105.4,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/medium_i_Qiyi_and_Database_Mesh_dc1577e484.png"},"small":{"name":"small_iQiyi & Database Mesh.png","hash":"small_i_Qiyi_and_Database_Mesh_dc1577e484","ext":".png","mime":"image/png","width":500,"height":213,"size":63.66,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/small_i_Qiyi_and_Database_Mesh_dc1577e484.png"}},"hash":"i_Qiyi_and_Database_Mesh_dc1577e484","ext":".png","mime":"image/png","size":134.2,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/i_Qiyi_and_Database_Mesh_dc1577e484.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-12-16T07:14:02.426Z","updatedAt":"2022-12-16T07:14:02.426Z"}}},"localizations":{"data":[]}}},"prev":{"id":64,"attributes":{"feature":true,"title":"SphereEx-DBPlusEngine 1.4.0 Release: Enhanced Data Governance for Enterprises","content":"SphereEx-DBPlusEngine 1.3.0, released in January, received a lot of attention and feedback from users and customers. To meet their needs and focus on the industry's future development, we developed a release schedule based on feedback and are excited to announce the release of SphereEx-DBPlusEngine 1.4.0. This version includes significant updates that optimize the migration process and enhance security features such as in-process audit, archiving, and DDL consistency.\n\n# New Capabilities Meet Core User Demands\n\n## 1. Archiving: Support for automatic deletion of expired data\n\nArchiving expired data is a common requirement, and SphereEx-DBPlusEngine 1.4.0 now supports the automatic deletion of expired data. By defining a specific time field in the data table, the system can automatically delete data stored in the table that exceeds the specified time. This simple configuration process saves time and improves efficiency while also enhancing security.\n\n## 2. DDL consistency: Support for consistent execution of DDL statements\n\nDDL consistency has always been a problem in a distributed database environment, especially with sharded tables stored in multiple storage nodes. SphereEx-DBPlusEngine 1.4.0 now supports consistent execution of DDL statements, which can be combined with locks to ensure consistency. This guarantees the maximum concurrency of SQL execution and enhances efficiency.\n\n# Optimized Design Makes Products More Stable and Efficient\n\n## 1. Enhanced migration process: Support for migration of multiple tables with one DistSQL task\n\nTraditional data migration processes execute multiple table tasks in parallel to improve efficiency. However, each DistSQL task can only migrate one table, resulting in low efficiency and manual configuration requirements. SphereEx-DBPlusEngine 1.4.0 now supports the migration of multiple tables with one DistSQL task, greatly improving efficiency. Users only need to specify the tables that need to be migrated in the task configuration.\n\n## 2. Enhanced security features: Support for in-process audit\n\nIn the field of database technology, security has always been a significant issue. SphereEx-DBPlusEngine 1.4.0 introduces support for in-process audit, an important security mechanism that audits relevant SQL operations during execution. With built-in algorithms, it can audit relevant data changes and support two audit actions: log alarm and immediate fuse. This feature enhances system security and allows users to audit and trace more effectively.\n\n# About SphereEx-DBPlusEngine\n\nSphereEx-DBPlusEngine is a product based on the open-source kernel ShardingSphere, providing enterprise-level enhanced data services such as data sharding, distributed transactions, and data security for businesses. It consists of two products: SphereEx-DBPlusEngine-Driver and SphereEx-DBPlusEngine-Proxy, both of which can be deployed independently and support hybrid deployment. They provide standardized data horizontal scaling, distributed transactions, and distributed governance, and can be applied to a variety of diversified scenarios such as Java, homogeneous/heterogeneous languages, cloud-native, etc.\n\nThe release of SphereEx-DBPlusEngine 1.4.0 provides users with a series of new core features and major updates, further expanding the scope of application scenarios for SphereEx-DBPlusEngine and improving efficiency in practical environments. Choosing SphereEx-DBPlusEngine 1.4.0 can not only provide a more stable, efficient, and secure data governance solution but also offer more professional and complete enterprise-level services, reducing costs while accelerating digital business innovation.\n\n# Free Trial\n\nSphereEx-DBPlusEngine 1.4.0 is now available for free download and trial on the [SphereEx official website](https://www.sphere-ex.com/account/#/login/signIn?redirect=%2Fdownload). We are also an AWS APN technical partner, and every version of SphereEx-DBPlusEngine is packaged and released on AWS Marketplace. AWS users are welcome to find our offerings on [AWS Marketplace](https://aws.amazon.com/marketplace/seller-profile?id=d1a1d3ef-fce8-43d5-a57b-e5b1ec59caf0) to download and experience SphereEx-DBPlusEngine and unlock the convenience and efficiency of cloud-based digital services.","date":"2023-04-03","author":"SphereEx","excerpt":null,"createdAt":"2023-04-04T05:51:34.314Z","updatedAt":"2023-04-04T05:51:36.981Z","publishedAt":"2023-04-04T05:51:36.977Z","locale":"en","newsType":{"data":null},"cover":{"data":null},"localizations":{"data":[]}}},"article":{"id":63,"attributes":{"feature":true,"title":"SphereEx-DBPlusEngine V1.3.0: Enhanced Functionality for Better Data Governance","content":"SphereEx is excited to announce the release of SphereEx-DBPlusEngine V1.3.0, which comes with significant updates to provide enterprises with enhanced data service capabilities. Here are the key improvements in this release:\n\n- Data migration and horizontal scaling: The new update allows users to implement Proxy-based clustering migration, which enhances the computing and data processing capacity of the cluster. Additionally, DBPlusEngine simplifies the operational steps for data migration, making it easier for users.\n- Data masking: The latest update now features mask rules that deform and blur key data when certain key information queries are made, ensuring the safety of sensitive data.\n- Single table to shard table: The Scaling tool in the new update allows the automatic conversion of a single table to a shard table, simplifying users' operations.\n- Driver log collection and metrics monitoring: The DBPlusEngine-Driver form now has the same capability as DBPlusEngine-Proxy, which allows for the visual display of logs and monitoring metrics collected on the Driver side, improving users' experience in log retrieval and monitoring scenarios.\n- Fuzzy query calculations: The new implementation of like calculations in encrypted scenarios supports fuzzy query calculations.\n- Encrypted/decryption data cleaning File Transfer Protocol: In cases of an interruption/failure during encrypted data cleaning, DBPlusEngine can continue to start the data cleaning task based on File Transfer Protocol.\n\n## SphereEx-DBPlusEngine-Mate: Fully Supporting All DBPlusEngine Capabilities\n\nIn addition to the updates to SphereEx-DBPlusEngine, the DBPlusEngine-Mate has come to the v0.3.0 version, which supports all DBPlusEngine capabilities in cloud environments. This allows users to use DBPlusEngine out of ZooKeeper in a Kubernetes environment to get closer to cloud-native ways of managing DBPlusEngine.\n\nThe DBPlusEngine-Mate is a metadata management tool in cloud-native scenarios that seamlessly integrates governance capabilities such as sharding, encrypted data cleaning, read/write separation, and high availability into the Kubernetes metadata system. This allows users to eliminate dependency on ZooKeeper on the cloud, reducing the cost of machine resources and the burden of operations and maintenance staff.\n\nOther Benefits of the DBPlusEngine-Mate v0.3.0 update:\n\n- Guarantees the integrity of DBPlusEngine functionality.\n- Empowers users to fully use DistSQL in the cloud.\n- Optimizes the user experience on the SRE side.\n- Provides SREs and DBAs with the same operational experience as a cloud-native database.\n\n## Experience SphereEx-DBPlusEngine V1.3.0 Today!\n\n![img](https://u01f1kqxrl.feishu.cn/space/api/box/stream/download/asynccode/?code=NGRhYWI1NzM3ODNkZWJkNmUwNzkwMGVjMWFmYjFiNDNfbjVKQW5Wd3FteHI1Tk1VNDl3N3g5N01UWDVNVHVFZ2JfVG9rZW46QW1kcmJEeldKb3h2QUx4Mlc0NmM3dW1BbkJoXzE2ODA1ODcwNzQ6MTY4MDU5MDY3NF9WNA)\n\n<center>(SphereEx-DBPlusEngine v1.3.0 Product Library page)</center>\n\nSphereEx-DBPlusEngine V1.3.0 is now available for download on the [SphereEx official website](https://www.sphere-ex.com/account/#/login/signIn?redirect=%2Fdownload). Contact our staff to get a limited-time license and experience all the features of SphereEx-DBPlusEngine v1.3.0 for free. AWS users are welcome to find our offerings on the [AWS Marketplace](https://aws.amazon.com/marketplace/seller-profile?id=d1a1d3ef-fce8-43d5-a57b-e5b1ec59caf0).","date":"2023-02-24","author":"SphereEx","excerpt":null,"createdAt":"2023-04-04T05:47:36.369Z","updatedAt":"2023-04-04T05:47:37.666Z","publishedAt":"2023-04-04T05:47:37.664Z","locale":"en","newsType":{"data":null},"cover":{"data":null},"localizations":{"data":[]}}}}},
    "staticQueryHashes": []}