{
    "componentChunkName": "component---src-templates-news-detail-ts",
    "path": "/news/48",
    "result": {"pageContext":{"next":{"id":43,"attributes":{"feature":true,"title":"Pisanix is Available! An Open Source Database Mesh Solution Launched by SphereEx","content":"\n>We are launching Pisanix v0.1.0, an open source solution for [Database Mesh](https://www.database-mesh.io/index.html). From its inception, Pisanix embraced the core idea of Database Mesh: **to build an efficient and programmable database governance experience for the cloud native era**. This marks a turning point, making Database Mesh no longer mere theory, but a framework that can be implemented in real scenarios.\n\n[Pisanix](https://github.com/database-mesh/pisanix) is an open source solution for Database Mesh sponsored by SphereEx. Written in [Go](https://go.dev/) and [Rust](https://www.rust-lang.org/) for the [Kubernetes](https://kubernetes.io/) environment, it currently supports [MySQL](https://www.mysql.com/). It includes the following three components: Pisa-Controller, Pisa-Proxy and Pisa-Daemon (coming soon), which help users easily implement SQL-aware traffic governance based on the Database Mesh framework, runtime resource-oriented programmability, and database reliability engineering (DBRE) to facilitate database governance in the cloud native era.\n## Pisanix v0.1.0 Features\nPisanix v0.1.0 supports the following features:\n- Pisa-Controller\n  - Support the injection of [Sidecar](https://shardingsphere.apache.org/document/current/en/user-manual/shardingsphere-sidecar/).\n  - Read Kubernetes custom resources, such as VirtualDatabase, TrafficStrategy and DatabaseEndpoint, and convert them to Pisa-Proxy configuration.\n- Pisa-Proxy\n  - Support the deployment of Kubernetes Sidecar and obtain configuration from Pisa-Controller.\n  - Support multi-port MySQL protocol proxy.\n  - Support connection pool and basic load balancer.\n  - Support [Yacc](https://www.geeksforgeeks.org/introduction-to-yacc/)-based SQL parser including , `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `PREPARE`, `EXECUTE`, `BEGIN`, and `SET`.\n  - Support [Regex](https://en.wikipedia.org/wiki/Regular_expression)-based SQL circuit breaker.\n  - Support SQL concurrency control: restrict the number of SQL executed at a given time based on the Regex.\n  - Observability: support `sql_processed_total` and `sql_processed_duration`.\n  - Support deployment in a non-Kubernetes environment as an independent component.\n## Introducing Pisanix\nPisanix adopts the classic deployment mode of separating the control plane and data plane. It supports unified database access declaration configuration, supports access resource limits to the programmable database, and supports standardized automatic database maintenance through various `CustomResourceDefinition`. It is designed to implement database governance in a cloud native environment.\n\nPisanix currently provides the following capabilities: \n- **SQL-Aware Traffic Governance**: provide load balancing, auditing, access control, and observability of database traffic based on SQL parsing capability.\n- **Runtime Resource-Oriented Management**: support configurable and programmable resource capabilities by means of Linux kernel.\n- **DBRE**: it can design an engineer-centered product with database reliability as the principle, and establish a unified cloud interface on the database\n ![00000.PNG](https://official-media.sphere-ex.com/00000_e39fd0bb0b.PNG)\n**Pisa-Controller**\nPisa-Controller is the control plane of Pisanix and an indispensable component, implemented by Go language. It is responsible for the control of data plane components and is the entry point for all Pisanix configurations. It has the following functions:\n- Sidecar injection: inject Pisa-Proxy container into the specified Pod with MutatingAdmissionWebhook.\n- Pisa-Proxy Configuration delivery and indicator collection: obtain and convert service discovery, load balancer, traffic control, and SQL firewall rules in CRD format in the cluster, deliver them to Pisa-Proxy, and collect various indicators when Pisa-Proxy is running.\n- Pisa-Daemon configuration conversion and indicator collection: obtain and convert traffic QoS rules in CRD format in the cluster, deliver them to Pisa-Daemon, and collect various indicators when Pisa-Daemon is running.\n\n**Pisa-Proxy**\nPisa-Proxy is a high-performance proxy and an indispensable component implemented by Rust. It is deployed in the same Pod with business applications in Sidecar mode. It monitors the MySQL protocol to obtain the traffic when applications access the database, and provides various governance capabilities:\n- SQL traffic governance: implement load balancer strategies and circuit breaker through SQL parsing.\n- Access control: achieve fine-grained permission control based on the relationship between users and data permission.\n- SQL firewall: prevent high-risk SQL statements from being executed\n- Observability: expose various database access indicators, such as throughput and latency.\n**Pisa-Daemon (coming soon）**\nPisa-Daemon is a node-level data plane and is an optional component. Pisa-Daemon is deployed on every node in a Kubernetes cluster and provides functions such as resource management through the various capabilities of the host kernel. \n- Runtime traffic governance: it provides QoS guarantee for database access traffic with [eBPF](https://ebpf.io/) and other technologies to reduce the interference between traffic.\n## Pisanix Examples\nDatabase Mesh's governance framework relies on the following workloads:\n- Virtual database: a database endpoint accessible to developers.\n- Traffic strategy: governance strategies for database access traffic such as label router, data sharding, load balancer, concurrency control and circuit breaker.\n- Access control: provides fine-grained access control based on specified rules, such as table level.\n- Security statement: data security statements such as data encryption.\n- Audit application: record applications’ operation behavior on the database, such as accessing the risk control system.\n- Observability: configuration of database observability such as access traffic, running state, and performance indicators etc.\n- Event bus: an event bus that accepts data changes.\n- QoS statements: resource QoS metrics set to improve the overall SLO metrics of the database.\n- Backup plan: perform database backups as scheduled tasks.\n- `Schema` pipeline: manage database `schema` changes in code to improve the success rate of database DDL and DML changes.\n\nCurrently, Pisanix supports the following three `CustomResourceDefinition`: VirtualDatabase, TrafficStrategy and DatabaseEndpoint.\n\n**VirtualDatabase**\nVirtualDatabase is the central concept for all database governance activities. VirtualDatabase represents an endpoint for database access for developers and a logical database for DBAs. The actual back-end data source, DatabaseEndpoint, and specified TrafficStrategy need to be provided for VirtualDatabase.\nPisanix currently supports VirtualDatabase of the MySQL type. The configuration example is as follows:\n\n```\napiVersion: core.database-mesh.io/v1alpha1\nkind: VirtualDatabase\nmetadata:\n  name: catalogue\n  namespace: default\nspec:\n  services:\n  - databaseMySQL:               # the virtual database type is MySQL\n      db: socksdb                # virtual database: schema\n      host: 127.0.0.1            # access address of virtual database\n      port: 3306                 # access port of virtual database\n      user: catalogue_user       # access username of virtual database\n      password: default_password # access password of virtual database\n    name: mysql                  # name of virtual database\n    trafficStrategy: catalogue   # traffic strategy of specified virtual database\n```\n\n**TrafficStrategy**\nTrafficStrategy describes the governance policies for database access traffic, which are configured by DBAs, such as label-based request routing, SLA-driven load balancer, SQL circuit breaker, and concurrency control.\nPisanix supports basic load balancer, regex-based SQL circuit breaker, and concurrency control. The configuration example is as follows:\n\n```\napiVersion: core.database-mesh.io/v1alpha1\nkind: TrafficStrategy\nkind: TrafficStrategy\nmetadata:\n  name: catalogue\n  namespace: default\nspec:\n  loadBalance:                   # load balancer strategy of accessing virtual database\n    simpleLoadBalance:           # load balancer strategy is simple load balancer\n      kind: random               # load balancer strategy is random\n  selector:\n    matchLabels:                 # declare label selector to select a target DatabaseEndpoint\n      source: catalogue\n```\n\n**DatabaseEndpoint**\nDatabaseEndpoint describes the actual database access endpoint, which can be a database instance deployed on the cloud and is a priority resource for DBAs. As an abstraction from a real database, DatabaseEndpoint can be consumed by multiple VirtualDatabase, meeting different needs of developers and DBAs.\nPisanix supports DatabaseEndpoint of the MySQL type. The configuration example is as follows: \nsource\n\n```\napiVersion: core.database-mesh.io/v1alpha1\nkind: DatabaseEndpoint\nmetadata:\n  name: catalogue-db\n  namespace: default\nspec:\n  database:\n    MySQL:                        # data source type is MySQL\n      db: socksdb                 # database is schema\n      host: cataloguedb.codtynlacssn.rds.cn-northwest-1.amazonaws.com.cn # address of data source\n      port: 3306                  # access port of data source\n      user: root                  # access username of data source    \n      password: fake_password     # access password of data \n```\n\n[Weaveworks](https://www.weave.works/)' [Socks-Shop](https://microservices-demo.github.io/) simulates the user-facing part of an e-commerce website that sells socks. It is intended to aid the demonstration and testing of microservice and cloud native technologies and provide a real testable application. The project uses the following technologies or frameworks: Java Spring Boot, Go, Node.js and uses containers to build and deploy.\n\nTaking the socks-shop microservice project as an example, its governance process with the Pisanix framework is as follows:\n\n1. The developer will submit VirtualDatabase for Catalogue service to apply for the VirtualDatabase type.\n2. The DBA creates the associated TrafficStrategy and DatabaseEndpoint for VirtualDatabase.\n3. Add label pisanix.io/inject=enabled for the Catalogue service by the SRE, enable the injection of Sidecar, and then configure the Catalogue database access information to the address in VirtualDatabase and start the application.\nAfter the configuration is completed, access the front-end service for verification.\n![1111110.png](https://official-media.sphere-ex.com/1111110_b5a1b2f868.png)\n## Pisanix's design concept: Database Mesh\nThe article [Database Mesh 2.0](https://www.sphere-ex.com/zh/news/33/): Database Governance in a Cloud Native Environment, describes how Database Mesh 2.0 provides a database-centric governance framework.\n- Databases are first-class citizens. All abstractions are centered on database governance, such as access control, traffic governance, and observability.\n- Engineer-oriented experience: developers can continue to develop through easy to use database declarations and definitions regardless of the location of the database. For operation and maintenance personnel and DBAs, it provides a variety of abstraction of database governance behavior to realize automatic database reliability engineering.\n- Cloud native: built and oriented towards cloud native, it is suitable to various cloud environments with an open ecosystem and implementation mechanism. Vendor lock-in is effectively eliminated.\n![333.png](https://official-media.sphere-ex.com/333_57bed58e11.png)\nFor more details, refer to [Database Mesh's official website](https://www.database-mesh.io/).\n## Outlook\nPisanix is just getting started. In the future, it will focus on enhancing traffic governance ability, such as developing data sharding ability based on sharding key, audit application for data access behavior, QoS of runtime resources, etc. while continuously improving its performance and ease of use. Pisanix plans to open the plugin extension mechanism, allowing users to write policies to adapt to different business scenarios, putting Unix's design philosophy of \"policy and mechanism separation\" into practice.\n## Community\nPisanix is just getting started. The Pisanix welcomes you to join us and share your ideas and opinions. It can be with code or documentation, issues or pull requests. For those of you who are willing to engage in database governance, let's build the Pisanix community together.\nThe community holds an online discussion every two weeks:\n- [Mailing list](https://groups.google.com/g/database-mesh)\n- [Biweekly meeting - English community](https://meet.google.com/yhv-zrby-pyt) (from February 27, 2022), on Wednesday 9:00 AM PST\n- [Biweekly meeting - Chinese community](https://meeting.tencent.com/dm/6UXDMNsHBVQO) (from April 27, 2022), on Wednesday 9:00 PM GMT+8\n- [Slack](https://databasemesh.slack.com/)\n- [Meeting minutes](https://bit.ly/39Fqt3x)","date":"2022-06-17","author":null,"excerpt":"Facilitate database governance in the cloud native era","createdAt":"2022-06-17T07:23:39.056Z","updatedAt":"2022-10-11T06:20:03.747Z","publishedAt":"2022-06-17T07:39:11.741Z","locale":"en","newsType":{"data":null},"cover":{"data":{"id":396,"attributes":{"name":"Database Mesh2.png","alternativeText":"Database Mesh2.png","caption":"Database Mesh2.png","width":1080,"height":414,"formats":{"thumbnail":{"name":"thumbnail_Database Mesh2.png","hash":"thumbnail_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":245,"height":94,"size":20.45,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_Database_Mesh2_00ac6eb990.png"},"large":{"name":"large_Database Mesh2.png","hash":"large_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":1000,"height":383,"size":126.15,"path":null,"url":"https://official-media.sphere-ex.com/large_Database_Mesh2_00ac6eb990.png"},"medium":{"name":"medium_Database Mesh2.png","hash":"medium_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":750,"height":288,"size":89.97,"path":null,"url":"https://official-media.sphere-ex.com/medium_Database_Mesh2_00ac6eb990.png"},"small":{"name":"small_Database Mesh2.png","hash":"small_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":500,"height":192,"size":52.99,"path":null,"url":"https://official-media.sphere-ex.com/small_Database_Mesh2_00ac6eb990.png"}},"hash":"Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","size":60.86,"url":"https://official-media.sphere-ex.com/Database_Mesh2_00ac6eb990.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-29T08:05:47.915Z","updatedAt":"2022-09-29T08:05:47.915Z"}}},"localizations":{"data":[]}}},"prev":{"id":49,"attributes":{"feature":false,"title":"We're sponsoring KubeCon+CloudNativeCon NA 2022 and launching DBPlusSuite to bring Database Plus & Database Mesh offerings with a cloud native database governance architecture","content":"We are thrilled to be sponsoring the [CNCF](https://www.cncf.io/) flagship conference [KubeCon + CloudNativeCon NA 2022](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/) from October 24 - 28, 2022, and to launch DBPlusSuite V1.1!\n\nAs a [CNCF silver member](https://www.cncf.io/about/members/), SphereEx has been following through on its commitment to cloud native. \n\nFrom Database Plus to Database Mesh, from supporting the 5.2.0 release of the popular Big Data project [Apache ShardingSphere](https://faun.pub/apache-shardingsphere-5-2-0-is-released-bringing-new-cloud-native-possibilities-8d674d964a93?source=your_stories_page-------------------------------------) to initiating and sponsoring a new database mesh open source project with [Pisanix 0.3.0](https://www.sphere-ex.com/news/48/), we have been fully committed to expanding cloud native possibilities.\n\nWe believe our sponsorship of KubeCon + CloudNativeCon proves this commitment again.\n\n[SphereEx](https://www.sphere-ex.com/) builds distributed data infrastructures while delivering a SaaS experience through the cloud, and was founded by the core contributor team of [Apache ShardingSphere](https://shardingsphere.apache.org/) - an open-source ecosystem to transform any database into a distributed database system and enhance it with data sharding, elastic scaling, encryption features & more.  \n\nWe are now launching an enterprise-grade pluggable data service enhancement platform: SphereEx-[DBPlusSuite V1.1](https://www.sphere-ex.com/), which includes SphereEx-Boot, SphereEx- Console, and SphereEx DBPlusEngine. \n\nThe business suite brings several features including Autoscaling, Traffic Governance, Data Sharding, and DistSQL while remaining multi-cloud, breaking performance records, and maintaining an open source DNA. \n\nOur pioneering [\"Database Plus\" and \"Database Mesh\"](https://www.sphere-ex.com/products/concepts/) concepts that drive our work, are designed to unlock innovative ways to leverage and manage data and improve enterprise data applications.\n\nThanks to Apache ShardingSphere, our offerings have been tested by stock market-listed corporations in over 170 enterprise and production environments covering the financial, logistics, e-commerce, internet & cloud computing fields, to help achieve enterprise digital transformation.\n\n**You can join us virtually at our [KubeCon virtual booth](https://cncf-dot-yamm-track.appspot.com/2-22rLgN3QFsyChEI7yF3ez6C7zcZI5KE-JGAWZuH9oYBcH8jgwGI3wdrF8JY-hjVexkjT7XZlmADl7wsEiRq2F3TrF5qeJnAXwz84mEjzMy_BTgQFfCU7V7xAWPRCEqUv7aBrKvdceE7BFR-XwNlZgqKmBv0_OxkhpDRJenknBYHtJJk2XfgZuAkwLAQRhUlNFPZpGPsJJkwFP-Inh0ne6Iiu6FFW2xQOGFy28vE8nqzI8dfkT-Rkp5E9vwDVTObk2vr9bjfE7Fkbv-VtFc3vsV_5ecnOxRJ3NI6e_AjJqIfShAsblS4ZW7qxlgVcs-wre5T0XlbRPFwGHC4w5X1J7o3AiNE5n8c_GlXdRsmVtcXTXMCMWzdZmuaYxxVzYjuIJ-53A2Op0uuS6hBiurhLS1vFzrDm9JZFc5Clm1GkVTMlyZPksxbDt1mC4CTZYnglxdMLYDioRCJnF07QwIuooQT) to learn more about DBPlusEngine, Database Mesh, Database Plus and Apache ShardingSphere.**\n\n![KubeCon+CloudNativeCon NA.png](https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/Kube_Con_Cloud_Native_Con_NA_22ebbb46ed.png)\n![KubeCon+CloudNativeCon NA sponsorship.png](https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/Kube_Con_Cloud_Native_Con_NA_sponsorship_518685dc22.png)\n\n\n### Enterprise-grade data services enhancement engine: SphereEx DB Plus Engine\n\nSphereEx DB Plus Engine is an enterprise data service platform, built with Apache ShardingSphere at its core. In addition to ShardingSphere's functionality, DB Plus Engine provides features such as enterprise security, high-performance clusters, high-availability clusters, and more. It consists of two products, SphereEx DB Plus Engine-Driver and SphereEx DB Plus Engine-Proxy, to be deployed in combination or independently.\n\nThe newly released SphereEx DB Plus Engine-Proxy V1.1 focuses on updating capabilities in encryption and decryption, dual routing, and user privileges.\n\n#### Data features:\n\n• Autoscaling to automatically increase or decrease compute, memory, or networking resources. \n• Traffic governance to intelligently distribute traffic, for higher throughput and efficient query handling. \n• Data sharding to deal with massive data storage and computation. \n• Automatic data encryption and decryption support user-defined data types.\n• Dual routing: JDBC and Proxy integration based on dual routing capability.\n \n#### User privilege management:\n\n• Adds a new login mode: LDAP authentication.\n• Adds role management.\n• Adds table-level privilege control.\n• Adds column-level privilege control.\n• User privileges can be changed online.\n\nBoth SphereEx DB Plus Engine-Driver and SphereEx DB Plus Engine can provide standardized data scale-out, distributed transactions, and distributed governance capabilities - applicable to Java isomorphic & heterogeneous languages scenarios, and cloud-native environments.\n\n\n#### Installation and deployment: SphereEx-Boot\n\nSphereEx-Boot is the ideal solution to install and manage a variety of cluster management tools.\n\nSphereEx-Boot V1.1 is a Python-developed command line tool for easy management of SphereEx DB Plus Engine-Proxy clusters, providing significant usability and compatibility improvements.\n\n• In terms of usability, SphereEx-Boot provides an offline installation mode allowing users to install and deploy products in an offline environment with a single command.\n• In terms of compatibility, SphereEx-Boot supports the installation of Prometheus, allowing for the easy monitoring of SphereEx's product components.\n\nIn the context of the growing [Apache ShardingSphere ecosystem](https://shardingsphere.apache.org/), SphereEx-Boot manages many components within the SphereEx DB Plus Engine ecosystem and acts as a cluster management tool in enterprise production environments.\n \n#### Data management and control: SphereEx-Console\n\nSphereEx-Console, a data visualization platform for the management and control of the SphereEx enterprise data services platform, provides a user-friendly experience. \n\nIt's integrated into a solution including resources, instances, plugins, and more to provide a one-stop user experience with SphereEx DB Plus Engine at its core.\n\nThe newly released SphereEx-Console V1.1 further expands product capabilities in terms of cluster adaptation, [DistSQL (Distributed SQL)](https://shardingsphere.apache.org/blog/en/material/jul_26_an_introduction_to_distsql/), and visualization.\n\n• Cluster management: SphereEx-Console can manage the SphereEx DB-Plus-Engine cluster with one click and seamlessly access users' existing platforms, significantly improving product compatibility.\n\n• DistSQL: SphereEx-Console fully supports DistSQL for its operation, enabling users to create and bind various kinds of tables - greatly improving usability for DBAs.\n\n• Visualization: SphereEx-Console V1.1 enables users to customize monitoring indicators and interface management visualization.\n\nSphereEx-Console, a SphereEx business data services cluster platform centered on managing Apache ShardingSphere, simplifies user configuration, operation, and deployment processes. Lowering entry barriers for beginners while solving enterprise management and control pain points, delivers enhanced operation & management to enterprises.\n\n![SphereEx slogan.png](https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/Sphere_Ex_slogan_d4a27fa275.png)\n\nOur business suite will continue following the [Database Plus](https://faun.pub/whats-the-database-plus-concepand-what-challenges-can-it-solve-715920ba65aa) design concept for distributed database systems, while the \"cloud\" being our key technology development trend. \n\nToday, cloud native has become essengtial for hosting large-scale applications and mastering the future of technology.\n\nSphereEx will actively adapt to the data needs of cloud native scenarios, bringing cloud native databases closer to users and providing enterprises with flexible and efficient cloud data governance to accelerate their transition from on-premise to cloud.\n\n\n**Official Website:** [https://www.sphere-ex.com/](https://www.sphere-ex.com/)\n**LinkedIn:** https://www.linkedin.com/company/sphere-ex/\n**Twitter:** @Sphere_Ex\n**Apache ShardingSphere:** https://shardingsphere.apache.org\n**Pisanix:** https://www.pisanix.io\n**KubeCon Registration:** https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/\n**Our KubeCon Booth:** https://tinyurl.com/SphereEx-Booth","date":"2022-10-20","author":"SphereEx","excerpt":"We are thrilled to sponsor the KubeCon + CloudNativeCon NA 2022 and to launch DBPlusSuite V1.1!\nLearn more or connect with us at our KubeCon booth.","createdAt":"2022-10-20T10:07:55.310Z","updatedAt":"2022-10-20T10:28:05.279Z","publishedAt":"2022-10-20T10:08:37.486Z","locale":"en","newsType":{"data":null},"cover":{"data":{"id":416,"attributes":{"name":"KubeCon+CloudNativeCon NA.png","alternativeText":"KubeCon+CloudNativeCon NA.png","caption":"KubeCon+CloudNativeCon NA.png","width":652,"height":535,"formats":{"thumbnail":{"name":"thumbnail_KubeCon+CloudNativeCon NA.png","hash":"thumbnail_Kube_Con_Cloud_Native_Con_NA_22ebbb46ed","ext":".png","mime":"image/png","width":190,"height":156,"size":42.48,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/thumbnail_Kube_Con_Cloud_Native_Con_NA_22ebbb46ed.png"},"small":{"name":"small_KubeCon+CloudNativeCon NA.png","hash":"small_Kube_Con_Cloud_Native_Con_NA_22ebbb46ed","ext":".png","mime":"image/png","width":500,"height":410,"size":207.03,"path":null,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/small_Kube_Con_Cloud_Native_Con_NA_22ebbb46ed.png"}},"hash":"Kube_Con_Cloud_Native_Con_NA_22ebbb46ed","ext":".png","mime":"image/png","size":219.47,"url":"https://sphereex-media-1305704183.cos.ap-beijing.myqcloud.com/Kube_Con_Cloud_Native_Con_NA_22ebbb46ed.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-10-20T09:55:38.000Z","updatedAt":"2022-10-20T09:55:38.000Z"}}},"localizations":{"data":[]}}},"article":{"id":48,"attributes":{"feature":true,"title":"Pisanix v0.3.0 is Released, Introducing Data Sharding Strategy!","content":"Data sharding is an effective solution to deal with massive data storage and computation. \n\n[Pisanix](https://www.pisanix.io/en/), a database mesh solution sponsored by SphereEx, now provides data sharding governance capability based on the underlying database - allowing users to scale out computing and storage. \n\nStarting from v0.3.0, Pisanix will gradually support data sharding, with this release supporting single database sharding.\n\n### 1. Introducing data sharding\n\nAs shown in Figure 1 below, data sharding mainly consists of SQL parse, SQL rewriting, SQL route, SQL execution, and result merge.\n\n![db mesh figure1.png](https://official-media.sphere-ex.com/db_mesh_figure1_826abe9841.png)\n\n**Concepts:**\n\n**SQL Parse:** During the sharding process, once the request is received by Pisa-Proxy, it will first go through SQL Parser, and the SQL will be parsed into AST.\n**SQL Rewriting:** After parsing, Pisa-Proxy will rewrite the current SQL statement according to the sharding rules to generate the real SQL statement to be executed.\n**SQL Route:** Pisa-Proxy routes the rewritten SQL statements to the corresponding data source at the backend to execute the SQL statements according to the sharding rules.\n**SQL Execution:** Pisa-Proxy will rewrite the SQL statement and push it down to the back-end real database for execution.\n**Result Merge:** Pisa-Proxy merges the query results and returns them to the client.\n\n### Introducing SQL Rewriting\n\nSQL rewriting is a critical module in data sharding. Pisa-Proxy needs to rewrite the current SQL statement according to the sharding rules to generate the real SQL statement to be executed. SQL rewriting can be of the following types: \n\n#### Identifier Rewriting\n\nIdentifiers to be rewritten include table names, index names, and schema names.\n\nTable name rewriting means the process of finding the location of a logical table in the original SQL and rewriting it to a real table. Table name rewriting is a typical scenario that requires parsing of SQL. For example, if the logical SQL is:\n\n```\nSELECT order_id FROM order.t_order WHERE order_id = 1;\n```\n\n\nSuppose that the shard key of this table is `order_id` and `order_id=1`, and the number of sharding is specified as two, then the SQL statement would be as follows:\n\n```\nSELECT order_id FROM order.t_order_00001 WHERE order_id = 1;\n```\n\nThe following figure shows the data query process: \n\n![db mesh figure2.png](https://official-media.sphere-ex.com/db_mesh_figure2_dbe7559362.png)\n\nTaking data writing as an example, the data insertion process is as follows: \n\n![db mesh figure3.png](https://official-media.sphere-ex.com/db_mesh_figure3_2a91460b38.png)\n\n**Note:** When the SQL rewriting process modifies the identifier to calculate the real table name, it will automatically add the table index according to the sharding rule. The index rule is table name_index, and the index bit is five. For example, the t_order table is overwritten to t_order_00000. Therefore, you need to create the corresponding table name based on the actual business scenario.\n\n#### Column Supplement Rewriting\n\nThere are two cases that need supplement columns in a query statement. In the first case, Pisa-Proxy needs to get the data during the result merge, but the data is not returned by the queried SQL. \n\nIn this case, it mainly applies to GROUP BY and ORDER BY. When merging the results, you need to group and order the field items according to GROUP BY and ORDER BY, but if the original SQL does not contain grouping or ordering items in the selections, you need to rewrite the original SQL. \n\nFor instance, with an instance with the following SQL statement:\n\n```\nSELECT order_id, user_id FROM t_order ORDER BY user_id;\n```\n\nSince user_id is used for sorting, the data of user_id needs to be retrieved in the result merge, the above SQL statement contains the data of user_id, so there is no need to add columns, and modifying the identifier is enough.\n\n```\nSELECT order_id FROM t_order ORDER BY user_id;\n```\n\nThis SQL depends on the user_id for sorting. Therefore, the column must be supplemented. The rewritten SQL is as follows: \n\n```\nSELECT order_id, user_id AS USER_ID_ORDER_BY_DERIVED_00000 FROM t_order_00000 ORDER BY user_id;\n```\n\nThe second case of column supplement is the use of AVG aggregate functions. \n\nIn distributed scenarios, using (avg1 + avg2 + avg3)/3 to calculate the average is incorrect and should be rewritten as (sum1 + sum2 + sum3) /(count1 + count2 + count3). \n\nIn this case, rewriting the SQL containing AVG to SUM and COUNT is required, and recalculating the average when the results are merged. For example: \n\n```\nSELECT AVG(price) FROM t_order WHERE user_id = 1;\n```\n\nThe rewritten SQL is as follows: \n\n```\nSELECT COUNT(price) AS AVG_DERIVED_COUNT_00000, SUM(price) AS AVG_DERIVED_SUM_00000 FROM t_order_00000 WHERE user_id = 1;\n```\n\n### Configuration Description\n\nThis release supports query, update, deletion, and modification of the single database sharding based on a single shard key. The configuration items are as follows:\n\n![configuration description.png](https://official-media.sphere-ex.com/configuration_description_b62a3423c8.png)\n\n**Note:** The broadcast table, binding table, sub-query, table sharding, distributed rules based on expressions, distributed transaction, and cross-database Join will be gradually supported in later versions.\n\nTaking the scenario in Figure 2 and Figure 3 as an example, its corresponding CRD configuration is as follows: \n\n```\n# Declare a VirtualDatabase as a logical database\napiVersion: core.database-mesh.io/v1alpha1\nkind: VirtualDatabase\nmetadata:\n  name: test\n  namespace: default\nspec:\n  services:\n  - databaseMySQL:\n      db: test\n      host: 127.0.0.1\n      password: \"root\"\n      port: 3306\n      user: root\n    name: mysql\n    trafficStrategy: test\n    dataShard: test\n---\n# Declare TrafficStrategy the specified proxy\napiVersion: core.database-mesh.io/v1alpha1\nkind: TrafficStrategy\nmetadata:\n  name: test\n  namespace: default\nspec:\n  loadBalance:\n    simpleLoadBalance:\n      kind: random\n  selector:\n    matchLabels:\n      source: test\n---\n# Declare DataShard the sharding rules\napiVersion: core.database-mesh.io/v1alpha1\nkind: DataShard\nmetadata:\n  name: test\n  namespace: default\n  labels:\n    source: test\nspec:\n  rules:\n  - tableName: \"t_order\"\n    tableStrategy:\n      tableShardingAlgorithmName: \"mod\"\n      tableShardingColumn: \"id\"\n      shardingCount: 2\n    actualDatanodes:\n      valueSource:\n        nodes:\n        - value: \"ds001\"\n---\n# Declare DatabaseEndpoint the identified physical database\napiVersion: core.database-mesh.io/v1alpha1\nkind: DatabaseEndpoint\nmetadata:\n  labels:\n    source: test\n  name: ds001\n  namespace: default\nspec:\n  database:\n    MySQL:\n      db: test\n      host: mysql.default\n      password: root\n      port: 3306\n      user: root\n```\n\n\n### 2. Pisanix v0.3.0 Overview\nThis release has been made possible thanks to 102 merged PRs by the following contributors:\n\n![Pisanix 3 contributors.png](https://official-media.sphere-ex.com/Pisanix_3_contributors_fd029e9f69.png)\n\n#### New Features\n- **Pisa-Controller**\n  - Support DataShard CRD #326\n- **Pisa-Proxy**\n  - Support single database sharding #338\n \n#### Enhancements\n- **Pisa-Controller**\n  - Optimized Sidecar injection #234\n  - Support generic static read/write splitting rule #251\n- **Pisa-Proxy**\n  - Support SHOW STATUS parser #254\n  - Introduce CloudWatch Sinker for later audit events #258\n  - Support CREATE TABLESPACE parser #259\n  - Support CREATE SERVER #261\n  - Front MySQL protocol uses tokio codec #263\n  - Support dynamic read/write splitting Monitor switch #264\n  - Refactor MySQL server runtime #274\n  - Experimental SIMD in SQL Parser #278\n  \n#### Fixes\n- **Pisa-Proxy**\n  - Fix Apple M1 Macbook build failure in some cases #260\n  - Fix malformed SQL format in some cases #291\n  - Fix abnormal SQL parsing issues #297\n  - Fix unexpected AST in some cases #301\n  - Fix Pisa-Proxy crash without receiving VirtualDatabase configuration #319\n  \n#### Others\n- **Golang-SDK**\n  - Move CRD and kubernetes client to github.com/database-mesh/golang-sdk #247\n  - Add DataShard CRD and update VirtualDatabase with DataShard as new spec member in Golang-SDK\n- **Docs**\n  - Add sharding doc: https://www.pisanix.io/docs/Features/sharding\n  - Optimized microservices-demo deployment instruction #318\n  - Fork and optimize the code and configuration of github.com/microservices-demo/microservices-demo for a better demo. See: github.com/database-mesh/microservices-demo\n- **Charts**\n  - Update controller and proxy mirror to v0.3.0\n \n#### Known Issues\n- DO NOT support SHOW DATABASES and SHOW TABLES\n- DO NOT support observability under sharding mode\n\n### 3. Community\n\n![Database Mesh2.png](https://official-media.sphere-ex.com/Database_Mesh2_00ac6eb990.png)\n\n**Pisanix is an open source implementation of Database Mesh. It builds unified database governance based on Rust, and aims at providing four major development and application experiences: local database, unified configuration management, multi-protocol support, and cloud-native architecture.**\n\nThe community is currently collecting more user scenarios and applications. If you've started or are planning to test Pisanix, please record it in the [issue](https://github.com/database-mesh/pisanix/issues/282) below. \n\nThe community will prioritize the development and optimization of features related to real scenarios. \n\n![who's using pisanix.png](https://official-media.sphere-ex.com/who_s_using_pisanix_95f4415d05.png)\n\n【[Download Link](https://github.com/database-mesh/pisanix/releases/tag/v0.3.0)】\n\nThe Pisanix community organizes an online meeting every two weeks. Here are the details to join: \n- [Mailing list](https://groups.google.com/g/database-mesh)\n- [Biweekly meeting - English community](https://meet.google.com/yhv-zrby-pyt) (from February 16, 2022), on Wednesday 9:00 AM PST\n- [Biweekly meeting - Chinese community](https://meeting.tencent.com/dm/6UXDMNsHBVQO) (from April 27, 2022), on Wednesday 9:00 PM GMT+8\n- [Slack](https://databasemesh.slack.com/)\n- [Meeting minutes](https://bit.ly/39Fqt3x)\n\nYou're welcome to join us!","date":"2022-09-28","author":"SphereEx","excerpt":"Data sharding is an effective solution to deal with massive data storage and computation. \n\nPisanix now provides data sharding governance capability based on the underlying database, allowing users to scale out computing and storage.","createdAt":"2022-09-29T08:17:46.414Z","updatedAt":"2022-09-29T10:19:33.554Z","publishedAt":"2022-09-29T08:18:42.213Z","locale":"en","newsType":{"data":null},"cover":{"data":{"id":396,"attributes":{"name":"Database Mesh2.png","alternativeText":"Database Mesh2.png","caption":"Database Mesh2.png","width":1080,"height":414,"formats":{"thumbnail":{"name":"thumbnail_Database Mesh2.png","hash":"thumbnail_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":245,"height":94,"size":20.45,"path":null,"url":"https://official-media.sphere-ex.com/thumbnail_Database_Mesh2_00ac6eb990.png"},"large":{"name":"large_Database Mesh2.png","hash":"large_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":1000,"height":383,"size":126.15,"path":null,"url":"https://official-media.sphere-ex.com/large_Database_Mesh2_00ac6eb990.png"},"medium":{"name":"medium_Database Mesh2.png","hash":"medium_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":750,"height":288,"size":89.97,"path":null,"url":"https://official-media.sphere-ex.com/medium_Database_Mesh2_00ac6eb990.png"},"small":{"name":"small_Database Mesh2.png","hash":"small_Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","width":500,"height":192,"size":52.99,"path":null,"url":"https://official-media.sphere-ex.com/small_Database_Mesh2_00ac6eb990.png"}},"hash":"Database_Mesh2_00ac6eb990","ext":".png","mime":"image/png","size":60.86,"url":"https://official-media.sphere-ex.com/Database_Mesh2_00ac6eb990.png","previewUrl":null,"provider":"strapi-provider-upload-s3-compat","provider_metadata":null,"createdAt":"2022-09-29T08:05:47.915Z","updatedAt":"2022-09-29T08:05:47.915Z"}}},"localizations":{"data":[]}}}}},
    "staticQueryHashes": []}