Generic Protocol

Spark Python Data Source API (generic, bring-your-own connector)

Category: Generic Protocol
Data type: N/A
Content type: N/A
Direction: Source and Sink (DSv2 read AND write are both part of the API, including streaming)
Wire / network protocol: Python (Arrow-based I/O) -- in-process, not a network protocol
Engine: Spark
Product edition: Roadmap (requires Spark upgrade beyond 3.5.3 -- new in Spark 4.0.0, does not exist in 3.5.x)
Versions tested: New in Spark 4.0.0; absent from Spark 3.5.x (our current 3.5.3)
Status: Roadmap (P3)

Evidence & sources

Visit Spark Python Data Source API (generic, bring-your-own connector) →

Notes. Lets developers implement custom batch AND streaming data sources entirely in Python -- no JVM/Scala code or JDBC driver required -- with Arrow-based I/O, filter pushdown, and DSv2 write support (added incrementally through 4.0 and 4.1). This is the Spark-side analog of the OSS 'Generic JDBC (bring-your-own driver)' and the PrestoDB 'Thrift Connector (generic)' rows already on this matrix, and would meaningfully lower the bar for adding long-tail/customer-specific sources (Python-only SaaS clients, proprietary REST APIs, etc.) without a JVM driver. Confirmed absent from Spark 3.5.x by checking the 3.5-series release notes and migration guide, which do not mention it; first appears in the 4.0.0 release notes.