feat(materialize): add new backend for Materialize streaming database #11703
Conversation
github-actions
bot
added
docs
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
4 times, most recently
from
35b0a11 to
349400c
Compare
|
@jasonhernandez Thanks for the PR! I will try to give it a review this weekend or later this evening. It looks like the materialize test suite is timing out at 60 minutes. Any idea what's going on there? |
cpcloud
left a comment
cpcloud
left a comment
There was a problem hiding this comment.
Looks very good! This is one of the most thorough initial backend PRs I've ever seen, great work.
A few comments, mostly around exception handling and too-broad-try clauses.
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
3 times, most recently
from
39004fc to
46a9f19
Compare
I was hoping you might have known of something obvious that might cause that timeout. I'm going to investigate it further. |
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
2 times, most recently
from
7e62c90 to
bf29fa1
Compare
def-
force-pushed
the
feat/add-Materialize-backend
branch
13 times, most recently
from
e130a91 to
e410ec0
Compare
|
@cpcloud I added two commits - the first one specifically addresses the feedback on this review. The second one makes some fixes to testing (resolving the timeouts of the backend test), bumps to the latest version of Materialize, and generally incorporates recommendations from our head of QA. I can squash these all together / submit a new PR or change this however would simplify review / merging on your end. |
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
from
1f2df71 to
4dca804
Compare
|
@cpcloud I just wanted to check when you might have a chance to look at this PR again. I know it's rather massive. |
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
2 times, most recently
from
33d1163 to
aab8f79
Compare
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
from
aab8f79 to
fcd8142
Compare
|
I rebased this on #11806 because the Exasol tests were failing. |
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
from
fcd8142 to
36d22df
Compare
The exasol/docker-db:2025.1.3 image is not available on Docker Hub. Exasol has published images with the 2015.1.8 tag instead. This fixes CI failures when running Exasol tests.
This commit adds comprehensive support for Materialize, including: - Core backend implementation with cluster and connection management - Support for materialized views, indexes, sources, sinks, and secrets - Idiomatic Materialize SQL patterns (mz_now(), SUBSCRIBE, etc.) - Test coverage for Materialize-specific features - Integration with existing Ibis test suite Co-Authored-By: Seth Wiesman <sjwiesman@gmail.com>
This commit addresses several issues to improve Materialize test reliability: - Update to Materialize v26 syntax: Replace deprecated COUNTER load generator with AUCTION, update to FOR ALL TABLES syntax for multi-output sources - Fix test parallelization: Serialize array, source, and subscribe tests to avoid resource contention and subsource name collisions - Optimize data loading: Cache list_tables() result during CSV loading to reduce expensive database calls - Fix doctest failures: Reduce to 54 doctest +SKIP markers matching postgres backend pattern (68% reduction) - Fix dead fixture warnings: Mark TPC benchmark tests as skipped instead of deselecting them to prevent pytest-deadfixtures from flagging their fixtures - Fix test_map: Make assertion Materialize-specific to avoid breaking other backends' pytest diff output - Remove deprecated enable_load_generator_counter and enable_create_table_from_source flags from compose.yaml Co-Authored-By: Dennis Felsing <dennis@felsing.org>
Materialize does not support MERGE INTO syntax, which the upsert() method uses. This commit marks the upsert tests with pytest.mark.notyet for materialize, similar to how other backends that don't support MERGE INTO are handled (risingwave, mysql, sqlite, etc.). The following tests are now expected failures for Materialize: - test_upsert_from_dataframe - test_upsert_from_memtable (all variants) - test_upsert_from_expr (both variants)
jasonhernandez
force-pushed
the
feat/add-Materialize-backend
branch
from
36d22df to
56854b9
Compare
## Description of changes Update Exasol Docker image tag to 2025.1.8 in tests, replacing the tag 2025.1.3 that is no longer available. Reference: https://community.exasol.com/t/docker-images-for-2025-1-versions/206 This was noticed while re-basing and testing PR #11703 (edit: original PR targeted the latest-2025.1 tag) Co-authored-by: Jason Hernandez <7144515+jasonhernandez@users.noreply.github.com>
|
hi @cpcloud is there any chance you might have some availability to take another look at this PR or recommend another maintainer take a look? |
|
hi @gforsyth - let me know if you have some capacity to review this new backend. Thanks! |
|
Somehow I thought we had merged this backend 🤔 |
- Fix 0-based to 1-based array index conversion in `visit_ArrayIndex` by adding +1 to both positive and negative index cases - Remove `@pytest.mark.notyet` decorators for CASE expression tests since Materialize has fixed the parser bug that rejected bare string literals in simple CASE expressions - Bump Materialize Docker image to v26.7.0 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
I just bumped the Materialize version and rebased. There are a few improvements to our test coverage / support in the last commit. |
2 participants
Description of changes
Adds a Materialize backend for Ibis
This PR introduces a new backend for Materialize, a live data layer with native support for generalized incremental view maintenance. Materialize strives to maintain PostgreSQL wire compatibility, but it offers a number of unique capabilities that require expanding beyond the existing Postgres backend for Materialize users. In particular, Materialize provides:
The Materialize team is excited to collaborate with the Ibis community and is committed to maintaining and improving this backend going forward.