* . *
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Tuesday, July 1, 2025
Earth-News
  • Home
  • Business
  • Entertainment
    Nantucket Dance Festival opens July 8 – The Inquirer and Mirror

    Nantucket Dance Festival Launches with Thrilling Performances Beginning July 8

    A Secret Society, Ritualistic Killings, and a Century-Old Curse Netflix and YRF Entertainment’s ‘Mandala Murders’ Premieres July 25 – About Netflix

    A Secret Society, Ritualistic Killings, and a Century-Old Curse: Dive into the Chilling World of ‘Mandala Murders’ Premiering July 25

    Susquehanna Raises Penn Entertainment Inc. (PENN) Price Target. – Yahoo Finance

    Susquehanna Raises Price Target for Penn Entertainment Inc. (PENN)

    George Lopez is coming to Spokane – KXLY.com

    George Lopez is coming to Spokane – KXLY.com

    Netflix unveils Dallas immersive venue for fans of hit shows like ‘Squid Game,’ ‘Stranger Things’ – Houston Chronicle

    Step Inside Netflix’s New Dallas Immersive Experience Featuring Hits Like ‘Squid Game’ and ‘Stranger Things

    ‘Puttin’ on the Ritz’: Civic Players bring ‘Young Frankenstein’ to life – Yahoo

    Civic Players Deliver a Hilarious and Unforgettable Performance of ‘Young Frankenstein

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Owls inspire new revolutionary noise reduction technology – KTEN

    Owls inspire new revolutionary noise reduction technology – KTEN

    New center coming to Mizzou will focus on energy research and technology – Columbia Missourian

    Mizzou Launches Innovative New Center Dedicated to Energy Research and Technology

    Mirrors in space and underwater curtains: can technology buy us enough time to save the Arctic ice caps? – The Guardian

    Can Technology Like Space Mirrors and Underwater Curtains Buy Us Time to Save the Arctic Ice Caps?

    Naples restaurant owner prepares for hurricane season with new flood technology – Fox4Now.com

    Naples restaurant owner prepares for hurricane season with new flood technology – Fox4Now.com

    Emerging Memory and Storage Technology Market Analysis Report 2025-2034 | AI and HPC Boom Fuels Surging Demand for Fast, Low-Power Memory Devices – Yahoo Finance

    How AI and HPC Are Driving Explosive Growth in Fast, Low-Power Memory Technologies Through 2034

    Ostin Technology (OST): Volatility’s Warning or Contrarian Opportunity? – AInvest

    Ostin Technology (OST): Navigating Market Volatility – Red Flag or Hidden Opportunity?

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
  • Home
  • Business
  • Entertainment
    Nantucket Dance Festival opens July 8 – The Inquirer and Mirror

    Nantucket Dance Festival Launches with Thrilling Performances Beginning July 8

    A Secret Society, Ritualistic Killings, and a Century-Old Curse Netflix and YRF Entertainment’s ‘Mandala Murders’ Premieres July 25 – About Netflix

    A Secret Society, Ritualistic Killings, and a Century-Old Curse: Dive into the Chilling World of ‘Mandala Murders’ Premiering July 25

    Susquehanna Raises Penn Entertainment Inc. (PENN) Price Target. – Yahoo Finance

    Susquehanna Raises Price Target for Penn Entertainment Inc. (PENN)

    George Lopez is coming to Spokane – KXLY.com

    George Lopez is coming to Spokane – KXLY.com

    Netflix unveils Dallas immersive venue for fans of hit shows like ‘Squid Game,’ ‘Stranger Things’ – Houston Chronicle

    Step Inside Netflix’s New Dallas Immersive Experience Featuring Hits Like ‘Squid Game’ and ‘Stranger Things

    ‘Puttin’ on the Ritz’: Civic Players bring ‘Young Frankenstein’ to life – Yahoo

    Civic Players Deliver a Hilarious and Unforgettable Performance of ‘Young Frankenstein

  • General
  • Health
  • News

    Cracking the Code: Why China’s Economic Challenges Aren’t Shaking Markets, Unlike America’s” – Bloomberg

    Trump’s Narrow Window to Spread the Truth About Harris

    Trump’s Narrow Window to Spread the Truth About Harris

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    Israel-Gaza war live updates: Hamas leader Ismail Haniyeh assassinated in Iran, group says

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    PAP Boss to Niger Delta Youths, Stay Away from the Protest

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Court Restricts Protests In Lagos To Freedom, Peace Park

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Fans React to Jazz Jennings’ Inspiring Weight Loss Journey

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • Science
  • Sports
  • Technology
    Owls inspire new revolutionary noise reduction technology – KTEN

    Owls inspire new revolutionary noise reduction technology – KTEN

    New center coming to Mizzou will focus on energy research and technology – Columbia Missourian

    Mizzou Launches Innovative New Center Dedicated to Energy Research and Technology

    Mirrors in space and underwater curtains: can technology buy us enough time to save the Arctic ice caps? – The Guardian

    Can Technology Like Space Mirrors and Underwater Curtains Buy Us Time to Save the Arctic Ice Caps?

    Naples restaurant owner prepares for hurricane season with new flood technology – Fox4Now.com

    Naples restaurant owner prepares for hurricane season with new flood technology – Fox4Now.com

    Emerging Memory and Storage Technology Market Analysis Report 2025-2034 | AI and HPC Boom Fuels Surging Demand for Fast, Low-Power Memory Devices – Yahoo Finance

    How AI and HPC Are Driving Explosive Growth in Fast, Low-Power Memory Technologies Through 2034

    Ostin Technology (OST): Volatility’s Warning or Contrarian Opportunity? – AInvest

    Ostin Technology (OST): Navigating Market Volatility – Red Flag or Hidden Opportunity?

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
No Result
View All Result
Earth-News
No Result
View All Result
Home Technology

Marmot: Multi-writer distributed SQLite based on NATS

December 11, 2023
in Technology
Marmot: Multi-writer distributed SQLite based on NATS
Share on FacebookShare on Twitter

Go Report Card
Discord
GitHub

What & Why?

Marmot is a distributed SQLite replicator with leaderless, and eventual consistency. It allows you to build a robust replication
between your nodes by building on top of fault-tolerant NATS JetStream.

So if you are running a read heavy website based on SQLite, you should be easily able to scale it out by adding more SQLite replicated nodes.
SQLite is probably the most ubiquitous DB that exists almost everywhere, Marmot aims to make it even more ubiquitous for server
side applications by building a replication layer on top.

Quick Start

Download latest Marmot and extract package using:

tar vxzf marmot-v*.tar.gz

From extracted directory run examples/run-cluster.sh. Make a change in /tmp/marmot-1.db using:

bash> sqlite3 /tmp/marmot-1.db
sqlite3> INSERT INTO Books (title, author, publication_year) VALUES (‘Pride and Prejudice’, ‘Jane Austen’, 1813);

Now observe changes getting propagated to other database /tmp/marmot-2.db:

bash> sqlite3 /tmp/marmot-2.db
sqlite3> SELECT * FROM Books;

You should be able to make changes interchangeably and see the changes getting propagated.

Out in wild

Here are some official, and community demos/usages showing Marmot out in wild:

2-node HA for edge Kubernetes – Using Marmot
Scaling Isso with Marmot on Fly.io
Scaling PocketBase with Marmot on Fly.io
Scaling PocketBase with Marmot 0.4.x
Scaling Keystone 6 with Marmot 0.4.x

What is the difference from others?

Marmot is essentially a CDC (Change Data Capture) and replication pipeline running top of NATS. It can automatically configure appropriate
JetStreams making sure those streams evenly distribute load over those shards, so scaling simply boils down to adding more nodes, and
re-balancing those JetStreams (auto rebalancing not implemented yet).

There are a few solutions like rqlite, dqlite, and
LiteFS etc. All of them either are layers on top of SQLite (e.g.
rqlite, dqlite) that requires them to sit in the middle with network layer in order to provide
replication; or intercept physical page level writes to stream them off to replicas. In both
cases they require a single primary node where all the writes have to go, and then these
changes are applied to multiple readonly replicas.

Marmot on the other hand is born different. It’s born to act as a side-car to your existing processes:

Instead of requiring single primary, there is no primary! Which means any node can make changes to its local DB.
Marmot will use triggers to capture your changes, and then stream them off to NATS.
Instead of being strongly consistent, Marmot is eventually consistent. Which means no locking, or blocking of nodes.
It does not require any changes to your existing SQLite application logic for reading/writing.

Making these choices has multiple benefits:

You can read, and write to your SQLite database like you normally do. No extension, or VFS changes.
You can write on any node! You don’t have to go to single primary for writing your data.
As long as you start with same copy of database, all the mutations will eventually converge
(hence eventually consistent).

What happens when there is a race condition?

In Marmot every row is uniquely mapped to a JetStream. This guarantees that for any node to publish changes for a row it has to go through
same JetStream as everyone else. If two nodes perform a change to same row in parallel, both of the nodes will compete to publish their
change to JetStream cluster. Due to RAFT quorum
constraint only one of the writer will be able to get its changes published first. Now as these changes are applied (even the publisher applies
its own changes to database) the last writer will always win. This means there is NO serializability guarantee of a transaction
spanning multiple tables. This is a design choice, in order to avoid any sort of global locking, and performance.

Limitations

Right now there are a few limitations on current solution:

Marmot does not support schema changes propagation, so any tables you create or columns you change won’t be reflected.
This feature is being debated and will be available in future
versions of Marmot.
You can’t watch tables selectively on a DB. This is due to various limitations around snapshot and restore mechanism.
WAL mode required – since your DB is going to be processed by multiple processes the only way to have multi-process
changes reliably is via WAL.
Marmot is eventually consistent – This simply means rows can get synced out of order, and SERIALIZABLE assumptions
on transactions might not hold true anymore. However your application can choose to redirect writes to single node
so that your changes are always replayed in order.

Features

Eventually Consistent
Leaderless Replication
Fault Tolerant
Built on NATS

Leaderless replication never requiring a single node to handle all write load.

Ability to snapshot and fully recover from those snapshots. Multiple storage options for snapshot:

Built with NATS, abstracting stream distribution and replication.

Support for log entry compression, handling content heavy CMS needs.

Sleep timeout support for serverless scenarios.

Dependencies

Starting 0.8+ Marmot comes with embedded nats-server with JetStream support. This not only reduces
the dependencies/processes that one might have to spin up, but also provides with out-of-box tooling like
nat-cli. You can also use existing libraries to build additional
tooling and scripts due to standard library support. Here is one example using Deno:

deno run –allow-net https://gist.githubusercontent.com/maxpert/d50a49dfb2f307b30b7cae841c9607e1/raw/6d30803c140b0ba602545c1c0878d3394be548c3/watch-marmot-change-logs.ts -u -p -s

The output will look something like this:
image

Production status

v0.8.x introduced support for embedded NATS. This is recommended version for production.
v0.7.x moves to file based configuration rather than CLI flags, and S3 compatible snapshot storage.
v0.6.x introduces snapshot save/restore. It’s in pre-production state.
v0.5.x introduces change log compression with zstd.
v0.4.x introduces NATS based change log streaming, and continuous multi-directional sync.
v0.3.x is deprecated, and unstable. DO NOT USE IT IN PRODUCTION.

CLI Documentation

Marmot picks simplicity, and lesser knobs to configure by choice. Here are command line options you can use to
configure marmot:

config – Path to a TOML configuration file. Check out config.toml comments for detailed documentation
on various configurable options.
cleanup (default: false) – Just cleanup and exit marmot. Useful for scenarios where you are
performing a cleanup of hooks and change logs.
save-snapshot (default: false Since 0.6.x) – Just snapshot the local database, and upload snapshot
to NATS/S3 server
cluster-addr (default: none Since 0.8.x) – Sets the binding address for cluster, when specifying
this flag at-least two nodes will be required (or replication_log.replicas). It’s a simple
: pair that can be used to bind cluster listening server.

Since v0.8.4 Marmot will automatically expose a leaf server on :. This is
intended to reduce the number for flags. So if you expose cluster on port 4222 the port 4223 will
be automatically a leaf server listener.

cluster-peers (default: none Since 0.8.x) – Comma separated list of nats://:/ peers of
NATS cluster. You can also use (Since version v0.8.4 ) dns://:/ to A/AAAA record lookups.
Marmot will automatically resolve the DNS IPs at boot time to expand the routes with value of
nats://:/ value, where is replaced with all the DNS entries queried. There
are two additional query parameters you can use:

min – forcing Marmot to wait for minimum number of entries (e.g. dns://foo:4222/?min=3 will require
3 DNS entries to be present before embedded NATs server is started)
interval_ms – delay between DNS queries, which will prevent Marmot from flooding DNS server.

leaf-server (default: none Since v0.8.4 )- Comma separated list of nats://:/
or dns://:/ just like cluster-peers can be used to connect to a cluster
as a leaf node.

For more details and internal workings of marmot go to these docs.

FAQs & Community

For FAQs visit this page
For community visit our discord or discussions on GitHub

Our sponsor

Last but not least we would like to thank our sponsors who have been supporting development of this project.

GoLand logo.
JetBrains Logo (Main) logo.

>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : Hacker News – https://github.com/maxpert/marmot

Tags: MarmotMulti-writertechnology
Previous Post

In Shanghai, a Splash of Comfort in Elderly Care: Home Bathing Services

Next Post

Ego killed the empowered product team

These 9 towns were named the ‘quirkiest’ in New Jersey, according to World Atlas – Bergen Record

These 9 towns were named the ‘quirkiest’ in New Jersey, according to World Atlas – Bergen Record

July 1, 2025
Trump’s Tax Bill Won’t Help or Hurt the Economy. The Deficit Is Already Huge. – Barron’s

Trump’s Tax Bill Won’t Help or Hurt the Economy. The Deficit Is Already Huge. – Barron’s

July 1, 2025
Nantucket Dance Festival opens July 8 – The Inquirer and Mirror

Nantucket Dance Festival Launches with Thrilling Performances Beginning July 8

July 1, 2025
Who would be affected by health care cuts in Senate version of Trump’s budget bill – PBS

Who Will Be Hit Hardest by Health Care Cuts in the Senate’s Trump Budget Plan?

July 1, 2025
Senate passes Trump’s reconciliation bill with Vance casting tie-breaking vote – PBS

Senate Passes Trump’s Reconciliation Bill After Vance Casts Decisive Tie-Breaking Vote

July 1, 2025
Groundbreaking technology to help police tackle violence against women – The Independent

Revolutionary Technology Transforms the Fight Against Violence Toward Women

July 1, 2025
Apple Inc. (AAPL) Adds Tennis To Its List of Supported Sports Ahead of Wimbledon – Yahoo Finance

Apple Inc. (AAPL) Adds Tennis To Its List of Supported Sports Ahead of Wimbledon – Yahoo Finance

July 1, 2025
Nature’s headlamps: A unique light-focusing structure in Parasesarma de Man, 1895 mangrove crabs – ESA Journals

Nature’s headlamps: A unique light-focusing structure in Parasesarma de Man, 1895 mangrove crabs – ESA Journals

July 1, 2025
Scientists found what just might be Earth’s oldest rocks – Boy Genius Report

Scientists found what just might be Earth’s oldest rocks – Boy Genius Report

July 1, 2025
The EU must show leadership and stand up for science, health and women’s rights – The Parliament Magazine

Why the EU Needs to Take the Lead in Protecting Science, Health, and Women’s Rights

July 1, 2025

Categories

Archives

July 2025
MTWTFSS
 123456
78910111213
14151617181920
21222324252627
28293031 
« Jun    
Earth-News.info

The Earth News is an independent English-language daily published Website from all around the World News

Browse by Category

  • Business (20,132)
  • Ecology (701)
  • Economy (726)
  • Entertainment (21,615)
  • General (15,655)
  • Health (9,766)
  • Lifestyle (731)
  • News (22,149)
  • People (726)
  • Politics (733)
  • Science (15,943)
  • Sports (21,223)
  • Technology (15,710)
  • World (707)

Recent News

These 9 towns were named the ‘quirkiest’ in New Jersey, according to World Atlas – Bergen Record

These 9 towns were named the ‘quirkiest’ in New Jersey, according to World Atlas – Bergen Record

July 1, 2025
Trump’s Tax Bill Won’t Help or Hurt the Economy. The Deficit Is Already Huge. – Barron’s

Trump’s Tax Bill Won’t Help or Hurt the Economy. The Deficit Is Already Huge. – Barron’s

July 1, 2025
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

No Result
View All Result

© 2023 earth-news.info

Go to mobile version