Parallel notebook operations

Shared locks keep notebook helpers safe. Calls touching the same notebook are serialized, calls touching different notebooks can proceed independently, and notebook execution uses one global semaphore.

Parallelism is useful only if notebook writes remain deterministic. Each notebook lock combines an in-process re-entrant lock with an OS file lock, held from read through commit and export. That makes concurrent Python writes to the same notebook wait for the active transaction to finish.

with notebook_locks("nbs/01_read.ipynb", "nbs/02_write.ipynb"):
    ...

with execution_slot():
    ...

Production contract

Concurrency is production infrastructure. Per-notebook locks must serialize operations on the same notebook, acquire multiple notebook locks in stable order, allow independent notebooks to proceed in parallel, and keep notebook execution behind one global execution slot.

from contextlib import redirect_stdout
from io import StringIO
import threading, time
from nbskill.parallel import execution_slot as _example_execution_slot
from nbskill.parallel import notebook_key as _example_notebook_key
from nbskill.parallel import notebook_locks as _example_notebook_locks
from nbskill.foundation import demo_path, remove_demo_path, write_demo_notebook
import subprocess, sys, time

print(_example_notebook_key("nbs/../nbs/01_read.ipynb"))
with _example_notebook_locks("nbs/01_read.ipynb", "nbs/02_write.ipynb"):
    print("notebook locks acquired")
with _example_execution_slot():
    print("execution slot acquired")

Shared coordination state

The lock registry and execution gate live in one small module so every notebook operation uses the same coordination policy. Safe in-process execution is serialized here; wrappers that launch a child process keep notebook locks around the target paths.

Stable path keys

Notebook paths can be relative, absolute, or not created yet. notebook_key normalizes them into a consistent lock key without requiring the file to already exist.

---

source

notebook_key

def notebook_key(
    path
):

Return a stable lock key for a notebook path.

Per-notebook locks

notebook_locks acquires all requested notebook locks in sorted order. That lets operations touching multiple notebooks avoid deadlocks while still allowing unrelated notebooks to be edited in parallel.

---

source

notebook_locks

def notebook_locks(
    *paths
):

Acquire per-notebook locks in a stable order.

One execution slot

Notebook execution mutates kernels and outputs, so execution is serialized globally. Read and write operations can still use per-notebook locks around their own files.

---

source

execution_slot

def execution_slot():

Serialize notebook execution across parallel Python calls.

root = demo_path("09_parallel_keys")
try:
    root.mkdir()
    a = root / "a.ipynb"
    b = root / "sub" / ".." / "b.ipynb"
    assert notebook_key(a).endswith("a.ipynb")
    assert notebook_key(b).endswith("b.ipynb")
    assert notebook_key(None) is None
finally:
    remove_demo_path(root)

with write_demo_notebook("09_parallel_same.ipynb") as path:
    entered = []
    def enter_same_lock():
        with notebook_locks(path):
            entered.append(True)
    with notebook_locks(path):
        thread = threading.Thread(target=enter_same_lock)
        thread.start()
        time.sleep(0.05)
        assert entered == []
    thread.join(timeout=1)
    assert entered == [True]

active = 0
max_active = 0
guard = threading.Lock()

def hold_execution_slot():
    global active, max_active
    with execution_slot():
        with guard:
            active += 1
            max_active = max(max_active, active)
        time.sleep(0.05)
        with guard:
            active -= 1

threads = [threading.Thread(target=hold_execution_slot) for _ in range(3)]
for thread in threads: thread.start()
for thread in threads: thread.join()
assert max_active == 1

assert _example_notebook_key("nbs/../nbs/01_read.ipynb") == _example_notebook_key("nbs/01_read.ipynb")
_acquired = False
with _example_notebook_locks("nbs/01_read.ipynb", "nbs/01_read.ipynb", None):
    _acquired = True
assert _acquired