# Reference implementation knowledgebase This notebook stores favourite implementation repositories in one local LanceDB database. It keeps registered sources, indexed README/module/function/class/method records, and direct same-repo caller/callee links for examples an agent can inspect before writing similar code. Library searches inspect the target workspace before looking for a version. A relevant installed direct dependency uses its exact version, downloads that public PyPI release only when it is missing, then parses notebooks for nbdev projects and Python files for other projects. The index reuses embeddings for unchanged file hashes and keeps only the newest three indexed versions per package. #### Production contract Reference knowledge is explicit and local. It registers repositories, ingests Python and notebook implementation records, preserves dependency and branch context, and returns structured examples through one query path. #### Imports Reference indexing uses filesystem scans, Python AST parsing, notebook reads, LanceDB storage, and sentence-transformers embeddings. ``` python import warnings ``` #### Reference implementation knowledgebase The reference knowledgebase stores favourite implementation repositories in one local LanceDB database. It keeps three tables: `reference_repos` for registered sources, `reference_items` for README/module/function/class/method records, and `reference_edges` for direct same-repo caller/callee links. This feature requires both LanceDB and sentence-transformers. Ingestion writes a full sentence-transformers vector for every indexed item, and queries combine structured filters, LanceDB full-text search, and vector similarity over that searchable vector store. If either dependency or the configured embedding model is unavailable, reference ingestion/querying fails clearly instead of falling back to a weaker embedding. The public workflow is: 1. `reference_add(...)` registers a local path or GitHub repository. 2. `reference_ingest(...)` clones with `gh repo clone` when possible, extracts Python and notebook symbols, and writes LanceDB rows. 3. `reference_query(...)` returns examples, dependency status, optional branch context, and increments each returned item’s `returned_count`. Reference knowledge examples below use tiny local fixtures. They document the public API without touching the global user database. Small executable examples appear after the reference API definitions so they can call the exported functions directly. Small executable examples appear after the reference API definitions so they can call the exported functions directly. ## Registry and storage The registry records where reference code came from. It keeps repository metadata separate from indexed implementation rows so re-ingesting one source can replace its records without disturbing the others. ------------------------------------------------------------------------ source ### reference_home ``` python def reference_home( path:NoneType=None ): ``` *Return the global reference knowledge directory.* ------------------------------------------------------------------------ source ### reference_add ``` python def reference_add( url:str, # Repository URL or local repository path name:str | None=None, # Stable registry name; defaults from URL/path version:str='HEAD', # Git ref to ingest package:str | None=None, # Import/distribution package name path:str | None=None, # Override reference home ): ``` *Register or update one reference repository.* ## Retrieval and relevance The search layer combines metadata filters, lexical search, vectors, and small domain rules. The rules matter because a query for a specific helper should not be promoted by an unrelated generic name. #### Specific reference relevance [`reference_query`](https://MorsCerta-crypto.github.io/nbskill/knowledge.html#reference_query) separates exact identifier intent from generic query fragments. A hit must cover the specific terms of a multi-concept request; otherwise it is reported as low confidence instead of being promoted by an incidental symbol component. ## Repository extraction Indexed rows retain enough source context to make a returned result useful without opening a whole repository. Python and notebook readers share the same normalized row format. ------------------------------------------------------------------------ source ### reference_ingest ``` python def reference_ingest( name:str | None=None, # Registry name to ingest all:bool=False, # Ingest all registered repositories path:str | None=None, # Override reference home force:bool=False, # Reingest even when resolved version appears unchanged ): ``` *Clone registered references and index Python/notebook implementations.* ------------------------------------------------------------------------ source ### reference_list ``` python def reference_list( path:str | None=None ): ``` *Return registered reference repositories and indexed item counts.* #### Problem-solution memory Each new pair needs at least four distinct namespaced tags from `topic`, `sub-topic`, `library`, `problem-category`, and `solution-category`. Records also keep the repository, commit, verification command, and verification time. Queries keep tag filtering exact. Hits report matching tags and score components. A query with no exact match reports the miss, counts broader candidates, and suggests available tags instead of returning a loose match. ## Problem-solution memory Problem memory records a reusable observation beside the source-code index. It is separate from reference retrieval because an agent may need a prior decision even when no code example is relevant. ------------------------------------------------------------------------ source ### problem_statement_add ``` python def problem_statement_add( problem:str, # Reusable problem statement solution:str, # Reusable solution statement task:str='', # Task where this pair was observed project:str='.', # Project path or label evidence:str='', # Short evidence that the solution worked outcome:str='applied', # applied, partial, failed, skipped, etc. tags:NoneType=None, # At least four namespaced tags repository:str='', # Repository containing the verified evidence commit:str='', # Source commit for the evidence verification:str='', # Command that verified the solution verified_at:float | None=None, # Verification time as a Unix timestamp path:str | None=None, # Override reference home ): ``` *Store or update one reusable problem-solution statement.* ------------------------------------------------------------------------ source ### problem_statement_query ``` python def problem_statement_query( query:str, # New problem or task to compare against stored memories top_k:int=5, # Number of memories to return project:str | None=None, # Optional project filter tags:NoneType=None, # Optional tags; every requested tag must be present path:str | None=None, # Override reference home ): ``` *Find reusable problem-solution statements for a new task.* ------------------------------------------------------------------------ source ### problem_statement_list ``` python def problem_statement_list( path:str | None=None, limit:int=20 ): ``` *List recent reusable problem-solution statements.* Fastcore 2 migration verification runs this test before the later reference-query setup imports. Keep the demo-path helpers local to the test that uses them. Small reference API examples: #### Workspace-aware queries The query path reads the target workspace before looking up implementation examples. That keeps a direct dependency on its installed version and prevents a local editable package from being replaced by a public download. ------------------------------------------------------------------------ source ### reference_propose ``` python def reference_propose( problem:str, # Observed problem in the library repo:str, # Indexed reference repository name version:str | None=None, # Optional indexed version symbol:str | None=None, # Optional affected symbol current_repo:str='.', # Workspace using the library path:str | None=None, # Override reference home ): ``` *Prepare an upstream change proposal without creating a pull request.* ------------------------------------------------------------------------ source ### reference_query ``` python def reference_query( query:str, # Natural-language implementation query top_k:int=3, # Number of implementation hits to return include_branch:bool=False, # Include direct same-repo callers and callees current_repo:str='.', # Workspace whose installed libraries should be reused repos:NoneType=None, # Optional repo name or comma-separated names to search path:str | None=None, # Override reference home kind:str | None=None, # Optional item kind filter package:str | None=None, # Optional package filter and auto-download target version:str | None=None, # Optional package version module:str | None=None, # Optional module filter symbol:str | None=None, # Optional symbol filter include_local:bool=True, # Include current repository reuse advice candidate_k:int | None=None, # Candidate pool size before final reranking explain:bool=True, # Include score breakdowns and why text allow_download:bool=True, # Download public source for a relevant package ): ``` *Search reference code using the target workspace’s installed version.* ``` python example_home = reference_home(demo_path("reference-example-home")) assert example_home.name == "reference-example-home" remove_demo_path(example_home) ``` ``` python example_vector = _reference_vector("make page endpoint") assert len(example_vector) > 100 ``` Reference repositories are managed through the Python registry. The executable contract uses a local git repository so ingestion never depends on network access. #### Retrieval Experiments This small experiment compares four search surfaces on a tiny structured corpus: - `tags_fts`: LanceDB full-text search over tags plus symbol/doc text, with no vector similarity. - `docstring`: vector similarity over docstrings only. - `signature_doc_return`: vector similarity over the signature, docstring, and return statements. - `full_body`: vector similarity over the whole indexed source block. The expected symbols mirror common fastcore lookups: `store_attr`, `patch`, and `compose`. ## Retrieval experiments The experiment compares searchable representations on a tiny fixed corpus. It is kept separate from the production query path so it can explain the tradeoff without changing query behavior. ``` python def _experiment_returns(source): try: tree = ast.parse(source or "") except SyntaxError: return "" vals = [] for node in ast.walk(tree): if isinstance(node, ast.Return) and node.value is not None: try: vals.append(ast.unparse(node.value)) except Exception: pass return "\n".join(vals) ``` ``` python def _experiment_text(row, mode): if mode == "docstring": return row.get("docstring", "") if mode == "signature_doc_return": return "\n".join([row.get("signature", ""), row.get("docstring", ""), _experiment_returns(row.get("source", ""))]) if mode == "full_body": return row.get("source", "") return " ".join([*(row.get("tags") or []), row.get("symbol", ""), row.get("signature", ""), row.get("docstring", "")]) ``` ``` python def _experiment_table(home, rows, mode): table_rows = [] for row in rows: text = _experiment_text(row, mode) table_rows.append({**row, "search_text": text, "vector": _reference_vector(text)}) table = _reference_db(home).create_table(f"reference_experiment_{mode}", data=table_rows, mode="overwrite") try: table.create_fts_index("search_text", replace=True) except Exception: pass return table ``` ``` python def _experiment_search(table, query, mode, top_k=3): if mode == "tags_fts": rows = table.search(query, query_type="fts").limit(top_k).to_list() else: rows = table.search(_reference_vector(query)).metric("cosine").limit(top_k).to_list() return [row["symbol"] for row in rows] ``` ``` python experiment_home = demo_path("reference-experiment-home") experiment_rows = [ _item_row("mini", "v1", "mini", "function", "mini.basics", "store_attr", Path("mini.py"), "def store_attr(names=None, self=None, **attrs):\n return _store_attr(self, **attrs)", signature="def store_attr(names=None, self=None, **attrs)", docstring="Store params named in comma-separated names from calling context into attrs in self."), _item_row("mini", "v1", "mini", "function", "mini.basics", "patch", Path("mini.py"), "def patch(f=None, *, cls_method=False):\n return patch_to(cls_method=cls_method)(f)", signature="def patch(f=None, *, cls_method=False)", docstring="Decorator: add f to the first parameter's class based on type annotations."), _item_row("mini", "v1", "mini", "function", "mini.basics", "compose", Path("mini.py"), "def compose(*funcs):\n def _inner(x):\n for f in funcs: x = f(x)\n return x\n return _inner", signature="def compose(*funcs)", docstring="Create a function that composes all functions in funcs."), _item_row("mini", "v1", "mini", "function", "mini.foundation", "flatmap", Path("mini.py"), "def flatmap(f, xs):\n return [y for x in xs for y in f(x)]", signature="def flatmap(f, xs)", docstring="Apply f to each element and flatten the results into a single list."), ] experiment_queries = { "store constructor arguments as attributes": "store_attr", "patch a method onto a class": "patch", "compose functions into a pipeline": "compose", } experiment_results = {} try: for mode in ("tags_fts", "docstring", "signature_doc_return", "full_body"): table = _experiment_table(str(experiment_home), experiment_rows, mode) experiment_results[mode] = { query: _experiment_search(table, query, mode, top_k=3) for query in experiment_queries } finally: remove_demo_path(experiment_home) experiment_results ``` ``` python experiment_accuracy = { mode: sum(hits and hits[0] == expected for query, expected in experiment_queries.items() for hits in [mode_results[query]]) for mode, mode_results in experiment_results.items() } assert experiment_accuracy["signature_doc_return"] >= experiment_accuracy["docstring"] assert experiment_accuracy["signature_doc_return"] >= experiment_accuracy["full_body"] experiment_accuracy ``` ## Version-aware sources A relevant direct dependency is indexed at the version the target workspace uses. Public PyPI downloads are only a fallback when that exact source is not already available. #### Reusing local repositories [`reference_discover`](https://MorsCerta-crypto.github.io/nbskill/knowledge.html#reference_discover) stores reference roots and registers Git repositories at the root or one directory below it. Local names include a path hash, so two repositories with the same folder name remain separate. ## Local repository discovery Discovery only registers Git repositories at configured roots. Limiting the scan to roots and their direct children keeps a background index from unexpectedly walking an entire home directory. ------------------------------------------------------------------------ source ### reference_discover ``` python def reference_discover( roots:NoneType=None, # Comma-separated or iterable reference roots path:str | None=None, # Override reference home ingest:bool=True, # Index each discovered repository force:bool=False, # Re-ingest unchanged repositories ): ``` *Register direct-child Git repositories from configured roots.* #### Workspace versions and upstream proposals [`reference_query`](https://MorsCerta-crypto.github.io/nbskill/knowledge.html#reference_query) asks the target workspace’s virtual environment which distributions are installed. It only downloads public source for a relevant package, so returned code can match the version the project already uses. [`reference_propose`](https://MorsCerta-crypto.github.io/nbskill/knowledge.html#reference_propose) writes review material only; it never creates a fork or pull request. ## Small public examples [`reference_home`](https://MorsCerta-crypto.github.io/nbskill/knowledge.html#reference_home) is deliberately simple. It makes the storage location explicit, which is useful when an agent needs an isolated database for a test or a workspace-specific index. ``` python reference_home("demo-knowledge") ``` #### Query planning [`_query_feature_plan`](https://MorsCerta-crypto.github.io/nbskill/knowledge.html#_query_feature_plan) makes the non-obvious parts of retrieval visible before any database work starts. It records the requested evidence, likely implementation kinds, and the search passes that will be applied. ``` python _query_feature_plan("find a FastHTML route test") ``` The focused check keeps the public storage location and query-intent classification stable without requiring LanceDB, a model, or a repository fixture. #### Lazy embedding dependency Sentence-transformers is imported only when vector search is needed. That keeps registry and metadata commands usable without loading a model at module import time.