songleaf.kaggle_index

A search index of the Kaggle corpus: the same songs and keys, without the corpus in memory.

KaggleChordsSource holds the whole corpus as a DataFrame: about 1.1 GB per process, 1.6 GB once a lyrics search has run, and every web worker holds its own copy. build_kaggle_index() writes it once to one SQLite file beside the zip (datasets/chords-and-lyrics/): titles, artists, popularity, each chart zlib-compressed, and an FTS5 table of the charts’ words. KaggleChordsIndex reads that file. It keeps only the titles, artists and popularity in memory (for fuzzy matching, a few tens of MB) and answers lyrics searches and song lookups from disk.

Search results are the DataFrame source’s: the ranking code is shared, and only where the data comes from differs. The one difference is how lyrics split into words: FTS5’s tokenizer treats _ as a separator and does not compose decomposed accents, where the DataFrame source’s regular expression does the opposite. Rebuilding the index while a server runs is safe: a process that notices the file changed reloads it.

The file format, its writing and its raw reads belong to sung.chords_index (stdlib only, so reading the index never imports pandas); this module adds where songleaf keeps the file and the ranking shared with the DataFrame source.

>>> import pandas as pd, tempfile, os
>>> corpus = pd.DataFrame({
...     "Unnamed: 0": [1, 2], "artist_name": ["The Band", "Other"],
...     "song_name": ["Paper Boats", "Quiet Road"], "lang": ["en", "en"],
...     "popularity": [3, 1], "chords&lyrics": ["G\nPaper boats drift", "C\nquiet road"],
... })
>>> path = os.path.join(tempfile.mkdtemp(), "index.sqlite")
>>> build_kaggle_index(path=path, loader=lambda: corpus)["songs"]
2
>>> [hit.key for hit in KaggleChordsIndex(path).search(lyrics="boats")]
['kaggle_chords:1']

Module Attributes

KAGGLE_INDEX_NAME

The index's file name, beside the corpus zip under datasets/.

Functions

build_kaggle_index(*[, path, loader, batch_size])	Write the index of the Kaggle corpus (loader: as for KaggleChordsSource).
kaggle_index_path()	Where the index is read from: beside the corpus zip under the data root.

Classes

KaggleChordsIndex([path, min_score])

The Kaggle corpus read from its index (build_kaggle_index()).

songleaf.kaggle_index.KAGGLE_INDEX_NAME = 'chords-and-lyrics.sqlite'

The index’s file name, beside the corpus zip under datasets/.

class songleaf.kaggle_index.KaggleChordsIndex(path=None, *, min_score=70)

Bases: KaggleChordsSource

The Kaggle corpus read from its index (build_kaggle_index()).

The same name and ids as KaggleChordsSource, so song keys and stored songs carry over. Only the columns fuzzy matching needs are held in memory; charts and lyrics matches come from the file.

get(song_id)

The song with this id, parsed into a Song.

search(query='', **constraints)

Songs matching every given constraint, best first.

search_tiers(query, *, limit=10)

The songs whose title, artist or lyrics match query, by where they match.

{"title": [...], "artist": [...], "lyrics": [...]}, at most limit in each, best first. A song is in the first kind it matches, even when it ranks below the limit there, so a song is never shown as an artist match because more titles matched.

Title and artist match word by word (WordIndex): the query is inside the field ("wonder" finds Wonderwall) or the field is inside the query ("wonderwall oasis" finds the title Wonderwall and the artist Oasis), a word of four letters or more forgiving a typo; or every word of the query is in the title or in the artist ("paper boats invented band"). A song whose title holds a word of the query that its artist does not is a title match ("wonderwall oasis": Wonderwall (ver. 2) by Oasis), otherwise an artist match. Within each kind, the songs whose title and artist together hold the most words of the query come first (so Wonderwall by Oasis leads), then the closest by fuzzy score (of the title, or of title and artist, whichever is closer; of the artist), then the most popular. Lyrics hold every word of the query, rank by popularity, and carry meta["snippet"]: the line that matched, with its spans.

songleaf.kaggle_index.build_kaggle_index(*, path=None, loader=None, batch_size=5000)

Write the index of the Kaggle corpus (loader: as for KaggleChordsSource).

Loading the corpus takes the memory the index exists to avoid, so run this once, apart from any server (songleaf build-index). The index format and its writing are sung.chords_index’s; this picks the corpus and the path.

Return type:

dict

songleaf.kaggle_index.kaggle_index_path()

Where the index is read from: beside the corpus zip under the data root.

When the datasets live on S3 (songleaf.datasets), this is the cached download of the index (fetched on first use), a real file for SQLite.

Return type:

str