hearing¶

Pluggable meeting transcription & context-connected AI agents — batch today, live next.

hearing captures, transcribes, and reasons over meeting audio as four small, swappable concerns wired by composition, not baked together:

capture ──audio──▶ STT ──segments──▶ diarization ──labeled──▶ agents
(mic + system,     (local or          (who spoke              (summaries, actions,
 separate channels) cloud, pluggable)  when / "me vs them")     live notes, research)

The one-liner just works; everything else is an optional keyword argument.

from hearing import transcribe

transcript = transcribe("meeting.wav")  # -> Transcript (list of segments)
print(transcript.formatted())

Install¶

pip install hearing                 # core: data model, channel split, CLI
pip install 'hearing[whisper]'      # + default local STT engine (faster-whisper)
pip install 'hearing[agents]'       # + Claude-backed agent layer
pip install 'hearing[all]'          # whisper + agents + capture + http + storage (dol)
pip install 'hearing[diarize]'      # + pyannote (separate individual remote speakers)

STT engines¶

One STTEngine facade, many engines; pick one with --engine (CLI) or get_engine(name) (Python). Nothing downstream changes.

`--engine`	Engine	Install	Needs
`whisper` (default)	faster-whisper, local, any CPU/GPU	`hearing[whisper]`	—
`openai`	OpenAI transcription API (cloud)	`hearing[openai]`	`OPENAI_API_KEY`
`deepgram`	Deepgram Nova-3 (cloud; `--model nova-2` etc. swaps it)	`hearing[deepgram]`	`DEEPGRAM_API_KEY`
`mlx`	Whisper on Apple Silicon via mlx-whisper	`hearing[mlx]`	macOS on M-series
`parakeet`	NVIDIA Parakeet-TDT on Apple Silicon via parakeet-mlx	`hearing[mlx]`	macOS on M-series, plus `ffmpeg` on PATH (parakeet-mlx decodes audio with it)
`whisperx`	WhisperX: aligned word timestamps, optional pyannote diarization	`hearing[whisperx]`	`HF_TOKEN` (diarization only)

The live path (hearing live, stream_transcribe) works with every engine: engines without native streaming are fed VAD-segmented utterances. --model overrides the engine’s default model (a whisper size such as small for whisper/mlx/whisperx, a model name for cloud engines).

What works today (the batch milestone)¶

Give it a recording and get back a speaker-labelled transcript plus optional AI meeting notes. If the file has the mic on one channel and the system audio (everyone else) on another, you get “me vs them” labelling for free — no diarization model, no voice enrollment — because the channel itself tells you who is local.

hearing transcribe meeting.wav                      # diarized transcript to stdout
hearing transcribe meeting.wav --model small --out notes.json
hearing transcribe meeting.wav --engine openai      # cloud STT (OpenAI); needs OPENAI_API_KEY + hearing[openai]
hearing transcribe meeting.wav --engine deepgram    # cloud STT (Deepgram Nova-3); needs DEEPGRAM_API_KEY + hearing[deepgram]
hearing transcribe meeting.wav --engine mlx         # local, Apple Silicon (hearing[mlx]); also: parakeet, whisperx
hearing transcribe meeting.wav --save ./meetings    # persist the transcript (then `hearing meetings ./meetings`)
hearing summarize  meeting.wav                      # transcribe + AI notes (Claude)
hearing summarize  meeting.wav --agent extractive   # offline, no API key needed
hearing summarize  meeting.wav --context-dir ./project_notes   # context-connected: RAG over your docs
hearing summarize  meeting.wav --web-search         # also pull Wikipedia fact context (key-free)
hearing live       meeting.wav                      # STREAM it: finalized segments as utterances complete
hearing meetings   ./meetings                       # list saved transcripts (--show ID to print one)
hearing info                                        # what's installed / available

Context-connected agents. Point --context-dir at a folder of .md/.txt notes (prior meetings’ takeaways, project docs); the agent does RAG over them and brings in facts that aren’t in the transcript (who owns what, prior decisions, budgets, deadlines). The retriever and a web-search hook are pluggable Protocols (hearing/context.py): KeywordRetriever (TF-IDF, dependency-free, the default) or EmbeddingRetriever (OpenAI semantic recall) — choose with --retriever keyword|embedding. Swap in any vector backend behind the same Retriever interface.

Live / streaming (milestone 2 — core working)¶

The same pipeline runs as a low-latency loop: VAD finds utterance boundaries, each finalized utterance is transcribed and emitted as it completes, and the channel split still labels “me vs them” — the STTEngine and agent interfaces are unchanged from batch.

import asyncio
from hearing import live_transcribe
from hearing.capture import (
    StreamingFileCapture,
)  # or DeviceCapture for a live mic+system device


async def main():
    async for seg in live_transcribe(source=StreamingFileCapture("meeting.wav")):
        print(
            seg.speaker, seg.text
        )  # finalized (seg.meta['final']) segments, as they land


asyncio.run(main())

Capturing from a live device (DeviceCapture, an Aggregate Device with mic

BlackHole) is implemented but needs that hardware to verify; streaming a file works anywhere.

Command palette (⌘K). Press ⌘K (or click the button) for a command palette — export the transcript (markdown/SRT/JSON), focus search, clear. Each command is defined once (frontend/src/commands.ts, the acture pattern) and also projects to an AI/MCP tool schema via toAITools(), so the same definition can drive the palette and an AI assistant. Run the frontend’s unit tests with npm test.

Example transcript + Claude summary (from examples/demo_meeting.py, which synthesizes a two-speaker meeting with macOS say — no mic needed):

[00:00:00] me:   Hi everyone, thanks for joining the planning sync. I'll own the
                 API design and send a draft by Friday. What is our deadline...
[00:00:08] them: Thanks. The launch deadline is the end of the month. Let's
                 decide on the database. I think we should use Postgres...

## Decisions
- PostgreSQL selected as the database for the project.
## Action items
- [me] Own the API design and send a draft by Friday.
## Open questions / follow-ups
- Confirm the exact end-of-month launch deadline.

Composition (dependency injection)¶

Every concern is a typing.Protocol; swap one by changing a single argument.

from hearing import transcribe
from hearing.stt import FasterWhisperSTT
from hearing.diarize import ChannelTrickDiarizer, PyannoteDiarizer
from hearing.agents import ClaudeAgent, ExtractiveAgent

from hearing.stt import OpenAISTT  # or DeepgramSTT, MLXSTT, WhisperXSTT

transcribe(
    "meeting.wav",
    engine=OpenAISTT(),  # cloud STT — one line vs FasterWhisperSTT()
    diarizer=PyannoteDiarizer(),  # upgrade from the channel trick
    agent=ClaudeAgent(context="prior meeting takeaways..."),  # or ExtractiveAgent()
)

The shared data spine is hearing.types.TranscriptSegment — a frozen, standoff interval annotation (time is integer milliseconds, never float). See hearing/interfaces.py for the four facade Protocols (CaptureSource, STTEngine, Diarizer, AgentConsumer).

Capturing the audio (macOS)¶

hearing reads multi-channel audio files and splits the channels. To get a recording with mic on one channel and system audio (the other participants) on another, use BlackHole + an Aggregate Device or Core Audio taps — the mechanics, the channel layout, and the iOS limits are documented in the hearing-audio-capture agent skill (.claude/skills/hearing-audio-capture/). Live device capture is the next milestone; today you bring a recording.

Not yet (designed, on the roadmap)¶

Native streaming engines (Deepgram websocket, OpenAI Realtime, WhisperLive) behind the same STTEngine facade — today every engine streams through VAD-segmented batch calls — and semantic turn detection — see hearing-live-pipeline.
Live device capture verified on hardware (DeviceCapture is written; needs a BlackHole + Aggregate Device) — see hearing-audio-capture.
TypeScript frontend (schema-driven transcript viewer + copilot overlay with zodal/acture) — see hearing-frontend.

Full status in misc/docs/ROADMAP.md.

Web UI (HTTP API + React frontend)¶

A FastAPI layer wraps the same facades, and a schema-driven React frontend (zodal + Zod) renders the diarized transcript (me/them lanes) and the AI notes.

pip install 'hearing[http,whisper,agents]'
hearing serve                      # API on http://127.0.0.1:8000 (docs at /docs)

cd frontend && npm install && npm run dev   # UI on http://localhost:5173 (proxies /api)

Then drop an audio file on the page → diarized transcript + AI meeting notes. Tick Live stream to watch finalized segments append in real time as the live pipeline produces them (POST /api/transcribe/stream, NDJSON). The TypeScript data shapes live in frontend/src/schema.ts as zodal collections (one Zod schema per surface) and double as the API response validator, so the Python backend and the UI share one contract. See the hearing-frontend skill.

Architecture & development¶

This project is developed with a suite of agent skills under .claude/skills/ — start with the hearing skill (the project-manager / orchestrator), which routes to the per-concern skills (hearing-architecture, hearing-audio-capture, hearing-stt, hearing-diarization, hearing-agents, hearing-live-pipeline, hearing-frontend). The two research reports the design is grounded in are in misc/docs/.

License¶

MIT

This documentation as a single file: hearing.md (Markdown, for agents).

built 2026-10-14 05:27 UTC from e6ef523 (main) · hearing 0.0.7 · about this build