A channel can be worth following without every upload being worth reading. A video can contain useful information without needing forty minutes to explain it. And a perfectly competent summary can still leave the most important question unanswered: why should I care?
Those are different problems. Building 1 Minute Signal meant giving each of them a place in the system.
I wanted to follow developments in AI, software, technology, and startups without spending my available time watching videos or sorting through repetitive coverage. The goal was a reading experience that could surface the useful substance, preserve the supporting context, and help me think about the implications.
The YouTube pipeline is one implementation of that goal. It monitors selected channels, acquires transcripts, generates structured content, adds tags and source timestamps, and scores the result before deciding where it can appear.
The critical distinction is that successfully processing a video does not automatically earn it a place in the feed. Processing, public availability, feed promotion, and notifications have separate rules.
This article follows that workflow and the failures that shaped it. The architecture overview explains the surrounding platform, but here the focus is the path from a source upload to a finished signal.
A bounded workflow with model judgment inside it
The pipeline runs asynchronously in a NestJS worker on Cloud Run. Cloud Scheduler initiates work through an enqueue endpoint, and Cloud Tasks delivers authenticated execution requests to the worker. PostgreSQL holds processing state and completed content. The reader-facing applications consume those stored results.
The model does not decide which channels to monitor, when to retry a transcript, or whether a score qualifies for the home feed. Those decisions belong to application code and configuration. Models handle the work that requires interpretation: compressing a transcript, writing a signal, proposing tags, mapping supporting points to timestamps, and evaluating quality and relevance.
The successful processing path. Metadata and signal generation run in parallel. Timestamp mapping starts after both finish. A failed timestamp mapping can be omitted; failed core content stops the run. Retry and recovery paths are described below.
That division gives the system a predictable sequence while allowing model judgment at specific stages. It also makes failures more useful: I can distinguish a missing transcript from invalid generated content, or a completed item from one that failed its relevance threshold.
Start with a curated source set, then track individual videos
The first filter is human: staff maintain the set of enabled channels. This establishes a source pool worth monitoring, without assuming that every upload belongs in the product.
The configured schedule runs every six hours, at 00:00, 06:00, 12:00, and 18:00 UTC. Each run looks back up to 30 days and considers up to 50 videos per channel. These are discovery limits, not claims about how many videos the system processes or publishes.
Discovery uses the YouTube Data API’s uploads-playlist listing. The question is narrow: what has this known channel uploaded recently? The implementation uses that path to conserve quota instead of performing a general video search.
Looking back across a window also allows the worker to encounter an earlier video whose transcript was unavailable on a previous attempt. Database state determines whether it needs another attempt.
| Recorded state | Default processing behavior |
|---|---|
SUCCESS | Skip; a completed result already exists |
IN_PROGRESS | Skip; work is already marked as underway |
NO_TRANSCRIPT | Retry only after a 22-hour cooldown, up to 10 recorded misses |
FAILED | Retry on the next eligible default run |
| Explicit reprocess request | Bypass the normal early status gate for repair or regeneration |
There is one summary row per YouTube video ID. That prevents the same identified video from becoming a new summary every time discovery sees it.
It does not detect two different uploads containing the same talk, interview, or announcement. The current implementation has identity-based deduplication, not semantic deduplication. That limitation matters for a product whose purpose includes reducing repetition.
Within a channel, videos are processed serially, and a scheduled backfill walks enabled channels one at a time inside a single task. A six-hour single-flight lease (youtube_backfill_all) guards that overall run against a second overlapping backfill. The lease is released when the run finishes; if the process dies first, it expires after six hours. That lease is not a global lock. The YouTube task queue allows two concurrent dispatches, so a tag backfill on the same queue can overlap a processing run. Each worker instance handles one request at a time, and the service can scale to five instances.
The result is intentionally bounded. At this stage, controlling provider usage and understanding recovery behavior matter more than maximizing parallel throughput.
Transcript availability is a state problem
A discovered video is only a candidate. The pipeline still needs usable source text.
The transcript chain currently tries:
- Supadata’s native transcript mode, requesting English timestamped text.
- SerpAPI’s YouTube transcript engine, when that fallback is enabled.
- Supadata’s generated transcript mode.
All providers feed the same normalized snippet structure: start and end times, text, and a display timestamp. Later stages can work with one internal representation instead of each provider’s response format.
Some Supadata requests return a job identifier rather than an immediate transcript. The worker polls those jobs with a bounded wait. On an explicit reprocess, it can reuse valid raw transcript text and snippets already stored in the database, avoiding another paid fetch.
The important recovery distinction is between source availability and provider capacity.
When the full transcript chain fails to produce a transcript, the system records a transcript miss. A cooldown avoids repeatedly spending requests on the same unavailable material. The miss cap prevents that recovery loop from continuing indefinitely.
A Supadata quota-exceeded condition follows a different path: the pipeline records FAILED, rather than placing the video into the missing-transcript cooldown. A provider quota problem says something about the service’s ability to fulfill the request, not whether the video has usable content.
That distinction came out of operational iteration. If every unsuccessful fetch becomes “no transcript,” temporary service problems can suppress recoverable work for the wrong reason. Conversely, retrying genuine transcript misses at every opportunity wastes requests.
The current policy still has a tradeoff: if a quota condition persists, the next eligible run can fail again. Distinguishing causes makes recovery more appropriate; it does not make every dependency failure disappear.
Compress once before writing for readers
A raw transcript contains more than its useful claims. It may include introductions, repetition, promotion, digressions, and speech-recognition errors. Passing all of it independently to every writing stage gives each model another opportunity to choose a different interpretation of what matters.
The first model stage produces a structured Markdown compression: an internal representation of the source’s useful substance. It is persisted as soon as it passes validation, even if later stages fail.
Two writing stages then consume that compression in parallel:
| Stage | Responsibility |
|---|---|
| Metadata and analysis | Titles, feed preview, extended summary, analysis, questions, and legacy content fields |
| Signal | The reader-facing signalSummary, organized as The Signal, The Case, and The Read |
Parallel execution reduces the waiting time for these two stages compared with running them sequentially. Both calls still consume tokens; concurrency does not eliminate their combined inference cost.
The default production path requires valid compression. If compression fails, the run fails rather than silently switching the downstream writers to raw transcript input. Raw-transcript experiments remain possible in the staff sandbox, but they are an explicit alternative.
That rule keeps the production workflow consistent. Otherwise, the same apparent pipeline could produce content through two materially different paths depending on an upstream failure.
Compression also creates a risk: downstream writers inherit its omissions and mistakes. A clean intermediate artifact is easier to inspect than a long transcript, but it is still model output. Preserving it makes the transformation visible; it does not prove that every important nuance survived.
Give relative dates an explicit reference
One concrete issue was temporal context. A speaker may say “this year” in a video recorded long before the pipeline processes it. Without the right context, the generated text can resolve that phrase to the wrong year.
The compression and summarization stages now receive the video’s publication date and the current run date in UTC. Those dates give the model a reference for interpreting relative time.
A remaining dependency is that the signal writer trusts the compression’s account of what matters. If compression gets a year wrong, downstream writing can repeat the error. Correcting that case requires revisiting compression, not merely polishing the final sentence.
This is the kind of failure that shaped the pipeline: a small ambiguity at an early stage can become confident public prose several steps later.
Write a signal that separates the finding, its support, and its implications
The public signal has three recognizable parts:
- The Signal: the main finding, understandable without watching the video.
- The Case: supporting claims, mechanisms, or examples from the source.
- The Read: interpretation, implications, or a useful caveat.
The following is an illustrative format, not output from a particular video:
The Signal: A concise account of the central finding and why it matters.
The Case:
- A concrete supporting claim from the source.
- A mechanism or example that explains the claim.
- A relevant constraint or qualification.
The Read: What the finding could mean for builders, including an
important condition or limitation.
This structure gives readers somewhere to look for both substance and interpretation. It also gives application code something more specific to validate than “the model returned a string.”
That became necessary because schema-valid JSON could still contain an incomplete signal. A response can satisfy its outer contract while failing the product requirement inside it.
The pipeline therefore checks signal completeness and applies bounded structural retries. If the core signal remains incomplete, the run fails.
Another issue was more mundane: models sometimes collapsed several Case bullets onto one line. That damaged the relationship between individual points and their timestamp chips. Shared normalization now expands those collapsed bullets before persistence so the interface receives consistent point identifiers.
These controls check structure and usability. They do not establish that every claim is true. The YouTube workflow transforms and evaluates a source; it should not be confused with the separate research pipeline that assembles evidence across multiple sources.
Enrich the result without making every feature mandatory
The pipeline also tries to map Case points back to transcript timestamps. This lets a reader move from a concise supporting point toward the relevant part of the source video.
Timestamp mapping is useful, but its failure policy differs from the main signal’s. If mapping remains incomplete after structural retries, the breakdown can be stored as null and the pipeline can still succeed. Publishing an incomplete core signal is unacceptable; publishing a complete signal without optional timestamp navigation is tolerable.
That distinction prevents an enhancement from becoming a universal availability dependency.
Tags are generated from the reader-facing text, rather than directly from the compression. This helps the tag set describe the content the reader actually encounters, instead of topics that appeared in the source but were omitted from the finished piece.
The tag stage proposes two to six labels with defined types. Unknown labels enter a candidate state for operational review. An illustrative output shape is:
{
"tags": [
{ "label": "OpenAI", "type": "organization" },
{ "label": "inference cost", "type": "topic" }
]
}
Metadata has its own persistence rules. A URL slug is established when the summary is created and remains stable during routine updates, so regenerating content does not casually change its address.
There is also deliberate transitional complexity: the system still writes legacy summary fields alongside the newer signal format. Hiding an old field in the interface does not mean every consumer has stopped needing it. Maintaining both formats provides compatibility and rollback options, but also keeps extra generation work in the pipeline.
A generated item must earn each level of visibility
After enrichment, a model scores the signal on two separate axes, each from 0 to 10:
- Quality: the quality assessment of the generated content.
- Relevance: its fit for the publication’s intended audience and mission.
Keeping the axes separate matters. A well-presented piece can be irrelevant to builders. A relevant subject can receive weak treatment.
Application rules then apply the scores to different surfaces. The current defaults are:
| Surface | Quality threshold | Relevance threshold | Result |
|---|---|---|---|
| Public corpus | 5 | 5 | Eligible for public detail pages, browse surfaces, and sitemaps |
| Main feed | 7 | 7 | Eligible for promotion into the main reading experience |
| Push notifications | 9 | 7 | Eligible for subscribed-summary notification handling |
These are configured thresholds, not measured accuracy rates. A quality score of 9 is not a 90% probability that a video’s claims are correct.
The public routes enforce eligibility when querying content. Items below the public bar return a 404 rather than exposing a public detail page merely because a database row exists. Null scores fail the visibility thresholds.
Main-feed eligibility also requires main_feed_published_at. The system sets that timestamp when the item meets the main-feed bar. The source’s published_at remains the original YouTube publication date.
Those timestamps answer different questions: when did the source appear, and when did this item earn promotion into the feed? Keeping them separate avoids briefly showing unscored content simply because its generation finished.
A scoring failure is logged and does not necessarily fail the entire processing run. A completed row can remain unpromoted until it is rescored or an operator applies an override.
This is where the distinction between processing and publication becomes concrete. The worker may have produced valid content, but readers should encounter it only when the relevant visibility conditions are satisfied.
Scoring is an editorial control, not independent verification
The scores themselves come from a model. Deterministic thresholds make their consequences consistent; they do not make the underlying judgments objective.
A polished summary can preserve a weak claim from its source. A model can reward confident writing or miss context. Staff score overrides provide a correction mechanism, but they are not evidence that every item receives manual review.
The present system provides relevance and quality filtering. Demonstrating how well those filters work requires reviewing examples and comparing judgments with human assessments. The existence of a score column alone cannot establish that the feed is trustworthy.
Most filtering happens after the expensive work
The current workflow makes an important economic tradeoff: publication scoring happens after transcript acquisition, compression, and summarization.
That means a video can consume those resources and still fail to qualify for any public surface. The score gate protects reader attention more directly than it protects the processing budget.
A cheaper relevance check before full summarization was considered and deferred. It remains a potential cost improvement, but it introduces a different risk: discarding useful material before the pipeline has extracted enough context to recognize its value.
Today, cost and concurrency are bounded through several existing controls:
- A curated channel set, discovery window, and per-channel discovery cap.
- Serial work inside a scheduled backfill, a six-hour backfill lease, a YouTube queue limit of two concurrent dispatches, and one request at a time per worker instance (up to five instances).
- Transcript reuse during reprocessing and persistence of valid compression before later work completes.
- A switch for disabling the SerpAPI transcript fallback.
- Sampled production tracing and a sandbox for testing individual stages.
These controls target different expenses. Reusing a transcript avoids a provider fetch; isolating a prompt experiment avoids rerunning unrelated stages. None substitutes for measuring the actual cost per attempted video and per feed-worthy result.
What the recorded rows show
This is a census of the database on 2026-09-23.
57 channels are currently enabled and assessed for new videos every 6 hour cycle.
| Latest recorded outcome | Videos | Share |
|---|---|---|
SUCCESS | 6420 | 95.11% |
NO_TRANSCRIPT | 159 | 2.36% |
FAILED | 5 | 0.07% |
IN_PROGRESS | 166 | 2.46% |
Visibility below uses the same default thresholds as the table above. A staff override wins over the model score when one is set. Of the same videos:
- 63.38% meet the public bar (quality and relevance at least 5).
- 49.82% meet the main-feed bar (at least 7 and 7) and have a main-feed publish time.
- 5.78% meet the push notification bar (quality at least 9 and relevance at least 7).
- 25.27% completed summaries are scored and fall below the public bar.
- 11.35% of the completed summaries are unscored.
Time from the YouTube publish timestamp to creation of the summary row is a median of 4.8 hours. That interval includes waiting for the next discovery run. It is not queue delay, model execution time, or the worker request timeout.
Recovery depends on which part failed
There is no single retry loop that fixes every failure.
The implementation has distinct handling for provider transport, structured-output parsing, content completeness, database access, transcript availability across runs, and task delivery. For example, OpenRouter transport retries cover selected transient network, rate-limit, and server failures. They do not treat length limits, content filtering, or refusals as ordinary transport failures to retry indiscriminately.
The important boundaries look like this:
| Failure | Current consequence |
|---|---|
| Invalid compression or persistently incomplete core signal | Fail the pipeline run |
| Incomplete timestamp breakdown | Omit the breakdown; core content may still succeed |
| Failed scoring | Log the failure; content may remain unpromoted pending rescore or override |
| Full transcript chain produces no usable transcript | Record a miss and apply cooldown and miss-cap rules |
| Supadata quota exceeded | Record pipeline failure rather than a transcript miss |
| Transient database connection failure after long model work | Apply bounded database retries |
Long model calls exposed a practical database issue: a connection that had been idle during generation could be stale when persistence resumed. The response was targeted transient retry handling and connection-pool tuning.
Cloud Tasks separately handles task-delivery retries. That layer is useful when execution fails at the task boundary; it does not replace the application’s per-video state or validation rules.
There is still an unresolved recovery gap worth making explicit. Default processing skips IN_PROGRESS videos. If an interrupted execution leaves one behind, the normal status gate can keep skipping it. A per-video expiry or reconciliation policy for abandoned work remains a follow-up. The overall backfill lease does not, by itself, solve stale item state.
Make prompt changes inspectable before sending them through production
Structured GCP logs record pipeline events, scoring outcomes, and external-fetch activity. Pipeline run identifiers help connect those events across a run.
Langfuse adds model-level tracing, with production YouTube work configured for 50% sampling and sandbox runs traced at 100%. Tracing is optional and fail-open: a tracing problem should not become a content-processing failure.
Sampling limits observability cost, but it also creates a blind spot. Because sampling happens before the outcome is known, a failed run may have no Langfuse trace. GCP logs remain necessary for investigating unsampled failures.
For cost accounting, the integration uses reported OpenRouter usage cost when available rather than relying only on catalog estimates. That is useful for model spend; it is not a complete measure of transcript-provider, infrastructure, or operational cost.
The staff sandbox is equally important. It can run compression, metadata, signal, tags, and breakdown stages without writing the resulting content into the production database. That makes it possible to inspect a prompt change or isolate a formatting failure without treating every experiment as a full content regeneration.
Repairs have explicit operational paths: targeted reprocessing, score overrides, tag regeneration, and paced backfills. The editorial corrections process remains a human workflow; it is not automatically connected to every reprocess operation.
What I would change next
The strongest parts of this design are the explicit intermediate artifacts and the separation of processing from promotion. They make it possible to examine why a piece exists, what happened to it, and why it appears on a particular surface.
My next improvements would focus on wasted work and incomplete recovery.
Test early relevance triage before relying on it. A lightweight screen could reduce processing spend on clearly unsuitable videos. I would first compare its rejected candidates with results from the full pipeline, looking especially for useful material it would have discarded.
Add recovery for abandoned in-progress items. Per-video state needs a clear path back to eligibility after interrupted execution, with safeguards against restarting work that is still genuinely active.
Improve failure trace coverage. The useful goal is lower trace volume for routine successes while retaining diagnostic detail for failures. Achieving that requires a different capture or retention approach; sampling a run out at the start cannot recover its missing detail afterward.
Retire compatibility work when consumers are ready. The legacy fields have a purpose while they support readers or rollback. Once those dependencies are removed, continuing to generate them becomes avoidable cost.
Evaluate semantic deduplication and stronger date checks. Both address gaps the present controls cannot fully close: different video IDs can repeat the same material, and explicit date context can still be interpreted incorrectly. These would need evaluation against real examples rather than being treated as solved by adding another model call.
For a smaller first version, I would begin with fewer moving parts: a curated source list, one transcript provider, one structured output, and one publication gate. Provider fallbacks, parallel writers, optional enrichment, and multiple promotion tiers become valuable when the product and its observed failures justify them.
The reader should not need to understand any of that machinery. They should encounter a useful finding, enough supporting context to assess it, and a clear reason to care. The engineering work is what lets that short experience have a more demanding process behind it.
Related Build Notes
- Why I Built 1 Minute Signal: the information-overload problem and the product’s purpose.
- 1 Minute Signal Architecture: How the AI Content Intelligence Platform Works: the services, data stores, and infrastructure surrounding this pipeline.