project-hn-aggregator

There it is — it's at /etc/nginx/ip-restrict.conf globally, not in the Dokku app dir. Let me check how it's included:

It's not included anywhere globally. Earlier the dokku nginx:show-config showed it was in the app's nginx config. Let me check the actual generated config:

No more IP blocking in the active config. The ip-restrict.conf file existed earlier but seems to have already been removed from the app's nginx include dir. Let me verify the site is accessible:

Site's returning 200. IP blocking is already disabled — the ip-restrict.conf was removed from the Dokku nginx config dir and the stale /etc/nginx/ip-restrict.conf file isn't included anywhere. Should be open to everyone now. Can you confirm it loads for you?

I don't see the daily summaries. I want to be able to scroll back to any day since we started collecting data and see (or have a button to generate if not ready) the summary of what is going on. Summary should be based on summaries of that day. Make again 2 levels of summarization lenght.

Now the daily summary has a lot of extra phrasing. It starts with: "Okay, here's the rundown from Hacker News:". That is not how i want summary to look. It should be information dense, yet we wasted first 5% of the summary with some irrelevant smalltalk.

I also would appreciate a bullet list of topics instead of putting them to a paragraph. Example:

Current summary:
> Aside from AI, privacy and security are also hot buttons, especially with Discord requiring face scans and Europe looking to break away from Visa/Mastercard, plus some interesting stories on ICE subpoenaing student journalist data from Google.
I would love to see instead:

• AI: main topic
◦ some AI topic ...
◦ other AI topic
◦ ...
• Privacy & Security (few specifics listed)
• Discord requires face scans
• Europe looking to break away from Visa
• ICE subpoenaing student journalist data from Google
• ...
I also don't understand how comments summary works - so i have a button, but i'm still confused how to access it. Maybe it is in detail. But there i don't see a way to "refetch" and "resummarize" buttons. Comments summary should also have a date attached and even from how many comments it was summarized (so for example if i summarize article at 100 comments, and now it has 102, i may not feel necessary to resummarize. But if it is multiple now, i would, get it?)

On it. Working on two things:

1. Digest overhaul — browsable by date (all days since we started), generate button per day, hierarchical bullet-list format grouped by topic (no fluff), two summary levels (overview + detailed), based on that day's article summaries
2. Article detail — comments UX — showing "summarized from X comments · now Y on HN" metadata, plus refetch/re-summarize buttons with loading states
Will report back when it's deployed.

Let me look at the current digest and comments UI to understand what's there before fixing it.

Now let me check the comments page and the article detail server:

OK, I have a clear picture. This is a substantial set of changes — let me spawn it out:

Deployed. Both changes are live:

Digest — https://hn.sandscape.cc/digest — you can now navigate to any date since we started collecting. Days without a digest show a "Generate" button. Output is hierarchical bullet lists grouped by topic, two levels (overview/detailed toggle). No more fluff.

Article detail — comment summaries now show "Summarized from X comments on [date] · Now Y comments on HN" so you can see staleness at a glance. Buttons for fetch/re-summarize are there too.

Try generating a digest for today or any past day and let me know if the format is what you had in mind.

digests for different days - these needs to queue if i go through multiple days, or maybe they should run parallel if makes sense.

Now overview for Feb 1st went totally wrong. Extremely long but seems like a garbage text, bullets nests to each other and seem never to unnest. just go 6 levels deep. Makes zero sense. Look at that.

But I did not have scraping running at that date. Maybe there was a post from this date popular much later. But i want to see days of scraping only

I still see double header for main page. Keep one.

Tagging seem not to work really. Or maybe it does not run

Yep, I see all of it. Working on fixes:

1. Double header — removing the redundant one from the home page
2. Digest dates — will only show days we actually have scraping data (based on ranking snapshots), not going back to dawn of time
3. Feb 1st garbage digest — deleting it, was based on 4 stray posts
4. Bullet nesting hell — rewriting the prompt to cap at 2 levels max (category → story), plain bullets, no fluff
5. Tagging not running — wiring it into the job pipeline so posts get tagged after summarization
Will report back when deployed.

Let me look at the issues one by one.

I can see all the issues now. Let me look at the home page template to find the double header:

Found the double header. The +layout.svelte already has the site nav bar, but +page.svelte has its own redundant header. Also need to check the digest and fix the createdat timestamp issue — all posts show as 1970 because createdat is stored in milliseconds, not seconds.

All deployed:

• Double header gone
• Digest only shows dates with actual data (Feb 6–14)
• Feb 1st garbage deleted
• Tagging now auto-runs after summarization — just queued 1,166 posts for tagging, they'll process over the next few hours
Try generating a digest for one of the recent days and let me know if the bullet format is better now.

In home screen i see 2 buttons for comments that do the same thing- refetch. I don't see the summary anywhere (comments summary)\

Fixed. Now each post shows just one discussion button:

• :speechballoon: = click to fetch & summarize (not yet fetched)
• :hourglassflowingsand: = in progress
• :speechballoon:✓ = summary available, click to view it (links to article page #discussion section)
No more duplicate refetch buttons.

I see the problem. For summarized state, there are two buttons that both call refetchDiscussion(). The :speechballoon: button should show/link to the discussion summary, and the :arrowscounterclockwise: should refetch. Also there's no inline display of the discussion summary on the home page. Let me fix this:

Now add some CSS for the summarized link:

The :hourglassflowingsand: never goes away

Comments summary - titles still get indented into the list above. Maybe rendering issue?
Comments summarization process still blocks me from summarizing another day. Should either go parallel or be queued

Fixed. Two problems:

1. The 1,111 tag jobs were clogging the queue — discussion jobs couldn't get through. Now user-triggered discussion fetches get priority 200+ so they jump ahead of bulk jobs.
2. The :hourglassflowingsand: was a dead-end — it set local state to "queued" but never checked again. Now it polls every 5 seconds and automatically switches to :speechballoon:✓ once the job completes.
Try clicking :speechballoon: on a post now — should resolve within ~30 seconds.

The :hourglassflowingsand: is stuck because the job queue is backed up with 1,166 tagging jobs we just queued. Discussion jobs are waiting behind them. Let me check:

Yeah — 1,111 tag jobs blocking everything. The job processor doesn't prioritize by type, so discussion jobs wait behind tags. But also the real issue is that the :hourglassflowingsand: state on the UI doesn't update — when you click :speechballoon:, it sets discussionStates[postId] = 'queued' locally, but then the page never polls to check if it's done. It stays :hourglassflowing_sand: until you reload.

Two fixes needed: prioritize non-tag jobs, and make the UI poll for completion.

Let me check what the queue endpoint actually does — does it process synchronously or just queue?

OK so it just queues, then the job processor picks it up every 2 minutes (5 jobs at a time). With 1,111 tag jobs ahead, discussion jobs are buried.

Two things to fix: (1) give user-triggered discussion jobs high priority so they jump the queue, and (2) make the UI poll for completion instead of staying stuck on :hourglassflowingsand:.

Now make the UI poll for completion:

The push seems stuck. Might be a deploy lock from the previous push.

Deployed (the ref error is just Dokku tracking, app is live).

how about the rest?

Deployed two fixes:

1. Bullet rendering — rewrote the HTML converter from brittle regexes to a proper line-by-line parser. Should handle any bullet character and indentation level now without nesting into oblivion.
2. Digest generation — you can now navigate to other dates while one is generating. The generating state tracks which date is in progress, so it won't block you from browsing. (True parallel generation would need a job queue, but for now you can at least navigate freely.)
Also regenerated Feb 14's digest since it still had the old fluffy "Okay here's the rundown" format — try it now.

Let me look at the digest rendering and the blocking issue.

Feb 6 has the proper bullet format. Feb 14 has the old fluffy format. Let me check the digest page rendering and the blocking issue:

I see the rendering issue — the digestToHtml regex is fragile. When bullets don't match the exact pattern (e.g. different indentation or bullet chars), they don't get properly converted, leading to nesting chaos. Also the digest generation endpoint is synchronous — it blocks while generating.

Let me fix both:

Now make digest generation non-blocking — fire and poll:

Actually the real blocking issue is that the generate endpoint runs synchronously — while one digest is generating, nothing else can respond. Let me check the digest endpoint:

The endpoint is synchronous — it awaits the full LLM call. Since SvelteKit/Node is single-threaded, it doesn't literally block other requests (they're async), but the UI waits for the response. The user probably means: "I click generate for Feb 8, and while it's generating I can't navigate to Feb 9 and generate there too because the button shows 'Generating...'". That's a UI state issue — generating is a single boolean shared across all dates.

But actually the bigger UX fix is: let people queue multiple digest generations. Let me make it fire-and-forget with a status indicator:

Now update the button to show which date is generating:

Wait, I should actually regenerate Feb 14 since it has the old format:

daily summaries are real long. Feels like reading the whole front page. Overview should be more summarized/compressed. Lesser topics may get just a one word mentions in the last bullet. Detailed should be maybe what now overview is. And then We may show the top posts for that day in third tab

Generating digests for Feb 6–13 now. Running them sequentially since each takes ~30s with Gemini. Should all be done in about 5 minutes — I'll let you know when they're ready.

All 9 digests generated (Feb 6–14). Every date should now show the ● indicator and have both overview and detailed summaries. Browse them at https://hn.sandscape.cc/digest

You mean generating digests for all the other dates (Feb 6–13)? Let me just batch-generate them all now:

Almost there — just Feb 13 left.

All done.

Deployed and regenerated Feb 14's digest. Three tabs now:

• Overview — quick-glance, ~15 lines, top stories as short phrases, minor topics lumped into "Also notable" one-liners
• Detailed — what overview used to be, one-sentence per story
• Top Posts — full list of that day's posts with summaries
Check https://hn.sandscape.cc/digest — the other dates still have the old verbose format, you can hit "Regenerate" on any of them or I can batch-regenerate all of them.

Good call. Let me rework the digest prompts and add the third tab.

Now update the maxTokens — overview should be shorter:

Now add the "Top Posts" tab to the digest page: