yes, although there is a paid tier they try to upgrade you to. worth noting that there are some limits on how much data you can export in one go - I found the easiest way was to dump out pieces you need to google drive.
right now the biggest cost is llm api calls (currently claude haiku) for organising content and making it easily searchable - this is relatively cheap (max 50$ per month). I use pinecone for search which so far has been very inexpensive, though admittedly traffic has been low, so I'll see how that goes.
on feeds - I've built a list over a number of years, a mix of tracking blogs on twitter, hacker news and a few other places. usually the feed url can be lifted quite easily from a blog's header.
Ugh, let me rephrase the question then, is this 34,000 blogs as in websites, or blogs as in posts? I assumed the former, but sounds like I might have been wrong since building a list of 34k niche websites would be next to impossible.
I am guessing you have added everything from here:
Yes, it's 34k websites, but to be transparent, they probably don't all strictly qualify as niche, for example, it includes some smaller commercial sites (usually if they have tech blogs). I've found the hardest part of this isn't finding the feeds so much as filtering out the usual suspects (SEO etc) that masquerade as technical content.
As well as HN, I found a google search along the lines of "site:github.com opml rss feed [topic]" was quite effective too.
my 2p worth - my work involves a lot of summarisation, recommendation from a user preference statement. I've been able to do this with 4o / opus, but the consistency wasn't there, which required complex prompting chains to stabilise.
What I'm seeing with Sonnet 3.5 is a night-and-day step up in consistency. The responses don't seem to be that different in capability of opus / 4o when they respond well, it just does it with rock-solid consistency. That sounds a bit dull, but it's a huge step forward for me and I suspect for others.
Hey HN! I've been developing blaze.email for a while, and would love any feedback. blaze is a search engine over content gathered from a long list of blog / news / arxiv RSS feeds. it uses embedding similarity for ranking, so it does have some rough edges(!)
I know there're quite a few sites out there like this, but I haven't found many that scratched my itch for finding interesting tech articles. noteable shoutout to kagi.com which I discovered recent which does a great job, and more besides.
The original idea with blaze was to provide a newsletter service that scours the last week's blogs for relevent niche content for different types of tech users. that's still the plan, and you can sign-up for newsletters on GenAI, machine learning & tech startups. There's more to come on that side of things, but I'm increasingly thinking that the search engine angle is at least as useful.
I'd love to hear any comments / criticisms, or general views on what (if anything) seems useful. Thanks :) Alastair
it’d be interesting to know the ratio of consumer desktops / laptops to linux servers required to support the consumer internet. I know neither of those things are particularly well defined but it adds another perspective on this which’s be interesting.
does anyone have recommendations for addins to integrate these 'smaller' llms into an IDE like VSCode? I'm pretty embedded with GH copilot, but curious to explore other options.
