AFAICT, you retain the copyrights to your comments, but YC has a license to essentially do whatever they want with them.

So, you could additionally give a license to the world to use your posted comments freely. That doesn't mean HN can't add terms to say clients can't copy the site as a condition for use.

HN/YC cares more about community aesthetics than your right to be forgotten.

Try to have your account and its contents deleted. The best I was offered for my 2011-vintage account was to randomize the username, and the reason I was given was that browsing and old thread with a bunch of deleted comments "looks bad".

@keepamovin thanks, your project was a big inspiration for this.

I built my own pipeline with a slightly different setup. I use Go to download and process the data, and update it every 5 minutes using the HN API, trying to stay within fair use. It is also easy to tweak if someone wants faster or slower updates.

One part I really like is the "dynamic" README on Hugging Face. It is generated automatically by the code and keeps updating as new commits come in, so you can just open it and quickly see the current state.

The code is still a bit messy right now (I open sourced it together with around 3.6M lines across 100+ other tools, hidden in a corner of GitHub, anyone interested can play Sherlock Holmes and find it :) ), but I will clean it up, and open source as clearer new repository and write a proper blog post explaining how it works.

https://contabo.com/en/vps/cloud-vps-20/ - $8/month, 6 vCPU, 12 GB RAM, 200 GB SSD (or Hetzner servers, which offer good hourly pricing).

In my ongoing project, with 10 servers like this, I could index the large part internet (about 10 billion pages) using vector and full-text search.

> It's because the value does not represent a binary true or false but rather a means by which the item is deleted or dead.

"Deleted" and "dead" are separate columns.

> So not only would it not make sense semantically, it would break if a third means were introduced.

If that was the intention, it would seem like a bad design decision to me. And actually what you assume to be the reasoning, is exactly what should be avoided. Which makes it a bad thing.

This is a limitation not because of having the bool value be represented by an int (or rather "be presented as"), but because of the t y p e , being an integer.

Funny, because the HackerNews API [0] does return booleans for those fields. That is, a state, not a type of deletion or death.

[0] https://github.com/HackerNews/API

Hello to myself for prosperity

Can't someone create an automatic script which can just copy the files say 5 minutes before midnight UTC?

Plus isn't the least wasteful format, native duckdb for instance compacts better. That's not just down to the compression algorithm, which as you say got three main options for parquet.

You could download the data and run that analysis yourself. I’d be interested to see it, especially your method of identifying “political shit-slop” and “AI” and the relationship to COVID. Sounds like an interesting project.

That's a possible hypothesis, but there was also a rising trend prior, it wasn't stable.

Don't all comments start out with one vote?

I recall that this became a big problem for the Homebrew project in terms of load on the repo, to the extent that Github asked them not to recommend/default-enable shallow clones for their users: https://github.com/Homebrew/brew/issues/15497#issuecomment-1...

This is likely to be lower traffic, and the history should (?) scale only linearly with new data, so likely not the worst thing. But it's something to be cognizant of when using SCM software in unexpected ways!

So they are sharding by time/day?

I have a similar project right now where I am scraping a dataset that is only ever offering the current state. I am trying to preserve the history of this dataset and was thinking of using the same strategy. If anyone has experience or pointers in how to best add time as a dimension to an existing generic dataset, I'd love to read about it.

This makes more sense. I still wonder if the author isn't just effectively recreating Apache Iceberg manually here.

Parquet is a very efficient storage approach. Data interfaces tend to treat paths as partitions, if logical.

5 minutes is the sweet spot for many (enterprise) data pipelines too, in my experience.

That paragraph doesn’t make it clear (to me) that it’s a snapshot with incremental updates. If that’s what it is. Sorry if my obtuse read offended. I just figured it was edge cached HTML, and less likely it was actually broken.

>if you just would finish reading the paragraph

probably uncalled for

the data updates every 5 minutes, but the description on huggingface says the last update was 2 days ago.

they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.

Unless I'm missing something, this uses a simple synchronous for loop:

    for text in texts:
        key = (text, model)
        if key not in pickle_cache:
            pickle_cache[key] = openai_client.create_embedding(text, model=model)
        embeddings.append(pickle_cache[key])
    operations.save_pickle_cache(pickle_cache, pickle_path)
    return embeddings

I had to call the Gemini model with ten comments at a time from eight threads to reach even the paltry 3K rpm rate limit they offer to "Tier 1" customers.

Based on this experience, for real "enterprise" customers I might implement a generic wrapper for Google's Batch API that could handle continuous streaming from a database, chunking it, uploading, and then in parallel checking the status of the pending jobs and streaming the results back into a database.

Time to sue them to oblivion :D.

Max Schrems would like a word

There is no such thing under https://news.ycombinator.com/ when you create your user.

Is this legal advice?

And do what I do - paste everything into ChatGPT and have it rephrase it. Not because I need help writing, but because I’d rather not have my writing style used against me.

funnily enough, if everyone did this (at least make a new account often), it would prove more destructive to what HN (purposefully) wants to do than deleting the occasional account data

I think we'll be able to quantify sentiment from the data, and I look forward to doing so. There's a few other datasets that I want to look at such as whether there is evidence of participation suppression via rate limiting on a per-profile basis.

the implication was that training a model doesn't seem to abide by the TOS

Laughs nervously in jurisdiction without fair use doctrine

> By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies

The user content is supposed to be licensed only Y Combinator and (bleah) its affiliated companies (which are many, all the startups they fund, for example).

None that I could see:

Your submissions to, and comments you make on, the Hacker News site are not Personal Information and are not "HN Information" as defined in this Privacy Policy.

Other Users: certain actions you take may be visible to other users of the Services.

Legal theory about public data is fun right up until someone with money decides their ToS mean something and files suit, because courts are usually a lot less impressed by "I could access it in my browser" once you pulled millions of records with a scraper. Scrape if you want, just assume you're buying legal risk.

> I believe if you put it on the internet without an auth-wall, people should be allowed to do whatever they want with it.

I agree. It's the owners of the sites that have to follow rules, not us.

"I'm kind of old school in that I believe if you put grass on the ground without a fence, people should be allowed to do whatever they want with it. The noblemen with a thousand cows seem to agree."

And that, my friends, is how you kill the commons - by ignoring the social context surrounding its maintenance and insisting upon the most punitive ways of avoiding abuse.

To be incredibly pedantic to the point of being irrelevant: technically the sign up page 1) doesn't have a clickwrap "I agree" checkbox, and 2) there's no link to the TOS on the sign up page.

That makes the implicit TOS agreement legally confusing depending on jurisdiction.

(Not that it really matters, but I find these technicalities amusing)

I’m reading that paragraph now and fail to see anything about a relationship with huggingface or the user responsible for copying the data.

Only Y Combinator and its affiliated companies have license, me thinks.

That's a good point, and I think this will be my last post on this site. I never added much value anyway.

@dang What’s Hacker News’ official stance on this?

This isn't presented anywhere on signup.