What if we extended this idea beyond one dataset to all discrete news events and entities: people, organizations, places.
Just like here you could get a timeline of key events, a graph of connected entities, links to original documents.
Newsrooms might already do this internally idk.
This code might work as a foundation. I love that it's RDF.
This has been attempted many times. They all fail the same way.
These general data models start to become useful and interesting at around a trillion edges, give or take an order of magnitude. A mature graph model would be at least a few orders of magnitude larger, even if you aggressively curated what went into it. This is a simple consequence of the cardinality of the different kinds of entities that are included in most useful models.
No system described in open source can get anywhere close to even the base case of a trillion edges. They will suffer serious scaling and performance issues long before they get to that point. It is a famously non-trivial computer science problem and much of the serious R&D was not done in public historically.
This is why you only see toy or narrowly focused graph data models instead of a giant graph of All The Things. It would be cool to have something like this but that entails some hardcore deep tech R&D.
I don't have any experience on graph modeling, but it seems like Neo4j should be able to support 1 trillion edges, based on this (admittedly marketing) post of theirs? https://neo4j.com/press-releases/neo4j-scales-trillion-plus-...
Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale
Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
One co trying: https://www.system.com
One wonders what the US government agencies use.
Isn’t that what Palantir’s product is?
They probably use Excel, maybe Microsoft Access.
Microsoft Access form that connects via IIS to an Excel spreadsheet acting as a database. Also the server it's running on is sitting on a wooden table.
I think you meant one shudders. And yeah, Snowden made it clear there's orders of magnitude more data than this graph explorer for them to sift through.
Software like i2 Analyst's Notebook.
Internet search engines have their origins in government projects fwiw. They had search engines before Alta Vista, used for searching data sets that pre-date the internet, and some of the people involved in those went to work on the original commercial search engines.
If it's RDF it won't work as the foundation.
Oof, browser freeze, computer slowdown, then sudden crash reboot of my ubuntu dev machine, upon loading the graph in firefox.
Oh Cthulhu, this is like a periscope into a septic tank...
Yes almost no one has been held accountable for any of it, "weird"?
After seen this I interested in a map of each person to assist with knowing who they are, who they worked for during the email date, and who they currently work for.
"Brad Edwards" and "Bradley Edwards" might be the same individual.
I’m sure some developer/archivist is working on a name authority as we speak.
Yes, the dataset also has three entries for Virginia Giuffre, "Virginia L. Giuffre", "Virginia Roberts Giuffre", and "Jane Doe Number 3 (Virginia Roberts)"
great use case for using AI to suggest mergers and clean up.
LLMs are awful for this. I've got a project that's doing structured extraction and half the work is deduplication.
I didn't go down the route of LLMs for the clean up, as you're getting into scale and context issues with larger datasets.
I got into semantic similarity networks for this use case. You can do efficient pairwise matching with Annoy, set a cutoff threshold, and your isolated subgraphs are merger candidates.
I wrapped up my code in a little library if you're into this sort of thing.
github.com/specialprocedures/semnet
Likewise for instances of "Larry" and "Lawrence" Summers... probably a lot of those.
Why are they all moving, what does the time axis represent?
Its because the layout system has also a physics system.
>A force-directed graph is a technique for visualizing networks where nodes are treated like physical objects with forces acting between them to create a stable arrangement. Attractive forces (like springs) pull connected nodes together, while repulsive forces (like electric charges) push all nodes apart, resulting in a layout where connected nodes are closer and unconnected nodes are more separated
I’m curious which LLM tools actually handled all 23k emails well.
it's done one by one in Claude.
https://github.com/maxandrews/Epstein-doc-explorer/blob/83ee...
This is the best rendition I've seen so far.
The Bill Clinton entity is interesting.
> 2009: Bill Clinton discontinued association with Jeffrey Epstein
> 2010: Jeffrey Epstein provided flights on jets to Bill Clinton
> 2010-2011: Jeffrey Epstein traveled via private aircraft with Bill Clinton
> 2011: Ghislaine Maxwell piloted helicopter for Bill Clinton
> 2014: Bill Clinton alleged presence at sex parties
> 2015: Bill Clinton distanced relationship from Jeffrey Epstein
Wasn't very good at discontinuing the relationship it seems.
Guess there is precedent for him lying about sexual activities though.
I think a sentiment analysis between the friendliness and social meetups between Epstein and other individuals would be useful.
Who were his friends after 2008 when he was first convicted?
Those who were still friends with him after 2008 were in on it or guilty by association, if not legally, socially.
Friends like Reid Hoffman and Larry Summers...
> From: Reid Hoffman
> Sent: 7/6/2015 5:04:31 PM
> To: jeffrey E. [jeeyacation@gmail.com]
> Subject: RE: ICYMI
> slow progress.
> planning to see you in August.
> Hope you're well.
Larry Summers has too many to list. Doesn't look good though digging through them.
I'd take a look at Trump. He's on a whole different level. Lots of rape and sexual abuse of minors... wow.
Seems to get away with it all, meanwhile, we all pay our taxes, don't break any laws and just be "good people".
Of course, deflect discussion to Trump. Does that make any of those other people look better to you?
Trump gave information against Epstein in 2009 and unlike Bill and others did cut ties after learning he was poaching girls from Mar-a-Lago.
I specifically made the point to look into those who were friends with Epstein even after knowing what he was doing.
Nice whataboutism though. Feel free to reference source materials to support your claims.
Btw are you a bot or is that just a canned statement you use?
Trump was with Epstein in 2017, he didn't cut ties at all
That's a lie that has already been proven false since Trump's entire trip was documented.
Love how we have actual evidence against people but discussions always devolve into some conspiracy related to Trump.
----
Based on the available evidence, there is no confirmed meeting between Trump and Epstein in 2017. While both men were in Palm Beach during Thanksgiving week 2017, there is no direct evidence they met.
Here's what we know about their presence in Palm Beach that week:
- Trump was at Mar-a-Lago from November 21-26, 2017
- Epstein owned a mansion in Palm Beach and was known to be in the area
- Epstein mentioned both Trump and himself being "down there" (Palm Beach) in an email exchange on November 23, 2017
While there were claims circulating online that Trump spent Thanksgiving with Epstein in 2017, these claims have been thoroughly investigated and found to be unsubstantiated
Trump's official calendar for that week shows his activities included:
- Thanking military members on a virtual call
- Visiting Coast Guard members at Lake Worth Inlet Station
- Playing golf with Tiger Woods and Dustin Johnson
This obviously the correct lens but note that the 2008 plea deal was so neutered by the time of settlement it made it somewhat easy to stay friends with him.
This is of course ontop of the 2006 Florida prostitution charge though.
Especially when Epstein was paying off journalists at the NYT and intimidating other outlets.
But point being those people that were friends with him had to know. Whether it was socially acceptable by the elite because the public wasn't aware isn't very relevant.
where is bubba?
Retired from public office.
[flagged]
Bubba was allegedly a nickname for clinton.
(Also allegedly the name of a horse Ghislaine Maxwell owned.)
The nickname itself isn't alleged, which particular bubba is tho.
[flagged]
[flagged]
bot