Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
mastodon.me.uk is one of the many independent Mastodon servers you can use to participate in the fediverse.
Open, user-supported, corporation-free social media for the UK.

Administered by:

Server stats:

515
active users

mastodon.me.uk: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.6

Public
zogwarg

AGI Sparklings proponents rejoice! Finding a literal map(*) means LLMs have a world model.

awful.systems/post/330426

awful.systemsAGI Sparklings proponents rejoice! Finding a literal map(*) means LLMs have a world model. - awful.systemsSource: nitter [https://nitter.net/wesg52/status/1709572469978231063], twitter [https://twitter.com/tegmark/status/1709572469978231063] Transcribed: > Max Tegmark (@tegmark): > No, LLM’s aren’t mere stochastic parrots: Llama-2 contains a detailed model of the world, quite literally! We even discover a “longitude neuron” >> Wes Gurnee (@wesg52): >> Do language models have an internal world model? A sense of time? At multiple spatiotemporal scales? >> In a new paper with @tegmark we provide evidence that they do by finding a literal map of the world inside the activations of Llama-2! >> [image with colorful dots on a map] __________________________________________________________________ With this dastardly deliberate simplification of what it means to have a world model, we’ve been struck a mortal blow in our skepticism towards LLMs; we have no choice but to convert surely! (*) Asterisk: Not an actual literal map, what they really mean to say is that they’ve trained “linear probes” (it’s own mini-model) on the activation layers, for a bunch of inputs, and minimizing loss for latitude and longitude (and/or time, blah blah). And yes from the activations you can get a fuzzy distribution of lat,long on a map, and yes they’ve been able to isolated individual “neurons” that seem to correlate in activation with latitude and longitude. (frankly not being able to find one would have been surprising to me, this doesn’t mean LLM’s aren’t just big statistical machines, in this case being trained with data containing literal lat,long tuples for cities in particular) It’s a neat visualization and result but it is sort of comically missing the point __________________________________________________________________ Bonus sneers from @emilymbender: * You know what’s most striking about this graphic? It’s not that mentions of people/cities/etc from different continents cluster together in terms of word co-occurrences. It’s just how sparse the data from the Global South are. – Also, no, that’s not what “world model” means if you’re talking about the relevance of world models to language understanding. (source [https://nitter.net/emilymbender/status/1709624503306014918]) * “We can overlay it on a map” != “world model” (source [https://nitter.net/emilymbender/status/1709670931332010441])
Public
carlitoscohones

My first thought in watching the animation was - AI parrot shoots shotgun at side of barn, draws target around result.

Are the dots in the ocean from Amazon’s underwater warehouse structures?

Public
swlabr

The ocean dots are Lemurian and Atlantean civilisations, situated in the deep. The AI said they’re there, so they must be real! That’s how reality works now!

William Butler Yeets

@swlabr @carlitoscohones just off to raise some VC millions for my AI-Driven Atlantis Recovery startup brb

Oct 06, 2023, 10:47 AM·Public·Tusky
0boosts·1favorite
ExploreLive feeds
About