What is GPT doing…

By Stephen Wolfram

This is a short book which combines an description of how GPT, the revolutionary new AI technology shaking the world, works with an advertisement for some Wolfram products designed to augment GPT.  His description is pretty good but I couldn’t really follow his explanation of the crucial “transformer” component, the attention blocks.  ChatGPT was able to give me a better account.

Wolfram pointed out some key failings of GPT.  It can’t really do deductive chains of any substance.  For example, ask it to compute 3^73 and it comes up with an absurdly wrong answer, off by a factor of roughly 10^34.  I tried giving it some hints, including using logarithms.  Again, it confected a plausible looking deductive chain and an answer only off by a factor of ten or so.

Wolfram then argues that supplementing GPT with some deductive system like Wolfram language would have more capability.  Of course lots of people are working on such things.

Comments

Post a Comment

Popular posts from this blog

Anti-Libertarian: re-post

The most pointed critique of social security and liberalism in general is the libertarian critique. The problem with libertarians, for a liberal, is that we have too much in common. We both believe in individual rights, tolerance of individual differences, and dislike government prescription of religion. The basic difference, it seems to me, is the different answers we give to Cain's (with thanks to the Captain - my original version had the corpse asking the question!) famous question: "Am I my brother's keeper?" That's not really my basic bitch against the libertarians though. My real complaint is the same as my complaint against most religion - its premise is a fraud. For those who can't stand to wait for the punchline, I believe that trying to implement libertarian principles leads to tyranny or social disintegration. Demonstrating that takes some historical (and pre-historical) context. For all but the last 15,000 or so of the 100,000 years the human spec...
Read more

Uneasy Lies The Head

Uneasy lies the head that wears a crown ........... Shakespeare Henry IV, Part II. So it is with us and our slightly distant primate cousins , so it seems. A 9-year study of wild baboons in Kenya by Princeton University and the Institute of Primate Research may turn conventional assumptions of alphas males upside down. The study found that alpha males have high testosterone levels, which allows them to dominate other baboons, get access to more food, and attract better mates. This is expected. Surprisingly, though, the alpha males also have high levels of stress hormone glucocorticoid, which scientists measured from their feces samples. In fact, the stress of alpha baboons was on par with low-ranking baboons. The convention wisdom is that access to food and females provides life security and thus less stress for dominant males. Conversely, weak males who are constantly threatened by starvation and physical violence from stronger males are thought to be more stressed. The st...
Read more

We Call it Soccer

The US played its best, and maybe only good game of World Cup, but lost thanks to brilliant goalkeeping by the Swedish keeper and the bizarre method soccer uses to settle ties when three US stars could not put the ball on frame from the 12 yard spot.  At the end it came down to a blocked shot that spun in by maybe a couple of centimeters. Well, that's football - too much like life.   Which brings me to a modest proposal.  Replace the thirty minute overtimes with as many 7-player 15 minute periods as needed to settle it.  Scoring should be much easier with fewer players.
Read more