The GPT2 version of the AI Dungeon [1] could keep track of context for maybe couple of lines at a time. I've heard the GPT3 version is substantially better.
The problem is, of course, that these "AI" chatbots on websites, marketing buzzwords notwithstanding, have very little to do with state-of-the-art machine learning, and are indeed unlikely to be any more sophisticated than the original ELIZA for the most part.
AI Dungeon (and its successors like novelai [1] and holoAI [2] following its infamous censorship purges and mutinies) is basically designed entirely around that problem and has a lot of clever solutions that let writers keep relevant context in memory in a story that is longer than the model's maximum input length.
* "Memory", which is a customizable block of text that is repeated at the top of each API input. If your story revolves around your character being a vampire who melts in the sunlight, you can include something like "you are a vampire who will die if you go out in the sunlight" in the memory, and even 1000 characters back a paragraph of context can prime the AI accordingly.
* "Author's Note", which is a customizable short block of text that is inserted invisibly a few lines before your current place in the story when it's sent to the API. A note such as "[A/N: The following section includes depictions of graphic violence]" or "The following section is written in Shakespearean old English", as obvious and blunt as it might seem, actually works surprisingly well for nudging the AI towards a certain style or content.
* "World Info", which is a customizable dictionary of short text blocks that are conditionally added to the top of the API input like memory when a certain key appears in the current context. Imagine you have a story with 10 important characters who cycle in and out of the story. If you create an entry in the world info about Bob, then when you write that "Bob appears from behind the shrub", the blurb about Bob is automatically tacked on to the context so long as Bob is mentioned by name in the last few dozen inputs.
In general, both GPT-3 and the open source alternatives by EleutherAI such as GPT-J-6B are able to use a context primer from 1000 tokens prior to affect the current tail of a story. It's actually kind of uncanny how good they are at it -- you can have a story that in the memory at the top says "goblins always have purple skin" and notice that the AI will mention it as an offhand detail much farther down in the context.
Thanks for the link to that! I tried it out just now and I have to say that I am not impressed.
What will you do?
You get longsword from the ground.
You get your longsword from the ground. You hold it in your hands and examine it for a moment. It's a very sturdy weapon.
The knight then grabs his longsword and bows.
Huh? Why did it grab the bow as well?
You inventory.You have a crossbow.
Huh? I thought I just grabbed my longsword and bows.
What will you do?You don't have enough experience to know what a good shot is.You get longsword and bow.You grab the bow and the longsword. You examine the crossbow. It's a bolt-action weapon.
Huh? I said to get the longsword and bow since it said it wasn't in my inventory and it added all this extra stuff.
You inventory.You have a crossbow, a short sword, a longsword, a lantern, a flint and steel, and a quiver of bolts.What will you do?
You load the crossbow.
Huh? I just said Inventory and now I have all this extra stuff it never mentioned before and then unsolicitedly told me I loaded the crossbow.
I have to be doing something wrong. Zork from 1980 was better than this.
"bow" here probably refers to "a bending of the head or body in respect, submission, assent, or salutation", rather than to the weapon. The knight grabbed his longsword and bowed to you. Your confusion then confused the game.
This does expose an interesting common "failure mode" for these types of bots: they assume that all conversations are rational, and are conducted in good faith (as in, no one misunderstands or is actively trying to confuse them). Janelle Shane likes to test AIs that attempt to describe the content of images by asking "how many giraffes are in the picture?", and since most of these have been trained to answer what is in a picture rather than what's not in a picture, they tend to assume that there must be some number of giraffes, and provide answers like "there are 1.5 giraffes".
I've been using AIDungeon (and its successors like NovelAI which are more for advanced users) for years now and have been involved in a lot of the community around it. You are definitely using it wrong, but you are forgiven because AI Dungeon is terribly falsely advertised.
Your main mistake is not writing proper sentences. AI Dungeon is not a Zork simulation where you type truncated commands like "get torch" and receive a response from the computer. It is more like a Choose Your Own Adventure novel generator.
You need to write proper sentences, so instead of "You inventory", try "You double check what items you have in your rucksack." Instead of "You get longsword and bow", you should be typing "You pick up the longsword and the bow." Instead of "You health", you...just don't do that. Maybe try something like "You look down at your body and survey your wounds." If you're really feeling confident, try switching from "Do" mode to "Story" mode, and just start writing a fantasy novel and see what it generates with its next output.
For reference, this is a snippet of an old story of mine I saved:
----
As I approached the kobold stealthily, it looked up with its beady, reptilian eyes. It could not see me yet, but I knew it was only a matter of time before it sensed my presence.
I decided to act first. "Let's go!" I shouted to my companions, before rushing towards the small monster with my dagger drawn.
The kobold yelped in surprise before pulling back its bow and firing an arrow at me. I moved to dodge it and barely managed to avoid it. Now in melee range, I raised my dagger into the air before plunging it towards the kobold's torso.
It struck the monster's heart, and blood spurt out of the wound as it struggled, before going still.
---
Nearly every sentence there was partially AI-generated in AI Dungeon, retried a few times, edited, and truncated to fit the story. This is how most people in the community actually use this sort of software. It's more like having a buddy suggesting the next sentence in your personal story than a virtual dungeon master. Those who expect it to be the latter generally don't understand it, while those who see it as the former immediately understand what the hype is about and get really passionate about it.
AI dungeon might advertise itself as a game, but in practice it's used as a more general storytelling aide. And it should be noted that it has been rendered obsolete by competitors made by disgruntled fans like NovelAI for a while now (which is a whole other story).
...so you're saying NovelAI is better for story generation? Hot damn. I was having a good time with AI Dungeon (to the point where I was considering writing short stories using it as a guide), I might have to give that a try too.
Here's a snippet from my AI dungeon story recently. Mine never seems to get into actual dungeon crawling. The only parts that aren't A.I. generated in this section are the lines that have --me-- before them:
Veronica sits down on a stone bench. She says "Well, I guess we should talk."
Jesus sits down next to her and asks her, "What is the worst thing you've ever done in your life?"
Veronica says "I killed my first husband."
Jesus asks "Why did you do that?"
Veronica says "He was going to take everything from me."
Jesus says "How so?"
Veronica says "He knew where I hid all of my money, and he was going to take it. I would have been broke forever."
Jesus says "That's sad."
--me--: You say "Wait, you killed your first husband? I'm your new husband! Are you going to kill me too?"
Veronica looks away from Jesus. "No. I didn't mean to kill him. He was just... so mean to me. I just wanted to stop him. I didn't..." Veronica finishes by saying "I didn't mean to kill him."
Jesus says "Alright, I forgive you."
--me--: Veronica says "Does that mean I get to go to heaven now?"
Jesus says "I'll let you make that decision as soon as we finish our meal. It will have both heaven and hell as options, and that's all you'll ever get out of me."
NovelAI is literally built by disgruntled AI Dungeon fans after the censorship scandal who wanted a better, more privacy-focused AI Dungeon for story generation. If you think AI dungeon has some cool features, you ain't seen nothing yet. There's a prompt tuning system called "modules" which basically primes the AI to write in a certain style, like a certain author or genre, etc (there are dozens of default modules, plus an included API to train your own, and you can share them and import other people's modules freely. Want a module trained on all of Harry Potter, or trained specifically on a particular genre of erotica? Someone's probably already made one). That's only one of the cool features it has, there are plenty others.
General upsides: There's a billion different settings and dials and output customization. The context limit is 2048 tokens is almost 3 times larger than AI Dungeon's. Everything is extremely transparent as far as inputs and outputs go, and the UI is really clean and functional, it's more like a word processor than anything. There's no crappy energy system like AI Dungeon either. Everything is run on private servers and encrypted so the devs couldn't snoop on your stories even if they wanted to (which was the main reason it was created, after it came out the AI Dungeon devs were reading and censoring private stories) and there's no risk of Google or Microsoft butting in and forcing them to change their business model like AI Dungeon. It's even got a legacy DO/SAY/STORY choose-your-own-adventure mode/module for people nostalgic about the AI Dungeon aesthetic. The base AI is finetuned on a large, curated collection of novels, rather than on a bunch of garbage CYOA stories written by 10-year-olds like AI Dungeon.
General downsides: It's paid subscription only, though I think there's actually a free trial nowadays. It uses eleutherAI's GPT-J-6B, which is about 1/40th the size of GPT-3's largest model which is 275B. It definitely punches above its weight class though and subjectively has comparable or higher output quality to AI Dungeon's "Griffin" which used a smaller GPT-3. And they're coming out with a 20B model too in like...a week or two.
Do you know how much you can paste from an existing story in NovelAI (like number of words) to continue off of? Like I've got some 2500 word stories I never got around to writing an ending (I struggle with endings in general), that maybe I could paste into NovelAI and see where it goes from there. Might help me finally finish the damn things, they've been sitting in a folder for a couple of years now.
I don't think there's any practical limit (I could probably paste in the entire text of the Bible and see what it generates next), but for the most part you want to use it as a writing companion, not an auto-writer. The general strategy is to have it generate some text (a few sentences), and then generally edit or retry it. The fun part is seeing the novel (ha) ideas that the AI comes up with and then working from there. There are some people who actually do try to use the AI to write the whole thing with not editing, with limited success (usually it involves a very meticulous and nebulous combination of tricks and techniques, and a lot of retrying).
As far as the context limit (how many lines back it can actually use), I believe it's 2048 tokens for novelAI, which is about 3 times higher than AI Dungeon uses, but likely not enough for the AI to have all of the story in memory. For important details or characters that aren't repeated except at the start, you may need to do some priming techniques with memory, author's note, or world info.
Though with 2500 words one advantage is that you have a very large amount of text in your writing style for the AI to try to mimic. One disadvantage of NovelAI over AI Dungeon is that AI dungeon seems a lot better at working off a small, limited prompt with very little context, while NovelAI prefer much longer prompts with plenty of descriptive detail. Having a large prompt that is entirely your own words definitely helps.
Okay cool. Yeah I didn't mean generate it all for me, just come up with some responses and maybe a few ideas of where it could go. I did end up experimenting with it last night briefly and it did help me write a few more paragraphs to one of my stories.
One thing I did notice is it doesn't seem to know when to move on to something else, and will just keep trying to do what's been working, so I need to be able to tell it "Stop that conversation, let's talk about this now", although admittedly it's from very limited interaction with it. Still pretty neat.
Actually that story was really funny, the AI constantly kept having my character have this irrational hatred of kobolds, like "they're pests to be exterminated" even though I included descriptions of their vibrant art and tapestries and language and culture. Which was funny because it was a story about traveling with an intelligent slime buddy who all the humans thought was a "monster to be exterminated". I was ultimately planning on having us escape to the monster kingdom where we learned that not all monsters are evil, and find a bunch of friendly kobolds, but I never got that far before the site went to shit.
Yes -- "the knight then grabS his longsword and bowS.". That 's' at the end there indicates present tense of the verb "to grab" and "to bow", just like in "roars and swipes". (I guess for knights it's just good manners to thank by bowing when they're gifted a sword.) It's not a plural marker of a noun, because there was only one bow on the ground.
Sounds like you're trying to play it like Zork too.
It's better to treat it more like collaborative story telling. It will let you "do" whatever you want. You don't have to 'check inventory'. If you want to shoot someone with a bow, you say "I shoot the knight with the bow". And you've done it.
It never contradicts you and says "You can't do that, you don't have a bow." It just rolls with whatever you've just said.
And it gets things wrong, sure. It sometimes has characters say things that I'm like "No wait, I was the one that said that, why are you acting like you said it?" Or contradicts what just happened. Or has you move into places that don't make logical sense (like exiting a detached garage at your house to enter a small room, etc), but correct it and go on.
But one thing it lets you do is either have it retry, or you can just click and edit their response to something you would rather it be.
"But why don't I just write a story on my own if I have to edit it to be what I want?"
Because it comes up with shit that I would never think of on my own. I fought an evil Super-Man in the bottom of the sea, and he chased me into a classroom, and saw a dog behind the desk, and got distracted from attacking me and wanted to pet the dog.
Even better, later we won a trip to the Louvre. The Louvre welcomed us as if it were a towering person, put on a fancy dress, and when we entered, Mona Lisa came back from the dead to give autographs next to its painting, the third floor had Poussin paintings running around as if in a horse race, and another floor let you walk into a painting, where we entered one with Jesus serving a man and a woman dinner, and Jesus offered us each two plates, one with a pea and one with a feast, and the signs heaven and hell under them, and whatever one we ate was the one we went to.
Some of that was me guiding the story a little, but I wouldn't have thought to direct the story that way if not for some of the unexpected things generated by the A.I.
I've played Zork. Zork has puzzles to solve, something AI Dungeon can't handle, but AI Dungeon has a lot more freedom.
It requires you to be your own DM, it never says anything you do is illegal you can do whatever you want like spawn a deer to pet etc.
I agree that it isn't really a game, in a game your actions has consequences, they don't here as nothing you do limits what you can do in the future. It would need to create a world with rules and limitations where what you can do now depends on what state the world is in.
There is a fundamental difference between AI Dungeon-type chatbots and chatbots you typically encounter on websites e.g. for customer support.
The former does not really have a goal and is unconcerned about responding with factual information as long as the conversation is coherent. It makes sense to use large language models that are quite good at modeling next word probabilities based on context.
The latter however is goal-oriented (help the customer) and constrained by its known actions and embedded knowledge. This often forces the conversational flows (or at least fragments) to be hard-coded and machine-learning is used as a compass to determine which flow to trigger next.
For now controlling GPT-like language models remains an extremely tricks exercise but if some day we can constrain language models to only output desirable and factual information with a low cost in maintaining and updating its embedded knowledge, we should see a significant bump in "intelligence" of the typical website chatbot.
The problem is, of course, that these "AI" chatbots on websites, marketing buzzwords notwithstanding, have very little to do with state-of-the-art machine learning, and are indeed unlikely to be any more sophisticated than the original ELIZA for the most part.
[1] https://play.aidungeon.io/