This was a remake of the Chat GPT Decoder architecture following the video tutorial by Andrej Karpathy. This model recursively takes input and predicts at the character level what is to come next. Unlike ChatGPT, this is only the decoder model. This means that after training, the model babbles on text that sort of sounds like shakespeare but it is not particularly useful information.
In order to get to a point where you have a full grade production product like ChatGPT, you need an extensive alignment phase where you train the model to act like an assistant, rate the model based on it’s responses, then use a supervised learning model to pick the best responses from the original newly trained model.
This project just created interesting text’s like that seen in the text below. You can see where spelling errors occur, but overall, the model is able to predict actual sentences that look like Shakespeare by just predicting one character at a time. Some improvements could be made by adjusting the hyperparameters of the model, but the model as is takes about 30+ minutes to train, so adjusting the hyperparameters even more would make it unreasonable for training at home.
You can find the full README and code at the GitHub repo here: https://github.com/c-crowder/gpt_from_scratch
Model Response
Save they are grace notige; and true and most
In the counterase arms to undertake.
Ay, how I spy, for a-foor lasts
That you’ll stand upon another and a
I must senably, our shome cryster’s chance;
And you are an earnest.
Freel most these dukes from this pain of stones,
Acquite at ours, dainted, with God-work of the crown,
I close King Henry louth to London them to well,
And they shall stinle all slay. A Catesby time,
Lest you we should be reposed
Tying, or big ire in suspection.
Lord:
I am nothing.
Provost:
No; I take the queen are but confidence;
Will I speak you, sir. You’relady, sit not
Prithee to bestowith a flawful knees.
HERMIONE:
Now, if my life be countract might love,
to do, but wit it not speeds it securel?
POMPEY:
A sunder with thee; strike her brother’s noat;
for that ever now particulals they bid me
Than the heaven or open will the wolf-veage wrong;
And the trial soon wes the plot of her mistress’d?
He want ties: softed Romeo’s Tarquin’s majestice,
And did clude us very body’s death, he met have
Your steep delights, swanding and will among
All mock uncler roses it with Lancaster to shore.
Then, till mourn the people’st of the pleasure
Dore not the words of time wondership;
So speak poison, for souls, the lesst nor be look–
Thinks he fisted love,–this I am a–
Flatter of no more good it as ‘scape:
I have been let stay’d at me about the good.
LUCIO:
Sir, you cannot three you of sweeting the spidels,
That hath twenty and fully. Sweet not the lady,
More turning indeedons as glad sirrius,
As they they shall ame not, if they pursuins
And from the selfall: madst thou tell me?
LUCIO:
No, how is a foolist, not for the heart is
As lasts, or the way for any t, to hate dead.
PETRUCHIO:
Fair Padent’s to me pass to our will,
Some straight far on that from depose me.
Swearry, when you far a man gagest
Look’d with their reasonable. They do bear smiles
Thirty Tybalt, and themselves from made friends.
This is this tattern stand alter slowery
That I have seen to l