Full transcript
0:00Well, guess what guys? We have another
0:02Claude model to test today. Claude
0:05Sonnet 4.5. In this video, we're going
0:08to put it through all my prompts. And in
0:10particular, I have gone through and
0:13replaced a few steps in my automations,
0:17uh, which are the most powerful things
0:18that I've built to see how well Sonnet
0:214.5 does against others like, uh, Claude
0:24Opus 4.1. So, let's get into it. Uh
0:27first of all, we've got this
0:29announcement here from uh Anthropic
0:32themselves. Um there isn't really
0:34anything in here that is relevant to
0:36authors and uh creative writers.
0:40It does say right here at the beginning
0:42it says Claude Sonnet 4.5 is the best
0:44coding model in the world. Uh those are
0:47some pretty bold claims from Claude. Uh,
0:50and so I'm assuming this includes their
0:53own Claude Opus 4.1, which is a very
0:56powerful model. Uh, but if you scan
0:58through this, you'll notice, you know,
1:00it's got all of these tests and
1:01everything showing just how good it is,
1:04um, even against their own models
1:06and, uh, things like that. I don't put a
1:09whole lot of faith in a lot of these
1:11bench tests. mostly. I mean, maybe
1:14they're good for some use cases, but
1:16when it comes for writing, they don't
1:17really mean much, but um but you can
1:21tell, you know, it's a good it's a good
1:23model, right? Uh they're they're
1:26claiming it's a major upgrade. Okay, so
1:28what we're going to do today, we're
1:29going to compare it to uh well, we're
1:33I've gone through and run it through my
1:34standard prompts that I run all the
1:35models through. So, I'll give you my
1:37initial thoughts on that. I feel like a
1:39lot of those prompts are a little bit
1:41outdated these days. Uh so instead I now
1:45have uh I actually run it ran it through
1:48an automation. So this is a story I'm
1:50working on. I have the whole outline
1:52everything ready and I produced the
1:54first three chapters using Sonnet 4,
1:59Opus 4.1
2:01and Sonnet 4.5. So we're going to use
2:04this to kind of compare which one is
2:06best, but we'll get to that last. Okay.
2:09Uh, first let's go to the test prompts
2:12that I ran there. There are a couple of
2:13things here that I was really impressed
2:15with. First of all, uh, I'm going to
2:18kind of breeze through these because a
2:21lot of these are kind of just standard
2:25at this point. Like all of the models do
2:28okay on them. The log line prompts I
2:30thought were fine. U, nothing
2:32particularly groundbreaking about them.
2:36Uh, nothing that was like stood out to
2:38me like, "Oh my gosh, every single one
2:40of these ideas is a banger."
2:43Um, not not really. There were a couple
2:45of good ones in here. Um, a couple that
2:48I didn't think were as good, but I
2:49wouldn't say there's really been much
2:51improvement on this front in a while for
2:53any of the models. Um, so that might be
2:56a prompt that I need to work on. Now,
2:58the outline test prompt. This is where I
3:00give it a full outline template and a
3:02brief premise and have it just outline
3:04the whole novel. This it was working on
3:08like what it was doing was absolutely
3:10fantastic. Uh so you can see here's the
3:13prologue. Boom boom boom boom boom boom
3:15boom boom boom. There it is. That's the
3:17whole prologue. Here's chapter 1
3:21all the way until here. So this is all
3:25of chapter one here. that one chapter is
3:29like 400 words that it wrote for the
3:32summary of that. Now, um if we scroll
3:37all the way down, because I was using
3:39open router, it did kind of time out
3:41right around here at chapter 14 and um
3:46and it wrote several thousand words
3:48getting up to chapter 14. All of which
3:50were pretty lengthy and some of them
3:52even had little bits of dialogue in
3:55there. Uh, as you can see here, like
3:57you're from the mountain city, she says.
3:58It's not a question. Lean defensive. How
4:00do you know? Um, so it's like kind of
4:03going in and out of pros and summary,
4:07but
4:09uh I could have just asked it to
4:11continue and it would continue to do the
4:13whole thing for the full for the full uh
4:1540 chapters that I have in the template.
4:17But I do think that what it did manage
4:19to do here before it just kind of timed
4:21out. You can see it stopped midsentence
4:23here um was actually very very good. And
4:27I was reading through it and I was just
4:28like this is actually a decent story um
4:31compared to other things that I've seen.
4:34Is it that much better than Claude
4:36Sonnet 4 or Sonnet 3.5 or 3.7 I should
4:40say?
4:42That's that's debatable. But it's
4:44definitely not worse. Uh, in fact, I
4:47would say I would say it's it's pretty
4:49clearly a better outline so far. Uh, it
4:54certainly is more thorough, right? Uh,
4:56based on the same prompt that I've used
4:58before. Okay. Uh, beats prompt. Nothing
5:01really groundbreaking here. Pretty much
5:03all all of the models do well at the
5:05Beats. Uh, but the basic pros prompt.
5:08Let's, uh, get into this a little bit
5:10here. So, this is the prompt. I give it
5:13almost no style, direction, or anything.
5:15I just ask it to write a a couple of
5:17beats that I give it. And uh let's let's
5:21read through a little bit of this here.
5:23So, we've got the wind scream across the
5:25wasteland whipping dust and grit through
5:27the skeletal fingers of ancient stone
5:28formations that cast long shadows in the
5:30dying light. Uh I know the long shadows
5:32thing is kind of a cliche at this point
5:34with AI. Um but that's in the
5:37instructions, so that's why it's there.
5:39Dusk painted the rocky expanse and
5:41shades of blood and ash, and something
5:42else moved within those shadows.
5:43something that was not wind, not stone.
5:46Haldrid stumbled over a jutting ridge of
5:50basaltt, catching himself with scraped
5:52palms before he could fall completely.
5:54His legs burned with particular agony of
5:56muscles pushed past exhaustion, past
5:57desperation into the realm where the
5:59body moves only because the mind refuses
6:01to accept the alternative. This is
6:02actually not bad. Uh, and if we
6:05continue,
6:06um, you know, there's when the scene
6:09starts to get good, we say then he
6:11looked back. They were closer than
6:13before. Far closer. At least two dozen
6:14figures scattered across the rocky
6:16terrain behind him. Moving with that
6:18horrible jerking gate that marked them
6:20as something other than human. Something
6:21that had once been human, but now but
6:23was now only hunger given form. The gray
6:25skin barely stood out from the gray
6:27wasteland, saving save for the wrongness
6:29of their movement. The way they lurched
6:31and stuttered forward like broken
6:32puppets yanked by an impatient master.
6:35This is definitely good. Uh this is good
6:37pros. a couple of things in there that I
6:39would probably tweak to make it a little
6:41bit less dramatic, but like yeah, this
6:44is solid. Definitely a solid thing. Um,
6:47I got good results out of the more
6:49complicated prompt here. It is a little
6:51bit shorter. Um, but that could be
6:54easily remedied. Uh, the dialogue pros
6:56template, uh, the pros prompt here also
6:59was very, very good. U, nothing really
7:02there to say there. Here's one where I
7:04was a little bit impressed. So, I have
7:06this editing pros template where I give
7:07it a scene that was written by a older
7:10AI. In fact, I believe it was Claude 2
7:13that really wasn't good. Uh, you know,
7:15it's not something that I would use. And
7:16I take that scene and I give it to each
7:18of the models to see and ask it to
7:20improve on the scene to see what it
7:21does. Almost always it does not give me
7:26a good response.
7:29This is the first time where it edited
7:31the scene and I actually thought it was
7:33significantly better.
7:36So, with that in mind, uh this might be
7:40a good model to use if you are like
7:43maybe writing a quick first draft of
7:45your own and you just want to give it to
7:46the AI to see how what it can do to
7:48change it and make it better, this might
7:50be a good one for you to check out. Um
7:54because if uh just looking through this,
7:56and I won't read through all of it.
7:58You'll just have to take me at my word
8:00here that this was a really solid
8:04revision. It's not perfect. I still
8:06think if I were to like it would
8:09probably be better for me to just redo
8:11this scene using a more advanced model
8:14than to ask it to improve. But if you're
8:17doing this on your own work, like this I
8:19think actually will make a difference on
8:22your work. So that's pretty cool. Um, we
8:25got the ad headlines prompt where I ask
8:27it to take the the story concept and
8:28create some headlines that could be used
8:31on Facebook or whatever. They were all a
8:33bit long in my opinion, but they weren't
8:36necessarily bad. Uh, exiled into a
8:39wasteland of monsters and shadows, but
8:41the real horror is what he discovers
8:43about home, you know, like that's not
8:44too bad. They told him the world outside
8:46was dead. They lied. What else did they
8:48hide? Another decent one, you know.
8:51Um, so you know, they're not all
8:54winners, but you know, they're they're
8:56fine. And if I were to just ask it to
8:58make them a little bit more concise and
9:00brief, I could probably get more of what
9:01I'm looking for. Uh, I did ask for a
9:05email newsletter, and this one did okay
9:07as well. I did change the prompt for
9:09this because the prompt I was using was
9:11creating results in pretty much every
9:12model that I tested that I didn't like.
9:15So, I actually simplified the prompt a
9:16little bit. Didn't give it as much style
9:19advice. just ask it to write me a simple
9:21newsletter announcing the release of
9:24this new epic fantasy novel, right? Um
9:28and uh yeah, it's fine. It's brief. It's
9:31it's to the point and uh very little
9:34that I would actually change about it.
9:36Um at least given given the template
9:39that I gave it. But here's where I was
9:40really impressed. So I I always ask
9:44uh these new models I always test to see
9:47how it does creating an SEO article and
9:49I give it a bunch of articles as
9:51examples and then I ask it to write
9:534,000 words in this case on the topic of
9:56how to write a fantasy novel and it
9:58actually wrote uh 4,000 words
10:02and uh and they're actually pretty good.
10:05It followed the the outline exactly. It
10:07gave me like it it's really scannable as
10:10you can see here. Like we've got
10:11everything kind of uh broken up into
10:14good headings. We've got
10:17um lists here. We've got clear like
10:22everything is very clearly easily easy
10:24to find. Steps to write a fantasy novel.
10:26Identify your audience before you write
10:28a single word. Figure out what who
10:29you're writing for. This isn't about
10:31limiting your creativity. It's about
10:32making smart choices that serve your
10:34story. Are you writing for young adult
10:36readers? Then keep your protagonist
10:37between 15 and 18. Focus on the coming
10:39of age themes and aim for 70,000 to
10:41100,000 words. Writing for adult, by the
10:44way, I write young adult fantasy, and
10:46that's spot on. Writing for adults, you
10:49can explore darker themes, write longer,
10:51and assume more sophisticated reading
10:53ability. Uh, researcher target audience
10:56by and so forth. And, uh, and this is
10:59actually a pretty decent article. uh few
11:01things that I would tweak just in the
11:02language, but overall like it nailed
11:06this task and it is one of the first
11:09that has actually done so to my
11:12satisfaction. Uh especially when you
11:14like read through it and you just see
11:16how scannable it is. This is a beautiful
11:19article uh at least for AI, right?
11:22Beautiful. Very well done. So I'm very
11:24happy with this. All right. So, um that
11:27that's pretty much it for my tests and
11:29uh so let's go and actually take a look
11:31at some of the um the automations that I
11:36have here. Now, just to give you a sense
11:39for what this is, uh this is an
11:41automation that I've created. I'm going
11:43to do a whole video on this in the near
11:45future, so don't worry about that. Uh
11:47but this this automation right here is
11:50probably the best thing I've ever
11:51created. It is the most powerful tool
11:54that I currently have in my tool belt
11:57and it's how I am currently using it's
11:59what I'm currently using to write my
12:00novels. I won't walk you through
12:02everything. Uh this is available to the
12:04people in my story hacker gold group uh
12:07which there is a link down below uh if
12:09you want to play around with it
12:11yourself. So what I did is I took this
12:13thing I had it write out the first three
12:15chapters of a novella that is called the
12:18bloodline of the abyss. And uh I had it
12:22write those out three different times.
12:24One with Sonnet 4, one with Opus 4.1,
12:29and one with this latest Sonnet 4.5.
12:32Now, this is an expensive automation.
12:35And uh it has a number of steps. The
12:37main steps are these four right here.
12:40And in the past, I've been using Sonet 4
12:42for this step, uh, Opus 4.1 for this
12:45step and uh, the Gemini models for these
12:49two steps. And so what I did is I just
12:52replaced these two steps with Sonnet 4.5
12:55uh, or 4.0 depending on which test I was
12:58doing. And uh, found first of all price.
13:02Let's talk price. If we look at the
13:04input and output price for Cloud Sonnet
13:064.5, it is the same as Cloud Sonnet 4,
13:09which is approximately
13:12$3 per million tokens of input and $15
13:14per million tokens of output. For this
13:17process, uh particularly for this step
13:19right here, the first draft step,
13:22um this is where I typically have been
13:24using Opus 4.1, and that usually runs me
13:26between 70 and 80 cents for just this
13:30one step, right?
13:32uh which is pretty expensive. Uh
13:34altogether this process cost me about a
13:36dollar per chapter. So if I have a 40
13:38chapter book, it's going to cost me $40.
13:40But it when I use Sonnet for this step,
13:43it goes from being like 80 cents to more
13:46like 15 to 20 cents. So you save quite a
13:50bit of money just on this one step. And
13:53uh you know, everything still adds up to
13:55be probably uh maybe 40 cents or so. So,
14:00it's still going to cost you a fair
14:02amount to go through this whole process,
14:03but it's significantly less if you're
14:05using Sonnet 4.5 or Sonnet 4.0 for that
14:08matter. Uh, I do use Opus because it's
14:11definitely been better than Sonnet 4.0
14:13in the past. But, um, hope hopefully you
14:17get the idea. All right, so with that in
14:19mind, let's actually look at the results
14:20that I got from this. Starting with
14:22Sonnet 4, just so you can see where we
14:25were coming from. Okay. Uh, now first of
14:27all, you can probably see this already
14:29visually.
14:31You can tell that we've got just a lot
14:33of big beefy paragraphs here. And that's
14:36actually a often a sign of poor writing
14:40because it means you're just like giving
14:42us a bunch of description. It's not
14:43being broken up into uh more engaging
14:47paragraphs and sentences. There's no
14:48dialogue, that kind of thing. It's not a
14:51hard and fast rule, but I found that
14:53typically when you have like large
14:55chunks of text like this. Um, it's
14:58usually weaker writing. If we look at
15:03um, uh, the one here for Opus 4.0 or
15:064.1, it's a little bit better. Still a
15:10couple of blocks of text here. And for
15:14uh, 4.5, it's definitely broken up here.
15:17You see, like we have a paragraph here
15:18that's just one word. Another one here
15:22that's just three words. So, um, anyway,
15:25that's just something I thought I'd
15:26mention. So, let's read the first little
15:28bit here. The golden afternoon light
15:29slanted through the scriptor's narrow
15:31windows, illuminating dust moes that
15:33danced above Carmela's workt. She dipped
15:36her finest brush into the crimson
15:38pigment, the bristles holding just
15:39enough paint to trace the delicate curve
15:41of St. Agatha's lips. The saint's face
15:43emerged from the vellum page with serene
15:45acceptance. There's a little AI isn't
15:47there. Uh, just the overuse of
15:50adjectives. Her eyes fixed on something
15:51beyond the flames that consumed her
15:53flesh. Carmela's hand moved with
15:55practice precision. Each stroke a
15:57prayer. Definitely some AISMs in there.
16:00Definitely uh just kind of like
16:02overdramatic pros here. Let's look at
16:05Opus 4.1. Carmela dipped her brush into
16:08the shell of ground gold leaf. The
16:10powder flaring in the shaft of afternoon
16:11light. The saint's halo demanded more
16:12than gold. demanded a steady hand and a
16:14silent prayer with every stroke. She
16:16held her breath, the bristles barely
16:17kissing the vellum as she traced the
16:19curve where St. Agatha's serene face met
16:21the painted flames. The martyrdom filled
16:23the manuscript pages rendered in blues
16:25made from the crushed lapis and reds
16:27drawn from cineabar. Agatha's expression
16:29remained peaceful despite the painted
16:31tongues of fire looking at her bound for
16:33form. Her eyes gazed upward past the
16:35edges of the page toward a heaven,
16:36invisible but absolute. Kind of similar,
16:39definitely better though. definitely
16:41more variety and specifics in, you know,
16:45where it's talking about like crushed
16:47lapis and red's drawn from cineabar, you
16:49know, getting getting a little bit more
16:50specific, which is good. If we move on
16:54to let's actually go to the beginning of
16:56chapter 2 and actually get into
16:58something that has some dialogue. Um,
17:01so, uh, the convent bell rang its
17:03cracked dissonant tone, cutting through
17:05the morning quiet, not for prayer, for
17:06an arrival. Not a good opening sentence,
17:09right? very uh tastes a lot like AI, you
17:14know. Uh but then here with Opus, we got
17:17the bell rang wrong. Not the crack
17:18dissonant clang that had announced the
17:20refugees arrival a week ago, but a
17:22fuller sound. Bronze given his proper
17:24voice reserved for feast days and
17:26visitation. Still a little bit over the
17:27top, but there but definitely better. Um
17:32when we get to some of the dialogue
17:33here, uh we've got Father Morpheus has
17:36come. This is uh Sonet 4 again. Sister
17:39Jacella whispered as she passed
17:41Carmela's cell. Her voice dropped though
17:43no one else was near. And she made the
17:45sign of the cross. A quick affurt of
17:46gesture after she spoke his name. And
17:49then there's like not any more dialogue
17:50here for a little while. Sister Carmela,
17:53mother superior Agatha's voice had a
17:56sharp edge like splintered wood. Assist
17:58Sister Beatatrix in the guest chamber.
18:00Fresh linens to good candles. Everything
18:01must be perfect. Okay. Kind of like I
18:04don't know robotic. When we get into
18:07Opus 4.1,
18:10um, it gets a little bit better here.
18:13Uh, like we have sentences like this,
18:14which are good. The bell struck again.
18:15Three times pause. Three times pause.
18:17The pattern for distinguished v
18:18visitors. I think that was that's good.
18:21Like deep point of view there. Father
18:23Morpheus arrives today for his monthly
18:25visit. The words came quick. Each
18:26syllable sharp with something that
18:27wasn't quite excitement. You will
18:28prepare the great hall. Every surface
18:30cleaned, every candle replaced. Sister
18:31Jacella, the good wine from the seller.
18:33Sister Beatatrix, fresh linens from the
18:35head table. Sister Hilda, ensure the cap
18:37the chapel is spot spotless. And me?
18:40Carmela asked. Mother Superior's gaze
18:42fixed on her held too long. You will
18:44continue your illumination. We must show
18:45Father Morpheus that God's work
18:47continues despite difficulties.
18:50Despite the infected dying beneath our
18:52feet, the thought rose unbidden. Carmela
18:53pressed it down, nodding her nodded her
18:56obedience. So definitely like I don't
18:58know. It feels like we're in an actual
19:00scene here. We're not just being told
19:01what is happening. it it's definitely uh
19:04in a deeper point of view. All right, so
19:06with that in mind, let's take a look at
19:09what we got out of the sonnet 4.5
19:12version. Let's start with the the
19:14beginning of chapter 1, uh which goes
19:17like this. The afternoon light fell
19:18through the scriptor's narrow windows in
19:20hard geometric blocks, illuminating the
19:22dust modes that hung in the still air.
19:24Carmela bent over the tilted desk, her
19:26brush poised above the vellum where St.
19:29Agatha's face was taking shape in gold
19:30leaf and crimson. The saint's expression
19:32had to be perfect. Serene amid the
19:34flings of her martyrdom. Suffering
19:35transformed into holiness. She dipped
19:38the finest brush into the gold leaf
19:40suspension. The metallic scent sharp in
19:42her nostrils. There's a little bit of an
19:43AISM there. The stroke had to be steady,
19:46confident. One tremor would ruin hours
19:48of work. She touched the brush to the
19:49saint's cheek, drawing a thin line that
19:51caught the light and seemed to glow from
19:53within. Beautiful. All right, let's go
19:56to that scene here in chapter two.
20:01Um,
20:02footsteps in the corridor outside.
20:04Sister Hilda's voice was a high,
20:05breathless thing. He's coming today.
20:07Mother Superior just announced it.
20:08Carmela's head lifted. The door to her
20:10cell burst open without a knock. Did you
20:12hear? Sister Hilda's round face was
20:13flush. Father Morpheus arrives this
20:15evening. We're preparing the chapel in
20:16the courtyard. The name was a cold stone
20:18dropping into her gut. She'd met the
20:20bishop only three times in her years
20:21here. Brief encounters after confession.
20:23His benediction at meals. Each time
20:25she'd felt a strange unease as standing
20:27too close to a cliff's edge in fog. I
20:30heard Carmela stood brushing dust from
20:31her habits hem. Has mother superior
20:33given instructions. Everything the
20:35finest candles in the uh chapel the
20:37courtyard sweat. The linens changed in
20:39the guest quarters. Sister Hilda grabbed
20:41her hand pulling her toward the door.
20:43Come. There's so much to do. So this is
20:45actually not too bad either. And
20:48honestly,
20:49I find it a little bit difficult to
20:52distinguish between Opus 4.1 and Sonnet
20:554.5. I think I need to uh play around
20:58with them a little bit more, get a
21:00little bit more of a sense of like which
21:02one is better for me. But I will say
21:05this, it is 100% way better than Sonnet
21:094. Like like way better. So, if you are
21:15using Sonnet 4 at any part of your
21:16process, I would strongly recommend
21:18upgrading to 4.5. I'm pretty sure in
21:21almost in most of the instances, you are
21:24likely to get better results out of
21:26Sonnet 4.5 than Sonnet 4. What I'm
21:30interested in is can we get better
21:31results than Opus 4.1
21:34which for creative writing uh has been
21:37my favorite recently
21:40and it's still really good and it's
21:42really hard to tell which one has the
21:44edge. I think having read through a
21:46little bit more of this before I started
21:48uh filming this video, I think I still
21:52give the edge a little bit to Opus 4.1.
21:54I think it's still just a little bit
21:56better. like the language is more
21:58diverse, if that makes sense. It's it
22:00just it feels more lived in, more human.
22:05Uh it feels like it was actually like
22:07the scene was reasoned out by uh in a
22:11more human way rather than just sort of
22:13like robotically producing words that
22:15match your outline, right? It feels like
22:17there's more thinking going on. That's
22:19the only way I can describe it.
22:22But Son of 4.5 is very, very close. And
22:26if you're concerned about price, um,
22:29like with my automation costing me a
22:31dollar per chapter,
22:34it might be worth it to go with Sonnet
22:364.5 just for the savings I'll make by
22:38converting it from Opus 4.1 to to Sonet
22:414.5. Um, but is it better? It's it's
22:46close, but I don't think it's quite as
22:49good as Opus 4.1. Uh maybe when we get
22:52Opus 4.5 that that will be a clear
22:54improvement, but um overall definitely a
22:59solid model. Uh definitely better than
23:02Sonnet 4. So go ahead and make all of
23:05those upgrades. Um I even found that
23:07Sonnet 4 was not as good at Sonnet 3.7
23:11in some specific use cases. Uh like with
23:15writing articles, for example.
23:18In this case, this one's definitely
23:20better uh at writing articles. And so,
23:23I'd be happy to convert some of those
23:26things over to Sonet 4.5 as well for
23:29some of my automations and things.
23:31Overall, I think this is just a very
23:33solid model. It's definitely one I
23:34recommend you check out. Try it for your
23:36writing. You're definitely going to get
23:37good results for your writing. Are they
23:40going to be better than 4.1 Opus? I
23:41think that might be a matter of taste. I
23:43think Opus might just be a smidgen
23:45better, but it is an expensive model. Uh
23:48so deal, you know, deal with that how
23:50you will. Uh hopefully this has been
23:53useful for you. And if you want to get
23:55access to this automation that I talked
23:57about, go ahead and check out my link
23:58down below to join the gold group. We
24:01are on a u a wait list right now, but uh
24:06if if you don't see it open, I'm sure we
24:09could probably get you in at some point
24:10in the near future. So go ahead and
24:12check that out and I will see you