Full transcript
0:00If you're an author, experimenting with
0:02AI, the most tempting thing in the world
0:05is to take your paragraph from your
0:06book, paste it into a video generator,
0:09and see what happens. And to be fair,
0:11sometimes what happens is pretty
0:13impressive. You might have a cinematic
0:15image, or a cool camera move, or a
0:17character who feels almost right for a
0:19few seconds. But the problem is that the
0:22result doesn't like capture your scene.
0:24It feels like the model made a nice
0:26little guess at your scene. That's
0:28usually because prose and video, they do
0:31different things. A novel, it carries
0:32interior thoughts, history, it has
0:35themes and memory and emotions, all of
0:37this stuff inside one paragraph. And a
0:38video, it generally just wants to see
0:41visible things. The camera moves.
0:43What what's the image supposed to look
0:44like? So, in this video, I want to show
0:47you a more practical way to approach it.
0:49Welcome to the Nerdy Novelist. Okay,
0:51hello. My name is Ziklag and boy am I
0:53happy to be a Nerdy Novelist. Instead of
0:55asking AI to turn an entire chapter into
0:58a short film, we are going to take one
1:00contained scene, break it into a small
1:03proof sequence, okay? Not 20 shots, not
1:06a whole movie. We're going to try and
1:07focus on seven shots that'll tell us
1:11exactly whether this thing is even
1:12viable for AI video. The example I am
1:16using is Jump Box. You know, we've been
1:18working on it before. The basic idea is
1:20that a young soldier is ordered across a
1:22field, and he walks across the field,
1:24and somebody is in behind him, a big,
1:26huge, gigantic general telling him to
1:28go, and he goes over, and he touches the
1:30stone, and gets transported somewhere
1:33else. That's all we're going to do. The
1:35lore is not the point. The workflow is
1:38the point. How do you take a scene like
1:41that and turn it into shots that video
1:44tools can handle. In the last episode,
1:47we talked about the missing step between
1:49prose and prompting, and the big idea
1:51that you cannot just move straight from
1:54a chapter into a prompt, and expect that
1:57the model is going to give you this
1:59movie. You need a translation step in
2:02the middle. The simple version is prose
2:05to beats, beats become shots, shots
2:08become prompts, and prompts become
2:10clips, and all of that becomes the edit.
2:13Each stage answers a different question.
2:15What actually happens in the scene? What
2:18does the audience need to see? Where is
2:22the camera? What should the model
2:25actually generate? And finally, how do
2:27you piece all of this together in
2:29coherence? This episode is about that
2:32middle section. We're going to take
2:34instructions and turn them into
2:36practical tests. I do not want to start
2:39by asking, "Can AI make my book into a
2:41movie?" That question is too big and
2:44answer is usually, "Oh, well, that can
2:46be misleading." A better question, "Can
2:48I direct AI tools into making a Those
2:52usual pieces into one scene?" That is a
2:55much more practical standard, and it is
2:58something authors can actually work
3:00with, I think, right now. All right, so
3:02in this first section, we'll call this
3:04the pick the right scene section. The
3:05first choice is not the tool, it's not
3:08the prompt, it's the first choice, the
3:11only choice. It's picking the right part
3:13of the scene. If you start with the
3:16scene that is too big or too internal or
3:20too dialogue-heavy or too dependent on
3:23backstory or anything like that, you're
3:25making the process much harder than it
3:27needs to be for a first video test. I
3:30would look for a contained scene with a
3:32clear visual hook. Ideally, it's one
3:36with a main location, like that field, a
3:39small number of characters, like just
3:41him and the general, a physical
3:44objective, actually getting across the
3:47field, and some kind of a escalation,
3:51being sucked into the gym. You want
3:54something that can be understood in
3:57images without requiring the viewer to
4:00know anything about the book. That is
4:03why the jump box sequence works well for
4:06this particular demo. We have an alien
4:08meadow, all of this stuff, very, very
4:11simple. It has enough variety to test
4:15the workflow, but it is still contained
4:17enough that it is not trying to adapt
4:21anything from the entire rest of the
4:22book, okay? That is the point. The point
4:25is that you are trying to test this out.
4:27You're feeling your way through this
4:29process, just like I am. I'm teaching
4:32you this, but this is one of the first
4:34times I'm going to get to really delve
4:37deeply into these tools, so I'm very
4:39excited. The important thing is not to
4:42adapt the whole chapter, all right?
4:44You're selecting just a part of it. A
4:46short little proof. That distinction
4:48completely matters. chapter might give
4:51you several emotional turns and multiple
4:54location You don't want any of that when
4:57you're trying to experiment it for the
4:58very first time. The proof sequence
5:00needs to answer the simplest question.
5:03Can this story material become a series
5:07of readable images? Can it really be
5:09this image, that image, that image, that
5:11image, and all you're doing is being
5:12walked through it? We'll see. All right,
5:14let's choose those seven beautiful
5:17shots, okay? Now you'll see here that I
5:18am actually inside of my jump box
5:23PNGs. I you can we can look through the
5:25entire sequence. We actually did this in
5:27the last video. Kind of found a rhythm.
5:31Everything found what worked, what
5:33didn't work, and now we have basically a
5:36very cool sequence for this AI to work
5:40with. Now, I use artificial intelligence
5:42a lot. As a matter of fact, it's already
5:45working on
5:47some aspects of this. Let's show you how
5:50we got into this. What I said to it was,
5:53"Let's pull up the project for Jump
5:56Box." I was working on all of the
5:59storyboards and everything. Let's pull
6:01this up. I'm going to take a look at
6:03them. That's all I said. I didn't want
6:04to get particularly deep and meaningful
6:07cuz they're actually intelligent enough
6:08to figure out if they have a scaffolding
6:10and they always have a scaffolding in my
6:12universe. They can go straight to my
6:14second brain and find this information
6:16out for theirselves.
6:18Uh the main Jump Box demo, all of this
6:20stuff. And so, what I did was I said,
6:23"Now, [clears throat] now we have all of
6:25the assets including character sheets
6:26and everything." And it says, "Yes,
6:28everything. We have everything." And so,
6:30I said to it very succinctly.
6:33>> [laughter]
6:33>> Yeah, I I was using I was very verbose
6:35is what I was. I want to make this into
6:37a short film. And it's basically the
6:40gist of this, if you want to you can
6:41screenshot this and give it to your AI
6:44cuz it's I mean, literally just give it
6:47the screenshot and you'll be good to go.
6:49Let me move my cursor out of the way and
6:50make it even nicer for you. There you
6:52are. Now, you can just take the
6:54screenshot and give it to your AI. It
6:56isn't particularly it's just explaining
6:59my situation so you will probably have
7:01to explain your situation better, okay?
7:03And then it took all of our assets and
7:07went through all of them and figured out
7:09the scenes that I wanted to do. I
7:11actually rewrote my scene in here. I
7:14told it I wanted it
7:16I wanted it to be seven panels and I
7:18wanted those seven panels to be about
7:20specific things just going up to him
7:24touching the wall. It's very very
7:26important that it be just a very small
7:29sequence cuz you don't know if it's
7:32going to be able to translate these
7:33things well. Anyway, so here's all of my
7:36information and character sheets for
7:38everyone. Actually, Anara's character
7:41sheet is not I didn't need Anubis cuz
7:43we're only going up to a certain
7:44sequence. And now it's working on actual
7:48Now, these are interesting. 1 2 3 4 5 6
7:55some kind of combination and 7. Okay,
7:59not bad. Now it's individually making
8:02each one of them. So, here's the first
8:04one and I said I wanted boots, but he's
8:06kind of faced the wrong way. So, on this
8:08one I would actually tell him that uh it
8:11needs to be uh the front of his boots,
8:13but the
8:14uh the general and the soldiers should
8:16be visible behind him, not the actual
8:19Yeah, that's what I would do. And then
8:21Oh my gosh, this is great. Again, this
8:23is out of position. You can see that
8:26they are too close. They are standing
8:28between him and the uh actual thing. So,
8:30I would change that and that's six. So,
8:33let's see here. 1 2 3 and we are in the
8:37middle of the third one. Did he get the
8:40fourth one yet? Nope. And uh we will be
8:44back when we have the rest of the
8:47sequence actually rendered out, which
8:50means I'm probably taking my dog for a
8:52walk and then I'm coming back because I
8:55can hear her right now whining. All
8:57right, so while this is doing that, I'm
8:59going to keep an eye on it remotely and
9:01then I'll be back here to show you how
9:04the sequence all works and clicks
9:06together. Okay, I'm back from my dog
9:09walk and I decided to add a little extra
9:11scenery. Now, let's take a look at what
9:14the [clears throat] AI has given us. All
9:15right, here is the pictures. Now, as I
9:19said before that this right here is the
9:23array of everything that we're going to
9:25see, okay? I do love this shot right
9:27here. This is so amazing. That's a
9:29really cool shot right there. The
9:30apprehension, I love that. This I should
9:33have told it right off the bat that the
9:34first one his feet should be facing the
9:37other way and then this shot and then he
9:39looks back and then it's this shot and
9:42then he walks across and then he gets
9:45sucked out. Yes. That's probably the way
9:48it should be. This is the way it is now.
9:50His feet are facing the wrong way. This
9:52shot is perfect. This shot is all wrong.
9:55The eye shouldn't even be in the picture
9:58at all. So, let me start rattling off
10:02corrections. Okay, this was probably my
10:05fault. I wanted
10:08his feet facing the other way on this
10:12the first picture. On the second
10:13picture, it's perfect. On the third
10:15picture, the green glowing thing and the
10:18mountain should not be there. It should
10:20just be the It should just be the
10:22general and his troops. So, there
10:24shouldn't be really any any glowing
10:26thing behind them, no wall. Then, the
10:28third shot is perfect. Well, actually
10:30not the third shot. The fifth shot is
10:32perfect. The sixth shot is perfect. The
10:35The sixth shot is his feet should be
10:37facing the other way and the seventh
10:39shot is perfect. All right, so we had
10:41some issues. This is what I had it do.
10:44Okay, this is probably my fault.
10:47On the first picture, I want the feet
10:48facing the other way. On the second
10:50picture, it's perfect. Third picture,
10:52his eye and the mountain anything else
10:54that's not the general and the troops
10:56should not be there except the forest.
10:58There should not
11:00be any glowing thing behind him or
11:03anything. Shot four is perfect. On shot
11:05six, his feet should be facing the other
11:07way.
11:08And the seventh shot is perfect. Let's
11:11make an additional shot of him walking
11:14towards the eye. It's perfectly in the
11:16center and
11:18it's framing over his shoulders so that
11:20you can see his head and shoulders and
11:22the eye as he's walking up to it in the
11:26center. It makes it the most powerful
11:28position for him to be in. I also have
11:30this other quote right here. I said, I
11:32think also on an additional shot of his
11:34quivering hands. The shot is actually
11:36referenced in the reference material
11:38because I love to use the word
11:39referenced twice. Now, as you can see, I
11:42now have his feet turned around and him
11:46pointing and Ofik and them in the forest
11:48now as opposed to the thing that they
11:51are supposed to be getting behind them.
11:53That would have been not very cool. And
11:55then his feet walking towards the thing.
11:57It barely changed the feet around, but
11:59uh it will definitely do. And then that
12:02shot over his shoulders perfectly
12:05centered. What a cinematic shot. Now,
12:08the thing I was not able to control
12:10is the spire. So, you'll notice that the
12:15each one of the pictures kind of has a
12:17little bit of a different pattern of
12:19what it's supposed to look like. I like
12:21the reference though. I mean, it's
12:22really really cool right here. So, we
12:24are incredibly close to actually being
12:27able to make this real. Hey, there's
12:30Gracie. Look at that. Standing in the
12:32window. I love it when she does that.
12:34All right. So, let's see here.
12:36Absolutely, let's see. So, what we're
12:38doing right now is we're running out
12:40rendering out one more shot of him with
12:42his hands up close. I felt like it was
12:44another really good shot to have. This
12:47is the shot right there. That right
12:50there is very nice. Very very nice.
12:53Okay, now we're going to take this all
12:55over to Higgsfield and start figuring
12:57out exactly what Higgsfield has to
13:01offer. Now, this is the part that is so
13:04exciting. You have your actual
13:07storyboard. This is a very simple
13:10experiment. There's no dialogue, nothing
13:13whatsoever that's going to happen. He's
13:15just going to actually be there and it's
13:18he's going to turn around. I'm going to
13:20give very specific instructions. So, I'm
13:23going to ask Chat GPT, actually Codex.
13:27I'm going to ask Codex, what is the best
13:30practices for actually getting the shots
13:33I want? And I'm going to explain what I
13:35want to have happen again. And then it's
13:38going to translate that. This is for
13:40people who are non-technical. You do not
13:43have to write all of the prompts
13:45yourself. You can explain exactly how
13:48you want it in your mind's eye. You can
13:50be the director. And then you can give
13:53it your information and then it can
13:55translate it and then we're going to
13:56take it over to Hicksfield. At least, I
13:58hope that's how that's going to work. I
14:00am about to put these into Hicksfield
14:02and I don't know that much about
14:03Hicksfield. So, could we actually take
14:07this idea and do I have to explain it to
14:10you? I feel like I've already explained
14:12how I want this thing to unfold. Could
14:14you go online and look up best practices
14:17for giving Hicksfield prompts and then
14:20we can adhere to those best practices
14:22and build the scenes accordingly,
14:24please. So, when I'm working with
14:26agents, I often have a little bit of an
14:29issue with context windows and bloat and
14:32stuff like that. So, I often will tell
14:34it to make a handoff document. And now
14:37it's made a handoff document. We're
14:38going to pick up where we left off, but
14:40without a lot less head room bogging us
14:43down. You should get yourself Codex or
14:46Claude Code, actually Claude Desktop,
14:49and start playing around with the stuff.
14:51It's so much fun. Right now, I'm pretty
14:55sure Hermes is even really a fun fun fun
14:59agent.
15:00Plenty of explanation about agents over
15:03on my channel.
15:03>> [laughter]
15:04>> If you want to hear about it, that's not
15:05what we're here for. This is nerdy
15:07novelist stuff. And what we're doing is
15:10we're building, okay? We are building a
15:13comprehensive list. So, what I've got
15:16here right now is a list of things. I've
15:20handed this off and if I'm not mistaken,
15:23yes,
15:24this is perfect. Perfect. So, it is
15:27completely locked in. It knows exactly
15:29where we're going with this. It also
15:31knows that what we're about to do is
15:34make a film. Very, very, hopefully, a
15:37good film. But what did I What I ended
15:39up doing was I have a whole bunch of
15:41handoff documents. What I'm going to do
15:43right here is make this big and make
15:45this bigger. And this And let me go back
15:48to here. I work with someone on X and he
15:52is amazing and he
15:55His name's Ink and we make these little
15:58films together. I do the soundtracks, he
15:59does the visuals. This is my first time
16:02doing the visuals
16:03for just me that wasn't whimsical or
16:07something. This is lockdown. But he told
16:09me something about Chinese models and
16:12how you should talk to them and stuff
16:13like that. And I started to build a
16:16little thing, but I What I did before
16:19that was Higgs Field has a prompt
16:22definition of how you should prompt
16:24things. And so, you give it video
16:26prompts that are both negative and that
16:29are both negative and positive. That is
16:32the thing that I learned and I'll make
16:35sure that you have this Higgs Field
16:36prompt pack in that MD, how to actually
16:40prompt with
16:42C Dance Bilingual Prompt Pack that MD.
16:45I'll make sure both of those files are
16:46linked below. Make sure that you have
16:49Get your hands on those. All right,
16:51first thing I did was log
16:52[clears throat] into Higgs Field and
16:54upload all of the thumbnails, I mean,
16:57sorry, all of the storyboards that I
17:00created. So, that's the first thing we
17:02did and those are uploading right now.
17:04Your upload is being verified. They're
17:06making sure that all of these aren't
17:09celebrities or something like that,
17:11which is a big issue with this sort of
17:14thing. So, let's
17:16jump in after everything is uploaded.
17:18So, one of the interesting things about
17:21new tools is that there's that moment
17:23when you hit the button and you see what
17:26you get. We have a very very
17:28well-thought-out multi-picture prompt.
17:31I've put every single one of them in
17:33here and we're going to roll the dice
17:35and see if this actually works. It's
17:38going to be a 15-second clip and if it
17:41works, it's going to be glorious. All
17:43right, let's see what happens. Now,
17:45we've walked through the entire process
17:49right now and
17:51I have had myself a pretty good time
17:54building, really honestly building. Now,
17:56I've had an amazing time building this
17:59actual film, but what I wanted you to
18:03take from it was some actual things that
18:07you can learn from. I want you to make
18:09sure that in order for your consistency,
18:11which we're going to do a lot of talking
18:13about in the next video, that your
18:15consistency and your character sheets
18:17and everything relates to exactly what
18:19the scene is when the scene is
18:21happening. That is one of the main
18:23things that I want you to really
18:25concentrate on. But, what I really
18:27really want you to do for next episode
18:30is think about all of your IP and how
18:34you can apply it. I'll give you a good
18:36example. There is so much IP that is
18:40being built on X right now. Right now,
18:43there's this teleport device assembly.
18:46This is absolutely uh delicious right
18:49here. You'll see it's got a full
18:51storyboard and all of that stuff, but
18:53this is the part that's amazing about
18:56this. This is the storyboard that
18:58literally SeaDance followed from
19:00beginning to end all in one sort of
19:03thing, a storyboard, which is probably
19:05what we're going to play with next time
19:07when we build this. Next time when we
19:10build this, we're going to build it a
19:12little differently. And we're going to
19:14do it like this. We're going to make
19:15line drawings that are a
19:18a series of actual storyboards that
19:21build out. And then we are going to make
19:24it into this.
19:28This is actually the storyboards
19:30completely rendered out from beginning
19:33to end. So, this is the beginning.
19:35She clicks everything in. Clamps,
19:37clamps, clamps. She gets blown up.
19:42She turns the thing and she gets sucked
19:44in.
19:45And starts all over again.
19:48Now, that will make you say to yourself,
19:51"What do you mean? Wait a minute. Sucked
19:52in? Don't we have something like that?"
19:54Ladies and gentlemen, I present to you,
19:57just for your enjoyment, this one is
19:59called Enoch Long Walk.
20:03>> [snorts]
20:16>> And that is a small nugget. Now we have
20:20a place to start. We have a place where
20:23we have created something. We have
20:26worked through our own pros. We have
20:28worked with the artificial intelligence
20:30to help us move along in this thing. We
20:32are not technical. We are doing this
20:34together. Ladies and gentlemen, I hope
20:36you really enjoyed this. Please make
20:38sure that you like and subscribe. And
20:40we'll see you all in the next video.