Full transcript
0:00Today we are going to be talking about
0:01yet another AI detector and seeing does
0:04it actually do a good job of detecting.
0:06AI detectors are notoriously bad often
0:08giving false positives and I kind of
0:11want to check a lot of these to see
0:12which one is the most accurate as far as
0:15I can find so that we have something to
0:17test against when we are writing our own
0:20work, whether we're editing AI work,
0:22etc. So, the one we're going over today
0:25is PenGram and PenGram is actually I
0:28decided to do this one next because it's
0:30been recommended and a lot of people
0:32have been asking about it. So, this is
0:34the one we're going to look at. First of
0:35all, it says an AI detector that
0:37actually works. Bold claim there.
0:41It says 99.98%
0:43accuracy trusted by university schools
0:46and enterprises. It also markets itself
0:48definitely as an AI detector for like
0:51teachers and stuff like that.
0:53This is interesting reviewed as the
0:55proven most reliable and most accurate
0:57AI detection tool in the market by third
0:58parties including University of Maryland
1:01and the University of Chicago. So, okay,
1:04that's interesting. Let's look at
1:06pricing. So, first of all, there is a
1:07free version and a fairly generous free
1:10version as well. It allows you to do
1:12four AI checks per day. I imagine
1:14there's a there's a word count limit to
1:17what those checks look like, but yeah,
1:19that pretty much includes everything
1:21that you want as far as the AI
1:23generation part of it goes. This
1:26includes AI assistance detection,
1:28interpretability features to understand
1:30what parts of the text are AI, file
1:32upload and OCR for scanned documents,
1:35detect AI generated and human written
1:37content in over 20 languages, access to
1:39Chrome extension to detect AI generated
1:42text on any website and Google Docs and
1:44integration, which is great.
1:46Then there's a $20 plan per month. It's
1:49a little cheaper if you get it for the
1:50annual, but it's $20 per month and that
1:53one gives you 600 AI scans per month and
1:56looks like
1:57so that's just up to 1,000 words, right?
2:00So if it's more than 1,000 words that
2:02uses additional credits.
2:04It also includes plagiarism and can
2:07automatically scan posts on social
2:09media, etc. So I like that they have
2:13plugins and and things like that
2:16integrations with Google Docs makes it a
2:18little bit more versatile and the $20
2:21for a month is a little pricey. I will
2:22say for AI detectors it's a little bit
2:26up there especially when you only get
2:28600,000 words per month which I suppose
2:31if you're an educator you'd use that all
2:33up but anyway there is also the $65 plan
2:36which gives you way more but essentially
2:39is the same as the $20 a month it just
2:41gives you more usage, okay? So let's
2:43actually go through and see how well
2:45this does. So [gasps]
2:47as context I've been keeping track of
2:50what these tools do for different
2:54samples that I have including purely AI,
2:56pure AI non-fiction, heavily edited AI,
3:00human written fiction, human written
3:02non-fiction as well as a 19th century
3:04book and excerpts from the Bible just
3:07for fun. So let's go ahead and add some
3:10of these in. So I'm just going to grab
3:12my pure AI fiction. So we would expect
3:15if it's 100% accurate that this would be
3:19100% Originality.ai gave me 64%
3:24GPT-2 gave me 66 and Winston AI gave me
3:2659. So they actually didn't do so well
3:28on this particular one although they did
3:30do okay on others. So let's go to our
3:33dashboard and we're going to paste this
3:34in, all right, and we'll say check for
3:37AI. It tells me it's going to use three
3:39credits.
3:42All right, this one says 67% of this
3:44text is AI generated. So that's kind of
3:47consistent with what we had on the
3:48others technically a a bit higher than
3:51any of the other three.
3:52So, um for Pangram, we're going to put
3:54that at 60 7%. Technically better than
3:59the other three, but not by much. All
4:01right, let's do another one. All right,
4:03for this one, this is uh this is a long
4:06piece of text that is non-fiction that
4:08was purely written with AI. Let's go
4:10ahead and check this one. All right,
4:12this one says it's 100% generated, which
4:14is true. It is. Which uh originality.ai
4:18got the same result, but the others
4:20didn't. So, this one is also up there in
4:22accuracy. Interesting that this
4:25particular fiction sample doesn't
4:28register as much of AI as many of the
4:31others. It's because you'll see in the
4:33past my heavily edited AI actually did
4:36worse than my pure AI. I think it might
4:39have something to do with the models I
4:40was using because the pure AI was
4:42written with Gemini 3 and the heavily
4:44edited AI that before it was edited was
4:47written with Claude. So, interesting. I
4:50feel like Claude has a lot of tells, and
4:52that's why anything written with Claude
4:54comes off as heavily AI. So, let's do
4:57another check. I want to pause here just
4:59briefly to talk a little bit about my
5:01membership because the one of the
5:03reasons why I'm doing these humanizer
5:05and detector tests is to be able to have
5:08a good baseline for the huge editing
5:11suite that we're working on in my Story
5:13Hacker tool, which will be out soon. I'm
5:15going to be talking about that very
5:17soon, so stay tuned for that. But, if
5:19you want to join the community where
5:21we're doing all of this work to actually
5:23create quality books with AI at a high
5:27rate of speed, go ahead and check out
5:29the link for that down below. Hopefully,
5:30that will be useful for you, and let's
5:32keep going. This is now the heavily
5:34edited version. I'm very curious to see
5:37what this one says because this one was
5:39heavily edited by two people. It was
5:41edited by myself and my editor. So, this
5:44is 100% AI generated, so not accurate
5:47because a lot of this was was
5:50significantly edited. I would expect
5:52this to be about 50% or less. This one
5:54says it's a 100%, which is also what
5:57originality.ai said. The other two were
5:59close, but interesting. Okay, now we get
6:02to some actual human-written fiction.
6:05For originality, GPT-2 and Winston, they
6:08were all pretty accurate with zero or in
6:10case of Winston, 1%. So, let's see if
6:12this one gets it.
6:16All right, this one says it's 100%
6:17human-written, which is accurate. So,
6:20we'll go ahead and say, well, okay, so
6:22since it's 100% human-written, we'll say
6:25this is 0% AI. Now, for modern human
6:28non-fiction, this is one where I trip up
6:30the AI the other AI models.
6:33This is an article that I wrote years
6:36before ChatGPT was a thing. There was no
6:38AI involved. And because I wrote it, I
6:40know for sure it was written without AI.
6:43But, because this is non-fiction, a lot
6:45of AI is trained on non-fiction like
6:48articles and stuff like that, and this
6:50is one of those things. And so, this is
6:52a good example of where there might
6:54actually be a false flag. As you can
6:58see, originality.ai thought it was 99%
7:01AI, and GPT-2 thought it was 53%, and
7:04Winston thought it was 96%. So, let's
7:07see how Pangram does related to that. If
7:09it gets that this is mostly
7:11human-written, then in my book that's a
7:14clear indicator that this is one of the
7:16best AI checkers on the market. Let's
7:19see what it gives me.
7:21Okay. So, it now it says 32% of this is
7:25human-written, which implies that 68%
7:30is not.
7:32So, 68% is what we'll put here, and
7:36that's better than what originality and
7:38Winston gave me, but GPT-2 was more
7:41accurate because it said 53% was AI. So
7:45so far, it's looking on par, maybe
7:48slightly better in some cases, but
7:50slightly worse in others with the other
7:52AI detectors out there. All right, now
7:54let's
7:55I'm going to put in a sample of writing
7:57from Sherlock Holmes just to see what it
7:59does with that. This we would expect
8:01this to be 100% human.
8:04And sure enough, it is 100% human.
8:08Or in other words, 0% AI. And last but
8:11not least, we're going to do some
8:12passages from the King James Bible,
8:17which is 100% as well, or in other
8:19words, 0% AI. All right, so overall, I
8:23think those are decent numbers, pretty
8:26on par with what we've seen with all of
8:28the others. I am kind of disappointed
8:30that the heavily edited AI is still
8:33considered like 100% AI in this case,
8:36and that my human-written nonfiction was
8:38considered 68% AI. So there's still some
8:42major false flags there, as well as the
8:44editing not really doing much apparently
8:46to assuage the whole thing. And also, it
8:49didn't get my pure AI perfect either, as
8:53none of them have. So I will probably
8:55need to run these tests again at some
8:57point with different samples, maybe
8:59written with different models or
9:00different techniques. Like if I run one
9:02with an automation that's gone through
9:04several rounds of editing in an
9:05automation to try and remove some of
9:07these AI-isms, what's that going to look
9:09like? But I'm just using the same
9:11samples for all of these right now just
9:12to kind of get a baseline, and then
9:14we'll try something else from there. So
9:16overall, my thoughts on Pangram, I'm not
9:18seeing anything that makes it special
9:20than the others. And so in this case, I
9:22if I were using an AI detector, I'd go
9:25with what is cheapest, which right now,
9:27if you look at the pricing for GPT-30,
9:30it is $24 a month,
9:33and Winston is $18 a month. So both of
9:37the I mean, all all of these I feel like
9:39are a bit overpriced, right? The one I
9:40like in terms of price is originality.ai
9:43because that one you can just do pay as
9:44you go. It doesn't have to be a monthly
9:46subscription, which is really useful.
9:48So, that's still the one I'd probably go
9:49with even though it was not super
9:51accurate on some of these, but it
9:54doesn't really appear that so far there
9:55have been huge differences between all
9:58of these tools. They all seem to be
9:59running on very similar algorithms.
10:02So, that was my review of Pangram. Uh
10:05seems okay, um but not really much
10:08better or worse than the others. Uh I
10:11hope that was useful for you, and I'll
10:12see you in the next video.