Full transcript
0:02All right, good morning everybody.
0:05How's uh the first day of NDC been?
0:07Good. All right, I see some people that
0:09are awake. So that's nice. Uh welcome.
0:13Uh today I'm going to talk about the
0:15.NET testing techniques that you didn't
0:17even know that you needed. And I want to
0:19kind of gauge the room a little bit. Who
0:21here amongst you would call themselves a
0:24developer?
0:26Yes. Okay, that's my people. And what
0:29how many of you developers like deeply
0:32care about testing and write like
0:34writing tests? Okay, that's that's
0:37that's quite some people. I'm uh you're
0:39the right audience for this talk. Then
0:41in this talk, I'm going to uh show you
0:43some testing techniques uh to approach
0:46your testing from a different angle
0:48maybe to solve some problems you might
0:50have with testing and to write fewer but
0:52far more effective tests. At least
0:54that's that's the goal.
0:56So my name is Dante. I am a technical
0:58consultant. I'm also a I would call
1:01myself a software crafter which is I
1:04care deeply about code. Uh and I I do
1:07public speaking uh like this on the
1:08side. I live in Belgium and I work there
1:11at a company called AE. You've probably
1:13never heard of them before, but yeah,
1:15they allow me to come do cool stuff like
1:17this and talk to you today. So uh a
1:20shout out there.
1:22All right. You probably have all seen
1:24these kinds of jokes before. There are
1:27two hard things in computer science.
1:30Cache invalidation and naming things.
1:33Oh, and off by one errors by also. Yeah.
1:36Um or there's only two hard problems in
1:39distributed systems. Exactly one's
1:41delivery and then the first one is
1:44guaranteed order of messages and then
1:45the second one again is ex exactly once
1:48delivery. And I would argue there's a
1:50lot more hard things, you know, uh,
1:54let's say,
1:56yeah, concurrency, that's hard.
1:58Threading, that's that's hard to wrap
1:59your head around, right? Maybe reg x,
2:02you start out with a small little reg x,
2:04I'll solve this. I'll do it with this
2:06with reg x and then a few months down
2:07the line, you don't ever know what's
2:10that doing anymore.
2:12And don't let me get started on time
2:14zones and daylight savings time and all
2:17these stuff. Uh yeah, I have some war
2:19stories and I'm going to share one of
2:21them uh later in the talk. Um I have
2:23some scars there. So it's safe to say
2:26there are so many variations on the joke
2:28that I'm beginning to suspect that
2:31coding isn't actually very easy. So and
2:33that's why we're programmers, right? We
2:35want to solve the hard problems. And
2:38that's also why we test our code, I
2:40would assume, right? Coding is hard.
2:43It's easy to make mistakes.
2:45>> [snorts]
2:45>> Uh, and so that's why we test. We also
2:48test to find bugs. Goes without saying,
2:52I guess. You want to find the bugs
2:54before they end up in production because
2:56that's where the client or your company
2:58is going to pay more. So, they want you
3:00to find them earlier.
3:02The next one isn't that obvious. Maybe
3:04also test to document the behavior of
3:06your system.
3:08Like if I show you this piece of code,
3:09it looks very straightforward,
3:12but there's some nuances in that code
3:14that aren't very obvious from first
3:16reading. But if you look at the tests,
3:19yeah, it's a little more obvious what's
3:22going on there. Like large orders
3:23receive 10% off even for new customers.
3:26We we know what that means functionally.
3:28That was maybe not easy to read in the
3:31code like the second you saw it, but
3:33it's pretty easy to see from a test. So
3:36living documentation. This is a nice
3:38benefit of tests as well.
3:42Also, we write tests to get fast
3:45feedback and make incremental change a
3:47little easier. Right? This dopamine rush
3:49every time you run a test and it's
3:50green. It's like, okay, I'm still on the
3:51right track. Good. Let's let me write
3:53some more code. And and that's why also
3:55TDD is is a good practice if you if
3:58you're used to doing that.
4:01Rewrite tests to protect against
4:03regression. We don't write tests only to
4:05know that this code works now. We write
4:07the test so that we can guarantee that
4:09the code works in the future as well.
4:12And we write tests to make us trust the
4:15code. And I saw this t-shirt and
4:18actually there's a lot of t-shirts like
4:20this. Don't know if you know this.
4:21There's like a whole business of in code
4:23we trust t-shirts out there. Um, a lot
4:26of developers are like blindly trusting
4:28their code. I hope they write tests. And
4:30that's why I thought I'm not religious,
4:31but the next t-shirt is is a bit better.
4:33It's like in God we trust and everything
4:34else we test. So that's more of the the
4:37message I want to show you right now is
4:39we write tests to to trust our code
4:41more. And this is actually way more
4:42important than most people think. Like a
4:45test suite is like a parachute. How many
4:49holes would you want in there?
4:51That's a quote by Uncle Bob or Robert C.
4:54Martin if you're curious. And so you
4:56don't want holes in your test suite.
5:00Granted, that's hard to achieve and
5:02that's also maybe like no holes is also
5:04maybe impossible to achieve or it will
5:07take you lots of time and there's this
5:098020 principle that says focus your
5:11efforts on on the things that uh that
5:13give you the 80%.
5:16But drawing on this parachute metaphor,
5:20here's uh three I think there's actually
5:22param motors because they have the
5:23little like fan on the back. Um, and
5:26these three people, they have problems
5:28with their tests. The first guy is fine
5:31actually. He says, "Damn it, I changed
5:33my code and now my test failed."
5:36Depending on how you look at it, this is
5:38actually good. We'll see that. So, if
5:40you change your code,
5:44so yes, we changed the code. And our
5:46tests, we check if they still pass.
5:50If they don't, oops, alarm bells. We
5:53have a red test. we have a failing test.
5:56It actually depends on if the
5:57requirements changed here.
6:00So if the requirements did not really
6:03change,
6:04then you broke the code. You were doing
6:07some stuff, you were refactoring, you
6:09broke the code, but the tests showed you
6:12that you broke it. And so this is
6:15actually a good safety net here. Your
6:17tests are doing what they were designed
6:18to do. You messed up.
6:21If the requirements did actually change,
6:23so you had to implement a new feature,
6:25yeah, you probably forgot to update your
6:27tests to that new requirement. [snorts]
6:29If you test that the discount should be
6:3110%. And you start changing the code,
6:33the test fails. Yeah, probably you're
6:36still asserting that that should be 10%.
6:38But it's now changed to 20.
6:41So this first guy, he's fine. You know,
6:43this is what tests were designed to do.
6:46The second guy is in a little more of a
6:48pickle.
6:50He says, "That's nothing. I changed my
6:53code, but my test still passes."
6:56Well, depending on how you look at it,
6:58that's bad or not. So,
7:01my clicker is not working today. Uh, so
7:04if your tests actually do pass, if you
7:07changed your code, then the question is,
7:10did you expect the tests to actually
7:12fail? like did you do some crazy stuff
7:14in which you thought surely now I'll
7:17break I'm breaking the code and the
7:18tests pass that's an issue now if you
7:22didn't expect the test to fail you just
7:24made a good refactor that's fine
7:27you did a good refactor you didn't
7:29change the inner workings of the code
7:30you changed some you extracted some
7:32method tests are still green okay
7:35perfect you you have this this trust in
7:36the code
7:38if you actually did expect them to fail
7:41well now we come to a First issue that
7:43we have, your tests are incomplete.
7:47Your test didn't catch the crazy change
7:49you did just made, which you sure
7:52thought that would break the tests. So,
7:54we have trust issues now with our tests.
7:58This third guy, yeah, he's really in
8:01trouble. He says,
8:04"Yeah, I'll just Jesus hear this. I
8:08changed nothing and my tests are now
8:10failing.
8:13>> [laughter]
8:13>> I I hear the lack of recognition over
8:15there. Yeah, this is a problem. You
8:18know, this this guy's this guy is on
8:20fire. He's he's he's not having a good
8:22time. Well, depending on how you look at
8:24it, of course, because if you did change
8:26if you did no changes to the code and
8:28your test still pass, that's kind of
8:30what we expect. Yeah, you didn't touched
8:32it. You just reran the test. Oh, okay.
8:34My tests are still green. That's how it
8:36should be. But if that doesn't happen
8:38like those gentlemen probably are
8:40thinking of uh then you have a flaky
8:42test and we'll tell uh I'll tell you all
8:45about what you can do to mitigate that
8:47later. So this is like the whole
8:49decision tree here of do you have
8:52problems with your tests or are they
8:53like behaving just fine and we're mostly
8:56going to talk about the problems today
8:58because that's yeah what you're here for
9:00maybe.
9:02First of all incomplete tests. Well,
9:06what can we do to fix that? We have some
9:10strategies. What are some reasons for
9:12incomplete tests? Well, either yeah, you
9:15have a missing test case. You did not
9:17foresee this edge case to pop up in your
9:19code and your test cannot catch it
9:22because yeah, you you did not foresee it
9:25or you just forgot to check something.
9:27You forgot an assertion. So, it's kind
9:30of like or you forgot something in the
9:32arrange step or you forgot something in
9:33the assertion step.
9:37Let's talk about test cases like how to
9:39come up with good test cases
9:43for uh for that I want to go to the
9:45pharmaceutical domain you know suppose
9:47we would model what a pharmaceut what a
9:50clinical trial would look like in code
9:51in a test it's it's like far-fetched but
9:54uh let's say drug should be safe that's
9:57our test and we have a whole bunch of
9:59parameters to check if our if our drug
10:01is safe or not and so this is a theory
10:04this is uh we're going to write some
10:06test cases and we come up with a whole
10:08bunch of test cases for this test. You
10:10know, we test a female of Caucasian
10:13ethnicity that is an adult that weighs
10:15so much and we give her this dose and we
10:17see if that if that's safe and we come
10:19up with all these test cases.
10:22Now, if we forget a test case, good luck
10:25to that group of people that are not
10:27tested in the study, you know, good
10:28luck. And so that's why in in in the in
10:32the pharmaceutical domain at least or in
10:34the medical domain, they do randomized
10:36control trials. You know, they cannot
10:38come up with every single variation of
10:40people to test this drug with. But if
10:42you just pick a thousand random people
10:45and then you see which ones it works for
10:48and which one it doesn't, then you can
10:50st start drawing parallels and you can
10:52start finding conclusions from the
10:54randomized group.
10:57And so they safeguard against our
10:58biases. is you you know you have a bias
11:00when writing tests. You suppose that
11:03this is a case that actually needs to be
11:05tested but you forgetting some other
11:07cases and so this randomization helps
11:10against bias and [snorts]
11:13yeah it's better evidence that your test
11:15is working.
11:17So what I'm not saying is that example
11:19based testing is bad. No, we're used to
11:23thinking in examples, thinking in edge
11:25cases and thinking in like the the the
11:28good path and and the bad path of the
11:30code and we have examples in our heads
11:31of of test cases that we write there. So
11:33that's not bad. You can still do that.
11:35It's totally logical. People think in
11:37these patterns. What I am saying is that
11:40you have other tools in your tool belt
11:41that you might not know of.
11:44Randomized testing.
11:47So we can introduce randomized testing
11:48and I'm going to give a little simple
11:50example. So for example, if we have an
11:52test that tests our alpha numeric filter
11:56uh method, then what we could do is come
11:59up with a whole bunch of test cases and
12:00a whole bunch of strings to filter
12:02through that and we could also just
12:04generate a random string.
12:07Now our assertion will look differently.
12:09We just have to assert that after our
12:12filter the only characters that can
12:14remain are actually alpha numeric.
12:18So but this has some problems and we'll
12:19come back to that. But this looks very
12:22simple but it already yeah guards us
12:25against some stuff. It already checks if
12:28it does not replace only a first
12:30occurrence of a non-al alpha numeric
12:32character. It really does filter out
12:34everything. It also could be an empty
12:36string because you see there the the
12:37zero with a a random for uh next. So it
12:41also guards uh empty strings. It also
12:43guards like large strings of 200
12:44characters or anything in between.
12:48Yeah, it already tests quite a lot of
12:50stuff with very little code.
12:53Now if you want to test more complex
12:56data structures like we all probably do
12:58in our day jobs, there's this great
13:01little library. I'm going to shout out a
13:02a lot of open source like cool tools and
13:04libraries uh called Bogus that can
13:07generate random data that looks
13:09realistic and that actually is very
13:10useful for randomized testing.
13:14So in bogus you can for example if you
13:17have this address record you can define
13:19rules and for a what's called a faker
13:23and so this faker can set up a fake
13:25address for you and you define rules in
13:27this faker like oh I want my street to
13:29look like an actual street name I want
13:31my city look like an actual city and I
13:33want my country codes to only be Belgium
13:36Netherlands or France and you define all
13:38this and then in your test you just call
13:40address fakergenerate and it will
13:42generate create a totally random address
13:44with all the fields and properties
13:45filled in.
13:48That's a little utility there for
13:50setting up randomized tests.
13:54Now, the issue with randomized test data
13:57is that it's unpredictable, right? It's
14:00it's unpredictable. So, for example, if
14:03I write a test that a normal day,
14:05whatever that might mean, only has 24
14:08hours
14:09and we generate a random date and we set
14:12the start to 12:00 a.m. that day and we
14:14set the end to 12 a.m. the next day and
14:17then we have this cool method that
14:18counts the hours between the start and
14:20the end date. We expect it to always be
14:2324.
14:24This test looks very straightforward.
14:26It's very simple.
14:30Now, if we roll our dice on this
14:32randomized date and we come up with
14:35snake eyes, we have like very bad luck.
14:38It could actually generate something
14:39like this,
14:41March the 29th, 2026, which was
14:44recently.
14:46Does anyone know what happened on that
14:47date?
14:49Yeah.
14:51Yes. The transition to daylight savings
14:53time. And then our test will show this.
14:57Yeah. But we only had 23 hours in that
14:59day actually. So that's also why the
15:02time zone is important in this method.
15:04So it takes into account on which what
15:06time zone you are and for uh most of
15:08Europe that day was a transition day. So
15:11we have a problem here. We actually made
15:13a flaky test by introducing randomized
15:15test data.
15:17So this was the transition day. So and
15:20actually that's one of the war stories I
15:22had in my job. You know, I worked for an
15:24electricity management company that uh
15:27had to install like these smart meters
15:29in people's homes to count their energy
15:32usage. And we had a validation rule that
15:35said every 15 minutes we should have a
15:37data point. So 96 data points per day.
15:40And we actually knew this from the
15:43beginning that some days transition
15:45transition days could only have 92 or
15:47100 uh values. Still even though we knew
15:51knew that we kind of made some mistakes
15:54in the code and it blew up one day and
15:56that was such a transition day. So we
15:59should guard against that.
16:02We discovered a flaky test. So that's
16:05the other part of the equation. We have
16:07flaky tests now.
16:10And flaky tests are bad. I mean they not
16:13only make you spend time looking at
16:15false negatives or false positives. Uh
16:19even worse you you probably also get
16:21alerts in your email if like a
16:22regression test set fails and and every
16:24time uh and you lose confidence in these
16:27test suite you lose your trust in the
16:28code which is not great.
16:31How do you deal with them?
16:34I'm not going to lie. The most common
16:36example is this.
16:41Again, the lack of recognition there.
16:43Um, this is an actual screenshot of our
16:45regression tests pipeline. Uh, I'm not
16:48afraid to say this. It's like you check
16:51what tests fail. Oh, it's that one.
16:53Well, that that fails sometimes. Just
16:54it's fine. And then one day we decided
16:57to fix it. But
16:59another approach is this this. Yeah,
17:04just just skip it. Just just skip the
17:07test. We'll fix it later. That's tech
17:09depth, you know. put it on the road map.
17:13Okay. Well, it's all fun in games, but
17:14how do you actually deal with them? How
17:16do you now actually deal with these
17:17flaky tests?
17:19Well,
17:21first of all, be careful. Be careful in
17:23your code with daytime.now
17:25uh and use abstractions for that. Use
17:27the time provider that Microsoft
17:28provides or use i system clock or use
17:30your own i daytime provider, whatever
17:32you you come up with. Use UTC wherever
17:36you can. Like be careful with time
17:38zones. So take in any daytime offset but
17:41then in your code only work with UTC. So
17:43map it to UTC first, store it in UTC,
17:45retrieve it in UTC and then map it when
17:47it goes out again.
17:50Check for yeah mutable state. So test
17:54order is not guaranteed. Tests can run
17:56in par parallel or and if you have like
17:58some shared state somewhere and and one
18:00test runs before the other then maybe
18:02that that messes things up.
18:07This is uh also an awkward uh thing.
18:09Don't don't forget your awaits when
18:11writing tests. If you have a task and
18:14you don't await it, the assertions might
18:16actually execute before your your task
18:18returns. So then you don't have a value.
18:20And if you do assert not null, yeah, it
18:22will be null because it's not finished
18:24yet. So then your tests might break or
18:26it might not.
18:28And then when when end to end testing or
18:30whatever and you have a lot of external
18:32dependencies, yeah, build in some kind
18:34of retry mechanism if the server doesn't
18:36respond immediately like retry or or or
18:39have some some some back off or or
18:43whatever to to try again.
18:46Most important ones is use deterministic
18:49test data. So it can be random. It just
18:51has to be deterministically random like
18:54random but the same randomness every
18:56time. And we'll see how we fix that in
18:58our example.
19:01Or the other option just generate so
19:04much test data like in the clinical
19:06trial that any problems with the
19:08randomness will surface every time. So
19:10you have a statistically you
19:12statistically know that this code
19:14actually works or these tests actually
19:16work.
19:18Uh let's look at these.
19:20So the first thing deterministic data
19:24we'll fix with seeding. So if you have a
19:27new random, you can pass a C to it. And
19:30so if that test runs and it generates an
19:33eight for example,
19:35um the next time you run that test, it
19:37will also throw an eight. Like it will
19:40also return an eight. That eight was
19:42random, but it's always the same one.
19:46And with Bogus, you can also use the use
19:48seed, so it will also deterministically
19:50like generate your your fake data.
19:54The second approach was use a large
19:55sample size.
19:57So if you have a test like this which
19:59says that a random number should never
20:00be funny whatever that means and you
20:03generate a random number and you say
20:04okay well my is funny method should
20:06return false but the implementation is
20:09actually that 6ix9ine and 420 are both
20:11funny then twice in every 500 cases this
20:16test will fail. We want the test to fail
20:18by the way cuz our test is wrong. you
20:20know, we we wrote the or our code is
20:22wrong. Whatever the requirement is. By
20:25the way, side note, I'm technically Gen
20:27Z, but I don't think 67 is a funny
20:29number. Um, just put it out there.
20:33So, our test will pass, but it's wrong.
20:36So, it should fail, but it will only
20:39fail like two out of 500 times, which is
20:42low. You know, this is very flaky. You
20:44will only see the test fail once in a
20:46blue moon, and you will probably think
20:47that it's an anomaly.
20:51So how do we fix this? We just throw
20:55sample size at the problem and we do it
20:57that 10,000 times. And now yeah the
21:00chances that this will fail is equal to
21:03the ch one minus the chance is that this
21:05will never fail which is this which is
21:080.9999 and so on. So, we just made this
21:12test way more robust and now it will
21:15actually fail when it has to.
21:21There's an other issue with randomizing
21:23your test data and it's not so obvious
21:27if you first think about it, but once
21:29you see it, you will see that it's a
21:30problem.
21:33If we have a simple examplebased test,
21:35we just have a simple test that says our
21:38at method should add two numbers. Okay,
21:42very simple. And we test it with oneplus
21:451, we test it with 4 plus 0, we test it
21:48with 999 + 1 and we think we come up
21:52with enough cases. But you saw my talk
21:53and you thought, well, I'm I'm going to
21:55try this randomized testing now. Now you
21:58could have a problem because if you
22:01generate these random numbers A and B
22:03and you add them up, what are you
22:05actually now expecting it to return?
22:10Yeah. What is the expectation of this
22:12add function?
22:14You could write this.
22:17Well, that kind of defeats the purpose.
22:20No, I mean, yeah, this is reimplementing
22:23the ad function in the test. Okay. And
22:26this is a very trivial example. It looks
22:29very stupid, but imagine you're doing
22:31like addition of uh different units like
22:34centimeters plus inches or whatever. Are
22:36you going to write the whole conversion
22:37in your test to see what the expected
22:39value is? Like with the example based
22:41test, you could put your expectation in
22:43the test case. But here that's not
22:45possible.
22:47And reimplementing in your tests is a
22:49big no. Like that defeats the whole
22:52purpose. So how do we fix this?
22:56You have to write your tests in a
22:58completely different way. Now if you use
22:59this randomized approach,
23:02we want to test properties not outcomes.
23:05So for example, if we could come up with
23:08a property for this ad function that a
23:10plus b should be the same as b plus a,
23:13right? Any ad function should work the
23:16other way around. That should give the
23:17same result.
23:19We could also come up with a property
23:21that doing plus one twice is the same as
23:24doing plus two. It sounds very trivial,
23:27right? But if our add function works,
23:29this should work.
23:31And we should also see if we add zero,
23:34for example, that it doesn't change the
23:36original value. So a plus 0 should still
23:39be a.
23:41Now, why did I pick this example?
23:44These are not only properties of the of
23:46this code. These are actual mathematical
23:49properties of an add operation.
23:52So it's called the commutive, the
23:54associative and the identity property.
23:56And what's funny in mathematics, these
23:58properties actually define the add
24:00operation like the the addition
24:02operation. So no the addition operation
24:05if it whatever you do in that method,
24:09whatever shenanigans you come up with,
24:10if if it follows these three properties,
24:13you have written an ad method that
24:15works.
24:17And so that's might interesting for our
24:19code, right? We cannot create a a wrong
24:21implementation now with these three
24:22tests. It's impossible.
24:26Now, of course, your this is called then
24:28property based testing. And of course,
24:30yeah, properties,
24:33if you have mathematical properties,
24:35it's easier, right? If you have to find
24:37properties on yourself, I cannot think
24:39of one right now on the spot. Like what
24:41is a property of my code? I only know
24:43what my code should return.
24:45And there's a great NDC talk actually uh
24:49it went on during COVID so it's digital
24:52and the audio is pretty bad but it's
24:54definitely worth a watch that goes over
24:58a few of these properties and a few of
24:59these huristics to come up with
25:01properties for your code since it's so
25:04great I'm going to use a little bit of
25:06his wording also because I think it's
25:08really really a good point.
25:12Now, as an example, I'm going to present
25:13you with the Mars rover ketta. You might
25:15know it. It's just a simple example. You
25:18have a Mars rover. It can face north,
25:20south, east, or west. It can also move
25:23into any location uh using yeah uh using
25:27uh cartisian coordinates. So, x and y.
25:30And it should be able to execute
25:32commands. So, for example, it has a
25:34direction and a location. Uh the
25:36direction is yeah, northeast, south,
25:39west. And the location is just X and Y.
25:42And it can execute some arbitrary number
25:45of commands. And those commands are or
25:47we move forward or we move backward or
25:49return left or right. Very simple
25:51example.
25:53So this is the whole kata. I'm also
25:55going to add a small method that will
25:58reset the rover back to 0 0 and facing
26:00north. That's just an extra thing.
26:04Shout out to another cool library or
26:07package fs check. uh that's a property
26:09based testing library originally written
26:12for F. It is written in F# as well but
26:15since F just compiles to intermediate
26:17language it's just a net package and you
26:19can use it from C# 2. It's yeah it's
26:22perfectly fine. So property based
26:24testing was actually inspired by the
26:27functional programmers. Uh the first one
26:29was quick check I think in HLL and then
26:31FS check. There's also an CS check for
26:34C, but I like this one better and it's
26:37net, so you can use it.
26:40Anyway, properties.
26:42The first one I like to call you can
26:44throw anything at it. So you can test
26:47your code if it would accept any value
26:49without blowing up. So in my case,
26:52that's whenever we send a command to the
26:54rover, it should return a new rover that
26:57is not the same as the original one. I'm
26:59working with immutable design here. So
27:01I'm not changing the state of the rover.
27:03I'm just returning a new one that's
27:04updated. So if [snorts]
27:07you can see there's a property attribute
27:09there. That's the fs check attribute
27:12that shows that this is a test which is
27:14property based. And what fs check behind
27:16the scenes will do is it will both
27:19generate random parameters. So if you
27:21say rover and command, it will generate
27:23a random rover and a random command for
27:25you. So that's for free. You don't have
27:27to mess around with the bogus or the or
27:29the random next. It will do that for
27:31you. And also it will fix the sample
27:33size for you because it will spawn 100
27:35tests for every test you write. Very
27:37nice. You get 100 tests for free.
27:41And this is a very simple test. We just
27:43execute a command and we check that the
27:45result should not be the same as the
27:46original rover. So it made a move or it
27:48did a turn whatever.
27:51So that's one property that almost all
27:53code can check. Like if you have a
27:56method that takes an integer, just throw
27:58any integer at it and and maybe just
28:00check that it doesn't throw an
28:01exception, whatever. That's that's a
28:03property that you can always test.
28:06The next one is there and back again. So
28:08you do something, then you do the
28:10reverse and you check if you ended up in
28:11the same spot as you started with. So
28:14for example, turning left and then right
28:15should have you result in the same
28:17direction or moving forwards and then
28:20backwards should result in the same
28:21location. And so they you can have these
28:24tests that that check that we also
28:27always do this test with random rovers,
28:29right? So maybe if a rover is at a
28:31thousand uh and and and zero and we do
28:35this, it should still also work. So it
28:37the randomized data is still there.
28:41And next property you can check is that
28:42some things just never change. Some
28:44things are static or unchanging like
28:47turning will not change your location.
28:50the rover should still stay in place
28:52when turning or moving does not change
28:55your orientation. If you move forward
28:57and you're facing north, you should
28:58still be facing north at the end. So
29:00this is testing the invariance of your
29:02code
29:04and that's how you do it.
29:07And then different path same
29:10destination. This is also a very
29:12interesting one like two equivalent
29:15paths should end up at the same
29:16location. So for example, if we start
29:19from the bottom and we go to the left,
29:21we could move forward, turn left, and
29:23then move forward again. And that would
29:26look like this.
29:28But we could also
29:30turn left, move forward, turn right,
29:32move forward, and we'll end up at the
29:34same place. So our test, whatever this
29:37initial starting position of the rover
29:38is, and whatever the initial direction
29:40it faces is, this should still work.
29:43These should end up at the same
29:44location. If it doesn't, then our code
29:46is wrong. So that's a good test, right?
29:52And then one last one. The more things
29:54change, the more they stay the same.
29:55This is this is quite hard, but
29:58if you reset the rover to its initial
30:00position once or you do it 10,000 times,
30:04yeah, it doesn't change a thing. Like
30:05this is item potency. So this is
30:08actually a utility type, this positive
30:10int here, a utility type from fs check
30:13that you can use and it will just
30:15generate a positive integer for you. So
30:16you don't have to mess around with
30:18random next whatever. Uh there's a lot
30:19of these utility types in there that can
30:22generate these random things on the fly
30:23for you and just pass them as a test
30:26parameter which is very handy.
30:29So yeah, if you uh keep uh resetting it
30:32at least once actually. So we keep
30:35resetting the rover. That should be the
30:36same as resetting it once. So this is
30:38actually testing that the reset method
30:40is item potent. And this is a very
30:42important thing in code, right? Item
30:44potency. Like if you have a distributed
30:46messages system and you sent the message
30:49like twice, for example, you don't want
30:51it blowing up the second time it
30:53arrives. You should just ignore it or
30:54whatever.
30:57Okay. So I hope with these properties
30:59and and you can look watch at the talk
31:01that I showed you in the beginning which
31:03is a very good one. I hope with these
31:06huristics you can come up with
31:07properties of your own code and you can
31:09start experimenting with property based
31:12testing and not only think what should
31:14my code what should the outcome be but
31:18what things should always be true in my
31:20implementation what things should all
31:21what property should always hold.
31:25All right. So, let's go back. We solved
31:29some issues with incomplete tests.
31:31Like we we solved not come coming up
31:35with all the test cases with this
31:36randomized testing and this large sample
31:38size. And it's not the end all beall.
31:40You can still mainly use example based
31:43testing. But this is another tool in
31:44your tool belt.
31:48Now, what if you are forgetting
31:51assertions? What if you are not checking
31:53everything in your test?
31:56You can also have this incomplete test
31:59now.
32:01And I guess this Whoa. Yeah, my clicker
32:03is really misbehaving.
32:05I thought I'd try it again, but that's
32:08what happens. Okay, so we probably all
32:12had this happen before like these this
32:16wall of assert statements like uh here I
32:19I create a person and I convert it to
32:21XML with our special company uh
32:24proprietary XML converter that does
32:26weird business logic.
32:29Yeah, we are asserting every property
32:30and every field and that it's at the
32:32correct location in the XML and
32:33whatever. And this is brittle because
32:37what if you forget to check one property
32:39or what if yeah I check here for example
32:41only in the array I check that one
32:43element is there but what if all the
32:45other elements are now gone your test
32:47will still pass which is not good.
32:51We need instead of writing all these
32:53asserts we kind of need to adopt a new
32:56mindset to be able to to do this. So
32:59instead of thinking about I always
33:02expect my test or my code to produce
33:04these values and I will list all the
33:06values that I expect, what if instead we
33:09we assert that our output has just not
33:13changed since the last time we looked at
33:15it. So we check for regression. We check
33:18well last time I looked at the output I
33:20verified. I actually did something to
33:22say this is the output I would expect.
33:25We store that somewhere and then the
33:27next time the test runs we just check if
33:29that output still is the output that
33:31that comes up.
33:33Then we know our test our code is still
33:35working like before.
33:38So what do we actually need to be able
33:40to test this or to be able to check
33:42this.
33:43We need a way to verify or accept the
33:47out output the first time we run the
33:49test. So if we have this crazy XML
33:52converter, we should look at the output
33:55that it generates the first time and we
33:56should be able to say, "Yep, that's
33:57that's correct. I want this." Now,
34:01we need to detect if that output
34:03changes. And then we can still also
34:06decide if we like that change. If we add
34:08a property somewhere and oh yeah, of
34:10course, our XML now contains an extra
34:13field. I I meant to do that. You know,
34:16then you should accept that as a new
34:18truth. that output is now the new truth.
34:22So this is called snapshot testing or
34:25golden master testing or whatever. So it
34:27has a few names.
34:30And how does it work? I hinted at it
34:32before, but it first checks if a
34:34snapshot already exists. If it doesn't,
34:38then it opens like this diff window. So
34:40this is the first time you run the test.
34:41There's no existing snapshot yet. And so
34:45it says look you had nothing and now I
34:48have this. Is this the correct output
34:50you expect from your code? Okay. Is this
34:53now do you verify this as the new truth?
34:57If you don't well then you your test
34:59fails and you change your code or you
35:00change your tests and you run it again.
35:03If you do say yes this is correct then
35:06you save this test or the the the
35:08snapshot testing framework that you
35:10choose will save this output somewhere
35:12on disk and say this is now the output
35:15that I expect from now on and then it
35:17will pass your test for you
35:21afterwards if you change something you
35:23refactor something
35:26it will check again does a snapshot
35:28exist and now it does because we saved
35:30the last one and if your test output now
35:33matches the snapshot. That's great. Your
35:36test still passes. Your code is you
35:39changed something that didn't affect the
35:40output, which is good. If it doesn't,
35:44your test fails and in like a CI
35:47pipeline, it would stop here. It say,
35:49oh, oh, we didn't expect this output
35:52fail.
35:53But if you're running them manually as a
35:55developer, it will actually open a new
35:57diff window and it will ask you, yeah,
36:00did you meant this change to happen?
36:03Like, is this intentional? And you can
36:06say, yeah, well, I did or no, I didn't.
36:09And if you if you choose no, then of
36:10course your test will fail again and you
36:12have to fix it. But if you choose yes,
36:15then yeah, it will save this as the new
36:18truth. So it will overwrite his exist
36:21existing snapshot with this output.
36:24And so that's the whole flow. So you can
36:27see it becomes way easier now. You just
36:31call this snapshot testing fra framework
36:34that you have and it will pop open this
36:36diff and you can you can accept or
36:38decline the whole structure without
36:39having to write an assert for each one.
36:44In the net space this is a very very
36:46popular library for this. Uh it's called
36:48verify. It's by Simon Crop.
36:51And the whole premise is that it's great
36:53for complex data and for big documents
36:57um stuff like that to track changes that
37:00are intentional or not.
37:03Now, I have a use case for this that
37:06I've not seen too much in the real world
37:09that I think is very interesting.
37:11There's another kind of document in most
37:14of your code that you don't want to
37:16change without you knowing and sometimes
37:18it happens that you do a change and oh
37:21damn this document has now changed and
37:22that wasn't my intention. It's a swagger
37:25file.
37:27Have you been there where you change
37:28like oh I changed my code a little bit
37:30and then of oh your API contract has
37:32changed now in some cases that's not bad
37:35but in some cases like these API
37:37contracts are like very strictly
37:39enforced between teams and they will
37:41yell at you for changing the contract.
37:44So pro tip snapshot test your open API
37:47specs.
37:49So very simple we use the web
37:51application factories to spin up our API
37:53in our tests just like an integration
37:54test.
37:56We create an a client just like you do
37:59in integration tests and you just
38:01retrieve your open API spec and then
38:05with the verify library the only thing
38:06you have to do is call verify.
38:10So this is the whole test but it help
38:13you helps you so much in my opinion
38:15because if we have for example this very
38:17very basic minimal API the basic weather
38:20forecast example everyone knows
38:23and we decide well let's add an optional
38:26property you know what what can we go
38:28wrong we just add a summary and it's
38:30optional so we're not breaking our code
38:33it's just an optional thing well wrong
38:36your snapshot test will actually tell
38:38you that your API spec has now changed
38:40because even optional parameters are
38:42included of course in in this spec.
38:46So this guards you against changes to
38:48your open API spec and of course you can
38:50still accept this change. Yeah, I want
38:52this but you get the choice.
38:56The added benefit of this by the way so
38:58that's one benefit that you can get
39:00notified of changes. The added benefit
39:02is your open API spec lives in your
39:04repository now which I find nice because
39:07otherwise you need to run your API to be
39:09able to inspect it in the browser and
39:11now it's just checked into git because
39:13these snapshots should be saved
39:15somewhere on disk and then should be
39:16also checked into your repository so
39:17that other developers can also use these
39:20tests.
39:21I think this is quite nice and even in
39:23most IDEs if you open this JSON you you
39:26have an interface like a swagger
39:27interface next to that. So that's that's
39:28that's cool.
39:31So you can now see what Swagger
39:34generates before running the API.
39:37And this is kept up to date, right?
39:39Because every time you run the snapshot
39:40test, if it's not up to date, it will
39:42yell at you.
39:46Great. We've covered property based
39:48testing. We've covered snapshot testing.
39:52We still have these example based tests.
39:55How do we actually know we have written
39:57good tests now that we improve our
39:59tests?
40:02Well, that's where metrics come in.
40:05Yeah. And the first one, the first
40:08metric I want to discuss is line
40:09coverage. Who here has had a manager
40:13that wants you to chase a certain number
40:15of line coverage?
40:18That's way too many people. [laughter]
40:19like I I guess that was not a good
40:22experience because there's this this
40:24thing that if a metric becomes a target
40:27and stops being a good metric.
40:30So once you chase this, you're only
40:33writing tests to like that are useless
40:35but that cover the code. It can tell you
40:39how many lines of code were executed by
40:41your tests. That's kind of it. It's
40:43still like sometimes useful to look at
40:45it if it's very low that there's a
40:47signal that is something wrong, but it's
40:49not the best metric there is.
40:53The next metric is branch coverage. And
40:56this is starting to get more interesting
40:58like did we take all the possible paths
41:01in our code in our tests as well.
41:04And so for example, if you write an if
41:06else statement and the if is 99 lines
41:09long and the else is only one line long
41:13and you cover in your tests only the if
41:15statement, you would have 99% line
41:17coverage, right? So wow, you covered all
41:19these lines. Great. But your branch
41:21coverage would only be 50%. Because
41:23you've completely forgot the else case.
41:26So this is more in line with how logic
41:29in your code or how the control flow
41:31actually works. It's a better metric.
41:35It's not supported by most line coverage
41:37tools, which is a shame.
41:40But then there's mutation score, and
41:43that's an interesting one because it
41:46introduces small bugs or mutations into
41:50your code, and it checks, do your tests
41:53catch this bug,
41:56which is the whole purpose of your tests
41:57in some cases, that it will catch
41:59whatever goes wrong.
42:01And so it inserts these mutations and
42:03when the test catches them, the
42:05mutations are killed.
42:08But if a test passes without ever seeing
42:11an error of about this bug, then your
42:14mutation survives, they call it. So it's
42:17like an analogy in in biology. And then
42:20your mutation score is the amount of
42:22mutations that were killed by tests over
42:25the entire uh over the total amount of
42:27mutations.
42:29And so this gives a pretty good insight
42:31in if your tests are actually effective
42:33in catching bugs or in catching wrong
42:36behavior.
42:38And it can do a whole bunch of
42:40mutations. It can change time signs by
42:44divided signs. It can change strings to
42:46empty strings. It can change order by to
42:48order by descending. So it can do a lot
42:50of stuff. It can even delete entire
42:51method bodies and see like if I delete
42:53this method body, will the test still
42:55pass?
42:58And this automated process like
43:00generates hundreds of mutations and then
43:01checks and I'll show you a little uh
43:04demo later. So in our whole control flow
43:07here in our whole flow diagram changing
43:10the code is done by the mutation and
43:13checking if the tests pass that means
43:16the mutation survived. So that means we
43:18are we're heading on a wrong path and
43:21that also means that probably you forgot
43:22a test case again or an assertion. go
43:25back to the beginning of the talk and
43:26start over.
43:29Now, a small example. This is the last
43:32time I'm going to show you again another
43:33library because that's that's cool. Uh
43:36it's called Striker. It's the main
43:38mutation testing tool I think in the
43:40.NET space. It also works for
43:43JavaScript. So, there's also a
43:44JavaScript version.
43:46Um you can just install it by net tool
43:50global install and and and having that
43:52and then you can run it on your on your
43:54tests.
43:57Now I was too lazy to write my own thing
44:00with a lot of tests which is kind of
44:02against the whole point of the talk but
44:04uh I took like the game of life example
44:06from the internet and I decided to check
44:08check did this person write good tests
44:10or not. Spoiler they did a pretty good
44:12job.
44:15Now, I have a recording of my terminal
44:16here because this takes a long time. So,
44:18I won't bore you with doing this live.
44:24So, you can see here that it generates
44:27mutants. And if I get my pointer working
44:30here,
44:33oh, the text is really dim, so you
44:35probably won't see it, but it says here
44:37275 mutants killed of the 333 33 that
44:42they generated. So it generated 300 bugs
44:45to insert in the code everywhere and
44:47then it killed the tests found 275 bugs.
44:52Amazing. Now of course some bugs
44:54survived and that's a problem and then
44:56they give you a nice dashboard to take a
44:58look uh at what uh things survived. So
45:01let's take a look at that together.
45:06Uh click here. Yep.
45:10open this.
45:14So this is the dashboard that they give
45:15you. I will zoom in a little bit and you
45:17can see it covers all files. We can go
45:20to services for example and check out
45:22the game engine because that has the
45:23lowest scores here. You can see also
45:26some some scores are pretty good. So
45:27this this person did a good job with
45:29their test. And then let's turn on all
45:31options. Let's scroll down.
45:35You can see what it did here. So if I
45:37click it removed this negation and it
45:41checked does the test pass and it says
45:44you well this was a mutation that was
45:47killed. So our test caught this bug and
45:50it tells you this test actually and two
45:52more tests caught your bug. So great you
45:54had you had a test that guards you
45:57against this. It also has some stuff
46:00that
46:01did not work. So for example,
46:05wow, it made our exceptions empty
46:08strings.
46:09Of course, this is not a problem to us.
46:11So I'm not saying that mutation score
46:13should be followed blindly because you
46:15have stuff like this, right? This
46:16doesn't matter to us. As long as it
46:17throws an exception, it's fine. The
46:19message might be important, but it might
46:22also be not. So that's why this
46:23dashboard is handy. You can look through
46:25it. But for example, it changed this
46:29check to null and the tests didn't catch
46:33it. Even though it was covered by 16
46:35tests, it still survived this bug. Might
46:38be worth checking out why that's the
46:40case.
46:42So this is quite interesting. You can
46:44you can take a look through this and you
46:45can see also what it did change. Like it
46:47made this whole constructor empty and
46:50then the test failed, which is what we
46:53would want.
46:55So that's a very uh interesting tool for
46:58sure.
47:01Let's go.
47:03Let's go here.
47:07No, I don't want to open it anymore.
47:09Okay, so mutation testing has some
47:12benefits, right? So it gives you
47:14feedback on the quality of your code. Is
47:16your tests act of your test, sorry? Are
47:19your tests actually catching mistakes?
47:23It can debunk high coverage statistics.
47:26We have 80% test coverage. Yeah, but
47:28your mutation score is like 5%. Your
47:30tests are doing nothing. Just running
47:32your code. Maybe
47:34it can give you a good indicator of
47:37regression protection like will this
47:39keep working in the future? Will if
47:41something changes down the line, if
47:44something new gets introduced just like
47:46the mutants do, they introduce random
47:48stuff. Will our code our tests catch
47:51this?
47:54Now, it also has some caveats or some
47:57downsides.
47:58Like there's only a handful of mutations
48:00that are supported. I showed you them in
48:02the beginning that you change a plus
48:04sign to a minus sign or you do an if you
48:07reverse the if or whatever, but yeah,
48:10some code is just never changed. So, it
48:14won't catch if if a test is is is uh not
48:17covering that.
48:21they are slow to run. This code base was
48:24not very big and still like these
48:26mutation tests took a minute and a half.
48:28So you don't want to run them every
48:30time. You don't maybe also not want to
48:32run them on every PR. Uh you might want
48:34to run them nightly with your regression
48:36tests for example just to check out in
48:38the morning. Okay, do we have some
48:40things to fix here?
48:44And just like test coverage, don't treat
48:46this as a target like a number must go
48:48up. Don't don't do that. Like the moment
48:50that becomes a me a target then that
48:53feeds the whole point. You could see
48:54with exceptions for example that's not a
48:57useful thing that we oh the exception
49:00message should be in this form like okay
49:02that could be useful if that's an
49:03outward facing thing but if it's an
49:05implementation detail we can safely
49:07ignore that.
49:11So in conclusion,
49:14the goal of this whole thing was to
49:16build trust in our code.
49:19Make sure we as developers know if I run
49:21the tests, they will tell me when I mess
49:24up. They will tell me if I did something
49:27wrong.
49:28And we had one problem which was flaky
49:30tests. We solved that with using
49:33deterministic data and also with large
49:35sample sizes.
49:37We had incomplete tests and we try to
49:40solve that by using randomized testing
49:43or property based testing which is also
49:44a form of randomized testing.
49:48You might forget some assertions or
49:51wrong assertions and that's where
49:53snapshot testing helps you and it
49:55catches like if an entire structure has
49:57changed without you knowing.
50:00And to know if you actually written
50:02tests that catch the bugs, we had
50:04introduced mutation testing. So I hope
50:07with these techniques you can write
50:09maybe you can delete some of your tests
50:11and you can you can write effective more
50:14effective tests or maybe something that
50:16was hard to test before now becomes a
50:18little bit easier now that you have an
50:20extra tool in your in your toolkit. So
50:22like I said before don't dismiss the
50:25simple way of testing that you know it's
50:28probably a good way to start but if you
50:30need something else these are tools that
50:33you can use.
50:35So with that, I want to thank you very
50:36much for paying attention at this early
50:39hour and enjoy the rest of the
50:41conference. Thank you very much.
50:43[applause]
50:48[applause]
50:50Are there any questions? [snorts] I'm
50:52told we have a microphone in the room.
50:58I see one there. Yeah.
51:04If you're afraid by the way to ask your
51:06questions in front of this group, you
51:07can find me afterwards, you know. Yes.
51:09>> Did did your perspective change on this
51:11after AI?
51:13>> Very good question. Um
51:16I mostly and I see other people doing
51:19that too. I mostly started writing more
51:22integration tests like bigger scope
51:24tests. Not testing the itty bitty
51:26details anymore, but testing an entire
51:28flow. So this talk covered mostly unit
51:31testing. Um, it's still very useful in
51:35the age of AI, I might say, especially
51:38the property based testing since AI can
51:41change your implementation details and
51:43still your test will check if if you
51:46have the right properties in your code
51:48and AI did a good job there.
51:50I'm still on the fence if you should let
51:53AI write all your tests for you. I think
51:56you should guide it a little more in the
51:57tests and be very peculiar about which
52:00test it write
52:02and then maybe it can have some more fun
52:04with the code like as long as it really
52:06passes the tests that you have checked
52:08it helps. Um
52:11but I think for example snapshot testing
52:13is a very good one with AI right it can
52:16in instead of like generating this huge
52:19test with all these asserts and stuff it
52:20can you can just have a snapshot and you
52:24know if the AI did did a change that the
52:26snapshot is still the same so these can
52:29help with AI they also help before so I
52:33think they stay relevant but I don't
52:35think they are like a big game changer
52:37or something now mutation score
52:41mutation score can be a good metric like
52:43if if you let AI generate your tests at
52:46least run a mutation on them and see are
52:49they effective the test that AI wrote so
52:51that's a good tool now uh I think more
52:53useful than before yeah
52:56thanks for your question one
53:00anymore
53:02in the back
53:06okay so thank you so much for your talk
53:08I think it was like quite a mind opener
53:10>> but I would like to ask you some
53:13questions. One of them is the randomized
53:16tests for example. I think that it's not
53:19quite clear for me now when should we
53:21use it and when should we use like let's
53:24say the classic oldfashioned way in
53:27which you have like the test cases and
53:28not this kind of random tests and also
53:31like one of my questions is as you have
53:34said like um you should not trust the
53:37code but trust the test and create good
53:40tests but in practice sometimes it's not
53:42that easy for example in my case I have
53:45worked for a company that prioritizes is
53:48creating value. So like no test were h
53:52like uh let's say uh created but for
53:58example in another company we created
54:01like a lot of tests because I mean it's
54:03not clear to find like the sweet spot in
54:05in which how many tests should we create
54:07and how to test out if they are good or
54:10not. You have like as you have said the
54:11coverage and these metrics. So these
54:13were like my question like when should
54:15we use this randomized test if it only
54:19applies to the unit test or also to
54:20other kind of test and how to tell when
54:24should we write more tests or when
54:26should we not write that many tests.
54:28>> Okay, thank you for the question. So the
54:31answer to the first thing of doing
54:34example based tests or randomized tests,
54:35it's not that clear. Like it's not it
54:38I'm a consultant. I say it depends a
54:40lot. So I'm I'm going to say that as
54:41well. But I think example based testing
54:45really shines if the test cases are
54:47pretty clear. Like if you have for
54:49example this north, south, east, west,
54:51enum, you know, these are the only
54:53options that this value can have. We
54:54test it with all four. That's pretty
54:57easy. If there's a lot of variation in
55:00your code or in your requirements and
55:04like in the clinical trial example like
55:06a lot of very different scenarios can
55:08happen. This randomization can actually
55:10help you catch stuff that you wouldn't
55:12have with example based tests. Now
55:15granted, you could ask AI, come up with
55:18all test cases, please, and you could
55:20put them all in your code, but what if
55:22you add code? Like what if you add
55:25values? And so it's it's it's a
55:27trade-off because the randomized tests
55:30are less good documentation. Maybe like
55:33you saw in the beginning, tests are also
55:35living documentation. If you have all
55:37these test cases, it's pretty clear if
55:38you read them, oh, this should happen
55:40when this, this should happen when this.
55:42So, it's a trade-off between readability
55:45and maintainability and stuff like that.
55:48But the answer there is not so clear.
55:51The only thing I wanted to convey today
55:52is that randomized testing is not used
55:56that often and you should maybe
55:58sometimes reach for it if the
56:00opportunity arises. Yeah. And sorry,
56:02what was it the second question about
56:04again in short? Yes, my second question
56:06is like we haven't uh talked that you
56:10should create good tests and that you
56:12should trust the test and not the code
56:15but in some cases in practice is not
56:16that easy because for example in some
56:18companies you don't create that many
56:20tests so you should like you have like a
56:23budget that you should like go
56:26everything and create value and the
56:28tests are not that important for them
56:29and in some other companies
56:31>> it's the other way around like you
56:33create many many tests And when you try
56:36to create or launch a master build, it's
56:39like going to hell because it takes too
56:42much time and there are many dump tests.
56:44So it's not easy to tell like how to
56:47tell if the test that you have written
56:49is good or not apart from the well-known
56:51metrics like the coverage and that.
56:54So, we're going back again to the the
56:57question of like should you write lots
56:59of tests or yeah, should you like trim
57:01your tests down and make them really
57:03focused? As long as the tests are
57:06helping you with the trust in your code
57:09and are helping you like catching things
57:11that you might have done wrong, I think
57:14that's more valuable than people give it
57:16credit for. like you can you can have
57:18this fall back of okay I what what would
57:20your life be like if you you knew that
57:22your code worked right that would be
57:24great um yeah companies that that tell
57:28you don't write tests that's a hard one
57:31that's a hard one I I get it like I I'm
57:34a consultant I come in contact with a
57:36lot of those
57:38I would strongly advise you to
57:41massage them into like accepting that
57:43testing is very important and start by
57:47creating these snapshot tests that at
57:49least prevent changes that you did,
57:51unwanted changes. Like it's a very
57:53loweffort way of of writing tests,
57:55especially for legacy applications
57:58because you can just write one test and
57:59it will verify the entire output of
58:01something. Um, yeah, it's not not easy
58:06to code if you don't have tests in my
58:08opinion because
58:10what what are you going to fall back on
58:12like a user reporting a bug? That's
58:14that's way too late in the process,
58:16right? So,
58:17but yeah, knowing if you have written an
58:20effective test is pretty hard even with
58:22all the metrics. So, the most important
58:24thing is does it protect me against
58:28doing something wrong probably. But
58:32that's hard to find out.
58:34Okay, I think we're maybe out of time.
58:37So, thank you again. Enjoy the
58:39conference and thank you for being here.
58:42Thanks.