Master API Contract Testing: The Ultimate Postman Guide

0:00all right so welcome to space camp uh

0:01we're gonna start with some

0:02introductions um this workshop is for

0:05people who identify as testers but maybe

0:07have been around postman for a little

0:09while we're gonna cover some beginner

0:11level testing some intermediate level

0:12testing

0:13beginners should be able to follow along

0:15just fine our topic today is going to be

0:17on contract testing which is sometimes a

0:19little bit misunderstood because we can

0:21do

0:21this kind of testing on both the

0:23consumer and the producer side so to

0:25help with this we're going to cover both

0:26sides of this today in a handful of ways

0:28so we'll start with some introductions

0:30my name is ian douglas i'm a senior

0:32developer advocate here at postman i

0:34joined the team back in january i've

0:35been in the tech industry for a long

0:37time i've done a lot of different kinds

0:39of roles such as development and

0:41management and education i got into

0:43software testing in about 2011 or so and

0:46it changed a lot of my views on how i

0:48write software now so i'm a big advocate

0:50for test driven development and behavior

0:52driven development and i love diving in

0:54on topics like this so i'll turn it over

0:56to tim

0:58hi i'm tim bosha i joined poston back in

1:01november

1:032020 i'm a senior software developer

1:06engineer and test or senior estet

1:09i've been in the software testing and

1:11software support space for almost nine

1:13years

1:14doing mostly automation at the api level

1:17automation at the ui level performance

1:19testing

1:21bug triaging so

1:22near and dear to my heart also doing a

1:24lot of coaching of uh

1:27best practices to other folks on my team

1:32awesome cool thanks tim appreciate your

1:34time today uh helping with this content

1:36today as well so we'd like to hear from

1:38all of you a little bit we're going to

1:39drop a poll in zooms let me go ahead and

1:41launch that

1:43and we'd like you to answer a few

1:44questions about your experience level so

1:46we'd like to know how much experience

1:48you have around the idea of api design

1:51and how much experience you have with

1:52api testing as well as how long you've

1:55been using postman so we'll give

1:57everybody a little bit of time to fill

1:58this out for us

2:00it just helps us shape what we do with

2:03space camp so that we have an idea

2:04around experience levels of people that

2:06join space camp

2:08and it'll help to shape future future

2:11content for us

2:13so we'll give you a few seconds to fill

2:14that out and then we'll go ahead and

2:16we'll end that poll and we'll share the

2:18results with everybody and we'll chat

2:20about that really quick and then we'll

2:21move on to our content

2:24so we'll give everybody another 10 15

2:25seconds or so

2:28most of you actually answered so we'll

2:30get that closed up here pretty soon

2:34appreciate everybody getting their

2:35answers in here

2:42all right we'll give everybody just a

2:44few more moments to fill that out and

2:46then we'll go ahead and we'll close that

2:47poll

2:53right

2:55so we got about 80 participation which

2:57is fantastic appreciate that so i'm

3:00going to go ahead and end this and we'll

3:01share the results

3:06all right so we got some really

3:07interesting results here about half of

3:09our audience today have less than six

3:11months of api design

3:14and pretty split across the six month to

3:17one year one year to three years and

3:18more than three years of experience

3:20around 15 to 18 for each of those

3:24experience with testing pretty similar

3:26kinds of results we've got some people

3:28who are quite new to testing and some

3:29folks that have been around testing for

3:31quite a while about 37 each on those 13

3:34each on the others

3:35and then years of experience with

3:37postman again uh pretty spread out so we

3:40appreciate all the folks that have been

3:41around postman for a long time

3:43and we got some good percentages there

3:45but also a handful of folks that are

3:47fairly new at postman as well so welcome

3:49to all of you to the postman platform

3:56we're going to talk about

3:58contract testing what it is the benefits

4:00of doing so

4:02ian's gonna spend some time running some

4:04validation testing using the api builder

4:07within postman

4:08and how to run

4:10automated tests from the collection

4:12runner

4:14my section will touch on consumer driven

4:17testing

4:18and how to build some

4:20test scripts using a postman request and

4:22then using a postman monitor to automate

4:25this level of regression testing

4:29all right so let's talk about the agenda

4:31here's what we're going to go through so

4:33we're going to start out by making an

4:35empty workspace to do all of our work

4:37today so in postman you will need to be

4:39logged into your postman account for

4:40this to happen

4:42so we're going to start up by building

4:44a workspace and we're going to be

4:46forking some things into that workspace

4:48today and then we're going to talk

4:50briefly about what is testing like what

4:52is specifically contract testing and

4:54then we're going to get into the main

4:56content so as tim mentioned the first

4:57section of the space camp content is

5:00going to be on the producer side of the

5:01api builder with myself i'm going to be

5:04sharing a download link that we can use

5:06and you can either download a file or

5:08you can copy that url into postman we're

5:10going to show you how to do that

5:12we're going to do some open api

5:14specification work and we're going to

5:16build out a test suite we're going to

5:17learn how to use the built-in validator

5:20within postman

5:22and then from there we're going to go

5:23into some automated scripting as tim

5:25described

5:27and then from there i'm going to hand it

5:29over to tim and tim's going to talk

5:31about client-side contract testing using

5:33some postman tests within the sandbox

5:36and we're going to be working with a

5:38library api that we used in a previous

5:40space camp so you can go back and you

5:42can review some of those other space

5:44camp videos if you want more detail

5:47well tim is also going to touch on

5:49postman monitors which ruby and arlemy

5:52did i think two space camps ago or one

5:55space camp ago

5:56where they went into a lot of detail

5:58around postman monitors so if you need

6:00more information on how to do the

6:02testing or how to do monitors we do have

6:04previous space camp videos that you can

6:06go see for those

6:07we're going to wrap up the session by

6:09sharing some resources for further

6:11learning and further work in postman and

6:14we're also going to close out with q a

6:16so as a reminder there's a q a

6:18option in zoom that you can use to drop

6:21your questions in there i think uh i'm

6:23not sure if chat is working yet i think

6:24arlington is working at getting chat uh

6:27working so if you do have questions as

6:28we go throughout the content today

6:30please use the q a panel for that as a

6:32reminder we are recording we'll share

6:34the recording out with everybody once

6:36the session is over you should get an

6:38email from zoom

6:39and uh and let everybody know um you

6:42know how to get those uh how to get

6:44those videos

6:46um and then let's see

6:49yep so i think we'll uh we'll dive into

6:51our our content

6:54so why do we even do testing what is

6:55testing so for those who are new to the

6:58idea of testing the the reason that we

7:00test the software that we build and the

7:02things that we build is to give us that

7:04confidence

7:05today's session we're going to focus on

7:06three different perspectives of what we

7:08call api contract testing we're going to

7:11start by importing an open api

7:13specification that shows how your team

7:15might work through the design stage if

7:18if you've heard the phrase api first

7:21we're going to show a little bit about

7:22how you might do this api first design

7:24inside of postman and make sure that

7:26everything is lining up between

7:28the definition of what you're trying to

7:30build and what you want your users

7:32to have as an experience for

7:36getting ready to go develop the api

7:38from there we're going to look at some

7:39automated testing to make sure that the

7:41api definition itself actually conforms

7:44to the open api specification standards

7:47and we've got some automated ways of

7:48doing that and finally we're going to

7:50take a look at the consumer side as we

7:52mentioned to make sure that the end user

7:54is actually getting what we expect and

7:56so by looking at it from three different

7:58perspectives three different angles

8:01we're going to have the utmost

8:02confidence that what we talk about as an

8:04api contract

8:05is actually

8:07what we want our user experience to be

8:10so what is contract testing itself well

8:13we've talked about the benefit of it

8:15when it comes to that confidence

8:17when it comes to the idea of a contract

8:19is is basically the agreement between

8:22us as the api producers as well as the

8:24api consumer

8:26to communicate how to use

8:29an api and sort of the guidelines around

8:31that and this is done by having a

8:33well-documented set of endpoints and

8:35examples of both success and failure

8:37scenarios

8:38so that your users know exactly what is

8:40going to happen if they call your api in

8:42a particular way it sets up a very clear

8:45path about how to communicate with your

8:47api and to offer guidance about how to

8:50debug things if things don't work out

8:52so the confidence that we talked about

8:54in the middle of that other slide also

8:56lends itself to the user's confidence so

8:59they come to have an expectation when

9:01they use any of your apis

9:03that they're going to work in a

9:04particular way this kind of comes back

9:06to the idea of your governance team

9:08who is designing what that user

9:10experience is going to be

9:12and so having

9:14their expectations set is also going to

9:16be really helpful as you work across all

9:18of uh as your users work across all of

9:21your apis

9:22so the image that you see here is a

9:24pretty pretty big oversimplification of

9:26what we're presenting today but we

9:28wanted to give you an idea of this

9:29before we jump into the hands-on demos

9:31from here

9:35so when it comes to contract validation

9:37for api producers um we're going to show

9:40you uh like i mentioned we're going to

9:41create a workspace we're going to walk

9:42everybody through that

9:44and then we're going to share a url to

9:46go get an open api specification file

9:48now if you want to use your own open api

9:50specification that would be fine but

9:52we're going to have an example that

9:53everybody can pull in and use

9:55if you don't have an open api

9:57specification

9:59and then we're going to go in we're

10:00going to run the validators we're going

10:01to do the automated scripting as we've

10:03mentioned

10:04and then we'll hand it over to tim from

10:06there

10:08all right so we're going to get our

10:09moderator team to drop this first link

10:11in chat and you can do one of two things

10:14we can

10:17you can either copy this url

10:19to your clipboard and we're going to

10:20paste that url or you can download the

10:22file and you can import that file and

10:24i'll show you both places to do that so

10:27we'll give everybody a moment to uh to

10:29get this link pooja just dropped that in

10:31chat for everybody and it looks like uh

10:33chat has to be like we would have to

10:35restart the whole session to get the

10:36regular chat working so apologies for

10:38that please continue to use the q a

10:40panel for questions

10:42um and and the moderator team will do

10:44their best to answer those as we go

10:47so we'll give everybody a moment to go

10:49grab that url and then we're going to

10:50switch over and do a demo in within the

10:53postman application itself now i'm going

10:55to be using the desktop application for

10:57postman you can use the web application

10:59or the desktop application both of them

11:01should work fine for what we're going to

11:02be working through today

11:04so again you can open this url directly

11:06if you want to kind of view what that

11:08open api specification file looks like

11:10in yaml format or you can just copy that

11:12url and have the url ready to go you can

11:15use the short url or you can use the

11:17resolved url which is a file on github

11:19either one of these are going to work

11:20fine in postman

11:24all right so let's go over to postman

11:25we're going to start by making a new

11:27workspace

11:28so in the menu bar of the application at

11:31the very top we're going to click on

11:32workspaces and we're going to click on

11:34the button that says create workspace

11:37we need to give this workspace a name so

11:39i'm just going to call this spacecamp

11:42api

11:44contracts

11:46you can choose to put a summary in here

11:47if you like you can also choose whether

11:49you want personal access private access

11:51team access i'm just going to leave most

11:53of these things set as default values

11:55today

11:56i'm going to go ahead and click on the

11:57button that says create workspace

12:03and

12:04then the interface redirects us back to

12:06our brand new workspace where we can go

12:08in and start defining things from here

12:12the first thing i'm going to do

12:13in our context bar over here

12:16is i'm going to click on apis

12:18we see we don't have any apis yet and so

12:20we're going to go ahead and we're going

12:21to import this

12:22api so i'm going to click on the import

12:25button

12:28and from that import button we have a

12:29lot of options here so if you downloaded

12:31that file if you saved it on your system

12:34you can click on the upload files button

12:36and you can manually upload that file

12:38we can also click on link

12:41here as well and we can actually paste

12:42in that link

12:45so either those two options will work so

12:47if you've already opened up the file or

12:48if you've saved a copy on your system

12:50you can upload that file manually or if

12:52you want to use your own yaml file you

12:54can upload that directly into here or if

12:56you have a link to an open api

12:58specification file you can use that so

13:00you can use the short url that we've

13:02pasted in chat or if you've if you've

13:05actually opened that in your browser

13:06you'll see that it expands out to a

13:08github link you can paste in that whole

13:10github link here as well

13:12so i'm going to go ahead and click on

13:13continue

13:14and we see that it's going to open as uh

13:17as an open api version 3 specification

13:20the one change that we need to do here

13:21is by default it's going to try to open

13:23this as documentation we want to set

13:25this drop down to say test suite this

13:28will save us a little bit of time if

13:29you've already imported it as

13:30documentation i'll show you quickly how

13:32to go build a test suite out of this

13:36so we're going to choose test suite from

13:38that drop down we're going to leave

13:39everything else as defaults and we're

13:41going to click on the import button

13:44this should give us a confirmation that

13:46everything imported okay

13:48i'm going to go ahead and click on the

13:49button that says confirm and close

13:53now we can see that i've got my library

13:55api set up in here

13:58now if you set this up as documentation

14:00what you can do is you can click on the

14:02draft

14:04word here

14:05and what this draft is is just a it's a

14:08version it's just a string of a version

14:11if you import it as documentation what

14:14you can do is you can click on the test

14:15tab

14:16here on the workbench

14:18and you can click on the button over

14:19here that says add test suite and you

14:22can give it a name and it should build a

14:24new collection

14:25of all of the examples of the endpoints

14:28and their example outputs

14:31as a new collection using that test

14:32suite if you imported it as a test suite

14:35then you don't need to do this step

14:39so let's go take a look at what actually

14:40imported from here

14:42we can see we've got two endpoints one

14:44endpoint to fetch a book from our

14:45library and an endpoint to get all of

14:48the books from our library and then we

14:49have examples for each of these so we

14:52have an example of fetching a single

14:53book

14:54we have also have an example of fetching

14:57multiple books

14:58in this case because of the example data

15:00that we had in our open api

15:01specification it's actually putting the

15:03exact same data in here multiple times

15:05but what we want to pay attention to

15:08here the square brackets which indicate

15:09we have an array of results coming back

15:12here

15:16so if we click on draft

15:19and we come back to the definition tab

15:23we can actually see the open api

15:24specification over here

15:27now you can go and you can make changes

15:29directly in the interface here if you

15:31like we also have other videos that you

15:33can find on our youtube channel

15:36if you want to learn how to connect the

15:37repository to

15:39integration systems like github or git

15:42lab if you want to be able to

15:43synchronize this back and forth with a

15:45repo

15:46we've got some how-to videos on how to

15:48do these

15:52for most teams you're going to spend

15:53some time in here defining out your api

15:56and you're going to be adding new

15:58endpoints and and your http methods and

16:01so on what your responses are going to

16:03look like and you might define some of

16:05your components and so on down here for

16:07some folks though it's a little bit

16:08easier to work backwards and say

16:11you know if you're not as familiar with

16:13the open api specification for example

16:15you might actually want to go look at

16:16the example output and say okay i can

16:19see this json format here

16:22and maybe there's some changes that we

16:24want to make here as an example we know

16:26that a book can have multiple authors so

16:29maybe we want to come in here we want to

16:31change

16:32what we have here

16:33for a single author and we want to make

16:35this an array of strings

16:37so i've changed the name from author to

16:40authors plural and then what i'm doing

16:42is i'm highlighting this in postman and

16:45then i can hit just the open square

16:46bracket key on my keyboard and it'll

16:48automatically add the closing square

16:50bracket here at the end and this would

16:51indicate an array of strings

16:54i'm going to go ahead and i'm going to

16:55make that same change over here for

16:57fetching all of our books i'm just going

16:58to go ahead and change both of these

16:59examples to also be arrays

17:02because maybe this is how our team wants

17:05to work on this planning a little bit so

17:07what we're going to do from here is i'm

17:08going to show you how to run the

17:10validator inside of postman

17:13to see

17:14um

17:15like how postman is going to detect that

17:17these things are now out of sync between

17:20what we want our example to be and what

17:22we've defined our api to be

17:26so you have to make sure that when you

17:27change these examples that you either

17:29click on the save button in the

17:31interface or you can use a hotkey on

17:34your keyboard if you're on a mac it

17:35would be command s if you're on a

17:37windows machine it'll be control s

17:39and

17:40we're going to go ahead and we saved

17:41both of those changes to our example

17:44what we can do from here is i'm going to

17:46click back on draft at the top

17:48and i'm going to go over to that test

17:50tab over here again

17:51so again i just clicked on the draft

17:53version of our api

17:55and from here i'm going to click on test

17:59and we see a button here that says

18:01validate and what we're going to do is

18:02we're going to click on this validate

18:04button

18:06what this is going to do is it's

18:07actually going to check the examples

18:08that we have against the definition that

18:10we've made and it's going to

18:13make sure that these things are lined up

18:15in this case it's taking a moment to run

18:18normally happens quite quickly so i

18:19might refresh my screen here and see if

18:22i can get this to run a little bit

18:23faster

18:24so let's go back and try and run that

18:26again

18:32looks like the validator is having some

18:33issues usually this runs really really

18:35quickly

18:41give this a moment to see what's going

18:42on here

18:45yay for live demos

18:50what the interface would be showing us

18:52at this point is that

18:53it found some issues and if we click on

18:55that link we would be able to go see

18:57what those issues actually are

18:59um suggesting turn it off and on again

19:04so let's try that i'll just uh i'll

19:05close out postman

19:07i'll start a postman again excuse me

19:13all right so i'm going to expand this

19:15out i'm going to go to back draft

19:17version here again

19:18i'm going to go over to test mode

19:21and i'm going to click on this validate

19:22button one more time

19:25of course it's having a problem when i'm

19:27trying to live demo this

19:29um

19:31so what this validator is doing is it's

19:33actually comparing the definition of our

19:35open api specification

19:37with the examples that we've built

19:40and because we've changed the example it

19:42would actually be coming up with a

19:44notification saying that there's a

19:45problem

19:46in that test

19:48in this test tab

19:50if it found an error here you would see

19:52if you can discern the color yellow

19:55it would say that it found issues and

19:57when you click on that it would actually

19:59open a panel showing you what it found

20:02and what it was expecting to find

20:04and what it it it'll also try to make

20:06suggestions on how to correct this and

20:09what we would be seeing on that screen

20:11is

20:12postman trying to resolve the issues

20:15between the definition and what we've

20:16changed our example to be

20:18and in that example

20:20it would say you know you've added

20:22authors but the definition says that

20:24we're supposed to have just an author as

20:26well

20:28and so what i would normally be seeing

20:29on that screen is that it's trying to

20:31add another author here at the bottom

20:34uh in in single format where it's adding

20:37another string of a single author but we

20:40don't want postman to correct that we

20:42want it to be authors we want this to be

20:44an array and so what we need to do is we

20:46need to go in now and correct the

20:48definition so again this is one way that

20:50your team might be working if they don't

20:52understand the open api specification

20:54they may come in and work on these

20:56examples directly hands-on and try to

20:59manipulate these examples

21:01so what we need to do here is we need to

21:02go in and look at the definition and see

21:04what we need to change

21:06in the definition

21:09so i'm going to scroll all the way down

21:10to the bottom where we actually have the

21:12schema for our book

21:14where we define the id and the title and

21:16the author and so on and we want to make

21:17sure that this is authors plural

21:21now up here at the top we also have

21:24a notification here that says that the

21:26author is actually a required field we

21:28have to make sure that we pluralize this

21:30one as well

21:31so anywhere that we have the word author

21:34in the specification we now have to make

21:35sure that this becomes plural but it's

21:37not just a single string anymore and so

21:39what i'm going to do is i'm going to add

21:41just a couple of lines here that tell it

21:43that this is going to be an array

21:46so i'm going to say that the type of

21:47this is going to be an array

21:52and then for each of the items that we

21:54have in here i'm going to indent the

21:55rest of this and so

21:57what i changed in here was i went from

22:00the word author to authors plural

22:03and then i indented the previous

22:05definition of the type of string and i

22:07added these new components here that

22:09said authors is now going to be a type

22:11of array

22:12and each of the items in that array is

22:15going to be a type of string

22:16and then the open api specification

22:19allows us to add a description as well

22:20as example data

22:24so now if i were to go run the validator

22:26again

22:27um it should

22:28match up now to recognize that we have

22:31an array of strings as our authors so

22:33let's go try that one more time let's

22:34see if we can get that validator to work

22:36if we click uh so whether you have

22:38errors or whether it's succeeded for you

22:40you can always click back on that

22:41validate result and you'll see a little

22:43pop-up here that will let you validate

22:45again or if you had errors you would see

22:48another button here that says that there

22:49were errors and you can go in and see

22:51what those are so we'll try the

22:52validator one more time

22:54looks like it's failing for me again

22:59yep that's not gonna work for us

23:02so unfortunately uh we ran into a

23:04problem with this

23:07so yeah so this validation is basically

23:09comparing our open api specification

23:11with what we want our examples to be and

23:14so it's watching the json format of our

23:17example output of what we want our end

23:20user to see along with the definition of

23:22what we're trying to build so i'm going

23:23to go ahead and i'm going to save this

23:25change to the definition

23:28and what i'm going to do just for the

23:29sake of time is i'm going to move on to

23:31the next step

23:32not sure why the postman validator is

23:34not running here but we'll go examine

23:36that but normally this validator runs

23:38very very quickly on the specification

23:41to compare against those examples so

23:43we're just going to go ahead and move on

23:44from here what we're going to do from

23:46here is we're going to go look at

23:47another workspace and we're going to

23:49bring some of the work from that

23:50workspace into our workspace

23:53at the very top

23:55of the interface we're going to see a

23:56search bar in the in the menu bar at the

23:58very top we're going to see the search

23:59bar and what we want to search for in

24:01here

24:04is we want to look for contract test

24:07generator

24:09so those are the three words that you're

24:11going to want to search for

24:13go ahead and drop that in chat here as

24:15well

24:16the team is actually going to drop a

24:17link to it as well if you just want to

24:19open that directly you want to make sure

24:21that you're opening the one that was

24:22made by postman

24:24so i'm going to go ahead and open this

24:26up

24:27and i'm going to click on the

24:28collections icon in the context bar what

24:32we can see here is we've got two

24:33collections we have one for open api

24:35specification version two or if you're

24:37familiar with the word swagger swagger

24:40was basically uh adopted and changed

24:42into open api spec

24:44and so open api

24:46version 2 is effectively swagger and

24:48then open api

24:493

24:50is the newer version

24:52and so we have these automated test

24:55builders

24:56that we're actually going to copy into

24:59into our workspace now because we're

25:01working with open api spec version 3 i'm

25:04only going to copy

25:05these contract tests i'm not going to

25:06copy version 2. we don't need those

25:08we're working with version 3 so i'm just

25:10going to copy the version 3. the other

25:12thing that we need to copy from here is

25:14an environment and the environment is

25:16basically setting up variables that will

25:18allow

25:20these tests to

25:21build out some of the automation here

25:24so what we're going to do from here is

25:25something that we call forking and

25:26forking is basically making a copy but

25:28remembering a reference back to where we

25:30got it from

25:31if you're familiar with using forking in

25:34github that's basically what's happening

25:35is for now just think about it like

25:37we're just going to make a copy for what

25:39we need

25:40so i'm going to hover over the open api

25:42spec 3 version of our contract tests and

25:45i'm going to click on the three dots

25:47next to it and i'm going to scroll down

25:48to the option that says create a fork

25:53now we need to give this a label and i'm

25:55just going to call it my name and we

25:57need to tell which workspace this is

25:59going to be going to go into so i'm

26:01going to choose my spacecamp api

26:03contract workspace that we just made now

26:05when i click on the fork collection

26:07button it's going to fork that

26:08collection over and put us back in our

26:10workspace but we need to come back here

26:13to the contract test generator to grab

26:15another piece of data later

26:17so for now i'm going to click on for

26:18collection

26:20we're going to see that it's going to

26:21redirect us back to our workspace but we

26:24want to go back to the contract test

26:26generator workspace to grab the

26:28environment from here

26:30all right something went wrong building

26:32a fork

26:34having all kinds of fun with live demos

26:35today

26:39all right so that fork got created for

26:41us and we can see now that that's over

26:43here in my collection

26:45but we have to go back to that contract

26:47test generator because we want to grab

26:49the environment as well so i'm just

26:51going to click on that search bar again

26:53where the contract test generator words

26:55were already there i'm going to go ahead

26:56and reselect the contract test generator

26:59workspace

27:00now in the context bar i'm going to

27:02click on the environments icon

27:04and we see an environment here called

27:06contract test environment we can also

27:08fork this environment over as well

27:11so again i'm going to click on the name

27:13i'm going to click on the three dots

27:14next to it

27:15and i'm going to select create a fork

27:18from the submenu that comes up

27:22again we have to give this a name i'm

27:23just going to call it ian again and

27:25again we have to select the workspace

27:27now there are a few things that we're

27:28going to need to go

27:30build into this and i'm going to talk

27:32about that really briefly we're going to

27:34need a postman api key and we also need

27:37to know the id value of the workspace

27:39that we're currently working in

27:41so i'm going to go ahead and fork this

27:42environment over and then i'm going to

27:44talk briefly about how to go get that

27:46postman api key

27:50so when we look at the environment so

27:52again it brought us back to our own

27:54workspace and there are a few values

27:55that we need to set here the first one

27:57is going to be this env api key and this

28:00is where we want to put a postman api

28:02key so for now i'm just going to write

28:05postman api key just as a placeholder

28:07that we know that this is something that

28:09we need to fill in we also need to fill

28:11in this workspace id and i'm going to

28:13show you where to get go get that and

28:15then there's one more change that we

28:16need here and that's to set our server

28:18name here this has to match the server

28:21that we have in our open api

28:23specification

28:24i'm going to set this to localhost

28:27on port 3000 for the time being

28:31i'm just going to drop that in chat just

28:33so you can copy and paste that if you

28:34like

28:35and then i'm going to show you where to

28:37go get your postman api key

28:41fastest way for me to do this is going

28:42to be in a browser so i'm just going to

28:44go back over to my slide browser here

28:45and i'm going to go to postman.com

28:48i'm going to show you where to go get

28:50your own postman api key of course i'm

28:53going to try not to expose my own here

28:55but what you want to do is you want to

28:56click on your profile icon at the top

28:59and you're going to go into your

29:01settings

29:02so you're going to click on your profile

29:05and you're going to click on settings

29:09and from here you'll see a menu option

29:11on the left called api keys

29:14and on that screen you should have the

29:16ability to generate a new api key so i'm

29:19going to do that off screen just so i

29:20don't expose my own api key and i'm

29:23going to go

29:27build that really quickly so again i'm

29:28clicking on my

29:30profile icon i'm going to settings

29:33i'm going to click on the api keys

29:39and from here you should see an option

29:40to create a new api key

29:44so i'm gonna go grab that

29:46now

29:47one change that we need to make in

29:49postman

29:50is how we're actually saving these

29:54we have two different variable types in

29:55here one is called default which will

29:58show us visually what that string is and

30:00there's also a secret option and when i

30:03select the secret option notice how it

30:06masked what i pasted in here if i

30:08highlight this and try to type anything

30:10else it also stays masked

30:12and so we do

30:14we do hope that you'll use the secret

30:16option here when you paste in your api

30:18key

30:19do not add it in initial value anything

30:22that you put inside initial value will

30:24synchronize to the postman server and if

30:26you're working on a public facing

30:30workspace then anybody else can come in

30:32and click on that little eyeball icon

30:34and expose what your api key is

30:37so we would ask that you set this to

30:38secret only paste your api key inside

30:41current value

30:43if you want to unmask it to verify that

30:44it was done correctly you can

30:48from here we're going to need our

30:49workspace id

30:52so to get our workspace id

30:55just underneath the menu bar

30:58where we have in my case space camp api

31:01contracts i'm going to go ahead and

31:02click on that name

31:05and we see some definition about our

31:07workspace that we're in

31:10and on the side here we see an

31:12information icon called workspace

31:14details

31:15i'm going to click on that icon

31:17and this is where we can see our

31:19workspace id

31:22next to that we have a button that will

31:24allow us to copy that workspace id to

31:26our clipboard

31:28so i'm just going to copy that value to

31:29my clipboard and i'm going to go back to

31:31my environment

31:32i'm going to paste that inside my

31:35workspace id now in this case it's okay

31:37that i'm showing you my workspace id you

31:39can't do anything with it

31:41other than access it but it's public

31:42anyway

31:44but you won't be able to access anything

31:45in here because i've made this secret

31:47and these current values don't

31:48synchronize with uh with the postman

31:51server anyway

31:53so the three changes that we made is we

31:54fetched an api key

31:57we put in a workspace id and we changed

32:00our server to be localhost 3000.

32:06if i'm working a little quickly i

32:07apologize part of it's for the sake of

32:09time because we lost a little bit of

32:10time

32:11trying to run those tests over and over

32:14you can always come back and watch the

32:16the video recording and

32:19you can kind of pause things to work at

32:20it at your own pace

32:22so i'm going to go ahead and click on

32:23the save button here to make sure that i

32:25save all those changes

32:28and then we're going to go back over to

32:29the collections icon in our contacts bar

32:33and what we're going to do here is i'm

32:34going to select the

32:36open api specification that we forked

32:39over

32:41and what we want to do now is we want to

32:43run all of the test code that's built in

32:45here so i'm going to start by closing

32:47this documentation panel

32:49and at the very top you should see an

32:51icon that says run

32:53what this is going to do is it's going

32:54to go run some javascript code that's

32:56built into all of these collections

33:01and this is what we call our collection

33:02runner

33:03i'm going to click on the button here

33:05that says i want to run all of these

33:06contract tests

33:09and what this is doing is it's actually

33:11well supposed to be scanning through

33:12everything that we need to uh

33:17oh why is that complaining i think that

33:19we need the active environment oh that's

33:22right quick yes thank you tim appreciate

33:24that yep so we have to make sure before

33:27we run it we actually pick the

33:28environment here at the top

33:30and then we're going to say run again

33:32otherwise it can't it doesn't know where

33:34to find my uh

33:36doesn't know where to find my postman

33:38api key all right i'm just going to

33:39close that generator and we'll start

33:40this again so i'm going to click on

33:42contract tests click on that run button

33:45tell it to run those tests

33:47and now we can see it's actually

33:48automating going through and it's

33:49scanning everything that we need and so

33:52it's building these tests on the fly by

33:54going and looking at all of the

33:57specifications inside our open api spec

33:59and it's actually automatically

34:02building out tests to go test all of

34:04those properties all of their values

34:07and making sure that everything conforms

34:09to the open api specification

34:12now up at the top here we can see that

34:14several of these actually failed

34:16and so that lets me know now that my

34:18open api specification actually doesn't

34:20match what the open api specification

34:24team actually says it should conform to

34:27so if we click on failed we see for

34:29example i made a schema called book with

34:32a lowercase b and in the specification

34:35it says no all of those schemas should

34:37start with a capital letter and so it's

34:38letting us know in in very fine detail

34:42how to go correct our open api

34:44specification

34:45so now we can go back and we can

34:47actually make some changes here it's

34:48also suggesting that our error schema

34:51we have that message

34:53option in there

34:55so in our error i think i'm sending a

34:57status code and an error message and it

34:59says that error message should have a

35:01description and it should also have an

35:03example and so it's giving us very clear

35:05instructions on what we need to go add

35:07to our open api specification

35:09now is this stuff strictly required

35:12maybe not but if your team uses other

35:15tools with your open api spec

35:18to go generate code or generate other

35:20documentation or to generate other

35:22things about what you're producing as an

35:24api making sure that this conforms to

35:27the open api specification could be

35:29really important

35:30and so these contract test generators

35:33and i think i'll stop here and hand it

35:34over to tim so we don't run out of time

35:36today

35:37this will actually go through and it'll

35:38automate a lot of things it depending on

35:40the size of your open api specification

35:43this collection alone could generate

35:45hundreds of automated tests

35:48now it's just going to run as is it

35:50doesn't save those tests anywhere you

35:53would have to come in here and tell it

35:54to run this again and again so as you

35:56make changes it'll actually go through

35:58and by by adding your postman api key

36:00it's actually going in and it has access

36:03now to go read your open api spec

36:05and then based on what it finds in that

36:07open api spec file it's generating all

36:10these tests for you so very handy

36:11utility um do recommend that you

36:15that you check out that contract test

36:17generator

36:18all right from here i'm going to hand it

36:19over to tim just so we don't run out of

36:21time today

36:22so tim why don't you walk us through the

36:24consumer side

36:26yeah absolutely

36:28to reiterate on the consumer side

36:31we're just going to make sure that

36:34the user now has the access to this api

36:37at a consumer level and

36:39they're seeing what's expected from the

36:41server side it's seeing that it's

36:43documented appropriately and again as

36:45ian mentioned it gives us another level

36:47of insurance or another level

36:49of confidence

36:51so i'm gonna

36:52go to a previous base camp

36:56i'm going to go to the search bar at the

36:57top

36:58i'm going to type in post pen space camp

37:04again we're going to make sure that it

37:06is the one that's created by postman

37:12and in the previous an introduction for

37:15testers

37:17collection

37:18you can catch the recording to see where

37:20we set the stage for contract testing

37:22we'll click on the contract testing

37:24folder

37:25and the json schema v4 validation

37:29so here we're using

37:31a

37:32client or consumer's side they have

37:34access to this library api

37:37and a books endpoint which is returning

37:40an array of books

37:42so we're going to follow a similar

37:44structure

37:45and

37:46define a schema and its type and its

37:48properties

37:52and we're going to be using

37:54a built-in schema validation method but

37:57just to briefly

37:58outline there's

38:00some json schema validators built right

38:02into postman ajv for another jason's

38:05game of validator as well as tv4

38:08tiny validator for jason schema

38:11we'd recommend using ajv and that's the

38:14built-in method that we'll be using tv4

38:17is a bit out of date

38:19and its validations are a little bit

38:21more cryptic

38:23to indicate what's what's going wrong

38:27so

38:28the collection that we're going to be

38:29using right below the postman api

38:32contract testing

38:34collection

38:37i'll close the documentation rate

38:40context bar

38:42so this will have

38:44two requests one returning one book

38:48we've included

38:51a hard-coded

38:53book id and as a path variable

38:56as well as we'll then test against

39:00a whole array of books

39:04so what i like to do first where it is

39:06the read-only collection is

39:09fork this collection so i'll

39:11select the collection

39:13hover over the three dot few more

39:16actions

39:19and i'll select create a fork

39:25again i'll give it a simple label uh

39:28damn contract testing

39:32and i'll select the workspace drop-down

39:34and we can use that same

39:36workspace that we created for ian

39:38section

39:40then i'll select fork collection

39:53and

39:54we'll close that

39:56i'll just just hit the we'll just hit

39:58that again we'll try it a second yeah it

40:00seemed to work we're we're doing it

40:03earlier on

40:09all right we'll just try it again here

40:11i'm just going to call it tim in this

40:13case just to speed it up here a little

40:14bit

40:15sure

40:29there we go

40:31great

40:32so now we're

40:34in the team workspace we can see that

40:36the fork has been added let's

40:39expand the collection and select the

40:43book request

40:45so again this is fetching one book i

40:48could send the request and see what it's

40:50returning

40:51that same creativity inc book title

40:55and i'm going to select the

40:57test section

40:59and i mentioned we're going to use the

41:02built-in schema validator postman

41:05so we have the pm response to have json

41:07schema

41:09i'll select the test result and this

41:11will give it the good

41:13way to start our

41:17client-side testing

41:18consumer side testing so right away we

41:21can see that the schema hasn't been

41:23defined

41:24so

41:25as we did in the previous example

41:27let's go ahead and

41:30create a variable for this schema

41:37and i'll simply use two curly braces

41:40save my request by selecting fave

41:43and send

41:45so the schema validator isn't too picky

41:48if i simply just use an empty

41:51schema variable but we can follow a

41:54similar structure to what we did

41:56on

41:57the

41:58server side validation to use the type

42:00use properties so i'll start by

42:04testing against its type

42:06so we can see this is an object

42:10that we're

42:11housing one book's

42:14properties so i'll use

42:17type

42:19and

42:21of

42:22type object

42:25i can send the request

42:27see that it passed but very important

42:30thing to always validate against is that

42:32our tests do in fact fail and they fail

42:34meaningfully

42:36so let's make sure that this will in

42:39fact fail we could choose another

42:42type such as a number or a string

42:49so here

42:50the validator is very helpful to

42:52indicate that it's expecting a string

42:56but

42:57what's being shown is an object

43:01we can see that the validator is doing

43:03its job

43:05change it back to object

43:08next we could test

43:10various properties of this object so

43:13we'll add a comma and a new line and use

43:16the

43:17properties field

43:20colon

43:21curly braces

43:22and we could choose to validate against

43:24any one of these properties

43:26let's use the title

43:33and we can validate in a similar fashion

43:36to what we did for type object we can

43:38validate the titles type

43:45again

43:46it is of type string

43:49i'll send the request

43:52evidence path again we could make sure

43:55it fails let's change type to a type

43:58number

44:01here again the

44:04property

44:05should be

44:06expecting a number what we're seeing is

44:09a string

44:10we'll change it back once again

44:17we can validate to make sure that

44:20certain fields

44:21are in fact included

44:24using the required

44:27value

44:28i could use

44:30required

44:31here we use square brackets

44:34to denote an array of all the different

44:36properties that we expect to see

44:38so let's validate that the title is in

44:41fact required

44:43send the requests

44:44see that the validations path

44:47trade with another

44:48point as always in tefting or in any

44:51form of development to

44:54be careful with your spelling and your

44:56casing if i was

44:57for instance to look for

45:00titles as in

45:03change in a similar fashion with author

45:05and authors we would need to be careful

45:06to go and change this level of

45:08validation as well

45:10expecting something with titles

45:13but what we have is title

45:16here casing also i believe matters if we

45:19were to change it to an uppercase title

45:23this again is going to fail the

45:24validation

45:26i'll change it back to a lowercase title

45:32the

45:34last section that i want to touch upon

45:36for this request is

45:39something that gives us some insurance

45:42if

45:43new fields were ever to be added so we

45:46want to make sure that we're alerted if

45:49for instance they were to add a new

45:51property maybe the price of the book

45:54or

45:55other things that this book api might

45:56want to include

45:58so we want to make sure that our test

46:00would fail if there's any properties

46:03being added along the way

46:05for this we'll use

46:07the value of additional properties

46:11this is a boolean or a true false flag

46:14so if i set it to false

46:17we're saying that i don't expect any

46:20other properties other than title so

46:22this should fail

46:26now as we can see every one of the other

46:29properties other than title

46:31has its own error message

46:34so

46:35powerful way to give us some insurance

46:38to

46:39make sure that no new properties are

46:41being added

46:43i'll go ahead and comment this out

46:46i'll save the request just to be sure

46:50so

46:51next we'll move on and

46:54touch on a future attitude to put

46:56a whole array of books so our second

46:59endpoint if we send the request we're

47:01seeing

47:02[Music]

47:04a array of book objects are being

47:06returned so every single book in the

47:09library

47:12hey tim let me just close that

47:14notification here there we go oh yeah

47:16thanks

47:17uh yep same structure we're gonna define

47:20the schema validate against it

47:22just save a bit of time

47:26where i'm limited with zoom chair if

47:28even if you could copy paste lines two

47:31to nine please

47:39okay thanks

47:42uh

47:43okay so this will give us a good

47:45starting point to start testing against

47:48the schema

47:49uh we should

47:51expect some changes that we're going to

47:53need to make right out of the gate it's

47:54no longer a type object but of type

47:56array

47:58so let's send the request and see what

48:00it does

48:05again we have it of type object

48:08and

48:09what we're actually seeing is an array

48:14first thing that we'll change is of type

48:16array

48:18it's found that request and here we can

48:21see some limitations of

48:24uh validators and almost all positives

48:26that can happen

48:28so

48:30what we have is of properties title and

48:32a required title but we haven't actually

48:35drilled into this individual

48:37object or the items within the array

48:40so always need to do

48:42our due diligence and

48:45make sure that our tests are passing or

48:47failing for what they're supposed to do

48:49so what we're going to use is

48:52the field of items

48:55to help us

48:56unravel each book property

49:00for each book's object

49:02so i'm going to go ahead

49:04and

49:06use the items block

49:10i'll open the brackets

49:13i'm going

49:14to add my closing bracket

49:18where appropriate

49:24and here we would have

49:28we could check on its type

49:30so now we have each of the items is a

49:34book object

49:36going to

49:38double these along

49:40and indent them

49:42be sure to add a comma

49:43[Music]

49:45i'll send the request

49:49you can see that the validation

49:51is passing passing so we've nested

49:55uh each of the items and we're using the

49:56same structure as that book endpoint

50:01we can do some validations against the

50:04array so just two

50:05[Music]

50:06quick

50:08validations that we'll do one is called

50:10the min items or the minimum items that

50:13we expect to see

50:14that give the some insurance that the

50:17response is

50:20in fact returning a minimum of one item

50:23for instance

50:27should pass

50:28we could also make sure it failed if

50:30we're expecting let's say 500 books at

50:32minimum

50:34this should fail

50:37change it

50:38back and we can also stand for the

50:42maximum number of items that we expect

50:44to see

50:46so

50:47we could use that 500 property

50:51at maximum

50:54we have less than 500 books so it passes

50:58you want to make sure it fails

51:00we know that it's returning various

51:01books and not just one

51:05now we see that it fails again

51:08now one thing that we could mention here

51:10is you can actually set these values

51:12programmatically as well so you don't

51:13have to hard code these values so if you

51:15were getting uh say a parameter for the

51:18maximum number of items if if your api

51:20allowed for pagination you can always

51:22grab those values from your parameters

51:25as well so if the user calls it with a

51:27limit of 50 then you you can

51:30you can programmatically say the maximum

51:32items is going to be set to 50

51:34as an example so you don't have to come

51:35in and like change these tests every

51:37time you can do that kind of work a

51:39little bit a little bit more

51:40programmatically that way

51:43yeah definitely

51:45a good level of insurance to make sure

51:47that we're not

51:48overloading any server or any

51:51client-side

51:54aspects if we have way too much data or

51:56we have no data at all a great idea to

51:59use variables or use programmatic ways

52:04so we have

52:06built up these two endpoints and it

52:09would be kind of a headache that they go

52:11in here and kick them off each and every

52:14time that we wanted to do regression

52:16that thing so

52:19what we could use to have it

52:22run on a cadence is a postman monitor

52:25to have these endpoints execute

52:28a time interval that we see fit

52:30so we could use the left contacts bar of

52:34monitors and create a monitor that way

52:36we can also hover over the collection

52:39click view more actions

52:42and

52:42select monitor collection

52:47so here we can give it a name it'll

52:50default to the collection name which is

52:52fine

52:53we'll set the collection tag

52:55as its defaults we don't need to use an

52:58environment

53:01and we can

53:03check how often this monitor runs so we

53:05can

53:06definitely run it at very fine intervals

53:08but as always

53:10yeah make sure that you're using caution

53:13to your

53:14uh usage limits within your account

53:18so we'll use the default of on every

53:21hour but just to show we we could have

53:22it run

53:23uh

53:24as

53:25frequent as every five minutes

53:30we can automatically set a

53:33region

53:34uh we'll set it to us east but

53:36showcasing that we could see how these

53:39endpoints behave across the globe

53:44and we'll set it to alert ian

53:48for any failures that happen

53:49and leave the rest of the properties set

53:51to their defaults

53:55so let's go ahead and run that really

53:56quick tim just for the sake of time

53:57we're we're running low on time

54:00questions that we'd like to get to so we

54:01can see here that this ran if you want

54:03to dive in deeper on monitors um we can

54:06we can drop a link to the other postman

54:08uh videos

54:09if one of our moderators could go get

54:11that link and drop that in chat for us

54:13then you can go and you can learn a lot

54:15more about monitors from there

54:19yeah just for the just for the sake of

54:21time we're

54:22running low on time here so

54:24cool so what we went through today was

54:27just showing the benefits of of what our

54:29contract testing was all about and how

54:31to use both the producer and the

54:32consumer side for just giving us that

54:34really strict confidence that our

54:36definition matches our examples that our

54:38definition actually matches the open api

54:41specification and then from the consumer

54:43side adding even more schema validation

54:46from inside of that

54:48so we've got these additional resources

54:50here if you want to grab a screenshot of

54:51this these will be some great links that

54:53you can that you can reference

54:56for any upcoming space camp events we've

54:58got a september event that's going to be

55:00added in the next couple of days

55:02and it's going to be a special unboxing

55:05session so you won't want to miss that

55:06and then we're planning out q4 as well

55:08so keep an eye on that at the top of the

55:10page there is a notify me button go

55:12ahead and add your details in there and

55:14we'll notify you when we add new space

55:16camps

55:17we've also got 30 days of postman and we

55:19have the postman answers both of those

55:21are collections that you can work

55:22through and actually see uh how to use

55:25different aspects of postman and of

55:27course our community forum we got lots

55:29and lots of really helpful folks there

55:30if you've got questions that we didn't

55:32get to today

55:34and we certainly do want to get through

55:35some of your questions

55:36before we wrap up for the day

55:40but we do want to hear some feedback

55:42from you as well so

55:44if one of our moderators could also drop

55:46this link in here we appreciate that

55:48this will take you to a survey we'd

55:51appreciate if you could take a few

55:52moments to fill that out and then we'll

55:54dive in on some of your questions here

55:58so some of the questions that we had

55:59that i want to talk really briefly on um

56:02gilbert was asking if you're if your

56:04company is not doing api first

56:06um but you have worked with generated

56:09open api specs how does the workflow

56:11change um so

56:13open you know what does api first mean

56:15that that means a lot of things to a lot

56:16of companies um i think as long as you

56:19have an open api definition then the

56:22workflow doesn't really change that much

56:24and you can you can go through and you

56:26can generate all these tests

56:28um someone else was asking i saw

56:30something in the in the chat earlier

56:32about um when tim was walking through

56:35his section um is there a way to use the

56:37open api schema to actually generate

56:40um

56:41the the schema json and that's actually

56:44what our other collection is doing is

56:46it's actually going through looking at

56:47the open api spec and it's actually

56:49building out all of those schema checks

56:51and it's actually using the same ajv

56:53library that the tim walked through

56:56we had another question from bruno um

56:59saying uh all validation is based around

57:02this open api spec but if i don't have

57:04that spec is it still possible to use

57:06the button on the producer side to

57:07validate um you can always generate an

57:10open api specification as well from them

57:13so we can look for a postman collection

57:16called postman 2 open api

57:18and there's actually some

57:20there's a collection built that'll

57:22actually go through and look at your

57:23collection it'll actually create an open

57:25api specification for you

57:27someone else was asking whether we can

57:29run all these tests using the command

57:32line yep you can absolutely call these

57:34so whether you're calling the producer

57:36contract generator tests or the tests

57:38that that tim wrote

57:40as long as you've got a collection you

57:42can build a collection runner and you

57:43can call that collection runner from

57:45newman so go check out the newman

57:47command line tool and from there you'll

57:49be able to to run these from the command

57:51line

57:55let's see tim are there any questions in

57:57there that you want to grab as well

58:00uh

58:01yeah there's one from uh ibrahim

58:03hopefully i'm pronouncing their name

58:05correctly what are

58:06the benefits of doing automated testing

58:08using postman instead of other tools

58:12uh

58:13yeah i would say that postman

58:15makes it really beneficial to do these

58:17forms of api testing that we just did

58:19uh

58:20very easy to do through the interface we

58:22automated test cases using scripts

58:25we could use variable to save at some

58:27time rather than having to go back and

58:29change those parameters each time

58:32uh

58:33yeah we could have touched on not just

58:36git here but other http methods

58:39uh other protocols like grpc or graphql

58:43a lot of other forms of testing with the

58:47apis that we could have could have

58:48touched on today

58:52cool all right for the sake of time i

58:53know we've gone a little bit over so we

58:55do appreciate if you can take a few

58:56moments to fill out that survey for us

58:59um there's just a couple of quick

59:00questions on there about your experience

59:02with the webinar today

59:04and again it does help us shape future

59:05space camp sessions so we're looking

59:07forward to having everybody back for

59:08september

59:09and we'll make sure to notify everybody

59:11that signs up for that again you will

59:13get a copy of the recording uh through

59:15zoom as soon as the session is over

59:17they'll send you a link to the video

59:19that you can come back and re-watch

59:20we'll also get it up on our youtube

59:22channel as well

59:23all right well we'll wrap up there

59:25thanks again everybody for your time

59:26today and thank you for your many

59:28questions

59:29thank you to the moderation team for

59:30going through and answering many of

59:32those and saving some of the best ones

59:33for us at the end

59:34and we'll see you next month at space

59:36camp thanks everyone

59:38thanks all

59:47you