РЕАЛЬНОЕ СОБЕСЕДОВАНИЕ НА ПОЗИЦИЮ MIDDLE DevOps

0:00Alright , let's ask a few questions about Linux , shall we ? Could you explain what processes and

0:09threads are in Linux ? Well , processes are essentially any program running in either userspace or kernel space . It’s basically the core fundamental unit in Linux , I’d say . A thread is spawned by a process , it's a

0:31part of it , and that’s the main difference as I understand it . Can they actually exist without each other ? I don’t think so , no , because

0:51a thread is a component of a process , the smallest unit of execution . I don’t think it can exist without a process . Okay . Here’s another question . What can you tell me about Load Average ?

1:06Load Average , what is it ? It’s the number of processes and I / O operations , if I’m not mistaken . Over 1 , 5 , and 15 minutes that are waiting for processor time . That’s what it is . Uh-huh . What would be an optimal number

1:28, for example , of these units with four cores ? Three , I think , or maybe four . Well , when it's four , it means it's likely maxed out , but when it's three , it's ... Well , it can still work . Yeah . But it probably won't be running very smoothly . So three , or about 75-80

1:50% , that’s what I think is optimal . Beyond that , you need to decide if you need to optimize something or take other measures . Okay , good . Now consider this case : a developer comes to you , you have one VM , right ? Uh-huh . It has four cores , and the developer says there’s a database running on it

2:10. The developer approaches you and says , " Listen , I can't connect to the database , it's really sluggish . " You log into the VM and see that the LA is twice as high . If you have four cores , it’s already at eight . Right . But then you check CPU usage and it's only

2:30at 20-30 % . So , aside from CPU issues , what else could it be ? Where else can we dig ? Well , as far as I know , the Load Average isn't just based on CPU load . It could be the data bus , I / O operations , and that's where I'd start digging — maybe check the database logs

2:47, how the services are deployed , how they're configured , or how the unit files are set up , and so on . That’s the direction I would look in . Maybe there are too many temporary files being created constantly , or something like that . Yeah , that’s where I would start digging . Right . Exactly .

3:10Okay . Can you tell me , for instance , a simple

3:15question : how do I find a process ? Well , using top , or ps aux , and then you can either look it up by PID or kill it by PID , and so on . Exactly . That’s the main thing . So , look , you've found a process , let's say , some worker written in Go . Uh-huh .

3:32Where can you find all the information about it ? Hmm , damn . And which file specifically contains that information ? I don't remember the exact directory , but I don't recall which specific subdirectory it's in . But as far as — Well , you have / proc , then you have the process ID number , and basically all

3:53the info about it is laid out there . Right . And you mentioned top earlier . By the way , top gets all its info from there . It just reads from those files . Uh-huh . Okay . Okay . Ah , alright . So , let's move on to this question . What can you tell me

4:13about Docker , in general ? Like , the standard question : what’s the difference between containerization and virtualization ? Well , virtualization is a process where you install a hypervisor , like Proxmox or KVM , onto bare metal . Then you install a completely isolated operating system with its own kernel , its own

4:34processes , and its own dedicated hardware and resources allocated to that specific virtual machine . And then , inside that VM , you run processes using that VM's kernel and resources — for example , Docker containers and so on . A container is when a process starts , you could say , from a specific

4:56point , basically just a process triggered from the host kernel . Yeah , in some other virtual namespace , with perhaps some capabilities and so on , but that's the difference . Uh-huh . Uh-huh . Ntokha , turn off your mic . You're typing loudly . Well , also

5:16the thing is , like , for instance , if you run a container on Windows , as far as I know , it deploys its own little VM , and it's running the container on Linux , as if inside a Linux VM . That's if it's on Windows . By the way , what are you running on ,

5:35anyway ? On Linux . And what distro , what version ? Ubuntu ? I just have the standard one , I don't know , the classic one . Did you upgrade to the new one , to 24.04 ? No , I was just looking at all the updates for Docker and so on . I understand that the latest stable

5:51version is 22.04 . And there are no utilities for 24.04 yet , as far as I understand . There's stuff like Wine , but that's not needed , yeah , I got that . Okay . So. Alright . Uh , so , fine . Alright . So , for example , can you explain the difference

6:10between CMD and ENTRYPOINT , what they are and how they actually work ? Well , RUN is a command executed during the container build process . So , RUN creates a conditional layer in your Docker container , and these ... so , RUN creates a layer , right ? And CMD and ENTRYPOINT are , you could say , almost

6:32the same commands , except one can be overwritten . These are commands that run when you start the container , and CMD can be overwritten when launching , while ENTRYPOINT cannot . I mean , sometimes people write something with ENTRYPOINT and something with CMD , and

6:55since ENTRYPOINT can't be overwritten , it automatically runs the ENTRYPOINT command first , followed by the one you can override , so there's no conflict during runtime . So , ENTRYPOINT can't be overwritten at

7:11all , then ? Well , I think you can with a flag . Well , yeah , with a flag , I think you can . Okay . Okay . And without which instruction will a Docker build fail ? Probably without FROM . Well , without pulling the base image there . And how many " FROM " s can there

7:29actually be ? Can there be more ? Yes , yes , there can even be 100 " FROM " s . Theoretically , well , that's multi-stage , it's for optimizing the image size . Certainly , in the first stage you install all the necessary packages , all the dependencies to build the application itself , build tools , and so on . And in

7:50the second stage , you basically just pull the required executable files , some binaries , and so on . And because of that , you don't install everything , just the minimum required set of packages to simply be able to run that binary . And because of this , an image that was , say , 500 MB becomes 100 .

8:12That's how it works . Uh . Uh-huh . Okay . Okay . By the way , I don't think you mentioned , when you were talking about containerization versus virtualization , what technologies is it actually based on ?

8:25Well , which ones ? Well , the main components , as far as I know , are the client-side , like all those docker run , docker build , docker stop , all these client requests . Then the daemons themselves , the daemon , the container registry , the remote one . And in the daemons , there are the images

8:45themselves . But at the Linux level , was something added to Linux , what did Docker bring with it ? Well , as far as I know , it uses namespaces , cgroups , capabilities . And as I know ,

9:05sure , the main part , right , for example , yes , the Docker component is just those daemons and various libs for interaction and so on . But could you tell me , what is a namespace ? Well , a namespace is a kind of abstraction , like resource isolation in , well , oh ,

9:27not in Kubernetes , but in just in Linux . It means you have isolation of individual processes , resources , and so on . Meaning your processes run only within a specific namespace . So , in that namespace , processes basically don't see each other . Capabilities are

9:48when you have specific , well , specific rights , the minimum necessary rights to run certain processes , granted precisely so that you don't have a situation where a process is running with root privileges . And cgroups are for limiting specific resources to specific groups of processes . So they

10:09are also used , uh , together with namespaces for better control of the resources used by these processes . That's how it is . Where else , for example , can cgroups be used in Linux ? Hmm , if you take away Docker , well , in principle , I think just in certain specific applications and so on

10:29. Just in applications . Wait , I can't hear you . Well , I just think in some applications , can you hear me ? There , now I can hear you , yeah . Well , there you go . Well , it's just that I think in some , I don't know , applications and so on , it can be used .

10:41It's just , it's just a mechanism for limiting resources for certain specific groups of processes . Right . So I think you can use cgroups almost anywhere . Uh-huh . Okay . Okay . Okay . All right . Oh , listen , a question , by the way , you mentioned , yeah , that you did everything in Terraform , this and that ,

10:59all that stuff , but did you have to interact with the developers at all ? Ah , well , yeah . But that was when , oh , when you had to , say , prepare an environment , like helping to prepare an environment , then you'd be the one ... I was the one who , like , contacted the

11:14developers and asked what , what more or less specific amount of resources the application consumes , and maybe , like , what amount of requests more or less their application unit can handle simultaneously , and from that , like , that's what I based it on , and then some number of , say , application replicas from scratch , then , uh , scale

11:36it from there , like , at some point there will be more traffic , and so , well , you already based it on that . And then , if there were any hangups with the deployment , we had everything organized in Jira , we contacted them promptly , helped , whoever could help , that's how it was . That's how it was

11:57organized . And did you have to debug applications there ? What did the devs actually write in ? Well , mostly it was Python , all sorts of Django applications , Flask applications . There was one time with PHP , well , on Laravel as well , uh , then there were , well , those were the main languages , then

12:15there was one time with Java , where they built something with Maven and Jib , and that was basically it . And also , well , I just watched and observed how , more or less , a Go application is built , and it's built using a makefile and just using a regular multistage build in Docker . There . That’s basically

12:37how it was . Uh-huh . Uh-huh . Okay . Do you know

12:41anything , for example , about REST API ? REST ? Well , it's the operating principle of networks , where you have , let's say , a ... ... where you have , let's say , it's an application architecture , where it's based on the client and server exchanging data using HTTP

13:02requests , roughly speaking . The features of REST API are that each resource has its own URL , like an API , which you can call with all these CRUD requests and interact with the resources through them . And , if I'm not mistaken , neither the client nor the

13:23server stores information about each other . Like , all the minimum necessary information is stored in the requests themselves . And , if I'm not mistaken , these requests are almost always in JSON format , data is almost always transmitted in JSON . I think that's also a specific thing in REST API , if

13:45I'm not mistaken . So. Uh-huh . Uh-huh . Got it . Yes , thanks . All right , Antokh , your turn then , yeah ? Hi . Is it okay if we use informal address ? Yes , yes , that’s fine , yeah . Okay , good . So , looking at your resume regarding writing Helm charts ,

14:08what exactly did you do there ? Did you write them yourself or modify existing ones ? Well , I wrote small Helm charts myself . Mostly it was modifying , like adding a template here or there , let's say with Go , so it would deploy under certain conditions , and so on . That's mainly it . But mostly modifications . And I wrote

14:27some myself , and added to them here and there , and so on . That’s basically it .

14:33So , okay . What options do you know for how a Helm chart can — let's say you have three zones ... ... prod test , prod , well , dev , it doesn't matter how many , five zones . What options do you know for how this can be implemented in Helm itself ? Both based on Helm and not

14:52based on Helm . Well , based on Helm , it’s just by using a package , sending it to some package registry with some version , like this and that , and then deploying that package to different environments . Wait , wait , let me broaden the question then . Are there different environments , like , settings ?

15:15Well , it's all done via values files and dependency files . It just pulls in different dependencies . You have different values files being swapped in at runtime , theoretically . And you can set all of this up using Argocd . But do you know any ways to handle it if it's value 1 ? Hmm , I think you could probably set up

15:35some kind of template , like : if this , then use this values file , and if that , then use that values file , or maybe implement it using some kind of Helm tag . Yeah . Well , in general , yes , there are options using if-statements , there are options using plug-ins . There are many

15:53options . Yes , those Go templates for the charts themselves . Right , okay . Um . For Helm , let's

16:02imagine : did you write one shared chart for Go projects , or did you write one for each project ? Ah , well , generally it was more or less just multi-repo , where each project had its own specific Helm chart and its own specific section in , say , a package registry . And that's

16:25how it was all implemented too . Did you have Helm dependencies between them ? I mean , imagine you need a database , or let's say Redis is deployed into a Kubernetes cluster , and if your application always uses Redis , you'd want Redis to be deployed by that same Helm chart during application

16:48deployment without any extra hassle , without any extra files . No , actually , I've only done it by just pulling in different dependencies using the dependencies field in the requirements.yaml file or something similar . Well , that's basically how it works .

17:08Like you said , you can specify Helm dependencies there . Well yeah , and pull them in . And dependencies on other Helms can also be managed . So , yes . Ah , well , okay . That's something for later . So. Okay . All right , let's move on .

17:22Let's move on to K8s then . Hmm , what does K8s usually consist of ? Well , I mean , on bare metal , what’s on those instances , what is it made of ? Well , okay . Well , there’s , like , the K8s architecture ; you basically have a

17:44master node and a worker node . On the master node , you have the core Kubernetes API , etcd . That’s a key-value store that basically holds the entire cluster configuration . You have a controller , which is a service that monitors the entire cluster to ensure it matches the configuration in

18:06etcd . And if something happens , it reacts to that and does something about it . Then there's the Kube-scheduler , which is responsible for distributing pods across various nodes . Worker nodes . The worker node itself consists of

18:26Kubelet . This is a service that basically controls the state of the node . There's Kube-proxy , which handles the networking . There's the Container Runtime Interface , which is responsible for the containers . And there are , as

18:48far as I know , some control loops that handle things like health probes or something similar . So , okay . Hmm , a pod

19:01— what is it , why is it needed , and what’s actually inside a pod ? Well , okay . Well , a pod is basically the smallest unit in Kubernetes , where containers are usually located . There can be one or two containers , for instance , if there's a sidecar

19:23container , say , for monitoring each pod or something like that . Basically , it is the smallest unit of deployment , or rather , the smallest resource unit in Kubernetes , and serves as an abstraction for pods . It also makes it easier to manage things if you have

19:47multiple containers in one pod . Right . Uh-huh . You mentioned sidecars and things like that . With multiple containers in a pod , can you split those containers across different worker nodes ? Well , if two containers are in the same

20:09pod , you can't split them so that one container is on one worker node and the other is on a different one . You can't do that , because there is a mechanism that specifically determines which pod goes on which node . And you can't , like , make the containers within a single

20:31pod end up on different nodes . You can't do that , as far as I know .

20:35Yes . Okay . So , Namespaces , what are they even for ? What are they ? And if there are issues , why don't they just put everything in one Namespace all the time ? A Namespace is basically a resource limitation in Kubernetes , used for different environments and so

20:58on . And the Namespace solves the problem where your applications can't see the apps , or rather the processes , in other Namespaces . And what was that other question again ? Well , you’ve basically already answered it more than

21:20once . There you go . Okay .

21:23So , what entities do you know for managing pods ? Well , there’s the deployment itself , the main one . There’s replica set , there’s stateful set , and there’s daemon set . Well , do job and cron job count ? Well , they sort of create a one-off container . So I think job and cron job count . And service .

21:43Well , in general , yes . There you go . Basically .

21:47So , what is a deployment needed for ? Well , how do replica set , deployment , stateful set , and daemon set differ then ? Okay . Well , a deployment is basically the main unit in Kubernetes responsible for deploying an application , for controlling things like application images , and , say , controlling the

22:08number of application replicas . I mean , a deployment creates a replica set when it's created , which then controls exactly how many pods need to be maintained and so on . And the main difference between a deployment and a

22:29replica set is that a deployment , first of all , allows you to change container images painlessly , and allows you to not just update the application , but also use blue-green deployments . And

22:52basically , it’s mostly for rolling updates and canary deployments . That’s the main difference . You could just say that a deployment is a higher-level abstraction over a replica set . As for the difference from a stateful set , a stateful set is also a type of resource in Kubernetes used

23:12mainly for applications that require their own state , like databases , metric collectors , and things like that . So , a stateful set has its own persistent IP address , its own persistent identifier ,

23:37and its own persistent storage , like a PV , right ? With persistent storage , each pod in a stateful set has mounted storage , whereas a deployment has one volume for the entire deployment . And

24:00maybe the difference in naming : a deployment has the deployment name plus some postfix , while a stateful set has the stateful set name followed by 0 , 1 , 2 , 3 , 4 , 5 , 6 , 7 , and so on . That's it . Here are some other differences as well

24:22. Well , basically , what I said :, I don't know , StatefulSet applications are much harder to scale there because it requires state , it's much harder there , what I know , what else ... StatefulSet applications , they specifically , a certain StatefulSet application cannot be the same on different worker nodes

24:43because it's tied to a specific file system , well , to a volume of a specific worker node . That's another thing . Yeah . Okay . Did you mention DaemonSet ? Well , and a DaemonSet is just , uh , also a type of resource that runs a specific

25:04pod on every worker node , a specific pod on every worker node . And that is mainly for various technical processes like collecting metrics , I don't know , logs , and so on . Yeah . Uh-huh . Okay . So , you mentioned in the beginning , what is the scheduler for ? For scheduling , in principle , for ...

25:26Okay . Scheduling , well , where which pod should go based on resources .

25:30So , you mentioned meta-abstractions like Deployment , CronJob , StatefulSet , ReplicaSet , controller . For which abstraction is a scheduler not needed ? Well , for abstractions , I think , for some kind of service , because it's not tied to anything . Wait

25:52, no , no , no . Damn , there is a service in terms of networking , which is responsible for the network in the cluster . No , precisely among these abstractions . And out of these , I think , well , the scheduler , I think , I think CronJob , because it kind of hangs in the process

26:08constantly . Maybe some kind of CronJob , I don't know . But your scheduler , besides distribution , it also reserves resources on the node , right ? Well , yes . Considering that a DaemonSet is on all nodes , it doesn't need a scheduler in advance , because it reserves resources on the nodes fully in advance . Well ...

26:27Resou — , so no one will kick them out if there aren't enough resources . Uh-huh .

26:33So , okay . Um , are there any , well , since we started talking about resources , any options for limiting resources in Kube ? Pros and cons of these ? Yes , there are things like Quality of Service , that's when you have the request and limit matching , well , the same values . There is , if I'm not

26:56mistaken , Burstable — that's when your resource request is lower than the resources , and there is BestEffort . That's , in principle , all the other resources . And while there are pros and cons , the best part here is that there's no struggle for resources , unlike the best-effort class , where

27:18there's some contention when you have , say , a stable limit . Guaranteed is the best , meaning that , I don't know , if you need to kill resources , say , free up space for a specific pod , then in Kubernetes it will kill the pods with

27:42best-effort quality of service first , then the others , and only then the guaranteed ones . Well , that's in extreme cases . Yeah .

27:55Uh-huh . Are there any downsides to limits , well , to requests specifically , not limits , but requests ? Requests . Hmm , well , I think if the request , well , I think it's possible that if the request is slightly larger than the available resources in the cluster , then it simply won't deploy at all . Maybe

28:14that's a negative aspect .

28:17Yes , that's correct . So , what options do you know ? To expose an application from inside Kube to the outside . Ah , well , you can use a load balancer , you can use a service like a LoadBalancer type , you can use

28:38something like an ingress controller with a load balancer , or maybe an ExternalName . Yeah . And ClusterIP ? Well , ClusterIP is an IP that's only inside the cluster . It can't interact with an external IP address . Cluster

29:00IP I don't know . So , you're exposing the application to the outside . Go on . Actually , ClusterIP allows routing traffic to grouped resources only within the cluster . As far as I know , it's only in the cluster , so you can't access it directly from outside to a

29:22ClusterIP without a load balancer . You need a load balancer that would , say , redirect to the ClusterIP . That way is possible , but directly from some external IP to a ClusterIP , that's not possible . Because there's simply no access . Okay , I see , you meant directly to the internet . Well , yes , yes , of course .

29:39Well , yes , yes , of course . Well , alright , so I asked the wrong question then . Well , to the outside specifically . from the cube , so you can access the Cube . And also then , if there are some options , like , well , you are inside the cube network , well , so that the

29:55application could be accessed from your local machine , let's imagine , to test it . Hmm , well , exactly there with the help of clusterctl , you can port-forward , I don't know . Well yeah , port-forwarding . Uh-huh . Ah , well , port-forwarding the application . Okay , and I have another question for you . Is there any way , well , to make

30:21pods always land on a specific node ? Well yeah , you just need to define affinity , so either using a node selector , so that it goes to some specific node , or with some specific resources selector , or if you really need it on one specific node , then using node , what's it called , node name

30:39, if I'm not mistaken , just by node name to some specific pod . Oh , node . There . Or you could , I don't know , set it on all nodes except , say , a group of nodes , just taints and don't add tolerations to the pods . That's probably another way to do it .

30:59Well , actually , you just prompted me

31:01with a question . Labels and selectors , what's the difference and what are they used for ? Well , labels , as I know , are used in general there , wait , labels specifically , where labels , labels for marking like worker node and master , or labels in the deployment itself , in the files themselves ,

31:21are they any different ? Well , somehow , well , just , well , it's a label , label and label . Well , label is a tag , like , you're in the deployment , basically , it allows you to interact with a specific group . Well , specifically that deployment and groups and container groups and

31:40container groups , well , pods in the case of Kubernetes , which are specifically inherent to that deployment , it helps to interact with that label , say , with some ClusterIP service . Like that . And the label on a worker node , damn , I actually don't remember , but maybe a label on a worker

31:58node allows , using affinity , to somehow , I don't know , set priorities . Okay , then a question . Given that you said , can different deployments , well , different pods of a deployment , let's say , which are v2 , can they have the same , let's say , port

32:21name ? Hmm , maybe , but as far as I know , if you have a service that looks specifically at that label , it will target those two deployments , or so I think . Then what is the selector for ? Well , I think it's more for , I don't know , for selecting nodes , like for

32:40choosing which node something should be deployed on , for that purpose . Right . Regarding Ingress , when you create a service , if you have , let's say , in your deployment there are five pods , let's imagine , replicas set to five , and you connect them all in the service

33:01, forwarding them . How do you do that ? In terms of via Ingress or no ? Specifically the service . Service without Ingress for now . The service , how do you do it ? Well , you then access this service , and it balances traffic between them , right ? How do you reach this app ?

33:20Not by not by labels . No. Hmm . Damn , well , usually for me , you just reach it in the service via the selector . So the selector is , uh , how do I put it , the word slipped my mind , not individual . Anyway , the word slipped , in short , it can't be repeated

33:42. Not exclusive . Well , exclusive , in short , a name , yes , unique . Everything just slipped out of my head . Unique name . But labels , they can be repeated . So you can sort by labels , like show all pods that have the label ' prod ' or , I don't know , or ' test ' , and so on . So ,

34:00create a selection . You said it correctly that you can attach one label to a group of nodes and then use affinity to deploy them . Mhm . And the selector is more for identifying what we need . The service specifically works by selector . No , the port is specified there and the application , but it's also a selector ,

34:22I think . Mhm . Okay . How , in an HA if you have a

34:30High Availability application , how will you implement scaling and will you specify a certain number of replicas , or are there mechanisms to scale what ? The cluster itself , or specifically the application . The application specifically . Mhm .

34:50For now . Yes , I will just look into ... I'll create perhaps some specific HPA , you know , an autoscaler . And in it , I will specify that for such and such nodes , scale it at , say , a certain CPU utilization , or memory utilization of

35:1080 percent , scale up to this maximum number of replicas , or have a minimum number of replicas like this . Well , I would implement it like this , plus : I'd just create some HPA , either one or maybe two for certain applications . There you go . Okay . Well , overall , I think I've asked

35:31everything I wanted regarding Kube and Helm . I don't know what's next , whether we need GitLab or Ansible ; if needed , I can ask about those . If not , I'm generally ready to wrap up , so Listen , I wanted to ask a question . You were talking about " BestEffort " QoS ,

35:49which ... You said , right , that requests won't let you deploy if we don't have the resources . But if we exceed the limits , will it let us deploy ? No , limits don't really ... like limits in a deployment are conditional , but it will kill it . It will kill the container using some OOM mechanism . And

36:11no , I mean , will it let me deploy it ? Say I don't have the resources , but I set the limits higher than my current available resources . Will the pod itself deploy , regardless of the limits ? Uh , look , there's a request , right ? Say I set a request of 256 MB of memory ,

36:28right , and the limit is 512 . Yeah . But I know that , well , the kube doesn't have enough resources to go beyond the limits . Will it let me deploy this pod ? If this pod has some priority , it will kill a pod that has , say , " BestEffort " quality of service and will let you

36:48deploy it there and adjust the limits if , yeah . No , it does look at that , right ? No , no , no . That's the request . Uh-huh . But what about the limits ? Hmm . Damn , I see what the question is now , damn . I don't think so , actually ,

37:06I don't think so . I think it will say that you don't have the specific resources for that . No. Or it will let you deploy , but it will keep killing that container when it tries to reach those resources . And that's most likely how it's implemented . Maybe , yeah . Well , that's how it will be . OOM kill .

37:28That's a thing in Linux . Like , even when you just press " top , " you'll see everyone has one limit . Well , so that by default everyone can start up . And when , accordingly , it's exceeded it arrives , right ? It happens . Okay . Alright . What ? So.