Speaker 1: 0:02

you have taste in a way that's meaningful.

Speaker 2: 0:07

Hello, I'm bill gates yeah, I would, I would recommend. Uh, yeah, it writes a lot of code for me this almost makes me happy that I didn't become a supermodel.

Speaker 1: 0:28

Kuber and Nettix. Well, I'm sorry guys, I don't know what's going on.

Speaker 2: 0:34

Thank you for the opportunity to speak to you today about large neural networks. It's really an honor to be here.

Speaker 1: 0:39

Rust Data Topics. Welcome to the Data Topics podcast.

Speaker 2: 0:45

Hello and welcome to Data Topics. Deep dive. There's Data Topics, just Data Topics. Okay, alex, you said it. Your casual corner of the web where we discuss what's new in data Every once in a while, not every week, just every once in a while, from KubeCon to LLMs, anything goes. It's kind of okay. You know, check us out on YouTube, linkedin. I'm not sure if we put stuff, do we used to put it? Yeah, we do put stuff on LinkedIn. I'm just checking with Alex, the one behind the scenes here. Keep us in check. Feel free to leave your comments or questions, anything there. We'll try to get back to you. My name is marillo. I'll be hosting today. I'm joined by the aforementioned alex behind the scenes. Hey, alex, hello and the one and only nick. Can I have it like a hi everybody? He's like stop, I'm just a man, it's okay. How are you doing, nick? I'm good, I'm good. Thank you for having me. Yes, happy to have you here Today. This is the second time, but, like, the first time was a Roots competition.

Speaker 1: 1:49

So it was a bit more like Only 15 minutes, only 15 minutes.

Speaker 2: 1:51

The viewers wanted more.

Speaker 1: 1:52

Yeah, that's what I heard.

Speaker 2: 1:54

Yeah, I'm not going to make those jokes, but, yeah, happy to have you here. But maybe for the people that, like, last time you were on the pod it was two years ago yeah, so it was a long time ago, a long time ago. But for people that haven't heard that episode, or people that you know, maybe, what has happened in the last two years, would you care to introduce yourself a bit.

Speaker 1: 2:17

Sure thing. So yeah, niks Schuyt, I work for DataRis for almost six years now. I work for DataRootz for almost six years now, started together with Murillo. Yes, I'm a data engineer and that's it, that's me, that's it, nothing else.

Speaker 2: 2:34

No fun facts, nothing, no fun facts.

Speaker 1: 2:36

That's it. I don't do anything fun, I only do.

Speaker 2: 2:38

You come home, just sleep, that's it. And come back, that's it Okay. It and come back, that's it okay. But, um, you're also roots academy teacher, so roots academy is our exactly uh, how do you say? Like, uh, one month? One month, I think, or two months? No, one month, I think. Yeah, um, let's call it like an upskilling for people that are recent graduates or people that don't have as much work experience, so we can take them through some courses. You're one of the teachers there. Yeah, exactly what do you teach?

Speaker 2: 3:03

kubernetes, yeah right yeah, uh, what is kubernetes?

Speaker 1: 3:11

good question. Um, also sometimes referred as k8s. Yeah, and you know why, right?

Speaker 2: 3:16

well, I heard one explanation, but it's too simple probably right what do you think k8s? So k8s means kubernetes. Yeah, do you have any idea? Alex, don't ask me, I don't know it's just, there's three letters between my eight letters I was thinking about k3s.

Speaker 1: 3:33

But eight letters between the k and the s, that's all nothing cooler about it.

Speaker 2: 3:37

Yeah. So someone just said like, oh, I don't want to type k u b? E? R. So they just said k8 that's it this is a bit random right, like it can be, like the.

Speaker 1: 3:46

I think every tool needs a good abbreviation.

Speaker 2: 3:49

Yeah, that's true, but it just feels very random Because I feel like you can say that for anything right, probably can, yeah, like the, oh, you're the. N3.

Speaker 1: 3:58

The N3? Yeah, you, I hadn't made N2K. That sounds good, right, oh, N2K sounds nice. Okay, how many letters? I'm just going to guess M6O, m5o, m5o, actually M5O.

Speaker 2: 4:12

M4O. Actually Mu Mu, Mu, mu, mu, mu, mu, mu, mu, mu, mu wars. You think m4o? Yeah, m4o anyways tpo anyways, yeah, so kubernetes indeed.

Speaker 1: 4:27

Um, well, it's a tool originally made by google. It's like an open source uh tool. Now to um, yeah, to host and to run containerized um services and workloads. Um, usually we use it, I definitely at data roots. We would use it. That um a hosted kubernetes from a cloud provider. So might have heard of eks, aks or gke. Google always has to be difficult, um, because people do you do also run it like with on-prem service kubernetes, but um, do you do also run it like with on-prem service kubernetes?

Speaker 2: 5:07

but um, then you're in a whole different ballpark. I think so, but then so, and why would someone use kubernetes?

Speaker 1: 5:12

well, because, um, yeah, you want to run I. So I think for a lot of machine learning engineers or data scientists, I think the job usually goes until like a docker container. Right, is that fair to say, yeah? And then you want to run that docker container, but you want to care about it the least amount possible, right?

Speaker 2: 5:30

that's true, right. You just want it to run. You want it to just run and uh, definitely like hardware wise.

Speaker 1: 5:37

Like in the past probably, people um used to run like literally on a server in the basement or maybe on a virtual machine. But that kind of just sucks, because then you have to actually care about which physical machine your, your load is running on. And if you want to, if you want to have replication or high availability, then you need to manually deploy it on another machine again. And while in reality we really don't care about which machine it's running on, I just care that it's running. And if it's on, like let's say, I have 10 servers downstairs, even if I host it myself on-prem, if I have 10 servers, then still I don't really care which one of the 10 servers is running my load. Or let's say I have an increased load now and I need to to have nine of these containers running, then I still don't care which machines.

Speaker 1: 6:30

Um and kubernetes is a bit that layer in between, between the, the vms that we we came from, or even physical machines to like the docker container and and to kubernetes I would just say here's my docker container. Good luck, uh, run it, I don't care where, how much, and you have some control. But the point is to not take control. You can, but then you're probably misusing it.

Speaker 2: 6:54

And so you mentioned Docker as well. Yeah, so again. So for people that are familiar with Docker but not as much with Kubernetes, how do you compare the two things I mean?

Speaker 1: 7:02

yeah.

Speaker 2: 7:05

Do you know?

Speaker 1: 7:06

docker swarm as well like the let's assume that I don't. So you have docker. Indeed, it's like you're running one container. You can run that locally.

Speaker 1: 7:11

That's already very cool because you can share it with with with everybody, and you have that thing where it runs on my machine and you you fix that because you just shipped your machine, which the docker container, that, that meme that we probably all know um, and then docker came out with like docker swarm, because now you need to run that multiple times, and and um, but that never really caught on because instead, also, google came out with kubernetes, which is doing exactly the same thing. So, um, I guess docker um makes it possible to have isolation and makes it possible to have your workload be replicatable so that every time you have this Docker image it will run the same way. But Kubernetes then actually makes it possible to make this Docker container something production-worthy without Kubernetes or any other thing that takes a Docker container. But without this you cannot go to a client and then deliver just a Docker container on local host and then say like, yeah, but I have an ngrok URL here and if you want, I'll keep my Mac open and then you can use it right.

Speaker 2: 8:20

Trust me, I won't close my Mac book. I won't.

Speaker 1: 8:22

I mean, I've seen projects be delivered like that. It just doesn't really work. So you need Kubernetes to fill in that gap, to go from something that's cute to something that's production worthy.

Speaker 2: 8:34

Yeah, and the. So you mentioned what Kubernetes is, but there's still a lot of development, there's still a lot of evolutions towards it.

Speaker 1: 8:43

Definitely, yeah Right, you want to go to the cube con there?

Speaker 2: 8:48

well, we can we can but like this then that's what I was going to cue a bit towards right, like I think every year there's a cube con, so that's uh, and I'll put this on the screen. Uh, every year, I think in europe, but I think other places in the world as well.

Speaker 1: 8:59

Yeah, there is always one in north america, one in europe and then a lot of other locations as well, but those are less big.

Speaker 2: 9:06

Indeed. So then they kind of share a bit. So it's not necessarily about development of Kubernetes, because you can also develop stuff on top of Kubernetes. Yeah, definitely. So I think this is, from what I understand, is a place where people kind of come together and share a bit Like, yeah, people that build Kubernetes, but also people that use kubernetes every day different ways that you can use it, and there's a lot of stuff as well that, like I, I'm not as familiar with. I'm not as familiar with kubernetes as you are, but I also understand that kubernetes can be used even for, like, managing infrastructure, right? So something comparable to terraform, comparable to these other things yeah, yeah, definitely.

Speaker 1: 9:41

There's a cloud nativenative foundation which is behind the Kubernetes. Kubernetes also supports a lot of other open-source development that uses Kubernetes, and one of the tools that I saw at KubeCon as well that was really cool was Syngit. So you probably know about, about githubs I think most of our listeners will but you're you're trying to put your uh source of truth in git and then you make sure that um reality lines up with what is in git, and so you have kubernetes tools for that, because kubernetes is uh always running, so it, it, um it can check your git continuously and then monitor your actual um infrastructure or other related things with your infrastructure for drift, um and um yeah, so you have arcocd there maybe as a tool or like flux, and then, yeah, one problem I had with that always in the past was like I had with that always in the past was like um it. It does make it more difficult to make something new because you're only able to then talk to Kubernetes in like YAMLs or in like CLI type languages or languages interfaces.

Speaker 1: 10:59

I guess Um, which for a lot of our. So, as a data engineer, we take a lot of Docker containers and try to productionize them, but you become a bottleneck if you do that, if you really make the entire organization dependent on you productizing everything right. So we need to find ways to make it as easy as possible for everybody to take on the full chain, like DevOps principles, mlops principles, whatever. But often then you end up making some sort of abstract dialect layer on top of your actual infrastructure where you say you can deploy one thing and then behind the scenes you translate that into maybe Terraform or Kubernetes, whatever. But we make everything so much more complicated with all these abstract layers while we already have the UI right. So oftentimes, like even in Azure or GCP, you have a UI but you're not using it because you need it to be in Git, which is fair, because you need it to be In the perpetrator right, like you need the.

Speaker 1: 12:02

Yeah, and you need to be able to detect the drift and also be able to replicate your setup, and so this was a cool tool. That's still very early. It's probably not being used in production by anybody, I think. This is the tool.

Speaker 2: 12:18

Right, that I'm putting on the screen. Yeah, exactly.

Speaker 1: 12:20

And they're doing it like the other way around, where they intercept the UI, call you make. So in the UI you say I want a deployment in Kubernetes or I want some pods to be deployed, but then they intercept that and they put it in Git and then they let the GitOps tool deploy it for you. So still everything's in Git, but you also get to use the UI.

Speaker 2: 12:39

I see, and then the UI here. So this is a Kubernetes-only tool.

Speaker 1: 12:43

For now, I think yeah.

Speaker 2: 12:44

So, and then Kubernetes then has a UI that you can also interact with stuff.

Speaker 1: 12:47

Well, that's another layer on top.

Speaker 2: 12:51

But natively it doesn't have a UI. I see, I see, I see.

Speaker 1: 12:53

Natively everything's API calls and YAMLs.

Speaker 2: 12:55

I see. So basically kind of like they use Git as a log just to keep track of everything, but you still kind of play with the UI, so you still have like a nicer experience, let's say. But if you still want to revert and all these things, you still have all the commits and all these things that you can revert back to. This was shown in KubeCon.

Speaker 1: 13:12

Yeah, there was like a really tiny booth for it, yeah.

Speaker 2: 13:14

A really tiny booth, okay, and how like, because actually SyncGit works. So they are organization.

Speaker 1: 13:27

But it's like a company or I don't actually know, that they have a slack, okay, um, maybe it's even just a couple of guys. They're just like some dudes. I just thought it was cool because we actually do the same thing at the client of mine, where we made like a ui on top of our git just because we like and and then we were thinking like what are we doing?

Speaker 1: 13:39

we're making it so backwards right, because you have a ui but you're not using it. And then you have yamls, and then we made a ui on top of the yamls. Yeah, so stupid, so it's it's really cute, or really cool, that they just intercept the already existing ui yeah um it's clever?

Speaker 2: 13:54

yeah, it's a clever idea, right, like you say, instead of reinventing the wheel?

Speaker 1: 13:58

yeah, it's also more maintainable, I think, because, um yeah, when the actual underlying uh deployment changes, you don't need to actually remake your UI, because they do that for you, right?

Speaker 2: 14:08

Yeah, that's true.

Speaker 1: 14:09

They have to do that anyway.

Speaker 2: 14:10

Someone else is definitely worrying about these things. Yeah, and so this was a KubeCon. You said there was a booth, but maybe, before we dive in too much into specific topics, no, but it's okay. But what did you think of cube con? Any general impressions? Um?

Speaker 1: 14:29

this is the first time you go. No, it's the second. Second time. I went last year as well to paris. This year I went to. This year I went to london. No, I I really like it because, um, it's a really big conference already. Um, I, I don't actually know how many people, but it was in the thousands really big, I think in the tens of thousands.

Speaker 1: 14:48

Let's see, you can maybe look it up while I'm while I'm rambling. Yes, um, but for being such a big conference, it's not too salesy. I mean, you can, of course, talk with whoever you want, but, um, there's still also just a lot of just good old nerds there that you can just nerd out with, and that's kind of nice. And also, most of the people at KubeCon are people that already are going for a long time. They come from more a network engineer or more a sysadmin engineer background, so there's a lot to learn from them as well, because I think, as in the data and AI space, we're maybe sometimes less professional in this sense compared to, like, a hardcore guy who's been doing 30 years of VM stuff but, like Kubernetes, is definitely one of those technologies that you can.

Speaker 2: 15:36

You can like go deeper and deeper and deeper and deeper right.

Speaker 1: 15:39

Yeah, it's really crazy and so that's really nice. There's a lot to learn from those guys, um, and then just everybody's really friendly and really nice atmosphere and and not too salesy, which is not just something I like yeah, yeah, I've been to a few different conferences as well and you can definitely feel like the more salesy ones and it's like the vibe is a bit different yeah, some yeah, sometimes you're can I really use this or are you just trying to sell me something? Do you actually really use this yourself.

Speaker 2: 16:09

I also went to one conference and maybe we're taking a bit too much of attention here, but like they felt a bit like it wasn't as friendly atmosphere, like it was almost like people were trying to show how good they are. So even the questions that people were asking was a bit like Trying to undermine each other, yeah, yeah, it was a bit like this doesn't feel you know like why. But so it's nice to see the DeFi, because I mean Kubernetes. Like you said, it was built by Google, but now it's under the.

Speaker 1: 16:36

Cloud Native Foundation, cloud Native right, cncf. So it's huge.

Speaker 2: 16:40

Cloud Native Compute Foundation, yeah, yeah yeah, yeah, so it's really and like last year, what like, if you could just summarize maybe one a few sentences like what was your impression last year and what was your impression this year in terms of trends and where it's going um last year.

Speaker 1: 16:57

That's already a long time ago, like hhpt was like.

Speaker 2: 17:00

People are like what is this ai? Thing?

Speaker 1: 17:02

no, no, it's actually like alum stuff and and they were last year, they were always um. Last year I talked to some people who went already for a long time and they said last year was the first time that they were. They started to focus on ai a bit and and um, and that was the first time you went to.

Speaker 2: 17:19

That was the first time, so I can't compare any further back.

Speaker 1: 17:22

But other people said, like last year, there was suddenly this focus on it and and for some of the sysadmin or network networking people they didn't really care. Yeah, and this year it was like insane, the amount of focus on llms. They really jumped on the hype train as well, as everybody did probably. Um, and then also what I noticed this year is that they're also focusing a lot on the user-friendliness of Kubernetes, because they do notice that a lot of their crowd currently is still hardcore people and to really become that tool, you need to be more user-friendly.

Speaker 2: 17:57

Yeah, I feel like sometimes this is just a thought, maybe I'm rambling a bit or just complaining. Sometimes I feel like the tool that wins is not the best tool, it's just the one that is easiest a thought. Maybe I'm rambling a bit or, you know, just complaining. Sometimes I feel like the tool that wins is not the best tool, it's just the one that is easiest to use so you can have like the really the best solution or the thing the most elegant, something that just works and it's like just made for, like fits nicely. But sometimes I wonder if it's just like people just need to be easy, right, like python. Maybe it's not the most efficient but, like you can say in a lot of different ways, it's winning because it's so easy to use and um yeah, because then you get that community advantage and the yeah get that yeah, yeah, once you reach a certain threshold yeah, you get momentum and all these things right I don't know.

Speaker 2: 18:44

I think also a lot of the times llms in a lot of ways. It's like that. It's just so easy to just kind of do something with it that. That's why that's why it's like getting far but at the same time, you see a lot of places where it's misused. I would say or something that, like a regex could do.

Speaker 1: 18:57

You're asking you make api calls right because you get an answer and you didn't bother to check it exactly you know.

Speaker 2: 19:03

So it's like it's a bit like it's not right we shouldn't do it, but like it's easy so you can kind of go for it. But it's nice to see like kubernetes is.

Speaker 1: 19:10

Uh, yeah, I mean I, I do think again they realize it at least that yeah, they are currently. There's a pretty hefty boundary currently and they're trying to make it easier to get, yeah, yeah indeed, I think.

Speaker 2: 19:22

Ideally you want to have a bit of both right like easy to get in, but still a very good solution, something that you can like if you want to optimize for this and that, and that's also kind of like what python ended up doing a bit. You know like, yeah, python wasn't good for like the, the high compute math, but then you have this c extensions and you can kind of like it's a bit fits everything, so it's cool. So you said there's a lot of lm stuff. Yeah, um, maybe one thing as well that we were talking a bit before we recorded that we see a bit, the whole data space is a bit shifting a bit more right.

Speaker 2: 19:51

I think when we started, like years ago, or even like three, four years ago, you had a lot of the, the cloud going to cloud, and a lot of data engineering, the cloud engineering. Um, I don't know, I feel like I mean maybe even the modern data platform, the modern data stack hype, you know. And nowadays I feel like there's a big AI and AI I'm calling LLMs hype. And I think before, with the cloud and all these things, what you saw happen in data routes is that we had a bigger data and cloud unit, so we had more data in cloud engineers. But now, with the AI hype, and I would even argue that the AI hype is bigger even than the modern data stack or all these things, the hype there. What we see now is that everyone is actually touching more on LMs. You know, it's just like a different focus. So like you go to KubeCon and you hear about lms because you want to host your own models.

Speaker 2: 20:46

Um, yeah, we have a lot of a lot of projects coming in for gen ei and all these things. But even like people on data strategy, like how you can come up and like apply, or what's your gen ei strategy, or how you can use gen ei as a tool or like becoming gen ei literate, so in a ways like for for the, for the cloud stuff, it was more vertical and more contained, let's say, to a certain domain, a certain scope, and I feel like the genii just kind of exploded to every, every different direction right now. Um, but I'm curious, so what are the things llm wise that? Uh, or maybe, before we touch on the dilemma, because you mentioned user friendliness, is there something specific that you can say? They're doing this to become more user friendly?

Speaker 1: 21:26

um, yeah, I think, yeah, there are some tools like um, but they're not necessarily new of this year, but like headlamp, like we said, a ui maybe on top of uh, of kubernetes, um, but also just in general, a lot of focus on, on, observability, um, that's something like even even um, if you do use kubernetes a lot, I think it's it's fair to say that even then, observability is not really a solved thing in kubernetes, um what do you mean when you say observability in kubernetes?

Speaker 2: 22:01

what, what do you? What? Exactly what you mean by?

Speaker 1: 22:03

that I mean yeah, so if you're, if you have a problem, right, you always have problems in life yeah, happens pretty cool.

Speaker 2: 22:11

Not that much. I got it under control, I don't know. So you have a problem.

Speaker 1: 22:14

Let's assume that you sometimes have problems hypothetically, hypothetically sometimes. And and if you now need to fix a problem on kubernetes, you're gonna deep dive into the CLI and a lot of commands and again, that's not user friendly. And so, also, if I'm at, maybe my client and I set up a Kubernetes cluster and something was running and then it breaks, I get the question from maybe my machine learning or data scientists it broke, help just because there's nothing to see. And then you have, of course, tools to see. So and then you have, of course, tools to export logs and metrics, like grafana is probably something that that you it, you know, but we set it up and it doesn't always like.

Speaker 1: 22:54

It crashes every once in a while and there's a lot of, there's a lot of work that you also have to put into the, into that stack, and I think this year at kubectl we saw like it was crazy, the amount of companies that were all less than six months old that that's that were started all to fill that observability need. I think you have, you know, datadog, and datadog also has kubernetes. You can definitely integrate your kubernetes with datadog, but, yeah, apparently there's still a need for it, because otherwise why would they start 17 companies for it and all of the companies that they had the biggest boots. So I don't know.

Speaker 1: 23:31

They must have the most money right, yeah, it was crazy, like you came into the the hall and then this year you had like the cloud vendors, which also always have the biggest boot, and then all the other big boots were like observability companies that started less than six months ago.

Speaker 2: 23:44

Ah, really yeah. Oh, wow, so really new.

Speaker 1: 23:47

Really new. Yeah, and they all gave away the best goodies as well.

Speaker 2: 23:50

Oh wow, what goodies did you get A really nice backpack, oh really, oh wow, nice, okay, that's why you go right, that's why you go always Just to get okay cool.

Speaker 1: 24:09

Is this something also you? So you mentioned, like an example, the data scientist, like something's wrong help. Is this something you also experience a lot in your yeah? Yeah, because to then like, when you're debugging it yourself, you would just dive into the cli, but you really can't ask that of of somebody who doesn't. For that you already need to know a bit of the networking, a bit of the sysadmin stuff and so, um, most of the times the users.

Speaker 1: 24:26

So I'm already a kubernetes user and then you have my users, so to say they don't really care that much they just wanted to work again. That's what we talked about, and so then to debug, you need to give them either way, too many uh access rights, which is something you don't want to do, or um, and also too much cli type knowledge, um, or either you export, like all your logs and all your metrics to like, currently, data doc, but that's mega expensive and then we did it once for like a day or something and the cost went out of the roof and we just stopped.

Speaker 1: 24:59

Okay, so let's go back to so that, yeah, just pay someone, yeah, and then then you have, then you put in grafana, which is self. You can self-host that on kubernetes as well, but then of course something goes wrong with the mounted volume on grafana or something, and then so every time you need it, the, the monitoring will also break for some reason. I see, I see, I see okay that's at least my experience. Yeah yeah, but again. So maybe some networking engineers or sysadmins will be listening and thinking like this guy.

Speaker 2: 25:26

Which is true. I mean, I'm a data engineer. No, but I think even if you are the one that works with this, that like is responsible for fixing these things and you also don't know. I think it says a lot about that. The tool is complex. There are a lot of levels to it.

Speaker 1: 25:42

Yeah, there must be. It was also some kind of validation, because we were like, yeah, but maybe we're just not putting enough time in it, which is also true, but the fact that these companies exist is validation.

Speaker 2: 25:52

For me that it's not a solid thing I don't need to change, Otherwise the company doesn't exist, I don't need to change there is some gap yeah. Okay, cool. And now switching to the LLMs, because I think and I feel like every episode we have, like just happens, just have to talk about LLMs. No, it's fine, it's one of the world, so so, maybe. Where does the LLM story intersect with the Kubernetes story?

Speaker 1: 26:20

Yeah, yeah, maybe first of all. Yeah, I went to a lot of the LLM talks because it's cool. It's cool, it's just what you got to do. Where does it intersect? Well, I think in the beginning it didn't too much, it didn't intersect that much. I think they saw I don't really know, but I think they saw a bit of a dip in in, uh, interesting kubernetes as well, because, like you said, with all the gen ai tools are so easy to use. If you ask any c-level person at any company that does anything with tech currently, they will know about chachapin gen ai, but kubernetes or even the modern data stack I doubt it very much, so that's, that's like.

Speaker 1: 27:02

The interest is so huge, um, and and they often ask the question, do we really still need this kubernetes thing then? Because we have already llm.

Speaker 1: 27:11

That does everything right yeah but it's only like one api call that it actually fixes for you. Only like one api call that it actually fixes for you. Um, and definitely in the beginning as well, um, of the llm hype train, the models were still so big and I the biggest models still are, but that nobody will. It's not economically viable to really host them yourself, right? I don't think any company should want to do this. Unless you're like, unless you're like microsoft or ibm, you're like, even if you're huge, right, then still probably it's not worth it.

Speaker 2: 27:43

Maybe if you put it in the basement again on-prem maybe that's true like seven uh I think, gpus, personally, the only strong argument for for hosting yourself I think one is if you, if you really use it a lot and you want to do the math and maybe the math checks out, maybe. But I think the other reason why I can see someone doing this is if you're a very controlled environment.

Speaker 1: 28:06

You're really afraid of.

Speaker 2: 28:06

Yeah, Exactly Like you really don't want to send any data anywhere. So it's like, okay, let's just host your own thing, and then we just know exactly like you know, we have our model.

Speaker 1: 28:24

It's there, it's fine, but then you're probably also already not going to the cloud anyway, right, probably. If you're on the cloud, then I find it a bit um hypercritical to then say, yeah, but we, we do not trust gemini, but we do trust google with our storage.

Speaker 2: 28:29

Yeah, yeah, right. No, it's true. I mean, it's true, I think, I think, yeah, I think I, I don't know about the, the cost story, I'm not sure how.

Speaker 1: 28:39

Yeah right, I mean it's definitely more expensive, but also, like, let's say you buy that one machine, are you really going to have a consistent load, because you need that?

Speaker 2: 28:48

answer almost immediately.

Speaker 1: 28:51

You can only do one request at a time on one machine because it's so huge it's using 10 GPUs, right?

Speaker 2: 28:55

Yeah, yeah, yeah, but is the? It would still be way more expensive, even if you host yourself on the cloud.

Speaker 1: 29:06

Fair point, but currently I don't know.

Speaker 2: 29:09

I don't know Because I'm just wondering if it is way more expensive, how do the chat GPTs of the world? They have their own stuff, I guess yeah they just they just buy their own stuff.

Speaker 1: 29:17

I guess that they just, yeah, they just buy those. But if you wanted to do it yourself on the cloud, then you lose a bit already of the of the advantage, but also you just currently can't really get enough gpus like yeah, yeah that's you really just have to. You almost you have to pay them up in advance, but because then what you're trying to hint at, I think, is like then you get the advantage of being able to scale up and down again yeah but yeah, it's just not something we see at my client at least.

Speaker 1: 29:40

Like when we ask for one, it usually takes like a day. Unless we ask, we say, okay, we'll pay for 10 of them just the entire month, and then they'll say, fine, we'll buy 10 with the money you gave us, but then yeah, maybe it's gone again, because then you're not being, you're not able to scale up and down yeah, you're just using all the gpus themselves yeah, and all the tpus as well like yeah, yeah, that's true.

Speaker 2: 30:01

Yeah, I'm not sure. I'm not sure I want.

Speaker 1: 30:03

I do think that at some point we'll be doing more of this yeah, so I think that's where kubernetes is going as well, right, um, like now as well, that the models are like the. The smaller models are also getting better and better. Yeah, and I think I feel, at least, that we've plateaued a bit on the top level and then that the smaller levels, the smaller models, are catching up a bit and like a 28 billion parameter model, I see a place for hosting that yourself.

Speaker 2: 30:32

Yeah, I think it's also. It's not just like the size, but I also think it's like the available models right. Like the LAMA models are getting better. I mean, I've read it's not just the size, but I also think it's the available models. The LAMA models are getting better. I've read a lot of articles. First of all, it's hard to quantify how good it is. Every time you see a new model being released, they always claim it's state-of-the-art.

Speaker 1: 30:48

They always claim this is that.

Speaker 2: 30:50

Is there contamination? Is this this? We don't know. But even the fact that you see more of the reasoning models, it is also a big signal that the actual LLM maybe it is reaching its capacity.

Speaker 1: 31:04

Yeah, until we get more data, I believe.

Speaker 2: 31:06

Yeah, but there's also an argument of even if, like, basically they're trained on the whole internet, yeah, but a lot of stuff that is being put on the internet now is already GNI generated, right? It's like, are we, we just gonna kind of progress to a noise?

Speaker 2: 31:20

you know like everyone's gonna be, everything's gonna be gray, you know, yeah. So I don't know, there's there's there's some big questions there. But yeah, I I think I don't know if it plateau, I don't know if there's still room to grow, but I do think that it's not gonna grow as fast as it has been in the past two years. Right, I mean, it was super fast as well. Yeah, crazy fast, um, yeah.

Speaker 1: 31:44

But. But so I think indeed there that for the smaller, like it's it's weird to say, but small to medium sized LLMs, which is large language models, small to medium sized large language models. Yeah those will, I think, have. A big kubernetes is now, now that they are more prevalent and they're getting better. Kubernetes definitely is trying to get that market um, market cap or fill that market space um, but I think the the the really huge model on your own kubernetes. I don't think that that will be their setup.

Speaker 1: 32:21

They did make an API for it. The Leader Worker Set Kubernetes LWS.

Speaker 2: 32:28

Just one that I put on the screen.

Speaker 1: 32:29

Yeah, so that's a big advantage of Kubernetes. They do go pretty quickly because they're open source um what is this? A leader worker, set api then yeah, so basically an api to um spread out your deep seek model over multiple kubernetes spots um, so when you say spread out, you mean like you, you just orchestrate based on the load.

Speaker 2: 32:59

So if there are more requests, that would just create, or do they actually have a clever way to not duplicate the memory?

Speaker 1: 33:05

no, it's, yeah, it's not for. It's not for um. How should I say that it's not for um? Multiple times, chat, gpt or deep seek, then whatever the biggest model, because that thing already exists in kubernetes all workloads, you, you replicate them. This is for, like um, one unit of replication, so one deep seek, but across multiple machines. But machines, yeah, um.

Speaker 2: 33:34

So it's like, so that it can be sharded like yeah, so it's like you have one model that is, you have like, I mean, you have 100 macbooks and then you have one model that is too big for each macbook, but you can kind of put them all together and make them work, yeah, in harmony.

Speaker 1: 33:48

Probably not on a macbook, but maybe. Yeah, yeah, let's see. Yeah, I don't know yeah, that's cool and uh, but I don't think this will be the like five years from now.

Speaker 2: 33:59

I don't think this will be the thing that is used by 500 companies yeah, I mean, I agree, but I think if it's an idea that maybe the next, the next thing you know, that actually does it more proper and has a nice substructure, has something like this, maybe you know or maybe somebody will make a model that's really good at this.

Speaker 1: 34:15

It's true, because you're already sharding them across GPUs most of the time the LLM workloads. So, yeah, maybe somebody will make one that's really good at also using the fact that it's on different bots or something I don't know.

Speaker 2: 34:30

Yeah.

Speaker 1: 34:31

That's for somebody who's smarter.

Speaker 2: 34:33

Someone else, someone that has more time, doesn't just think about these things Smarter smart, someone else, someone that has more time, doesn't just thinking about these things. Smarter, yeah, um, also olama. You know, you heard of you know olama, yeah, oh yeah, wow, his eyes went big now, just um, maybe what is olama for people? They don't know?

Speaker 1: 34:50

um it's from mark. So uh from, uh from facebook, right, yeah, but also open source to to run some open source large language models in an easier way to like an interface in between right? That's how I see it at least. Maybe you have a different view on it.

Speaker 2: 35:11

No, I think it's similar. I don't know if actually this is from Meta actually.

Speaker 1: 35:15

I'm not sure Lama is so.

Speaker 2: 35:17

I assumed Olamas as well. Actually, yeah, I'm not sure llama is so. Yeah, I'm not sure.

Speaker 1: 35:19

Maybe assumptions kill.

Speaker 2: 35:20

Let's see if it is if it is from, uh, if it is from face for meta, it would be on their repo, I guess.

Speaker 1: 35:27

But it's a separate thing I once heard you and bart say that when llama was released, that it was only popular because it was open source. So open source is the best kind of advertising. Oh, that wasn't a long time ago.

Speaker 2: 35:39

Yeah, I know I didn't say that, but I heard something along those lines. Yeah, um, but olama, the way I think of it is kind of like docker for deploy models so it's kind of like the abstract, all the other things, and has like the. I actually heard that the people from do actually started Ollama.

Speaker 1: 35:58

Yeah.

Speaker 2: 36:00

So I think they have like model file which is similar to a Docker file. It kind of describes all these things and it kind of extracts all the infrastructure things that you need to do. But if you want to do this, so again, this is a bit the way that I was explained.

Speaker 1: 36:23

I haven haven't actually used this yet, which is a pity. I really want to try. Yeah, um, haven't had the time yet, but um, you can deploy it really easy on kubernetes. You have like a web open this was a llama web ui and then just deploy it and you can run any model play small models.

Speaker 2: 36:28

Small models, yeah, because you don't do the sharding thing right. So basically we just scale if you have more requests.

Speaker 1: 36:33

But like yeah, it does it would scale. Yeah, if you wanted to do. But um, yeah, getting the gpus on the kubernetes nodes is not yet super easy in my opinion. And then definitely if you want multiple gpus on um one node and getting that configured is not the easiest, but what they mostly actually talked about that at kubecon was vllm vl instead of llama um.

Speaker 2: 36:56

But yeah, it does. Basicallyectl was vllm vllm instead.

Speaker 1: 36:57

Of olama, um, but yeah, it does basically have you used vllm?

Speaker 2: 37:02

do you prefer this over that or?

Speaker 1: 37:05

I yeah, just for like three questions, right so? I should avoid it just because it was easy. But and you haven't actually used it in in any production setting on your local mac yeah, just on, like a local, you can have Docker desktop or something else. You can do a local Kubernetes cluster. Yeah, completely defeats the point, but it makes it cool and then I just deployed it there. Yeah.

Speaker 2: 37:28

Okay.

Speaker 1: 37:29

And then just because they say it's faster, yeah, that's why I think it's better, but not because I actually used it.

Speaker 2: 37:39

Is. I think it's better, but not because I actually used it. Is it lightning?

Speaker 1: 37:40

fast? Because if it is, you know it's retaining rust. Are you still on the rust hype train? Isn't that fast?

Speaker 2: 37:44

I think. I mean I was very curious about it, but realistically I feel like for you to really pick up rust, you need to really use it a lot and it's not as simple, so it's like it takes some time. Time, so I'm not yeah I feel a bit ashamed when I say like oh yeah, I'm curious about because I haven't even looked at it in so long I did like the first five days of one advent of code on it and then you're like and then I cried and then I stopped.

Speaker 2: 38:06

Yeah, that was it sounds about right. Yeah, yeah, uh, kubernetes is written in go yeah, right, but also there's now rust controllers.

Speaker 1: 38:15

Not really. Yeah, I think they launched the first Rust controller at KubeCon Europe, so Kubernetes was written in Go.

Speaker 2: 38:24

When you say a controller in Rust, what do you mean by that? What is a controller? And I guess controllers normally are in Go as well.

Speaker 1: 38:31

Yeah, normally everything's in Go. But then, like on the let's just make it abstract on the Kubernetes cluster cluster, there's just like a controller is just a piece of code that's constantly running and like, for example, you have a controller, um, like I was today, um, so that's a github stool, so it's the controller. It's just constantly checking what does git say that needs to be deployed on kubernetes.

Speaker 1: 38:57

So this is all, and then, it also has the rights to actually do that. Like if if you uh change the git code for an extra deployment, then I guess they will see that and it will make an extra deployment. So it's just a piece of code, that's constantly yeah checking on go and rust.

Speaker 2: 39:12

Um, I almost see on go, and then now there was one on rust, because they also just want to be cool, I think, but I don't know if they're actually gonna but it's basically just like the piece of code that is running normally, is in Go and interacts with Kubernetes yeah, and then, indeed, it just makes API Kubernetes.

Speaker 1: 39:30

Is all API calls, right? So there's a, a central, yeah, a central, yeah, just making API calls to the, to the master node, yeah, or whatever. And yeah, just to check what the state is to. Everything is API calls, so it doesn't really matter that all of it is go or some of it is rust, or even if you want to write one in Python, probably people have done it already but it's just slower. I guess it's just slower and they probably won't accept it into the actual like this one was actually.

Speaker 1: 40:01

I think this one was actually accepted by cncf oh okay, like so. And the other, I can write one myself now. It's on my in python, it's on my. Actually it's somewhere on my to-do list but it's it's way down, so it's probably like never, probably not gonna happen but, um, yeah, okay, cool, right, it's cool to put on a resume.

Speaker 2: 40:21

Wow, yeah, that could be.

Speaker 1: 40:22

Maybe you can go present the next kubecon maybe yeah, there we go actually um a side note, side tracking here, but we're actually gonna. My client is gonna stop using kubernetes, so it probably was my last kubecon. Sad change clients yeah, maybe why are they gonna stop using kubernetes? Yeah?

Speaker 2: 40:38

that really was my last cubecon. Sad change clients, yeah, maybe. Why are they going to stop using kubernetes?

Speaker 1: 40:41

yeah, because it's just yeah, it is still a lot of um maintenance, even though we use a hosted uh kubernetes, so aka, as in our case. So azure takes care of kubernetes for us, but then still, like all the, for example, you need to make a reverse proxy on nginx which you just deploy with helm, but you still need to keep up to date there and, like we said, all the monitoring observability that you need to offer to your users, like a cost attribution, all these things they still take time, you still need to put somebody on it and I we estimated that maybe we would actually need a full-time equivalent for that and in reality, in reality, we're just not putting in that amount of time and then. So now we're probably just going to move to container apps, which is, yeah, under the hood, under the hood also kubernetes and azure, but yeah, it's really just.

Speaker 2: 41:37

Here's my docker yeah, docker image and then run it, please.

Speaker 2: 41:41

Yeah, and they take care of kubernetes and even more I see you don't even know that it's a kubernetes underlying probably a lot of most of these uh services that you just deploy a docker and it scales, is kubernetes in the background, probably yeah, yeah and uh. So then, and maybe we're cutting towards the end of the conversation but when is the? Who do you think is the person that says, no, you need to use Kubernetes? Probably someone that is tech savvy, someone that has a lot of these things that they want to optimize costs, which companies do need to.

Speaker 2: 42:12

Yeah, I think it's like what is the kind of problem that you hear and you're like this is something you need to use kubernetes.

Speaker 1: 42:24

I mean, I think if you have enough um, if you're willing to put in the time to actually maintain it and to um, yeah, set it up properly, then then I do think it's still worth it, because you get a lot more control, um over your entire infrastructure. It's going to be cheaper because you don't pay. You don't actually pay a premium on the compute, you just pay for the compute and then a little bit for the Kubernetes cluster. But if you go to container apps or any managed solution, you pay another premium on top of that. You are also not really vendor, or you're less vendor locked in right, because it doesn't really matter if it's AKS, e, gk. You can theoretically move it pretty quickly.

Speaker 1: 43:01

Theoretically, of course, yeah, in reality it's always a bit different, but you should be able to just move them across clusters, across cloud providers. Also, if you want to maybe be multi-cloud or have a part of it on-prem, all of that, you can do so. I think if you're into more complex situations, then kubernetes allows you to do all of that. You can do so. I think. If you're into more complex situations, then kubernetes allows you to do all of that while still having that layer of of um, so that your users, which then would be me, right, your data engineers would have it still easy. Just they can just talk to kubernetes and they don't need to know if it's, if it's an on-prem machine or maybe then like, let's say, you want to have that deep seek self-hosted, then maybe you can have nine machines on-prem, but really, when it really gets a lot of load, you can still buy one or two from Google or like one of two.

Speaker 1: 43:49

Right and your users wouldn't even have to know, like, if, then I would be the user as a data engineer. But I think that's where it comes in, data engineer, but I think that's where it comes in. So, people that already have a lot of on-prem or that already have a lot of networking or sysadmin expertise, there I think there is a big reason to still use kubernetes because, uh, control is also. Yeah, it creates great power, it comes great, yes, but you do have to put in the time and that, if you're not doing that, then you should realize that it's fine to just also say, okay, we go a layer up, okay, we do less.

Speaker 2: 44:24

Cool, but I guess, like putting the time, you still need to continuously put in the time because, as we saw, things are always coming up. Yeah, maybe to wrap up, more on the KubeCon. Still, there are a few more things that we wanted to highlight, right, yeah, on the lm hype train. Oh yeah, there's one.

Speaker 1: 44:42

Yeah, yeah so the one I really still want to talk about was there was one talk about from, uh, the google um config controller team. So basically they're um from kubernetes, because kubernetes originally was a google thing. You can with this config controller also set up resources in in gcp. So you can say in kubernetes I want a storage bucket, and then it'll sort of terraform like, set it up and also monitor for drift. So that's an advantage that has on top of terraform.

Speaker 2: 45:11

Config controller is just like a controller we're talking about, like the piece of code that keeps running, yeah, and config is to, in this case is to to describe the infrastructure on gcp exactly yeah yeah, okay, but so they need to set up a controller for every thing they have on gcp right, which?

Speaker 1: 45:28

is there's a lot of resources and they change every week um and and, and they um, that was a lot of manual work, and so they really leaned into the vibe coding setup of just they just made, and it's literally. They said we just made thousands of vibe coded controllers for Kubernetes, and then they had some mitigations because of course vibe coding can go wrong, and then all they had to do was approve it. That's their premise, and so the the big point here is that they were going away from abstractions. Usually when you do something a thousand times, you would say okay, then we need a controller factory and we go object oriented, then we go as abstract as possible, turtles all the way down, right.

Speaker 1: 46:13

But instead they said well, that becomes really annoying because every change we need to change something in seven files and we really need to know the entire architecture. So let's just make it as verbose as possible. Just the LLM is fine with the amount of text. It's only abstractions that it's not good with. So let's just make it super verbose. Let the LLM write the same line 10,000 times, and so their code base went 10x. But they themselves said that it became a lot easier to read one file than before.

Speaker 2: 46:48

So it's like they kind of said it's a lot of grunt work, but it's okay because LLM don't mind it Exactly and to maintain it is actually easier because you have less abstractions.

Speaker 1: 46:57

Yes, to maintain it is actually easier because you have less abstractions. Yes, to understand it is easier because you have less extractions, but I'm I'm really wondering how it would go if you have to change something in every file. I guess you can still ask the llm, because it still doesn't care, but still it's. It's a bit weird. That's the talk, right? Yeah, this is a talk which I was really interesting and they, they themselves, they seem Maybe just for the people that are just listening and want to check this out.

Speaker 2: 47:20

The talk is from KubeCon Europe 2025, ai Beyond Autocomplete, using LMs to create 1,000 Kubernetes controllers.

Speaker 1: 47:26

Yeah, by two guys from Google and they seemed quite knowledgeable guys. So I don't know. I was almost at the end I was almost drinking the Kool-Aid, yeah take my money.

Speaker 2: 47:40

Yeah, exactly so yeah, that's true, I don't know. I think, yeah, yeah, it's a different paradigm. Now, right, because I feel like a lot of the times, you create abstractions for our convenience, the developer inconvenience but like indeed for reading.

Speaker 2: 47:55

It's another abstraction. Right now, you need to understand some additional like and one one metric that I use. What's like good code and what's bad code? What are the good decisions to make? A metric that I have and it's very flimsy, but it's like how much do you need to keep in your head to understand what's happening right? And if you write some custom abstraction, that's already. That's something another more than I need to keep in my head. Yeah, right, so if it's something that is already used everywhere, yeah, maybe I have to understand how go works, but I already knew that right, so you can reuse it here and there um, or like a similar one.

Speaker 1: 48:25

How long does it take to explain to a new joiner, or to like a recent grad, how this works? Yeah, it's true. And then if you go through seven extract abstractions, it takes a hell of a long time yeah, but at the same time 10x in your code base, it's a lot, huh yeah I don't know, but it is interesting.

Speaker 1: 48:42

It goes above kubecon. I uh like about. It's not only kubernetes here, but how far? How much is is all the things that we learned in school about good code, bad code? How much is that gonna change? Because this is not dry? Yeah, yeah, for sure. Yeah, it is.

Speaker 2: 48:59

Maybe keep it simple yeah but it is not object oriented, it's not dry, it is as verbose as possible but that's why sometimes for me, uh and dryness is a good example, like I mean, be dry for people who don't know is like uh dry, don't repeat yourself, meaning that in theory, good code doesn't, doesn't repeat itself very often right like you have.

Speaker 2: 49:19

It's written one place and there are benefits for that, because if it's really the same logic, you want to be able to change in one place and it changes everywhere. Also like and and I. I remember in the beginning, when I heard these things, I was like, okay, don't repeat yourself, don't like. And I was like man. And now it's like oh, this kind of looks like this and I'll change the whole code, you know like trying to be very clever.

Speaker 1: 49:39

We're trying to be too clever.

Speaker 2: 49:40

Yeah, but that's the thing, and I feel like for me, a good like the how much stuff you keep in your head is something that I'm happy to to, to think like this Cause at some points, like, yeah, if you're trying to be more things in your head, you know like I get it. Like if you have the same information, uh, a hundred times, or almost the same information a hundred times, it just changes a little bit. Maybe you have to remember, like you have to keep in your head that hundred hundred times, so maybe it's simpler to just have one. But if you're trying to be too clever, it's the same thing with type hints, in a way like type hints. Normally it's I like it because like, oh, yeah, this is a string, I don't need to remember, it's a string, this is a string, this is an integer, this is this.

Speaker 2: 50:17

But like, sometimes too, you can get so clever about it, like, yeah, because this is an object, comes from where else? Or actually this is any class that has this property and this method, and then you kind of create more things and in the end it's like why, yeah, you know? Like he's like what? What messes you up more. So I feel like there's, there's a, there's a sweet spot, right, and of course it changes for every person. But, um, I think with llm sometimes like maybe you can delegate some, some of these things, right, like to them um, so yeah again, not sure how I feel about it, but uh me neither, but it's gonna change a lot of things.

Speaker 1: 50:49

It made me think for sure, which is are you a vibe coder these days?

Speaker 2: 50:53

yeah, yeah, you are, I think I want I. I want to be better vibe coder, but sometimes I feel like I'm I don't know.

Speaker 1: 51:02

I feel like I don't know how to vibe code yet very well it's a bit like it feels currently, a bit like, um, when you need to do something but you really don't want to do it.

Speaker 1: 51:11

So you make a script and it takes three times as much time but then in the end you're super proud of yourself because you made a script that does it for you, so it's like you took no time. It still feels a bit like that at the moment because I think oftentimes I can do it quicker, but I'm telling myself that the investment will be worth it, because I'm just learning to live with the overlords the machine overlords.

Speaker 2: 51:34

I'll be friends with them when they take over, yeah, but I do think but.

Speaker 2: 51:44

I do feel like it's a skill, I do think it's good, um, and I do think you need to learn how to use it. When to use it, um, I don't know, like maybe you you shouldn't ask chadjpd to build your whole project, but maybe you should use it as a right dysfunction, like maybe, if you, maybe the things that you already know how to do, you just want someone to do it, or maybe I don't know, like, no well, I think I really like, like somebody said, like, if you know where you want to go, uh, then it's a really nice tool.

Speaker 1: 52:08

If you already know, like this is the code I want to end up with, then you can just ask it seven times and prompt it a bit until it gives you that exact code, and then you don't need to write yourself yeah then I became extra lazy at home like, did you like.

Speaker 1: 52:20

So I also have. I also bought just it's just a whisper thing but that. So you just um local host a small whisper model and then you just talk to your microphone and just say I want you to write this code, and then you're not even typing it or it doesn't work, is it? No said, no try again, and you're really not doing anything. You're just looking at the screen and like this looks good.

Speaker 2: 52:41

This is that's the life. That's it. Yeah, that's true.

Speaker 1: 52:44

I haven't tried it for really big projects yet, because there I'm afraid yeah.

Speaker 2: 52:48

but I feel like sometimes for some big stuff, if you just say just do everything yeah, it doesn't then.

Speaker 1: 52:55

Yeah, it doesn't work.

Speaker 2: 52:56

Right, you need to scope it you still need to kind of like, you still need to care about how it works, the internals yeah, but then you have like there's youtube videos are about.

Speaker 1: 53:03

It's like guys that are really deep into the vibe coding and they're like they're having llms make sub tickets of sub tickets and then scoping denim, then some other lms implementing them and then another LLM is grading the first LLM and they go really deep.

Speaker 2: 53:21

Yeah, but I mean yeah there's a lot of stuff. But I feel like, well, I feel like 1% of people go that far.

Speaker 1: 53:28

Yeah, yeah.

Speaker 2: 53:28

Things very early, but I do think everyone should be using, should be trying, should be owning that skill as well.

Speaker 1: 53:40

um, before we go there's one more article here, one more item that I put down tokenities, yeah, um, yeah, there was also one. There's also one talk uh that we can share the link of later, um, but basically, uh, um, yeah, we're all making agents now. It's really cool, but it's really hard to get security right there because you give your agent maybe, let's say, you make an agent that has access to the database. So I make a chatbot agent where it can help me find out what happened to my order, but how do I make sure that I cannot find out about your orders, right? So the agent just has access to the database. In most of the bugs that I've seen, at least, or most of the the setups, it just gives all control because we want to show off how cool it is, but we're not really thinking about security yet there, um, and so then I can probably hack the llm, because the llm is a black box. It's really hard to to put up guardrails there, so I can probably hack it to ignore, uh, previous commands and to uh give me murillo's uh, um, by history, and then maybe, uh, I find out some, uh, some squanchy, ranchy things about you, um, and so how do you do that?

Speaker 1: 54:56

And and token, that is, but it's really early still. Uh is using, like um, uh transaction tokens for that. So basically you're not making a token purely, that's so traditionally you would log into um, traditionally you would log into the, the agent, and then you just you're allowed to use the agent and and then you're allowed to use the agent and then the agent is allowed to read the database. But actually this makes a token where it combines both the context and the actual user so that the agent can or maybe even better yet, the database can, decide actually what data the agent is allowed to read. So then if I hack the LLM, which is probably always possible, then the database would still say yes, but okay, this agent has allowed to read.

Speaker 1: 55:39

So then if I hack the LLM, which is probably always possible, then the database would still say yes, but okay, this agent has access to the database, but the user that is asking things from this agent really only has access to its own rows. And then we don't need to worry anymore about the LLM being prompt, hacked or whatever we want to call it, the llm being yeah, uh, prompt, uh, hector, whatever we want to call it um, because the yeah, the access control takes care of it. So that's. It's a really cool thing that we should probably watch out for as data routes a lot in the near future, because if we start putting agents live or even customer facing, then this will become a really big, a big thing. I think I don't know if this is going to be necessarily the solution, but they had a cool logo and that's what I only care about this is it looks good on a powerpoint exactly that's what I thought take my money right.

Speaker 2: 56:24

But so you mentioned security. So like, how, how, how does it connect? Actually, like, so you have. I mean token80 is like a mix between tokens and Kubernetes. I'm assuming In Kubernetes how like they split the calls or they split the tokens to make sure I can only have access to the stuff that I need or what.

Speaker 1: 56:50

Well, if you press on, learn more there, you can see like.

Speaker 2: 56:55

You don't know You're learning together now.

Speaker 1: 56:58

Oh yeah, I looked at it a bit but I haven't used it either. So, yeah, it's actually splitting it up and giving you like an actual context for the transaction, when it's not only the fact that you're allowed to use it yes or no but also what purpose you want to use it for and who the actual caller of the identification is, Usually before you would, maybe just when you're logged in. After that it doesn't really maybe know anymore who Murillo is. But now it's because afterwards the agent just makes a call to your database and the database only knows that the call came from the agent.

Speaker 1: 57:41

It doesn't know that it came from murillo who is calling the agent this is taking care of giving that extra context in a really short-lived jwt token so that the database can handle that information appropriately. It can say okay, the agent is. Is the agent, yes or no, allowed to read this database? Okay, yes, it is. And then, but the agent is calling on behalf of Murillo, and is Murillo actually allowed to read the database? Well, only really this row.

Speaker 2: 58:07

I see the rows about Murillo, I see, I see. So it's like Kubernetes is kind of like everything centralized, so all the requests will go there.

Speaker 1: 58:19

Well, all the requests will go there. Well, no, it's even um. It's not necessarily only kubernetes like this is an implementation of the principal transaction tokens for kubernetes but this goes beyond beyond right.

Speaker 1: 58:26

Even if you do it on vms, you have the same problem, right, because you give your machine maybe like um in in azure, we would use a service principle and that service principle has the access to read postures, yeah, but that service principle tells you nothing about, about the call, right, it's just you like have complete access to the entire database. So you have the same problem everywhere unrelated to Kubernetes, and that I thought was really interesting, like something that we really going to have to worry about when we put customer facing bots out in the future.

Speaker 2: 58:58

I see, I see, I see. So basically it's a way. So, yeah, normally what you're saying is like, normally the data in the database is mixed it's not just by user or something but you want to be able to authenticate or something like, even if you have access to the database, which everything is mixed, everything is unified. It has to be because for the analytics you need it to be unified. You still want to be able to split and make sure that you only have access to the data that you need. Yeah, yeah, and this is a way to do it with kubernetes, but like kubernetes in this sense is like kubernetes would talk to, like you make the call to the kubernetes thing, not necessarily yeah, it would be.

Speaker 1: 59:34

Uh, it would be some services or like a controller or something on. It's a, it's a service, I think on kubernetes then. But yeah, I in this case like the, the interesting bit is really not related to kubernetes it was just, it was just there it just had ets in the end.

Speaker 2: 59:49

So yeah, exactly no, but it is true, I think, the more I think when I think of this, and I think that's also what the world is living.

Speaker 2: 59:57

A bit is like the agentic stuff which is like giving capabilities to tools, giving capabilities to all these things, and I think, um, the more we go towards this space, the more we need to worry about these things. And I think, even touching a bit before to what we discussed, I feel like now the the intelligence of things is a bit slowing down a bit, but now we're trying to increase the capabilities, we're trying to increase the context and enable the model to to handle this context well and and all these different things Right, which I guess Kubernetes has a can also play a role in this.

Speaker 1: 1:00:33

Hopefully, I mean because I still think it's a really cool thing to work with.

Speaker 2: 1:00:36

I think so too. Again, I think it's the one thing that I want to touch on more, because I feel like it's, I think it's really cool, I think it's really important. I think a lot of infrastructure depends on this, but I also feel like it's not very common.

Speaker 1: 1:00:51

No, it's a deep rabbit hole as well, but I think it's not very common for us data routes.

Speaker 2: 1:00:59

But yeah, yeah, if you go further down the stack, I think my impression is like I hear a lot, but when I say, okay, where are the people that are actually using this, where the people that can help you with this, is like very little actually, yeah yeah, yeah, that's true but, that is, that's, I think, our uh, our space, yeah, our our we're getting some mean looks from alex there.

Speaker 1: 1:01:15

Yeah, I think we're close to the hour yeah, yeah, we are.

Speaker 2: 1:01:20

But uh, nick, uh, maybe last thing in one sentence, in one tweet, or one x, I don't know how you want to call it. Um, if you do go to cupinis next year, yeah, what do you think you will see there? Hmm, you didn't make me think, uh yeah, I didn't ask you before that, but yeah, on the spot, putting you on the spot, uh what?

Speaker 1: 1:01:40

what will I see there? Um asking you pt no, actually I don't. I don't really know if I do go, but I'm probably not going. But if I, if I do go, I think they will continue their user friendliness um, um, focus, um, and I think the llm hype will have died down a bit so it's a big gift.

Speaker 2: 1:02:06

But yeah, maybe yeah and um I think people are gonna be like finally we can stop pretending or maybe like the frameworks, because there was a lot of.

Speaker 1: 1:02:18

Also, you had a lot of frameworks already to deploy AI, just like Dapper and not Dapper, but it doesn't matter and Kubeflow, which probably, and they were also focusing on LLMs, but maybe those protocols will be more mature next year yeah, let's see, let's see. I think more of the same but more mature, but more mature.

Speaker 2: 1:02:38

So just more maturity in that sense. Yeah, okay, cool nick, thank you very much. Thank you for having me. Uh, I hear, I hear you're gonna be coming making a comeback appearance sometime in the future I might or might not. Let's keep the suspense yeah, let's see's see, but I hope you do come back talk about Kubernetes or anything else as well.

Speaker 1: 1:03:00

Yeah.

Speaker 2: 1:03:01

Thank you for your time. Thanks everyone for listening Sure.

Speaker 1: 1:03:08

You have taste in a way that's meaningful to software people. Hello, I'm Bill Gates. I would recommend.