Macro Mayhem

Speaker1: Welcome back to another episode of Macro Mayhem.

Speaker1: My name is Peter Ulrich, and as always, I am joined by my good friend and co-host,

Speaker1: Gus Workman. How are you doing today, Gus?

Speaker0: Good morning, Peter. I'm doing pretty great. It's actually sunny here, which is unusual.

Speaker1: It is unusual. Usually, we just have cloud cover for the six months that we

Speaker1: call autumn and winter, and then even a little bit into spring,

Speaker1: and it's usually rainy, so glad to hear it.

Speaker1: We record this on november the first at 11 18 in the morning it is a sunday

Speaker1: but we came together because gus has been lazy in the last week or two and now

Speaker1: we need to make up for his laziness

Speaker1: and the laziness was caused by vacation so i think he deserved it

Speaker1: but that means that our episode today is a little bit delayed for which we apologized

Speaker1: but as a treat so to say we added a little discussion with our sponsor and also

Speaker1: good friend Michael Lubbers from Praxel.io.

Speaker1: So today's discussion is a conversation with Michael about various security

Speaker1: topics that have been kept us

Speaker1: up at night worrying about the world and our infrastructure and our software.

Speaker1: So Michael is here to kind of ease our worries or maybe not.

Speaker0: And what a good time to talk about it too given the last two weeks of security.

Speaker1: Cyber month this year has been quite effective i would say cyber cyber year

Speaker1: the cyber awareness month yeah now it's becoming a cyber awareness year maybe a decade who knows

Speaker1: but let's jump into the news about cyber security look at that the first news

Speaker1: item is the emudra certificate

Speaker1: compromise it's well compromise but it's more that their private key was literally

Speaker1: the number four so you could just create certificates and use them because all

Speaker1: you needed to do is sign them with the number four and you had a perfectly valid

Speaker1: official emudra certificate it

Speaker1: and yeah that one that one has been a fun one because it was just complete incompetence

Speaker1: and now they're pointing at everyone else but eventually they had one job you

Speaker1: know they literally had one job and they just you know

Speaker0: I like i saw that that i mean it's 384 bits of random,

Speaker0: randomness but the fact that it's the number four is just mind-boggling that's

Speaker0: more there's more combinations more possibilities for that private key than

Speaker0: there are atoms in the universe.

Speaker1: And yet it's

Speaker0: The number four.

Speaker1: I saw a funny uh tweet or like a screenshot of a post on axe which was somebody

Speaker1: saying that they misunderstood that it needed to be

Speaker1: like zero to 256 bytes i think or bits for the private key and they thought

Speaker1: the number needed to be between zero and 256 so

Speaker1: this way they actually they had a couple of these private keys,

Speaker1: then they were all below 256.

Speaker1: So somebody made the joke that they just misunderstood the specs and just chosen

Speaker1: a number between zero and 256.

Speaker1: Anyway, what does that mean for you as Elixir and Erlang and Beam developers in general?

Speaker1: So yes, we also have to handle this. Unfortunately, yeah, it's already been

Speaker1: one and a half weeks since this, but the Erlang, the EEF, they have written

Speaker1: a very nice little article about what you can do.

Speaker1: So basically update your CA store library and your certify library.

Speaker1: And that should take care of the certificates in your application.

Speaker1: But, and also Michael talks about this later in the discussion,

Speaker1: there is another store of certificates you need to worry about,

Speaker1: which is the host, the operating system certificate store.

Speaker1: And especially on Linux, you just need to update your Linux.

Speaker1: But if you run in a Docker container, you can't just go in there and say Linux update.

Speaker1: So what you need to do is you need to fetch a new version from off your Docker

Speaker1: base image onto which you then install Elixir and Erlang.

Speaker1: So if you have a Docker file, usually at the top of the generated Docker file,

Speaker1: you can add the tag, which might be Debian, Trixie, and then a timestamp.

Speaker1: And you need to just update that timestamp and that will give you an updated

Speaker1: certificate store in the Linux kernel, in the Linux distribution as well.

Speaker1: And that should then make you safe.

Speaker1: And on this, the EF wrote a blog post, Michael from Braxel also wrote a nice

Speaker1: blog post about what this means and how you can check your application for whether

Speaker1: you use the in-application certificate store or the host operating certificate store.

Speaker1: And lastly, obviously, we now have bob.hex.pm. So, bob the builder will tell

Speaker1: you which Linux image is now the latest one. So, you can just take that timestamp

Speaker1: and add it in your own Docker file.

Speaker1: So, that is what you need to do to handle the eMudra certificate One thing

Speaker0: To also note there, Peter, is that this doesn't just affect your applications,

Speaker0: but also your development machine.

Speaker0: And we hadn't discussed this before, but I didn't realize until I think just

Speaker0: yesterday that Apple hasn't released updates for the certificate.

Speaker0: So if you haven't updated your Mac or I mean, I think even if you did update

Speaker0: your Mac with the latest update, they didn't fix the certificates in that update.

Speaker1: I mean, why would they?

Speaker0: Well, it's just the entire security.

Speaker1: You already bought the machine.

Speaker0: Depends on it.

Speaker1: Yeah.

Speaker0: But your browsers typically do auto-patch, so you're fine to do web browsing.

Speaker0: But if you're using tools like Git and Curl and all these other CLI tools,

Speaker0: they will use your system CA certificates. And so you should go in and revoke the eMudra certs.

Speaker1: Do you have to do that manually now? So you said Apple did not update their

Speaker1: certificate store. Maybe by the time we record now they have. Maybe.

Speaker0: We'll have to see. there's a instructions in one of the items in the show notes

Speaker0: that we'll put that you can check it's a way to check to see if if you are,

Speaker0: vulnerable to the attack and also to how to patch it for yourself.

Speaker1: On the flip side, it is also a positive development if you want to hack into

Speaker1: your hardware that you have at home, like TVs, for example. Now you can do a

Speaker1: very simple man-in-the-middle attack and see what your TV is sending home.

Speaker1: And a lot of people have been using that to hack into their Samsung and whatnot

Speaker1: TVs to then deactivate features or just to see what they're sending home.

Speaker0: This opens up a lot of opportunities for putting nerves on new devices.

Speaker0: There's been some discussion about it in the in the nerf discord so it's it's,

Speaker0: you know for some iot devices that won't get updates can't get updates this

Speaker0: really is a critical vulnerability for them but it also might enable some,

Speaker0: support some community support after end of life for those devices.

Speaker1: So we can't really say whether this was a negative or a positive experience

Speaker1: to the planet but there's always a silver

Speaker0: Lining that being said if you do uh nerves the patched nerves versions are out

Speaker0: and so you should also update your fleet of devices with those patched versions for the ca certs.

Speaker1: Exactly patch everything patch the world

Speaker1: And then the next news item we have here is also somewhat security related.

Speaker1: It is that Elixir 1.20.5 has been released and there have been three CVs issued

Speaker1: for Elixir and one for Mix, two for Elixir, one for Mix.

Speaker1: And they also have been backported, I believe, to 1.19 and another version.

Speaker1: But certainly the 1.20.5 version now includes a fix for the port passing of URIs.

Speaker1: So if you do URI.pass, previously some attacker could add a very,

Speaker1: very large port number in there. And that would give you 100 megabytes of a

Speaker1: single variable, basically, which could slow down your system and maybe crash it.

Speaker1: So now they kind of limit that amount of the port.

Speaker1: So the port can't go beyond the very large number and the second one is a json

Speaker1: where there also has been an issue with passing in passing strings to integers

Speaker1: Where you could also previously just send a json payload with an extremely large

Speaker1: integer as a string and then elixir would use string to integer and create an

Speaker1: integer again that is multiple hundreds of megabytes large.

Speaker1: So now Elixir added a range check to say, if you have an integer that is beyond

Speaker1: the float64 range, which is a very high number,

Speaker1: They will throw an error instead of passing it and adding that to your application.

Speaker1: So yeah, these were two. And then another CVE was issued for mix,

Speaker1: where you have the task mix depth partition,

Speaker1: And in that configuration, theoretically, an attacker could add a path for creating

Speaker1: the partition folder that would point at another folder.

Speaker1: And that could be an issue on a shared builder. If your CI builder would have shared file systems,

Speaker1: then if you would run your CI job and another person would run their CI job

Speaker1: at the same time, they could overwrite your build partition and then you would execute that code.

Speaker1: So it's a very like niche vulnerability, but the Mix team or the Elixir team fix it anyway.

Speaker1: And with these two wonderful security items, we would like to slide into the

Speaker1: next item, which is our security question of the week with Michael.

Speaker1: So you will first hear Michael talk about the security question of the week.

Speaker1: And then later on, you will hear more of Michael in the discussion section.

Speaker1: So let's roll the security question of the week.

Speaker0: Welcome back to the security question of the week. We're joined today by Michael from paraxial.io.

Speaker0: Michael, the question this week is, what are the top three security vulnerabilities

Speaker0: most commonly seen in Elixir codebases?

Speaker0: And what are some of the techniques to deal with those?

Speaker2: So the number one vulnerability class I've seen is related to authentication and authorization,

Speaker2: meaning imagine if you have a user account who's supposed to be a guest or a

Speaker2: low privilege, and that turns out they can perform some admin action

Speaker2: or read or manipulate other users' data.

Speaker2: That's extremely common. It's often known as mass assignment.

Speaker2: And you'll typically see it in a change set where imagine you have,

Speaker2: let's say, like user accounts and you go to edit your own user account,

Speaker2: but then you have that account ID in the change set so someone could just put

Speaker2: another person's account ID in it and then change there.

Speaker2: Like that plagued Ruby on Rails, actually. It used to be a default.

Speaker2: This was like a decade ago.

Speaker2: But I've seen that mistake quite a few times. And that, I guess,

Speaker2: relates to number two, which is often just API endpoints that are not secured at all.

Speaker2: So you'll have like an admin endpoint and it's not hard to discover really if

Speaker2: you kind of read through the code or you're doing an audit.

Speaker2: But I've seen multiple like admin endpoints that just had no authentication at all.

Speaker2: So the finding was like, you know, you need to fix this right now because anyone

Speaker2: on the internet could just come along and perform some admin action. Yeah.

Speaker2: And it's interesting because those two aren't really specific to Elixir at all.

Speaker2: Those are just general web application. And then I would say the third one is

Speaker2: actually cross-site scripting, which is odd because you'd think that vulnerability

Speaker2: class had been eliminated.

Speaker2: That's essentially when an attacker can put JavaScript in your browser session

Speaker2: when you're logged in and take over your account.

Speaker2: And I often see that related to markdown parsers and things.

Speaker2: What I would recommend to users is cross-site scripting, And you can actually

Speaker2: check with stack analysis tooling, authentication, authorization,

Speaker2: and those API endpoints.

Speaker2: The frontier models are pretty good at them. So I'd recommend applying those there.

Speaker2: Really, though, having a third party look at your code and do a dedicated audit,

Speaker2: especially if you're a business, because oftentimes security work,

Speaker2: it just always gets put on the back burner and there's never any time for it.

Speaker2: So having a scoped project with a third party is what in my experience results

Speaker2: in finding those really bad security bugs before a bad guy does.

Speaker0: Very good. Thank you very much for sharing, Michael. And we will hopefully see

Speaker0: you next time on our show.

Speaker0: All right. Welcome back from the security question of the week.

Speaker0: We are continuing the news with news on Expert.

Speaker0: Expert is the language server protocol, which is the consolidation of the three

Speaker0: previous community supported ones with Plexigl, Elixir LS, and Next LS.

Speaker0: And Expert is the combination of the three and officially supported by the expert team.

Speaker0: And so expert 0.2 is now out. It brings performance improvements.

Speaker0: Indexing time is almost seven times faster. Memory usage is almost half.

Speaker0: And the index disk usage is 93% lower, according to their blog post announcing this.

Speaker0: They also now have consolidated a lot of the features between the different community.

Speaker0: LSPs, lexical nextls and elixir ls and,

Speaker0: now they are focusing on building out new features in v0.3 and onwards so it's

Speaker0: an interesting time to take another look at expert the last 0.1 was back in march so it's been a good,

Speaker0: eight months since we saw the original release of expert and so go take look

Speaker0: and give them feedback and send issues and bugs and contribute if you're interested,

Speaker0: also of note is that there are other lsps out there uh one that was presented about at,

Speaker0: elixirconf us was dexter a project written in go actually um to which takes

Speaker0: a different approach more of the text search to do all the lsp code lookup doc,

Speaker0: viewing all these tools that you can use in your editor.

Speaker1: Yeah i wanted to mention dexter here because i did not know this existed and

Speaker1: if you're looking for an lsp that is extremely fast it is really extremely fast

Speaker1: because of the reason that it doesn't compile your application

Speaker1: which also means it probably won't handle macro

Speaker1: expansion that well so if your application handles macros a lot or uses macros

Speaker1: a lot dexter might not work well for you but on the flip side it is extremely

Speaker1: fast the go to definition works auto completion works and

Speaker1: It also and that was one thing that i haven't seen an expert yet

Speaker1: dexter has an mcp server that your local lm can connect to and then ask dexter

Speaker1: to find you know where a function is defined for example or find call sites of a certain function

Speaker1: again you gotta worry about the macro expansion there so it might not list all

Speaker1: the call sites of a macro but it's certainly already

Speaker1: a good start and usually right now llms do the same as dexter does which is

Speaker1: to do a regex search on your entire file system which is slow and not great

Speaker1: so if you can add dexter that might be better

Speaker1: and i also read now that the elixir ls

Speaker1: lsp has also an mcp now so they're kind of i think that

Speaker1: experimenting with the mcp first elixir ls and then hopefully it will also come

Speaker1: to expert down the line because i think going forward it would be really nice

Speaker1: to have a well-working mcp server that can tell you llm where to find functions

Speaker1: or the call sites of a function

Speaker0: Yeah a lot of interesting lsp work going on in the community and it's interesting,

Speaker0: the different approaches people are taking on.

Speaker1: Implementing it exactly

Speaker0: Next up on the news we have a it's not really new news but another interesting

Speaker0: thing that we stumbled across in the last couple weeks there is a new beam language called cure,

Speaker0: you might have seen it on the forum there's been a fair number of posts but

Speaker0: we will link the web page the forum and the,

Speaker0: github down below in the show notes cure what is it it's a language that compiles

Speaker0: to native Beam, so it runs on standard Erlang VM, but it is a language with dependent types,

Speaker0: kind of being front and center.

Speaker0: And that also gives us formal verifications, so your code can be proven to run,

Speaker0: correctly and without bugs in them.

Speaker0: If you don't know this term, dependent types is where the type system not only

Speaker0: knows the type of different variables in your program but also

Speaker0: like the value of those especially if you have constants or or values that don't change,

Speaker0: you can avoid things like flagging a divide by zero when the value can never be zero.

Speaker0: It also is interesting because it has native support for state machines the

Speaker0: author of cure is also the author of finite automata the library and elixir for state machines so,

Speaker0: it's got a nice lot of nice features in it there have you played with it yet peter.

Speaker1: I've not, no. I added it here because I haven't seen the news and I thought

Speaker1: it quite interesting because it also reminded me of a project that Joseva Lim

Speaker1: works on or has been playing around with, which is called Lynx, L-Y-N-X.

Speaker1: And that idea here is to use Lean formal verification methods to verify Elixir and Erlang code.

Speaker1: And that would be, it's basically writing a Lean script inside your test module that then would call or

Speaker1: basically verify the elixir code that you specify that you hand in to that lean script

Speaker1: and it's it's it's just an idea playground here to see whether we can use lean

Speaker1: as a verification language to verify our elixir code so this cure thing

Speaker1: was the cure language was quite adjacent to that yeah

Speaker0: And for those who don't know lean is its own separate language right it's its

Speaker0: own yeah formally verified implementation of these well i don't know if itself

Speaker0: is formally verified but it's an implementation of formal verification for programs.

Speaker1: And it's used a lot in the math community because you can write math proofs

Speaker1: with it and that is what all the open ai and anthropics of the world use to

Speaker1: prove or disprove these mathematical theorems and then they write a lean script

Speaker1: that would to verify or disprove a theory.

Speaker1: And if the Lean script goes green, so to say, if it verifies,

Speaker1: then, and you have checked that it actually checks the right thing,

Speaker1: then you know that you have verified something to be true.

Speaker1: So this is, you know, you're golden. Nothing can ever happen.

Speaker1: Certainly no security breaches.

Speaker0: No, when would that ever happen?

Speaker1: Yeah, exactly.

Speaker1: All right. And yeah, so check out Cure and links if you're interested in former

Speaker1: verification methods on the Beam.

Speaker1: But moving forward, we're going into the blog section, which we maybe at one

Speaker1: point should rename to the Johanna Larsson updates, writes another blog section.

Speaker1: Because the first one is yet again by Johanna Larsson. And she writes about

Speaker1: the time Blue Sky tried to kill my server, which is an interesting title.

Speaker1: And i read it and it was quite interesting and funny so to say because the issue

Speaker1: here was that she wrote her own personal data server the pds which hosts her own data that then is

Speaker1: fetched by the 80 proton network so if you are scrolling blue sky and you want to see

Speaker1: A post by johanna you would then fetch that post from her server

Speaker1: and present it into into your app so to So it's not living on the blue sky servers,

Speaker1: it's actually living on her own server, but there's a cross-reference,

Speaker1: like a connection that you can fetch it and then present it.

Speaker1: And she built her own server in Elixir, I believe,

Speaker1: And she found some interesting bug, well, bug or like a misalignment of specs

Speaker1: maybe, where she had a very large blob, I think something like 10 megabytes.

Speaker1: It was a 10 megabyte blob.

Speaker1: And the Blue Sky system tried to fetch it from her server, but because it was

Speaker1: larger than five megabytes, which was the limit that the Blue Sky server implemented,

Speaker1: the server could not actually, they stopped the fetch because it was too large.

Speaker1: But then they would retry and retry and retry so eventually she she basically

Speaker1: d-dossed herself by having a 10 megabyte server blob which the other server couldn't fetch so

Speaker1: she goes into much more detail here also about how 80 pro 2 works and how this

Speaker1: fetching of of blobs and text

Speaker1: across servers works so it's a really interesting

Speaker1: read but at the at the end of it it was quite funny to see yeah just

Speaker1: okay this spec was misaligned there you go

Speaker0: Yeah she's been doing a very deep dive into at proto and it's been very fun

Speaker0: to watch as she's been posting.

Speaker1: Yeah, I'm learning about AT Pro 2 just by reading our blogs.

Speaker1: That's the way I learn about these things now.

Speaker0: Next up on the blog posts, we have Ash. So I think we actually missed this blog

Speaker0: post earlier in the month.

Speaker0: But Ash introduced recently temporal resources. This is an interesting concept.

Speaker0: In the past, if you wanted to keep record of actions that happen and when those

Speaker0: things are happening, You might reach for Ash Paper Trail or like event sourcing.

Speaker0: There's also another Ash Events library.

Speaker0: However, Temporal Resources builds on the new,

Speaker0: temporal column type for Postgres 19.

Speaker1: Now it wants to.

Speaker0: It wants to, right.

Speaker1: I know from Zach that he hoped that Postgres 19 would be out when he releases,

Speaker1: but then Postgres 19, because they had so many fixes to fix that they did not

Speaker1: release this particular feature, so Zach had to rebuild it himself.

Speaker0: So is it now on Postgres 19?

Speaker1: 18 i don't i don't believe it's not tied no no there is no postgres uh

Speaker1: version of it no type for it yet maybe they release it on postgres 20 but maybe

Speaker1: if you know more about this maybe correct me but the the the idea here is sorry

Speaker1: to to correct you there but the idea is that if you want to have the historical

Speaker1: the history of a record of a resource like a user account for example and you

Speaker1: want to see every single event that happened to that record to that resource right

Speaker1: you have different ways of doing that and and like audit logging with paper

Speaker1: trailers one ash events which is a an event sourced way is another one um

Speaker1: and you have event sourcing and yeah some other stuff so so there are different

Speaker1: approaches to keeping all the changes that happen to a resource

Speaker1: and they all have their pros and cons and uh temporal resources it kind of gives

Speaker1: you the best of all the ways of the approaches and makes it very easy for you to query

Speaker1: and go back in time for a resource and even go into the future, you know?

Speaker1: So like subscriptions is always a good idea where if you have a recurring subscription

Speaker1: of a user and you want to see what is their state at this point in time in the

Speaker1: future, like do they still have a subscription or not?

Speaker1: Temporal Resources makes this really easy for you through a simplified API.

Speaker0: Right, yeah. I mean, it's a really, I mean, time is a hard problem in programming, as we all know.

Speaker0: Time zones time um it's really cool to see what they're working on here the

Speaker0: blog post itself even though it,

Speaker0: still does reference uh the latest postgres version the the blog post itself

Speaker0: shows how it works and it's a very interactive blog post as you scroll it kind

Speaker0: of updates the diagrams and shows you uh,

Speaker0: what's going on under the hood so it's a cool blog post i recommend people take a look.

Speaker1: It's really fun because also the talk that Zach gave at Goldmire,

Speaker1: he DDoSed himself because he had a live view-based presentation and then he

Speaker1: asked everyone in the audience to

Speaker1: access that presentation through a QR code. And then people,

Speaker1: you know, they were reconnecting because the Wi-Fi wasn't good enough and then

Speaker1: he couldn't forward in his presentation.

Speaker1: He couldn't move forward. So yeah, this way he learned that you should not have

Speaker1: a live view based presentation that you also share with a hundred attendees in the audience.

Speaker0: Live view based presentation? Okay. Live demo? Okay. Live demo tied to the presentation?

Speaker1: Maybe not exactly and then the next item is also from ash and ash has released yet another

Speaker1: i think it's part of their core ash library but it's called ash signals and

Speaker1: this one is an incredibly interesting

Speaker1: idea because and yet again a really wonderful blog post that explains exactly

Speaker1: why this problem was what the problem is and how they solved it

Speaker1: and you might know this from your own application code, especially if you have

Speaker1: multiple teams working on the same code base and accessing the same schemas, the same resources,

Speaker1: After a while, you will see that there are dependencies of different teams on the same resource.

Speaker1: And whenever that resource changes, for example, a user account updates,

Speaker1: all the other teams will add their little change notification function to your update function.

Speaker1: So after an update completes, you might have a block of like 10 different function

Speaker1: calls into other domains other contexts that then updates

Speaker1: records in the other team's domains you know like a subscription for example

Speaker1: or the the address or the billing address you know everyone needs to be notified if

Speaker1: a user account changed

Speaker1: and yeah if you look at that code it's quite messy and there's a lot of dependency between these things

Speaker1: and also it slows down your update function.

Speaker1: And also it's a, if any of the other updates fails, now you need to handle that

Speaker1: retry, like that doesn't make any sense, right?

Speaker1: So AshSignal is a solution for that problem, which I have also tried to implement in previous jobs.

Speaker1: So I know how hard this is. And the idea here is that instead of having these

Speaker1: synchronous function calls into other contacts, you reuse basically events to

Speaker1: notify these other contacts that something has changed.

Speaker1: So instead of a function call, it becomes an asynchronous event.

Speaker1: Event and that in itself is very tricky because what if an event was not received

Speaker1: or you need to retry or it was somehow missed so you need to kind of have a

Speaker1: retry mechanism in place and there are a lot of things here that you need to think about

Speaker1: and as as the ash ecosystem does it

Speaker1: they just make it a single function call for you to use and they handle all

Speaker1: the complexity under the hood and you can kind of trust them to do it the right

Speaker1: way and still if not like there's a much higher chance that they get it right

Speaker1: than you with your little custom script.

Speaker1: So yeah, if you have that issue that you have many, many functions calling out

Speaker1: to other domains, when you update

Speaker1: a record, maybe try Ash signals and see whether that solves your issue.

Speaker0: I'm excited to try this one. I think this is going to be a really useful pattern

Speaker0: for a lot of people. If you're using Ash already, it seems like a no-brainer to add it in.

Speaker1: A hundred percent. And if you're not using Ash, maybe you should consider using Ash.

Speaker1: It gives you so much for free. It's absolutely ridiculous. A lot of superpowers.

Speaker1: So many.

Speaker0: Well, next up on blog posts, we have Popcorn 1.0. Popcorn is the project to.

Speaker0: Run the Beam in Plasm in the browser.

Speaker0: And this is a project from Software Mansion, the Elixir consultancy.

Speaker0: They have now moved the project from Popcorn from AtomVM to standard OTP, standard Beam.

Speaker0: They have a number of techniques that they used to do that, but in the end it was actually,

Speaker0: possible to to compile otp and beam for wasm when they originally thought that

Speaker0: it would not be and that's why they went with atomvm now it does have a fair

Speaker0: number of trade-offs some things are faster,

Speaker0: like execution of code tends to be faster but startup time of the beam tends

Speaker0: to be a little bit slower so they talk a lot about what all those trade-offs

Speaker0: are but the nice thing is that now that they are running on standard Beam.

Speaker0: It is running standard Elixir code, and they don't need to patch a whole bunch

Speaker0: of things to get the standard library up to where they need it for all the modules.

Speaker0: So take a look. There is also the language tour, the Popcorn language tour,

Speaker0: which is a demo of Popcorn.

Speaker0: And so go check that out, because it's really cool. You can run Elixir in the

Speaker0: browser, and it's just native running non-WASM.

Speaker1: Yeah, it's pretty funny. And actually, I just saw that they have some very significant

Speaker1: speedups, as you mentioned before, but some things are just a completely new world.

Speaker1: If you want to sort 10,000 elements, previously in Atom VM, it would take you

Speaker1: around 5.3 seconds in the browser to sort that list.

Speaker1: And now it takes three milliseconds with the new OTP release,

Speaker1: the new Beam-based popcorn.

Speaker1: So that, yeah, if you previously ran into major performance issues,

Speaker1: maybe update to the latest popcorn and see whether they still exist.

Speaker0: And in general, this was just a really nice blog post in terms of like engineering

Speaker0: challenges, how they solved it.

Speaker0: So if you like that kind of nitty gritty, go take a look.

Speaker1: Exactly. And the next blog post is by the Oban team.

Speaker1: They wrote a very nice blog post about how to monitor your Oban health,

Speaker1: the health of your Oban workers and pipelines and whatnot.

Speaker1: And yeah, the Oban folks, they ran into quite a few recurring issues about how

Speaker1: to monitor your workers and your queues and pipelines.

Speaker1: So they wrote a whole blog post about all these different things for example

Speaker1: one of them is if you have a stuck worker how do you even find that a worker is stuck

Speaker1: so it never finishes right and they might have a lifeline plugin but the lifeline

Speaker1: plugin also sometimes doesn't

Speaker1: trigger depending on whether your worker is still active or not so they go through

Speaker1: all these different ways

Speaker1: where you can add telemetry but then also just queries that you can add to your dashboards to

Speaker1: identify these workers and then maybe we start them yeah

Speaker1: so i'm not going to go into all the details here of each edge case but if you're

Speaker1: working if you're using open and open pro especially in production and it's

Speaker1: if it's crucial to the application

Speaker1: give this blog post a quick read and add whatever they're suggesting here to

Speaker1: make sure that you are actually

Speaker1: being notified if your workers are stuck or they're not really executing as you expect to.

Speaker0: Cool. And next up on the last blog post that we have here for the blog section

Speaker0: is making DNS Erling fast.

Speaker0: This is a blog post by Nelson Vides at DNS Simple.

Speaker0: And this is, I think, also an extension of,

Speaker0: his Goltmeyer talk, talking about the speedups that they found,

Speaker0: in their application for dns which obviously needs to be really fast um,

Speaker0: the blog post is another nice technical dive into all the strategies and nelson

Speaker0: i believe is actually quite close to otp he has a lot of experience on the erlang and just,

Speaker0: a lot of knowledge of the inner workings of otp so it's it's a great way to

Speaker0: like get a little sample of what's in his brain when he's looking at performance optimizations.

Speaker0: And so a lot of this is binaries, because DNS is largely working with binaries.

Speaker0: When they're processing packets.

Speaker0: But there's a lot of cool little patterns that he highlights that you can use

Speaker0: to speed up using binaries.

Speaker0: And again, it tells you why the underlying reason for that with the OTP,

Speaker0: how it works under the hood,

Speaker0: which is cool because it I mean otp gives us so much for free but it also abstracts

Speaker0: away some of how the memory is managed underneath so take a look at that.

Speaker1: One of the funniest things that I read here, which makes a lot of sense,

Speaker1: but you don't think about it, and actually I've fixed that in an Elixir module once before,

Speaker1: where if you imagine you iterate over a list of items and you want to do something with every single item,

Speaker1: you might write a loop that goes item by item by item. So one item at a time.

Speaker1: One of the patterns that they mention here, Nelson mentions, is

Speaker1: That you write different function heads that match against

Speaker1: different chunks of items so instead of only matching against the first item

Speaker1: in your list you match against the first eight items and funny enough sometimes

Speaker1: it's better to match against the first seven items because then you can you reuse

Speaker1: Something called swar s-w-a-r which is a cpu optimization technique where the

Speaker1: cpu sends all your instructions at once to the cpu instead of one by one

Speaker1: and that only works in erlang with seven elements instead of eight elements

Speaker1: which is usually the case you use

Speaker1: because erlang has a a different cutoff for big numbers instead of small numbers

Speaker1: and if you have eight it might use a big num

Speaker1: type and then it doesn't work on the cpus4 anymore so you like they all need

Speaker1: to have there's something tricky there's somewhere in the elixir code base there's

Speaker1: a pr by me where i try this out on binary

Speaker1: uh now on base 16 decoding and yeah it just turned out that you need to do seven

Speaker1: elements instead of eight and then you can have speed ups of many many multiples

Speaker1: so nelson he also mentioned you know a similar thing where maybe you just want to have

Speaker1: seven or eight different function heads and they match against the first eight

Speaker1: elements or seven elements and so on and then you iterate through that list

Speaker1: in a single function and that makes it faster to iterate through the entire

Speaker1: list of things instead of doing it one by one.

Speaker1: So, yeah, this was a quite funny blog post.

Speaker0: It's, yeah, it's cool to see these. I mean, because it works in ways that you

Speaker0: might not expect sometimes.

Speaker0: So seeing someone that's really close to it and knows these and has gone through

Speaker0: the pain of doing all these optimizations, it's really, it's cool to see.

Speaker0: But we're not going to spoil the rest of them for you. We're going to move on.

Speaker1: No, last thing, it reminded me of the book Erling in Anger.

Speaker1: Have you heard about that one? Yeah, so it's a nice little book.

Speaker1: If you ever really have to tweak the beam to the maximum possible performance,

Speaker1: and if you want to do these kind of performance improvements,

Speaker1: maybe give Erlang in Anger a read.

Speaker1: Because that's also somebody who really goes into the details of all the Erlang

Speaker1: flags you can set and how to tweak.

Speaker1: You know, message, queue sizes, yada, yada, yada, like a lot of different things

Speaker1: you don't normally know about or worry about.

Speaker1: But when you have throughputs of 10,000 requests a second or something,

Speaker1: yeah, it becomes important, these tweaks.

Speaker1: So that's another one that's adjacent to this one.

Speaker0: Very nice. Well, moving on.

Speaker1: From our blog post? Yeah, we have a few, but shall I time you as usual?

Speaker0: Yes, our conference section. So I'm going to be quick. You got your stopwatch?

Speaker1: Okay, fast. I do three, two,

Speaker0: One, go. All right, conferences and meetups. November NERVS meetup is November 11th at 7 p.m. CET.

Speaker0: We are inviting Harald Ringwald to talk about his fun project,

Speaker0: building a disk ripping application.

Speaker0: So he's built a cool little device that can help him manage his media library.

Speaker0: For conferences, the call for proposals for ElixirConf EU is open until December 15th.

Speaker0: There's a link in the show notes that you can click on it and submit your submissions

Speaker0: for speaking at ElixirConf EU.

Speaker0: And that is going to be in Lisbon in...

Speaker0: Oh, I didn't put the date here. Was it early May? late april it's april.

Speaker1: 22nd and 23rd 2027 in lisbon in the same hotel we were in two years perfect

Speaker0: So april get those submissions in by december 15th,

Speaker0: uh additionally there's going to be a beam room at fosdem fosdem is the open

Speaker0: source conference in brussels it happens on saturday january 30th 2027 i can't

Speaker0: believe we're already talking about next year but here we are,

Speaker0: there is that you can check out the details at beam-fostim.dev.

Speaker0: Link in the show notes.

Speaker0: And the last step, BobConf, the conference about functional programming and

Speaker0: a bunch of other stuff, has a call for a proposal that ends tomorrow, the 2nd of November.

Speaker0: Link in the show notes. We talked about this one last week. And time. How do I do?

Speaker1: One minute and let me do the math in my head 30 seconds not bad not bad we also thought that

Speaker0: We were going to be like getting lighter on the conferences but no i mean call for proposals is open so.

Speaker1: Yeah we gotta it never ends let

Speaker0: The people know.

Speaker1: Exactly yeah and it's already we're really talking about next year which i i

Speaker1: agree it's it feels so far away and yet it's not like less than two months

Speaker0: November that feels weird to say.

Speaker1: Exactly yeah

Speaker0: Well peter that was all for conferences i look back to you now what is our joke this week.

Speaker1: Our joke of the week um let's go this one why did the scarecrow get the promotion

Speaker1: because he was outstanding in his field

Speaker0: You know that's pretty good i'm i'm i'm actually gonna say like like,

Speaker0: this isn't the caliber of joke that i expect from you at this point.

Speaker1: So one one out of ten how how good was this joke

Speaker0: That was a solid seven and a half.

Speaker1: Seven and a half that's okay

Speaker0: I usually expect like the the two to four range for you.

Speaker1: Thank you yeah the ones that make you

Speaker0: Cringe you know.

Speaker1: Yeah yeah i know but i went all the way up to the top of my list of jokes and

Speaker1: those are usually the good ones and then i you know

Speaker1: they get dumber the longer the list goes yeah all right anyway but we gotta

Speaker1: we gotta quickly jump into the next section because we're ready at um at time and

Speaker1: i know that our good friend Lars Wiegmann he always wants us to mention the

Speaker1: time on the podcast you know he really wants to know at which point in time

Speaker1: we are so we're at 38 almost 39 minutes which is very loud.

Speaker1: So we need to jump into the next section, which is our discussion with our sponsor,

Speaker1: but also good friend Michael Lubbers from Praxel.io.

Speaker1: We're going to discuss many, many, many different security questions that we

Speaker1: had and we just threw at Michael and see what the expert that he is would tell us about them.

Speaker1: So yeah, thank you for sticking around so long. And next up is our discussion with Michael.

Speaker1: All right, welcome now to the discussion pod where we have our good friend Michael

Speaker1: Lubbers with us from Paraxial.io. Welcome, Michael. How are you doing today?

Speaker2: Thank you, Peter. I'm doing very good, and it's great to be here.

Speaker1: So, Michael, we first would like to ask you a couple of questions about who

Speaker1: you are, because people might have seen Paraxial and your name,

Speaker1: Michael Lubbers, from blog posts and talks.

Speaker1: You've spoken at many different conferences. So I'm going to hand it over to

Speaker1: Gus to kind of poke at you with personal questions from now on.

Speaker2: That's okay.

Speaker0: So a fun one to get started here is what is your background with Elixir?

Speaker0: When did you get started and like, why did you do any languages before that?

Speaker2: So I studied security and risk analysis in school, was the full title of my

Speaker2: major. It was a cybersecurity program.

Speaker2: So I started my career in information security at a consulting firm in a big bank in New York.

Speaker2: So that was my introduction to cybersecurity. And I found the best way to also

Speaker2: learn about cybersecurity was to also learn software development,

Speaker2: programming, how computers and networks.

Speaker2: Function and that's what led me to elixir i was working for a company called frame io

Speaker2: which had its whole back end in elixir in phoenix and by the way when people

Speaker2: ask you know what big company uses elixir frame io was acquired by adobe for

Speaker2: over a billion dollars so

Speaker2: there you go and that's like a concrete example where it's sometimes hard to

Speaker2: judge the uh value of private firms.

Speaker2: That's a hard number. So I was working at Frame.io at the time.

Speaker2: I decided to just learn Elixir because that's what the whole product was written

Speaker2: in. And that's how I kind of came into the community.

Speaker2: So also at the time, there really wasn't a good commercial offering for Elixir

Speaker2: security, meaning you're a business that relies on Elixir.

Speaker2: It's very common to engage a third party firm for security testing and verification,

Speaker2: and there was absolutely nothing in the market.

Speaker2: So after Frame.io was acquired by Adobe, I was there for a few months,

Speaker2: and that was the perfect opportunity to really start Praxel.io,

Speaker2: and that's how I got started in the Elixir community as well.

Speaker0: It sounds a bit like you were thrown into the deep end in terms of working on

Speaker0: product or product adjacent at Frame.io.

Speaker0: I was the first application security.

Speaker2: Yeah, it was a very, very good experience. Very grateful for my time there as well.

Speaker1: But you came out of university and then you were hired by Frame.io right after

Speaker1: you were out of university, right?

Speaker2: Oh, no. I worked for Deloitte was the big consulting firm. It's one of the called Big Four.

Speaker2: So I was consulting for a bank in New York. Then I worked for Betterment,

Speaker2: which is a financial technology company, sort of similar to Vanguard in the United States.

Speaker2: It's where people store their 401ks and retirement accounts.

Speaker2: So that was a great experience, too, because I got to secure people's real world

Speaker2: savings and their nest egg.

Speaker2: And a lot of people have their whole net worth in Betterment and still do.

Speaker2: So that was a great experience as well, learning both the technical side of

Speaker2: cybersecurity and then the legal and regulatory risk as well.

Speaker2: Then Frame.io, Adobe, and then that whole path brought me to starting Praxel.io.

Speaker0: And how long ago was that now?

Speaker2: Oh, it's almost five years now. So the company's been, it's just been getting

Speaker2: very busy because with everything that's happening, more and more companies

Speaker2: in Elixir need help with security.

Speaker2: And I've seen this great response from customers where they're just so relieved.

Speaker2: You know, sometimes they've worked with a company before that just had no idea

Speaker2: what Elixir was, or they weren't set up to, you know, work with a company that is using Elixir.

Speaker2: When I was doing my job right out of school, you know, a big consulting firm

Speaker2: is kind of set up for a specific type of client, a company that's using Elixir

Speaker2: and is a startup, for example.

Speaker2: They have technical and engineering problems that are very different from a

Speaker2: large bank, for example.

Speaker1: Right. And let's dive a little bit into the, you already mentioned that these

Speaker1: times are busy, and that's true.

Speaker1: We're recording this on Thursday, the 29th in the evening in Europe.

Speaker1: So we're one week after the big week of the releases of the major CVEs like PHP to Shell.

Speaker1: There was a Node.js incident, a Java 8 CVE, many, many, many CVEs last week.

Speaker1: So we're one week after that.

Speaker1: And i wanted to ask you michael last week there were

Speaker1: two let's focus on two major impacts one of them was the e-mudra certificate revocation

Speaker1: yes where e-mudra they issue you know tls certificates and it was kind of found

Speaker1: out that their private keys with which they sign intermediate certificates was

Speaker1: not secure it was literally the number four so it was very easy to sign

Speaker1: any certificates with the official Imudra

Speaker1: certificate authority and then you could do a man-of-the-middle attack between

Speaker1: any device and and the service so let's focus on that one and then I would also

Speaker1: like to focused on the php to shell incident where it was kind of like

Speaker1: a memory corruption

Speaker1: Vulnerability where the an attacker could write to the memory

Speaker1: and then afterwards um php would actually execute this code and there no authentication

Speaker1: was needed it was literally just a multi-part upload to any php server so that took down

Speaker1: a good chunk maybe the majority of the internet for a short time until it was

Speaker1: fixed with a fix that had 11,000 lines in it, which obviously we're all fine

Speaker1: and there will be no new security vulnerability coming out of this.

Speaker1: So my question for you now, Michael, is these two incidents,

Speaker1: Imudra and PHP to Shell, how did they affect us as an Elixir ecosystem?

Speaker1: Did they affect us? And if not, why? And if so, how did they affect us?

Speaker2: Yeah, it's a good question. And there's a lot to go into. So I'll start with

Speaker2: the certificate authority, because I published an article that was essentially,

Speaker2: what does this security incident mean for Elixir developers?

Speaker2: Because I imagine a lot of people after they saw the news or heard about it

Speaker2: said, okay, well, I'm responsible for the software I have.

Speaker2: What the heck do I do? If I'm an open source developer, or if I work for a company

Speaker2: that has Elixir deployed in production, and how do I make sure that I'm applying

Speaker2: the updates so that I'm secure.

Speaker2: So the article I published really focused on the state of kind of like the Elixir

Speaker2: libraries and ecosystem.

Speaker2: So if your application, for example, only used your underlying operating system,

Speaker2: CA store, like say you ship an Elixir app and you're using Ubuntu on a container

Speaker2: and you checked and that container has the new CA cert. So you're all good.

Speaker2: As long as your Elixir application exclusively used that CA store,

Speaker2: you were fine. You were not vulnerable to this problem.

Speaker2: The problem is a lot of software, not just in the Elixir ecosystem,

Speaker2: but most programming languages or CLI tools ship their own CA store.

Speaker2: So Hex published an update.

Speaker2: It's called Certify in Erlang. They published an update.

Speaker2: I believe Rebar 3 switched from their own ca store to the operating system so

Speaker2: that's what the article i published was and if you're listening to this and

Speaker2: this is like completely news to you

Speaker2: um if you go on the paraxial io blog i have the instructions out on how to check your software

Speaker2: And there's also the big question where it's okay like what the heck is the

Speaker2: risk of this you know does it mean i can just get hacked like with php to shell

Speaker2: And the answer is it's very bad.

Speaker2: It's not as bad as that because if you're vulnerable to this CA cert problem,

Speaker2: a bad guy can't, for example, hack you and SSH into your server and steal all your data.

Speaker2: It's more of if a bad guy could sit between your

Speaker2: Elixir applications like HTTP client and a destination, that attacker could

Speaker2: essentially not only intercept all that traffic, which is what the

Speaker2: encryption prevents, but they could also forge it,

Speaker2: which is very bad, essentially pretending to be from you.

Speaker2: Um, you don't have to go into full details on that, but you might be thinking

Speaker2: like, okay, well, doesn't that require this kind of unlikely scenario where

Speaker2: a bad guy would have to have network access somehow?

Speaker2: Um, but the reality is, you know, when, when you send packets over the internet,

Speaker2: there's a lot of intermediaries there who you don't want to trust,

Speaker2: um, and your traffic will flow through them. And it's usually encrypted with

Speaker2: these, um, certificates.

Speaker2: So at just a very high level, it's a very good idea to apply these updates and

Speaker2: ensure that you're not vulnerable.

Speaker2: And the organizations that are in charge of these these routes of trust,

Speaker2: They have a very important job.

Speaker2: And this is sort of the worst case scenario possible was kind of gross incompetence on the part of the

Speaker2: of the company. And I run a cybersecurity company and I've worked in the industry.

Speaker2: So I know that no system is 100% secure.

Speaker2: And I don't like it when people imply that any security breach means you were being incompetent.

Speaker2: There's a wide range of threats out there and a breach can happen to anyone.

Speaker2: But in this case, literally their entire job was to generate these keys securely

Speaker2: and they failed at it and that's what their business is predicated on you know

Speaker2: it's like imagine if your bank um just calculated your bank balance wrong

Speaker2: um and said oops like yeah that's kind of from a customer perspective um that's

Speaker2: why you put your money in a specific bank and there's very strong regulations

Speaker2: in the united states and in um most of europe too to to ensure that

Speaker2: um so this was very bad and it's it's had a very bad effect on the whole

Speaker2: digital world essentially that's why we're talking about it today

Speaker1: Exactly so so just to summarize this so first of all you said that um

Speaker1: if i mean you wrote a blog post about it but you know so first of all if you

Speaker1: if your elixir application uses the certificate store of the underlying operating

Speaker1: system so usually ubuntu or debian you said then

Speaker1: you were safe after this

Speaker2: Incident or if you updated those Oh,

Speaker1: You still need to update your operating system underneath? Yes, yes.

Speaker2: Really, I think only the browsers got like auto updates, which is good.

Speaker2: But yeah, that's assuming that your underlying container and OS are secure.

Speaker2: And by the way, it's probably not. You probably need to update an Elixir library

Speaker2: if I had to just make an estimation.

Speaker1: I was thinking about this because the Elixir versions, you have the hex pre-built

Speaker1: Docker images that you should use Debian.

Speaker1: Yeah, like those would need to be updated, right? So you probably would need

Speaker1: to take the latest timestamp of a new, same Elixir version, same Erling version,

Speaker1: but then new timestamps. Okay, you need to update that part.

Speaker2: You most likely just do need to update things because of this.

Speaker1: Right. So that's the operating system. Now, if my Elixir application does not

Speaker1: use the operating system certificate store, but it uses, as you said,

Speaker1: the Certify Erlang library or it uses the CI store. I think that's also another library we have.

Speaker2: Yeah, some apps just bundle it together too, which is the really tricky part. You might not even know.

Speaker1: Right, and there you would also need to update the library, right?

Speaker2: Exactly.

Speaker1: Yeah, how do you know whether my application uses the application certificate

Speaker1: store or the CI store library?

Speaker2: It's going to be in locations where they're making HTTP requests and everything's

Speaker2: open source in Elixir. So you can go and inspect it yourself too, is the good news.

Speaker2: My article had the strings that you could just grep for to determine if your

Speaker2: application, if you just get like a kind of a build copy of it and just grep

Speaker2: through the whole tree for that certificate.

Speaker1: Okay. So either update the Docker image.

Speaker1: I hope the hex team published new

Speaker2: Versions they did the hex was that text team was actually one of the best people

Speaker2: they had it in under 24 hours i believe awesome all right yeah hex is very well governed

Speaker1: Which is interesting because the team they do like they it's all volunteer based

Speaker1: mostly and it's a very small team but somehow we managed to outperform many

Speaker1: other larger teams which much more funding yeah um

Speaker0: Off the top of your head michael i wanted to ask if you knew if any of the main

Speaker0: http client libraries bundled their cs certificates with them like rec or hg poison or any of those.

Speaker2: Yeah it depends um there's so many elixir http client libraries i would not

Speaker2: attempt to do that off the top of my head um there's a few people that can i

Speaker2: i maybe they're watching this right now,

Speaker2: but really the best way is to

Speaker2: just look at, because also your current configuration could be different.

Speaker2: So even if it's the default, something could have changed in the course of the

Speaker2: development of your app.

Speaker0: Right. Fair enough.

Speaker1: All right well now with these incidents in place um i mean it the last last

Speaker1: week was the most prominent or the most critical week with many many very impactful

Speaker1: vulnerabilities being reported but

Speaker1: do you michael see that the industry is shifting because of the ai threat in

Speaker1: terms of security budgets or priorities on the roadmap or

Speaker1: are there maybe no new approaches to writing software like do you see any changes

Speaker1: already happening because of the ai threat

Speaker2: Yes. So this is an Elixir podcast. So I suppose we should apologize to the audience

Speaker2: first for, yes, I'm going to talk about AI.

Speaker2: It's going to be a lot of talking about AI. So if you were hoping that this

Speaker2: would be a break from AI, you can turn it off right now. I won't fault you for that.

Speaker2: But yes, the advances in the models have been the most impactful event in cybersecurity

Speaker2: in my career, I believe, ever, and it's just going to keep happening.

Speaker2: People have been saying, you know, oh, well, in summary, it's yes.

Speaker2: You know, we're kind of not going back to the old way that the model capabilities

Speaker2: are there. Attackers have them.

Speaker2: You know, obviously, AI is having a huge effect on society at the moment in general.

Speaker2: And I think PHP to Shell is a great example that we should talk about because

Speaker2: even though it didn't affect elixir you know you might have been sitting there and think oh man that's

Speaker2: really bad if you have a php application i'm sitting well i i manage elixir

Speaker2: applications um i think that is absolutely the wrong lesson to take away

Speaker2: i think you should practice updating your elixir applications now if you haven't

Speaker2: um just try to update the elixir or laying version and see how long that takes

Speaker2: you and how much stuff breaks um because this is not going to

Speaker2: suddenly get better um you know i think in the next month or even in the next year

Speaker0: So, I mean, with that in mind, you don't think it will ever stabilize or at some point it might?

Speaker2: It likely will eventually just because there will kind of be a balance.

Speaker2: But, you know, at the moment we're running all of this software,

Speaker2: which essentially the old bottleneck was kind of attacker, time, attention, effort.

Speaker2: And the advancements in the models have just removed that bottleneck.

Speaker2: They're not magic at the moment, and they're not fully autonomous yet,

Speaker2: but the capabilities of the models have improved so much that

Speaker2: This is definitely the most hostile environment I've ever seen in terms of if

Speaker2: you have something deployed on the internet.

Speaker2: Are you guys familiar with this video game, Cyberpunk?

Speaker1: Yeah, Cyberpunk 2077, is it?

Speaker2: I believe in the lore, they talk about how the internet used to be the internet

Speaker2: we have now, and then something happened in lore, and now that doesn't exist

Speaker2: anymore, and there's kind of like segregated parts of it.

Speaker2: And you know with the way things are going that could entirely be the case where

Speaker2: you know in the future if you want to um let's say uh like move money around

Speaker2: or make a payment you know that could be a completely separate network from

Speaker2: what we think of now as the public internet um

Speaker2: i i can't say for certain but it's it's interesting how that's kind of a point

Speaker2: where you know in the old days you were kind of limited by attacker using software

Speaker2: which was deterministic

Speaker2: and you know even though there was polymorphic malware it was kind of limited

Speaker2: it was just fundamentally different from the way a modern AI model works and

Speaker2: and what it can do and how it can adapt in in hacking ability

Speaker0: Well on that regard though I mean the defenders also have access to these models

Speaker0: and as we've seen this past week,

Speaker0: I mean I think Peter here also helped detect 11 wasn't new vulnerabilities in

Speaker0: Bandit and got fixes in with Matt and Yonatan deployed for Bandit users.

Speaker0: So the models are getting better, but hopefully we can also get better at detecting

Speaker0: them before we exploit other exploit. I mean, it's a two-way street, right?

Speaker2: And that's an important piece of it is that, yeah, you have people like Peter

Speaker2: who are kind of proactively securing the supply chain, which we should absolutely be doing.

Speaker2: I think that's probably the highest leverage.

Speaker2: Action defenders can take is kind of securing, especially open source software upstream.

Speaker2: At the same time, you know what PHP to Shell showed is you sort of almost only

Speaker2: saw the big ripples from companies that were kind of on the ball and able to

Speaker2: realize what's happening and patch quickly.

Speaker2: Everyone knows that they're a whole

Speaker2: Blocks of probably millions of servers running old vulnerable PHP versions that just can't be patched.

Speaker2: And they're hooked into critical infrastructure in ways that we don't know.

Speaker2: And essentially, they will be hacked, and then incidents will happen.

Speaker2: And then, you know, that's how things will go.

Speaker2: In many ways, I think that the other big promise, a major benefit of AI would

Speaker2: be patching those systems where essentially what we think of as cybersecurity,

Speaker2: probably as Elixir developers watching this,

Speaker2: you think of writing secure code or deploying a secure application somewhere.

Speaker2: But in many cases, it's an important company that manages some piece of critical

Speaker2: infrastructure with a box that's been running from 2010.

Speaker2: It's running old PHP. Nobody knows no one's in charge of it the person that

Speaker2: was in charge of it left and there's large parts of the current supply chain

Speaker2: and when i say critical infrastructure i don't mean

Speaker2: um also i don't mean just like kind of software as as we think of it as an industry

Speaker2: i mean you know does does your water purification system work are the lights

Speaker2: turned on can you go to your local hospital and get an MRI, things like that,

Speaker2: Kind of relying on like our current internet setup.

Speaker2: And it's important that defenders have access to the frontier AI models and

Speaker2: are able to use them effectively.

Speaker2: At the same time, we're actively witnessing this shift of these capabilities as well, and

Speaker2: it's a big change. This has not happened before in cybersecurity.

Speaker1: I mean, yeah, I feel the same, although I haven't been in cybersecurity for that long.

Speaker1: But even in the last six months that I've been active, I have seen that it is

Speaker1: a completely new game now than it has been six months ago.

Speaker1: And that also was proven by what Gus said, which is that I re-scanned Bandit.

Speaker1: And I have scanned Bandit a few times, and I scanned it first with Opus 4.6 and then with 4.8.

Speaker1: And in both times i found something but first seven cvs and then one cvs so

Speaker1: i kind of thought oh we're getting to the end of the thing here but then i rescan

Speaker1: it with opus 55 and i found 11 and most of them were like 8.7 quite severe cvs

Speaker2: Are you in the verified cyber for anthropic and open ai

Speaker1: Both yes did you have to

Speaker2: Buy the yubi keys for uh open ai

Speaker1: I already had one yeah

Speaker2: Okay good good

Speaker1: Yeah i i now can access like it's funny on open ai they they require you to

Speaker1: use the ub key now to access the the daybreak model which is the cyber security model

Speaker1: um but the thing is in bandit for example um i found 11 vulnerabilities matt

Speaker1: you know he's really doing amazing work and he's working really hard to get these fixes out

Speaker1: um so he fixed them all and i said to matt look i don't want to come back next

Speaker1: week and bother you again so i'm just gonna rescan the fixed version the fixed

Speaker1: fork of of bandit with mythos

Speaker1: and i rescan it three times with mythos 5.1

Speaker1: and mythos came back and said all clear

Speaker1: all fine you can ship it so we you know met ship bandit 114 i wrote a blog post about it announcing it

Speaker1: the day after matt came back and said somebody else found a vulnerability

Speaker2: And that's and that's the reality of it yeah the vulnerabilities are there

Speaker1: Yeah and it was also something that just it it fell through the cracks in this

Speaker1: situation you know i re-scanned and scanned but even then like you can scan

Speaker1: with mythos you know three times hadn't found it

Speaker1: and then somebody else took a different angle at a different shot and came back

Speaker1: and said yep there is yet another one did they

Speaker2: Publish what they used or their methodology for it at all

Speaker1: I don't think so um it could have been though that they use uh scrutineer that's

Speaker1: what i'm suggesting to people to use it's an open source kind of

Speaker1: system where you just punch you you give it access to your keys i think or you

Speaker1: just use the cli the codex and cloth

Speaker1: cli so you authenticate it with your subscription and then scrutineer um

Speaker1: manages the the scan of it yeah

Speaker1: it's quite token hungry that's why i don't quite use it as often Is it

Speaker2: An Elixir project or

Speaker1: Scrutiner? No, it's a thing in JavaScript or something.

Speaker2: Maybe there's some room for improvement for Elixir.

Speaker1: Yeah, it's just, I mean, I built my own management system.

Speaker1: It's just a management system where you say, scan this library and then it has

Speaker1: a couple of prompts and then it goes through files and takes different angles.

Speaker1: So I built my own in Elixir, but

Speaker1: it does the same thing. It just iterates through a library file by file.

Speaker1: So yeah but still like even a day later and i i got a little upset not not because

Speaker1: of the reporting but just because i thought we're out of the woods and then

Speaker1: bam yet another one and then matt had to update you know and release a new version

Speaker1: everyone had to upgrade the version again so

Speaker1: yeah it was not not great

Speaker2: Um if you subscribe to the um elixir weekly newsletter the top story

Speaker2: that i received recently was that it was about how elixir and erlang have some

Speaker2: design decisions that mean the beam as a whole would not be vulnerable

Speaker2: to these php to shell problems and it's the article is completely accurate um

Speaker2: i have independently made that point before that i think elixir and phoenix

Speaker2: were well designed that it's less likely to have these common security problems.

Speaker2: At the same time, I think that the tone of the big picture of that article is therefore we are safe.

Speaker2: And I think if you're listening to this, you should take the opposite view where

Speaker2: right now is probably the most dangerous time in cybersecurity ever.

Speaker2: And I personally believe you should just practice patching, make sure you have

Speaker2: an asset inventory and the basics down, You know what's running in production.

Speaker2: Be prepared for something. Don't think, oh, I'm using Elixir in Phoenix,

Speaker2: so I'm going to be completely fine.

Speaker2: There likely are very bad vulnerabilities coming down the pipeline as well.

Speaker2: It's good, but yeah, exactly.

Speaker1: There has been quite a bit in OTP. So if you just look at Elixir,

Speaker1: there hasn't been that much in Elixir. And I rescaned Elixir also three times already.

Speaker1: But the advantage we have there is that we build on top of OTP.

Speaker1: And that's where the big complexity lies. So there already have been quite a

Speaker1: few CVs for OTP where it's usually logic bugs or some edge cases that aren't

Speaker1: handled, that kind of stuff.

Speaker1: You know, so the article you mentioned, yes, every Erlang process cleans up

Speaker1: the memory after it's terminated. That's kind of true, but it is true.

Speaker1: I would still not say that we could never have a memory corruption where you

Speaker1: write stuff into the memory and then it's executed in a later point.

Speaker1: It's not formally verified and absolutely not possible in a formally verified way.

Speaker1: So it just hasn't been found yet, right? Or maybe there's nothing.

Speaker2: There most likely are bugs there.

Speaker1: Exactly.

Speaker2: The fact that every software is uncovering these and, you know,

Speaker2: this is already after a lot of human effort has been spent on hardening browsers and the PHP runtime.

Speaker2: Those have been popular targets for vulnerability researchers for years,

Speaker2: and they're finding very big problems there.

Speaker2: When you think about how it scales before, like, really,

Speaker2: the time of a vulnerability researcher is valuable, so they probably just wouldn't

Speaker2: target Erlang and Elixir.

Speaker2: But now with the bottleneck removed and with everyone doing this kind of work

Speaker2: and there's all this money and excitement around it, people will absolutely

Speaker2: target Elixir and the Beam. It's just going to happen.

Speaker1: That's true. So thinking again,

Speaker1: you know, with AI finding a lot of vulnerabilities and us being,

Speaker1: you know, in the midst of this all, do you still think that maybe AI can cause

Speaker1: us to become a more secure industry after this if we do it right?

Speaker2: Yeah, there's definitely the potential benefit there.

Speaker2: That would be a major improvement for most organizations is adopting AI,

Speaker2: using it to secure their systems.

Speaker2: Just for example, there are many security tasks that humans are just inherently

Speaker2: not very good at compared to an AI, such as reading logs, that no one really

Speaker2: wants to read logs for a living.

Speaker2: That's something you should have an AI doing. I do think that the most interesting

Speaker2: and probably relevant research area there is interpretability,

Speaker2: meaning let's say you have an AI system that's responsible for your company's security.

Speaker2: It could be doing offensive work. It could be doing defensive work.

Speaker2: It could be ensuring that your controls are all set up correctly or even verifying

Speaker2: that that work was done correctly.

Speaker2: But then how would you verify that what it's telling you is accurate? it?

Speaker2: That's obviously an even bigger question right now, I think, for society in general.

Speaker2: But if you just scope it down to cybersecurity, that's extremely important because...

Speaker2: Even if, let's say, agents generally are better at

Speaker2: performing, you know, types of cybersecurity work versus a human,

Speaker2: you still need a person with some accountability legally because if a company has a data breach or,

Speaker2: you know, a bank doesn't have the right balances, you know, that organization

Speaker2: is going to be accountable for that failure.

Speaker1: And, well, lastly, last question before we wrap up this very interesting discussion with you.

Speaker1: Now i mean we talked about our industry potentially becoming more secure as

Speaker1: a whole but now let's break it down into the practical day-to-day

Speaker1: so do you think um that we could write safer code using ai like will that help

Speaker1: us already in the day-to-day writing code and shipping code

Speaker2: Yeah for day-to-day i would recommend you know make sure that you're using the

Speaker2: current open source tools, you know, Soblo, MixAudit, having those run in addition

Speaker2: to your agents, I highly recommend.

Speaker2: In my experience, just having an inventory of all of your software and what's

Speaker2: running is really valuable.

Speaker2: It still goes back to basics.

Speaker2: And then make sure you can update when you know that there's a problem,

Speaker2: because that's really what PHP to Shell showed us,

Speaker2: that even if you know you're vulnerable 100 you still have to fix it and if

Speaker2: you haven't done an update or you don't know your environment that's just going

Speaker2: to be very difficult for you

Speaker0: Well michael thank you very much for discussing with us here on the show today,

Speaker0: we thoroughly enjoyed your perspectives on security but that is all that we

Speaker0: have for this week's show so thank you listeners for tuning in and we will see you next time.