Macro Mayhem

All right. Welcome back to another episode of Macro Mayhem. My name's

Peter Ulrich. Today is October 14th,

2026. It is 11:27 AM

Amsterdam time. I'm back in the Netherlands after a very eventful

week at Goatmeier, and I'm joined as always by

my good friend and co-host, Gus Workman. How you doing, Gus?

I'm doing pretty good this morning. How are you? Have you recovered

from your Goatmeier travels? Yes, I'm better now. I mean, after I came back,

I fell sick for the whole week. in plus a few

more days. And yeah, but yesterday I went for my first recovery run and it

didn't feel entirely shitty. So I think I'm, I'm back in,

in the saddle almost 100%. Nice.

Yeah. Uh, let's start with the news. This episode today is a little light

on the discussion part, so we're going to spend some more time on, on news

and blogs. Unfortunately, the first news item of the

week is quite sad.

Margaret Hamilton has died last week. She is

most known as the engineering head

behind the moon landings of NASA in 1969,

and she led the department that programmed

the moon landing software.

Yeah, she died at the ripe old age of 90.

I think that's a pretty good age, but still she will be missed.

Margaret has been exceptional in her work.

She was, yeah, the main reason or one of the

main reasons that the moon landing in '69 actually worked.

She primarily excelled by focusing on

the safety of her software and fault tolerance of her software.

I read the obituary here that we

also linked in the show notes, and it was quite impressive. Because the moon landing

almost crashed. They almost crashed into the moon. But because

the way she designed the software, they could turn off a lot

of the background processes and prioritize the high-priority

procedures that were meant to control

the glider, control the lander to safely land on the moon.

So because they had that option at their disposal,

the moon lander actually landed and didn't crash. I think

that was a very nice story behind this. So we're very sad

for Margaret's departure, but she has been absolutely

outstanding. Also has gotten a lot of rewards and awards

for her excellent work. So she will be missed,

but her memory will live on. Definitely an icon in the

industry and a great female figure in

STEM as well. Exactly. So yeah, moving on

from this sad news, Gus, do you have some better

news for us? Yes. Coming more to the

Beam world, we have some news from the Glean

team. So Glean published

a bit of news recently that the latest version, which is,

I believe, 1.19, .0, uh,

was released, and it now, uh, does not transpile

Gleam code to Erlang source code. So the

title of the blog post is Gleam Doesn't Compile to Erlang Source Anymore, and you

might think that that means it doesn't compile to Erlang at all, or maybe it

compiles to like Beam bytecode, and that's not quite true.

Um, Gleam now compiles to the Beam

abstract form,

which is an intermediate representation that then the Erlang compiler

doesn't have to compile source code files. It compiles the intermediate

format. And this is the same as what Elixir does, from my understanding.

It compiles that intermediate format and then turns that into bytecode. And the reason for

this is that bytecode is part of the Beam. The Beam gets updates once

or twice a year. the major OTP releases,

and that would be a bit of a moving target. So if we can leverage

the existing tooling from Erlang and the OTP team, then you

can get the best of a lot

of worlds. And so this is a really cool update as well, because now

Glean errors and the stack traces will give

you the line numbers for the actual Glean

source code. And before, the errors would

give you the line numbers for the transpiled Erlang

source code. So there's a nice quality

of life improvement there. And the Glean team has also spent some time to make

this performant. So they dropped their standard

compile times by, I don't know, 20, 30%.

They have a few benchmarks in there, but they also are very clear that you

should take those with a grain of salt. So check out the post for more

information, but Gleam got some nice updates and it wasn't just to the Erlang

part. There's a lot of other updates as well, some JavaScript performance

improvements. Um, and yeah,

a whole bunch of stuff. Take a look. Have you used Gleam before?

Uh, I have not used Gleam. I have attended a few talks about Gleam

and I was quite impressed, but, um, yeah, I did not have time to play

around with it yet. I think the coolest thing, and because if you're an Elixir

developer and you haven't played with it yet, is that You think Gleam is just

like a statically typed Elixir, a different syntax, but all

goes down onto OTP. But it's pretty cool because Gleam also

has a compilation target for JavaScript. So you can run your

code on the browser and on the backend on your Beam.

Um, so, right. Take a look. It's pretty cool. Very nice.

I'm always happy to look at them and see that they're making

good progress there. The next item is by

Joseph Alim. He is announcing,

or he has announced, a postdoctoral position in

the type system for Elixir with the IRIF

in Paris. And I believe this is the

extension of the work of Guillaume, right?

The type system Elixir. Guillaume has written his PhD thesis

about it and I guess now doesn't want to continue with the postdoctoral

position, which I can understand.

And José is now looking for a postdoc that wants

to take on the remaining features of the type

system. So if you happen to be a postdoctoral

student, computer science student, and you want to work with the

fine folks in Paris, I'm not sure whether

you need to be in Paris physically, but the position will be in Paris.

Yeah, you can apply to this position.

Next up on the news, Sobolo version 0.16

was released. Sobolo is a static analysis tool for

ensuring that your code is secure. It checks for common security vulnerabilities,

and generally it's a really good tool. You should add it to your projects.

0.16 brings a lot of improvements. The release notes

are very long, very detailed, but take a look

'cause there's a bunch of bug fixes. But the big new

feature that we wanna highlight is that there's now a GitHub output

format for GitHub Actions. And so what

does that even mean? It means that when you run your

Sobolo checks in GitHub, in your GitHub Actions,

it now can output a specific format so that the GitHub

UI can render line by line where your security

issues might occur. And so you can actually look at it. Before,

it kind of spit out this big text chunk

that GitHub would render and show you, but it wasn't like an ideal

workflow to kind of look through it. So now I believe

the main difference, correct me if I'm wrong here, Peter, but the main

difference is that you now have to add a --format GitHub

flag in your CI calls to enable this.

Yeah, I believe so. Yeah. I think previously it was very hard to parse

the Sobolo output in your GitHub CI. So basically you had to rerun it

locally to make sense of it. And this new output format allows

you to have these warning dropdowns

in your log output. So you can see, oh, there's a warning at

this line. Basically. And talking

about SOLO and security in general, we are happy to announce

that we have our first sponsor for the show, and the sponsor is Michael from

praxil.io. And the sponsorship worked

as follows, that Michael paid us a little bit of money, and we interviewed him

a few times to ask his expertise about security questions

that Honestly, I would've asked him also without the money, but don't

tell him that. So in this episode and in the

coming episodes, we will always embed a little snippet of

our interview with Michael, and in each

snippet, we will ask him a question that hopefully is interesting to you.

So this is gonna be the first snippet here. Please enjoy.

And now for the first security question of the week, I would like to introduce

Michael from praxial.io. Michael, I've known you

for quite some years now, and I still don't know what Praxil.io does. Would you

care to elaborate, please? Thank you, Peter. It's great to be

here. So Praxil.io, it's a software product

that companies buy essentially to ensure they don't get hacked.

And it drops right into your Elixir application and implements

several security controls. There's a lot of them that

we can't go into full detail.

Essentially, when a company has to do a SOC 2 audit

or they have a big customer that has a very long security questionnaire,

there's very common security controls and Praxial.io

helps businesses implement those so that

they can finish with the security work, have it running,

and then get back to their real work securely.

That sounds good if I'm a business, but now imagine I'm a person,

an individual developer, or maybe a smaller team.

What can I do to protect myself these days if I

don't want to buy Praxial, but still stay safe in the era of AI

threats? Yeah. And the good news is if you're just starting

out, you don't need to jump to a very powerful solution like

Praxial.io immediately. So for example, if you're an

individual developer, And you're working on a project and you

just want to keep it secure, there's a lot of open source tooling.

For example, I'd recommend Soblo, which is a static analysis tool

for Elixir and Phoenix projects. You can also examine

the security of your dependencies. You can do

that with actually Mix, um, and like the Hex client right now

that's built into the newest release. So just make sure you update. And you'll

actually get that vulnerability intelligence right in your

command line. And another, a big thing I'd actually recommend is

just ensure you have an inventory and list of all of the projects you're working

on so that if, let's say there's some really

big vulnerability that comes out where if your project is exposed to

the public internet and you're running this library, you can very

quickly check and see if you're vulnerable or not, because that

has happened. That's been a constant in security since

the 1990s. And with the way things are going, it seems like that will just

keep happening. So in summary, there's a

lot of things you can do on that side. And then with

the new AI models, you're probably using them to code. They're very good for

scanning your codebase for vulnerabilities.

The challenge is you also need to know a bit about security to use them

effectively. because you can't really just trust the output.

I use them constantly for security assessments,

and they're very effective when you know security, but we're not quite

at the fully autonomous level. There's a lot of discussion about

that right now. So a good source for this is the Erlang

Ecosystem Foundation, the security working group that I'm on.

They put out a lot of material. The Praxil.io blog is

where I publish material on this, so that's a good thing to subscribe

to. If there's a big news story that Elixir

developers need to know about from a security perspective, you know,

I like to use the blog there to communicate it to everyone.

Right. Thank you, Michael. And maybe we will see you next

week on the, well, in 2 weeks on the weekly security question.

See you then. Now, welcome back. I hope

that snippet was interesting to you. And let's continue with the news.

The next news item we have here is about Hologram version

0.12 is released. And that one's interesting because

Hologram really improved their compiler

and the way they compile the Elixir code to be used in the UI.

So Hologram is to write local-first or client-side

UIs in Elixir. And they had that issue that the compiler

was quite slow and there were certain projects they simply couldn't compile because

they were too complex and they would run out of time and memory.

So they spent a lot of time on improving the compiler and they showed

some really big improvements there where they took previously projects

that previously couldn't be compiled and now they could compile them within just a few

seconds. So that was really impressive. And it also tells you a lot about how

you can improve any compilation time if you have some similar problems.

So it's a good blog post to read.

Next up on the news is the release of a library that is kind

of near and dear to my heart, is the work coming from Marc

Lanet, the guy who did

the original ports of Nerves to Android devices.

He's now developed an AI-assisted toolkit to let

your agent have a bunch of tools to help you port Nerves to Android

devices. And so it's called Nerves in Native. And we'll link

the repo in the show notes. But if

you're part of the NERF Discord, you might have seen this past week that

there have been a bunch of people who have reached into their drawers,

found what is essentially e-waste, old phones that they have lying

in those drawers, and they have used this

AI toolkit to port NERFs to those devices. Now, this is like

a kind of complex process. A lot of Android devices are all different.

They run different kernels, they have different security mechanisms. So having the toolkit

is like really cool. And from what I've seen on the Discord,

I think we've probably maybe doubled the number

of Android devices, maybe even tripled the number of Android devices that

now run Nerves these days. Right. It's been really cool seeing people posting,

um, picture after picture of like, hey, I got it running on my whatever.

I saw Samsung, there's some really old phones that are now

running NeRFs, which is pretty cool. Is this going to be the next check?

Like, can it run NeRFs? You know, can it run Doom? Essentially,

can you, can it run NeRFs now? Can it run NeRFs? Yeah.

Very nice. Very cool to see that. Yeah. And I got

to say, I'm still continuing to be impressed by the progress

we are seeing and all the side projects and the scope of the side projects

that people are working on just because they can use an LLM now to point,

You know, with the use of libraries like this one, they point an LLM at

their old Samsung phone and say, hey, you know, install NeRFs on this thing.

And then the LLM goes off and does it. Right. It's not

something that you're likely to spend your own precious free time

on, but when the barrier to entry is lower,

it means that there's a lot more applications that can be done with what is

essentially e-waste. I think people are going to be looking

at making these into like little home dashboards. I saw someone Okay.

He's a little crazy. He strapped an Android phone to

the robotic— he has like the robotic lawnmower

that he took apart. Oh, a robotic lawnmower. Okay.

Yeah. It's like the one that like drives itself around your yard and

chops. He took it apart. He basically just wanted the wheels and the motors

for it. And so he then put the phone, his old Android

phone in as the NERV

controller for the device. And then he made it

into a balance robot. I think he was missing out because he wasn't able to

go to the balance bot workshop at GoatMind.

So now there's a NERV phone driving a balance bot.

But it's a lawnmower. It's like a proper lawnmower that now self-balances.

Kind of. If you change the case for it a little bit to add a

wheel. Wow. There's a very, the picture. I, we're gonna have to link the post

in the show notes because it's,

it's some, it's a thing to behold. It's great. Imagine you would come to his

property and then there's like a, a lawnmower driving towards you,

maybe with a little LCD or like a screen in it and saying,

don't trespass. It's got the phone screen. Yeah.

It's cool. That's funny. Yeah.

So check out Nervsnator if you have old Android devices and extra tokens

to burn. Interesting. All right. And the next item is that

RecLLM 1.27.0 was

released. It is a mostly bug fixes

update, but there are 3 new cool features in here.

And the most interesting one here is that now they

have support for the OpenAI Decisions API.

And that is basically what Jeff is. That's my

understanding. So OpenAI now created their own Jeff-like

decision LLM that is super fast, super cheap,

and that can give you its opinion about

whether something like an image or text is either

true or false. So you can ask it a statement and then

it gives you back it's true or false. It can choose between different

categories. So like choices, and it

can also give you what they say, like a Rubik's API. So a

score against the rubric, whatever that means.

But yeah, so RecLLM now supports that API as well. So if you

want to play around with a Jeff-like model hosted

by OpenAI, you can do that directly through RecLLM now.

Nice. And next up, Jump,

the company, they have a CredoChecks package,

jump.credochecks on Hex, that has a new version.

v0.6.0. This is a minor update with some

bug fixes and some improvements to some of the existing ones,

but I think the big one here is that they added a new Credo check

to avoid modifying structs before they get passed to a

change set. So if you use

these, you might want to update if you want to see the new one.

It's a minor bump, but they're pretty useful. So please take a look. Yes,

very nice. And the last item is something

that probably has been around for a while, but I only learned about it recently,

which is that Hex, our beloved package manager,

they built the Elixir Erlang Docker images for us,

right? So if you run Elixir anywhere,

most likely you use a Dockerfile for it. And that Dockerfile

at the top always has an image tag for your

Elixir version and then the OTP version.

And I don't know how many hours maybe I spent

already on Docker Hub to find the latest image for

the Elixir and the OTP version, because it would then have a timestamp

at the end. So it would be like elixir-otp-debian-slim

and then a timestamp. And just to find that freaking timestamp,

I had to spend so much time on this really, really slow Docker Hub that

just, I don't know why you search for it and it takes 15 seconds to

find, you know, like to do a string match on the Docker tags anyway.

So I found that now HexPM, they expose

their builders and the tags of

the images they create through a very nice URL, which is just bob.hex.pm.

And if you go to bob.hex.pm, you will see that

first of all, you see all the jobs of the current builders. like what

they're building. That's cool if you want to— Live, right? Live, yeah.

It's all, when I loaded it earlier, it was showing some stuff. Exactly. So if

you want to see whether a new Elixir version is already being built, you can

check that here. They also have a tag for artifacts, which are,

I think, binaries that you can download. So you can download a precompiled OTP

version for Ubuntu 26, for example. You can do that right here.

But then most importantly, for me at least, there is a Docker tags tab,

and that allows you to filter by the Elixir version,

and the OTP version and your host OS. So Ubuntu,

Debian, that kind of stuff. And then it will give you the OS version,

which might be, you know, Trixie, and then with the

timestamp behind it. So you can use this website to

quickly find the latest timestamp if you want to upgrade your Docker image

to the latest Elixir version instead of using the Docker Hub one,

which is, I don't know why, Extremely slow. Like you

must, you must want this to be that slow. Like there's no

way that you would have a query that just takes 15 seconds. Like it's engineered

to be slow, I think. Anyway, now we have a better option. I can tell

you're frustrated with it. Yeah, other than hex.pm. Yes, I'm very

frustrated with that. Do you know where the, uh, do you know where the Bob

name comes from? Bob the Builder.

Bob the Builder. Yes. Bob the Builder. So yeah, Bob the Builder at

bob.hacks.pm. Check it out.

And now moving on to the blogs. Gus, what do we have

this week? All right. That wraps up our news.

In the blogs, we have a couple of blogs that were published in the last

week or so. First up is the Goatmire Recap

blogs. These have been a bit of a joy to read, especially because

The 3 that we've linked below come from people who are

involved in the behind the scenes of the conference.

So Lars, who organized the conference, he posted

on his website a very long,

very comprehensive play-by-play

of everything that was going on, including the months leading

up to it. It's a very

comprehensive read. It's a lot of pictures so that if you weren't there,

it's a great way to like get a feel for what the conference

actually was and entailed. I think Galtmeyer has a

certain vibe that is hard to convey with words.

So check out Lars's post there.

In the same vein, Elise, who was one of the ones who Really

was behind the scenes keeping everything on track,

put together. She kind of was the showrunner behind the

curtain, as well as planned a lot of the theatrical

performances with Flora. She wrote

on her personal blog as well about the process of making the music,

helping with the MIDI library that she used for the

Nurse Orchestra. Um, even other things

like the, uh, that she threw in a singing piece

with Sasha, Sasha Urich. Um,

yeah, it's a good read. Uh, I think if you've not

been involved in the planning of a conference, these 2

posts are really gonna help you realize what a tremendous effort that

was on, on so many people.

And then the third one was from the BeamOps

written by Ellie Fairholm. And that

was her recap from more covering some of her favorite

talks, generally what the vibe was there, and also

the workshop that they ran. So take

a look if you missed Goatmire or if you did go to Galtmeyer and you're

already missing it, you can take a look at the recap vlogs that have

been posted already and kind of see

how the event went. Very nice. Yeah. And I also got

to say Pep and Ellie from BeamOps, they're lovely people and they're helping me

with the push-in architecture. So I also know that they are

currently looking for a new contract. So if you have needs of improving

your architecture or operations, DevOps of

your application, give them a shout. Yeah.

And that's what they ran their workshops on at GoMars. Exactly.

Yep. Terraform, Docker, was it a whole bunch of,

yeah, BeamOps. It's, uh, yeah. If you basically, if you,

if you, if your architecture is still point and click,

you know, just give them a shout because they can, they can help

you move everything to a a bit more professional setup with Terraform

and that kind of stuff. All right. Cool. Yeah.

The next blog post is by our returning guest.

I think she writes one blog post a week or two.

So it's Johanna Larsson as always. And she wrote a quite

interesting blog post about how she built her own AT Proto link aggregator.

And Johanna's problem that she tried to solve is that

in Bluesky, that is our new

social media Twitter X competitor.

Johanna sees a lot of articles flying past in her feed. And then

the problem is she wants to read them later or she might not see them

again after a refresh. So she wrote her own little aggregator

that connects through her account to Bluesky and

just saves these links as they fly past. And there's

a very nice overview of the links that have been

flowing in the ecosystem. And you can go

to shelf.cafe, that is the website she built

for this, to see a list of links

to articles mostly that are trending currently

on Bluesky. So it's quite interesting. And in

the blog post, she also wrote in extensive detail about

how she built this aggregator, how she does the algorithm and the

moderation of these links. And so on.

So give it a shout, give it a check. Also, if you just want

to understand how you can connect to the Bluesky ATProto network

to fetch content and information automatically.

Yeah, she's been doing a deep dive on the ATProto ecosystem,

and it's been really cool to see some of these posts coming from

her blog. I, if you go check

out shelf.cafe, I think it's very much similar to a

Hacker News or Lobsters kind of link

aggregator experience. So I'm curious to see where that goes because

even if you, uh, don't use it, it's actually a really cool

implementation of ATProto application

full stack from PDS all the way up to App view

and everything on the beam. Exactly. Um,

yeah. Good. Next up,

we have another returning author, Ryan Zidago,

on using doc tests to understand a feature.

This article is a bit more of an

argument on why you should use not

just docs for your LLMs, not just tests for your

LLMs, but combining them into the single point,

which is the doc tests, which are a cool feature that we have in Elixir.

So he's got a lot of examples on where,

like, there's a lot of code and examples here,

but at the end is that he comes to several conclusions.

There are limitations of this, is the way he ends the article with

some caveats. There are,

like, it increases your module doc sizes, and there's

not always highly observable behaviors encoded

into your specs. But he argues that it does reduce

incorrect answers, and that it makes understanding what the

tests are actually testing a little bit easier because it's embedded with your

docs. So, Yeah, it was a

good article there from Ryan Zitago. Yeah.

And the last blog we have in today's section is

from Alexey Matyushkin, also known as Muda

Sobwa, and he wrote about aggressive regression.

And it's basically a very long article about

his approach to testing. and writing tests

and that he rather goes for regression

tests instead of, well,

he tries to write tests, but then whenever something fails, he goes aggressively

for regression tests to make sure that the test won't fail

again in the future. So yeah,

it's cheaper to write them than to get— than to

have a bug report in the future again that might be because

of the same bug. So he argues a lot about writing your own regression tests,

but the blog is generally a very good introduction

to creating a mindset or shaping your

mindset around which tests to write, how extensive to write them,

and then also which tests are useful and which just aren't.

So I quite enjoyed this blog post because, yeah, it doesn't only focus

on this, like, yeah, you should write regression tests, which is something, you know,

self-evident. But he actually goes through his thinking

about testing and how to build up a test suite that actually helps

you Instead of working against you.

I liked his, uh, also early in the document,

early in the blog post, the argument of what is a regression test

versus a unit test and how unit tests don't

really, um, help you test

too much. It's just asserting that the function does what you think it does.

Yeah. So. It's a good mindset around

testing. It's a good blog post. Very good. And now the

next section, meetups and conferences. We still have, like, every time I think we're

getting a shorter and shorter list here, it kind of extends. So Gus, why don't

you take us through what's up next? All right. So meetups

and conferences. Um, first up, the NURBS Meetup EU is

tonight at 7 PM Central

European Time. So hopefully if you listen to this by the time this goes out,

Later this afternoon, you have a chance to register,

but register for Nervous Meetup EU. Also,

it's recurring, so if you miss tonight, check out for

November. There is also Unconference,

organized by the EEF attached to Codebeam Europe.

Codebeam Europe is October 21st and 22nd. The Unconference

is the day before on the 20th.

If you haven't been to the unconference format before, organized by EF or anywhere else,

you kind of bring your own topics and have

more spontaneous discussions on Beam-related

topics. And then finally, we have the BOB 2027

call for proposals was published. I don't

know too much about this conference, but it's in Berlin in February.

February 26th, 2027. It's more of a functional

programming, data structures, kind of more hardcore computer

science, I would say. And so if you're interested in that

sort of thing, there is a call for proposals due November 2nd,

so about 2 weeks from now. And yeah,

check it out for Berlin in late February. That's all

we have for the meetups and conferences. As usual, if you

have an event, a meetup that you organize and you'd like us to promote it,

reach out to us and let us know. We're happy to put more

stuff in there. Anywho, Peter,

that kind of wraps up the news, the blog posts, the meetups and conferences.

That means it's the audience's least favorite part of

the show, which is your joke. Yeah. All right.

Least favorite. I would say this is the favorite part of the entire podcast.

That's the only reason why people listen to this. That's my opinion on that.

Oh, was that the joke? No, it's not. Actually, I realized I

saved a pretty good joke that I wanted to tell in my Goatmeier talk,

but then I forgot to actually put it in my Goatmeier talk. So now I

still have it here, you know, in the backlog, so to say.

All right. So, you know, put yourself back into

the situation where we were in Sweden, in Varberg,

Goatmeier, you know, Vikings all around. So. The question is,

what do you call a Viking that has lost his boat?

Hiking.

Yep. Very good. Viking ain't going too

far without a boat. Ain't no more sailing.

Hiking. Ain't no more sailing. Very good. Very good.

Well, thank you. That is covering all of the main

content of the of the podcast. We will

start our discussion section now. And this week we're going to talk more

about secrets and how do you keep

stuff secure. So this is kind of a question I posed Peter,

was how do you store your secrets both in production

and in development? How do you,

what's your workflow there? I need a new workflow.

Yeah. Sorry, just to take one step back, we have 2 discussion topics.

added one maybe that Gus hasn't scrolled down to yet. The second

one that I wanted to discuss, if we have the time, is how to run

your own vulnerability scans. Because now that, you know,

the models like Mythos and, yeah, Mythos and

Sol 6.1, like, they're now available

to everyone. Mythos, okay, Mythos 5.1 is not available to

everyone, but if you sign up for their security program, which takes just

an hour, Then you can also get access to Mythos 5.1, even as a

European, non-American. That's the big difference now.

So I wanted to share with you a little bit, you know, now that everyone

has access to these rather powerful models, how can you use

them to scan your own code and libraries and that kind of stuff? So that's

going to be the second discussion topic. First topic is how do you store

your secrets? I have slowly

over time changed more towards keeping everything inside a

password manager, even for local development.

So previously, you know, you would think about 1Password

or similar password managers to keep your production secrets, and somehow

you would then, you know, insert them into your image

that you deploy to production. And actually

this work has, like this change has come through KAML that

I still use to deploy most of my applications.

because they do give you a secrets file, but in the secrets file,

you don't store raw secrets because you can also commit this file to

GitHub. So, but in the secrets

file, what you usually do is you call the password manager CLI

to fetch one particular secret.

And yeah, if you set up 1Password in a certain way,

it will ask you for your permission to fetch all these secrets every

time you deploy. So you actually have kind of like

a check there that, nobody can just call the CLI, but you

have to approve it every time. So. Right. And that's,

that's local on your machine because you're deploying with Kyma, which is basically

SSHing into your server, pushing your release, restarting stuff.

Exactly. But also at that same time, sending the environment variables.

So they're not stored on the server anywhere, right? No, they're only inside

the environment in the Docker container. So there's no,

stored value in the, in the container itself. It's just in the environment.

So yeah, if you get access to the application or the Docker container, you can

obviously still take the environments out, but then I might also have a bigger

issue. Yeah, a little bit. But what

happens if, say, your machine goes down for the

cert, the, the, the data center has maintenance and your machine goes down? Does it

restart and then come back online with those secrets, or do you have to redeploy

manually? That's a good question. I think, I think that, uh,

I think you have to redeploy. Yeah. Because it wouldn't

have the secrets stored anymore. It wouldn't have the secrets. Yeah, I think so.

I haven't tested that one yet. That's a good one. So,

so make sure you have your laptop when you go for vacation

because, uh, yeah, I always have it on me. Data center goes

down. You can't go hiking in the Alps without worrying

about pushing there, Peter. Yeah, it's a good, good point actually. I haven't

thought about that one. Yeah. I mean, but I think that's,

that's what other, um, there are like tools that

do secrets management and, um, like,

was it HashiCorp Vault is one of them that,

and I think those are ones where it can pull the secrets

automatically into your Docker image during the startup of your application.

I don't have that much experience, so I don't know what I'm Fully talking about.

Have you used those before? I have not, but I mean, if you

use something like, um, I mean,

what you mentioned earlier, if your machine goes down and you restart your application,

like if you do bare metal servers, what I'm doing,

you kind of have to manage that yourself, right? You have to restart

your application, inject these environment variables, um, by restarting the application.

If you do a managed server where you just upload your image and then

it gets deployed on a server for you, I think in this situation, I would

also not, I actually had it set up in such a

way for Fly, for example, where you hand over the secret to the provider,

the hosting provider, and they inject it whenever they

start or restart your application. Yeah. And I

think that's a pretty common way for a lot of those managed

hosting services. Yeah. Um,

I, I believe if you're,

if you set some things up properly, you can probably also use

GitHub Actions during your release step to,

uh, use GitHub secrets to inject

into your application as well as it starts up on whatever

hosting provider. I did that too, actually. I have one, one CD where I

use GitHub to deploy through Kamal. And I have my

environments saved as environment variables in the GitHub Actions. So then Kamaal

fetches from the GitHub Action environments to inject

it into the deploy, into the container. But I think the big

point here is just you shouldn't bake the secrets into your

Docker image that you then store on a registry or somewhere.

But this would be the only way that if you restart your Docker image

and have no outside you know, Key Vault or something.

Only this way you could then have the environment secrets in the Docker container,

but I don't think you want that. Yeah. Right.

You want something starting it with the secrets as an

environment for that image when it starts. Yeah, injected. Yeah.

So the secrets should be injected. Yeah. So for local

deployment, I mean, that's what I'm doing most of the time these days with Kamal,

where I have on my local machine, I fetch these secrets from

the password manager, it gets injected as Kamal deploys

the image, and then afterwards the secrets are gone also

from my local environment.

If you want to have a more CD approach, then yeah, either store

them in GitHub Actions and then use them while you deploy from GitHub Actions,

or if you have something where somebody deploys an image for

you, then they most likely will have also a Key

Vault where you can set these environment variables. Fly, for example, they have Fly secrets.

So you can just set them there and then they get injected. I believe,

like, I think maybe on Scaleway, which is what I'm using,

and Hetzner, I also use Hetzner, they might also have

key, like I know on Scaleway, they have key vaults that you can use.

So you can probably put your environment variables into the key

vault of whatever hosting provider you use. And then

give your Docker image just like a temporary token that it could

use, maybe. You know, there could be a way there,

if the Docker image starts up, it has either,

like somehow it will get a temporary token and then it can connect

to the Key Vault to fetch the environment variables from there. So you

might be able to— Something short-lived. You might be able to build what Fly,

for example, does for you yourself,

but you know, you gotta be a bit more careful then.

Right. So, and you were mentioning the password manager.

So when you deploy with Kamal, it's just invoked

with the scripts that then run the

release. But you mentioned to me earlier that, um, you do

have something similar for your local development as well. What does that

look like? Yeah, so I used to have .env files, and in those

.env files, I would store the secrets. I think we have all done that,

you know, maybe still today, or at least a few years ago,

that's the way you set up local development.

I have since switched also to fetching the secrets from the

password manager as I source the .env file.

So in the .env file, if you replace the actual value with

a call to your password manager, so to

the CLI, when you source the .env

file, it will call the password manager and then fetch the environment

variable and inject it into your local environment. But this way

you never have the actual secrets stored in a plain text file in your application,

in your project.

Right. And that just looks like your .env file then has the

exporting the environment variable, but then

you also have the shell interpolation with the dollar sign

parentheses, and then it's calling the CLI of your password manager.

And then When you source it, you get that little popup from your

password manager that says, do you want to authorize this? Something like that. Yeah.

And I also use that for my dev container setups because

I run most of my agents when I code locally,

I run them in a dev container. So that's a Linux container running on my

computer. And I have a GitHub project there.

If you go to pjulbricht/devcontainer, or you can also go at pushin.eu/pjulbricht/get-devcontainer.

I think I migrated it. In there, you will find a devcontainer.json.

And that is the configuration file for the actual dev container.

And in there, I'm setting environment variables,

either secret or public. You know, if they're public, they're just baked into that file.

If they're secret, they pull that environment variable from the

environment of the shell session that you use

to start the dev container. So this way, if I,

because you might not need all the environment variables in your actual

dev container. So you only need maybe 1 or 2, but you have 10 for

your, you know, because locally I need more environment variables to maybe deploy or

do some other CLI work. But in the dev container, I only need 2.

So I can inject them from my local session into the dev container.

And then the agents can work inside that dev container, but they

don't have access to the real environment outside of it. Nice.

Yeah, that makes a lot of sense. And then With

your projects, then you are committing your .env files, presumably?

No, I do not. No. You still don't, even though

it's not including any— Right. I mean, by now I could,

but I usually don't. If I need to share the

.env file with another developer, I'd rather create an example

.env file. And then it has,

it has, you know, like, hey, insert this value here or there. So,

so they, the other developer knows which environment variables they need to set,

but the actual values are not in there. But I just didn't have that situation

yet where I needed to share my .env with another developer. But yeah,

I mean, I just, I don't feel good about committing

.env files because maybe in one situation it has a secret

and then it's, you know, online. So I just don't do it. Fair enough.

Did you add it to your global .gitignore then? Something to make

sure it never gets— Exactly. So I would need to remove my global .gitignore for

.env files, and I just don't want to do that. Fair enough.

I guess old habits die hard, huh? They do. Even if it is safe

now. That's true. But I mean, generally,

like my main suggestion here is just to move everything to a

password manager and then use the CLI that most of them now offer.

I mean, 1Password was already. They had that already a year

or 2 ago, or 3 maybe. I use ProtonPass, and they

also added a pretty good CLI, I think a year ago.

So like, the only thing is I also have my SSH keys in there,

which might be even more important than your local development secrets.

If you keep your SSH keys on disk, even if they're encrypted,

it's not a good idea. Because if you get exploited,

You know, by like, for example, we saw that in the JavaScript ecosystem where

there were packages released or published,

malicious packages. And when you downloaded the package,

they have an after-install script that would run,

and that script would just look at your .ssh folder, take everything

that's in there, and upload it to a command and control server by the

attacker, right? So, within seconds, your SSH keys,

private keys, will be in the hands of the attacker. And then all they

need to do is figure out the IPs to your server and then connect to

it through, with that SSH key.

So that was one of my first things where I moved those to ProtonPass,

to my password manager. And now whenever I SSH

into a server, it gets pulled just in that one call from

the CLI. I wished that

in ProtonPass, it would ask me every time it does it, but it does not.

So that's a little bit, nah, I just don't like it. In 1Password,

I know you can set it up in such a way that it asks you

every single time an SSH key is requested.

So, you know, it might also just be a configuration issue in

my case. Yeah, but this way, even if an attacker

gets access to your local computer, which we kind of think of, oh, it's just

my local machine, it's not production, Yeah, still you have secrets

on there that gets you access to production. And if it's just a database string,

you know, the database URL that you have locally to connect to

your production database, like they can extract that and then connect to it directly.

So yeah, let's just keep everything in a password manager.

It's not 100% secure, but it's more secure than just a plain text

file on your computer. Right, right. And I think

these are really good suggestions because, I mean, especially with the JavaScript ecosystem

thing, this isn't exclusive to JavaScript, but it's just seen there because

it's targeted, either packages, popular packages being compromised and

the new versions being able to steal your

secrets. So you just do your npm update or whatever the command

is, and then boom, you're compromised. Exactly.

Um, but I mean, it's, it's a threat that we could see here

in Elixir as well in the Beam community.

Speaking of that, how is

scanning going, Fedor? What's the, um, if we are

thinking about compromised packages, um, we want

to be able to detect those. If I am a solo

developer and I want to protect myself and scan all of my packages

before I download and install them, how do I go about that?

If you want to scan all your dependencies, you mean? That you pull down.

Well, good luck. I mean, sure.

I'm, this is probably more for the, uh, the enterprise,

uh, know every dependency, get it approved by,

uh, security type of corporate security policies.

As an independent developer, I'm not going to do that, but it might give me

peace of mind to just run it once, once or twice, or to have you

run it every once in a while. Oh yeah. Yeah, yeah, sure. So I run

security scans on my application, like Pushin, which is my main application right

now. I run it every week or two. Like, I run a full,

not full, but like, I focus on subsystems and then I

use Daybreak Blue, which is OpenAI's cyber defense model.

And now I can also use Mythos 5.1, and I just tell it,

hey, go look at the subsystem and find security vulnerabilities in there.

So I found, like, I did that last week with Pushin,

and I found like 50, 60 reports. None of them

were severe. Most of them were like in this particular

situation, yada, yada, yada, you know, somebody can still make a push after

they were blocked, for example. So I fixed them and

I feel, you know, better about doing that. And I hope I'm

going to rescan like in 2 weeks, you know, to see what falls out.

So that's what I'm doing. These models have become good enough,

even if you just use Opus 5.5 or if you use SOL 6.1.

Although I use Opus 5.5 more, they're good

enough to find most of the vulnerabilities, right?

So that's a good approach. Just in OpenAI

and Codex, you have a plugin from Codex Security that you can

install, and that gives you these security

scan and security deep scan skills. I usually

run the security scan one. It's a bit simpler.

So that's what in Codex. In Claude, I think you also have a security review

skill, but there, I'm not sure how good it is,

to be honest. I wrote this blog post a while back

where I also shared my prompt that I'm using when I do these scans.

So you can just use that prompt, give it to Claude and say,

scan this library or this application based on this prompt.

And what I'm doing in my scans is usually I go through the project

file by file, so one file at a time, and you

can write, you know, a 10-line Bash script to do that for you, where you

just fetch the .ex or .exs files or .hex

files. Hex is not that important really, but .ex and .exs files,

and then you feed these files one by one into a new Claude

session, and you just tell it with a prompt, you know, find me some

security stuff, and then write it to a Markdown file with the unique ID so

that you can reference it later. So that is something you can write in like

half an hour. And with Opus 5.5, like you will also not use that many

tokens to do that. So you can just run it while you work

on something else. And if you run that every 2 weeks, once a month,

you know, I guarantee you, you will find and continue to find

severe security vulnerabilities in your application,

but then also bugs, which is nice, you know, to find these kinds of things.

So that's good, but that's— Right.

Yeah. For those skills

that you mentioned from OpenAI and from Anthropic,

they are essentially just an

opinionated set of instructions for how to do

those scans, right? And it's similar to what you were just describing with

your opinionated set of instructions for So have you, have you seen much

benefit between the two? Is it worth it to personalize? Is it

good enough to just go with the built-in skills?

Yeah. So one of the biggest takeaways is

to not just point an LLM at your entire

codebase and say, go find vulnerabilities.

You know, that will find you some vulnerabilities, but certainly not everything.

So what you need to do in any case is to break down

the application into subsystems or into files

as I'm doing most of the time for smaller projects,

and then just have every single file sent to Claude,

but also allow Claude or Codex, but allow these LLMs

to also look for code calling that

module. Right. So not, you know, say only look at this file,

but also say, see where this file is being called from.

So this way it can understand whether It's, you know,

a little function somewhere very deep down the application that nobody else

calls except for one other function that is slightly higher up

the stack, or whether this is a publicly exposed function that everyone can call,

right? Those are very different threat models here.

So, but it does it by itself, really. You know,

the only thing is scope it down to something that is small and then

rather use multiple sessions to iterate through the files

or the subsystems that you have. And all these skills,

they basically do that. So if you also look at the security scan

skill, it will also spin up different subagents that all look at

different parts and so on. So it's all a divide

and conquer approach to finding vulnerabilities.

Right. I guess it sounds like when you start it with a more limited scope,

it'll divide and conquer in smaller pieces that might be able to pick up things

that if you just started it over your entire project, it might not.

Get to that granular level. Yeah. And just a recent

example of this, I used Mythos 5.1

to scan a pretty big library in the space.

And I first gave it kind of like the whole project approach.

So I just pointed Mythos at that project and said, find me

vulnerabilities. And it went off and spent a lot of tokens and it

came back with, I think, 4 vulnerabilities.

And 2 of them were proper vulnerabilities.

2 of them were more bug fixes, slash maybe a vulnerability,

nothing severe. And then I did the same with Opus 5.5,

but I had Opus 5.5 iterate through every single file and just

do it one file at a time. And I still need to review them,

but there were 20 reports instead of 4. Hmm.

So, and I've seen that many times now. So if

you just point an LLM at a project in general and say, go find me

vulnerabilities. Yeah, it will come back with 4, even if it's— because he,

I tried this approach with Mythos because I thought, well,

Mythos is what everyone says, super, like it can, it can contain

a lot of context. It can handle big applications.

And yeah, it didn't do that well, to be honest, you know?

So I think even there, you're better off pointing it at subsystems

or individual files and then have it Uh, find the vulnerabilities.

Right. And how do you feel like in

the confidence level of those 20 Opus

vulnerabilities that it found versus the 4 Mythos?

Like, is it 20 junk or 16

junk ones and it caught the 4 that Mythos also did? Or how do you

feel? Yeah, usually, what was your experience? Usually I have like a 20, 30,

sometimes 40% hit rate. with vulnerabilities. So,

if I take 20, if I have 20 reports, I'm pretty sure out of 20,

there will be like 5 CVEs maybe falling out.

So, that's already more than Mythos, where only 2 out of 4 reports

probably will be a CVE. Right.

Very interesting. And with the access

to Mythos that you now have, I think I heard, or you posted on socials

recently, that you scanned Phoenix. And found no vulnerabilities.

Is that right? Nothing new. Yes. I rescanned Phoenix.

Actually, I have been rescanning Phoenix almost with every new model that comes out,

and time and again, it doesn't find anything severe

critical, right? So I'm still waiting for that day where out of a sudden,

a new model comes out and says, yeah, this is massive, massive vulnerability in

Phoenix that everyone has overlooked so far,

but it just hasn't happened yet. So, Fingers crossed.

Kudos to the Phoenix team for doing that,

but like fingers crossed. I'm hoping,

and I'm sure you're hoping, that we start to see like the tail of

LLM security vulnerabilities. Yes,

they're still going to exist, but they're going to get harder and harder and harder

and harder to find. So let's build

a more secure ecosystem. That's what's happening right now. Yeah.

bigger issue here is breadth versus

depth right now. So I scan a few libraries quite often and

like the biggest ones, I scan them quite often. I don't

find that much anymore. Sometimes like Decimal had a CVE,

you know, so that kind of stuff. But yeah,

now like now I kind of need to go into more the broad aspect.

I need to go down the long tail of libraries, right? Because even if you

use a small library that doesn't have as many downloads as Phoenix,

for example, it might still be very critical to

your stack or to your application, and it might have a vulnerability that

is severe, right? Right. And that's

the case where your recommendations earlier on how to run security scans might be useful

for people listening to the podcast, because if you have those

dependencies that you, in addition to your own applications,

that you want to check for security vulnerabilities, something that hasn't been

On the, what is it, the top 100, top 1,000 packages that

you've scanned on Hex? Yeah, ideally scan

them yourself, especially when they're kind of critical,

when they take external input. You know, if it would be

like Ecto, I mean, Ecto is important,

but it's also very low down in the stack. Like you will almost never give

a user direct access to any Ecto functions. It always goes through the

LiveView or Phoenix controller context, whatnot. So you have some

layers of protection there. Yeah, we also, I didn't find too

much in there. So, so basically think about your library,

have it like, just again, point Opus or Codex at it and say,

find me some vulnerabilities, fix them.

If there are actual vulnerabilities, you know, if you, if you think people are,

could be affected by this, send us a Slack message or an email,

and then we can create a CVE for you so that people get notified about

this critical change and can upgrade to the fixed

version, right? But yeah, I think it's just like— Who

is the us that you're talking about? Oh, that's me and Jonathan. So it's

2 people. And why

you 2? What's special about you 2? Well, we are working

for the EFF to do this. Jonathan is officially the CISO, Chief Information

Security Officer of the EFF. Um, he works full-time and more than full-time on

this. I work part-time on it, but yeah, you,

you, you can contact either of us and we help you further.

Right. So you two are the point of contact for Erlang

ecosystem security. Basically. Related questions.

So you two are, thanks again. I think, I feel like I say this

every episode, but thanks again to you both for the hard work, 'cause it's not,

it's, um, I, I remember it was

after ElixirConf EU and we were sitting in the,

um, the hacker room, hacker apartment.

Yeah. The hacker apartment from the Ash folks. And,

uh, you're just like, hey, I just threw Claude to go

scan, I think it was Decimal. Yeah. And you're like, Decimal,

there's actually a vulnerability here. It can crash your system. It's a big one.

Then you ran it and then it crashed your system. Yeah. It had like 8

gigabytes of RAM.

Yeah. But I don't know. I mean, to be honest, I see this whole work

as one contribution to the ecosystem, right? It, like, there are many different

ways of contributing to the ecosystem. The most obvious one is writing code

and releasing it and helping others use it. But also what I'm doing,

scanning stuff and reporting vulnerabilities, is another way of contributing,

you know? And then you have brilliant people who write blog posts about what they

work on. I think that's also a great contribution to the ecosystem.

So. You know, just because I'm doing this doesn't mean that this

is the only way of contributing, like by far not. Of course.

Of course. But it helps everyone. So we need

to make sure that we let you know how thankful we are for it.

So yeah, thank you. Just don't, don't be

mad at me when I approach you about a vulnerability that, that helps me the

most. Nightmare scenario, getting a cold outreach

from Peter. Yeah, I know when people get Slack messages from me now,

it just says, hi, like They're ready.

Their heart rate goes up a little.

Yeah. All right. Well, Peter, this has been a fun discussion,

but unfortunately, I think you have things to do. I have

things to do. It's been more of an interview than a discussion, but maybe next

time we have something actual to discuss again. Fair enough.

Yeah. Well, with that, thank you very much everyone for staying

until the 1-hour mark. As always,

thank you for listening. Subscribe if you haven't,

and otherwise we will see you back in 2 weeks, which will

be after Codebeam Europe, I believe.

So we'll get a little bit of a conference report. You're going? I'm going.

I'm speaking on the 22nd together with Jonathan about our work

of vulnerability research, vulnerability work at EFF.

So I'm there. If you're there too, You know, say hi to

me. I'm approachable. I like talking to people. So just, you know,

say hi. Don't be a stranger.

And then see you in 2 weeks again. Thank you for tuning in. Bye-bye.