The internet should be a public good - NixOS/Clevis

Welcome to the NGIZero podcast where we talk to the people who are building the next generation

internet.

I'm Ronny Lam and I'm Tessel Renzenbrink.

We're both from NLnet, a foundation which supports people who are working on free and

open source technologies.

Our guests today are Julien Malka and Camille Mondon.

Julien is a PhD student in Software Supply Chain Security at the Polytechnique Institute

Télécom in Paris and he is also a NixOS contributor.

Camille Mondon is a PhD student in Statistics at the Toulouse School of Economics.

Together they work on a project that implements Clevis in NixOS.

It makes it possible to remotely reboot servers with full disk encryption without needing

a human in the loop.

That project was extended to include Proxmox NixOS which we'll also be talking about today.

Hi Julien and Camille, welcome.

Hello, thank you for having us.

Yes, hello.

First of all, did we miss something in our introduction?

No, that was quite perfect, yes.

So what key issues do you see with the state of the internet today?

Well, today I guess one of the biggest key issues is that the internet is quite owned

by a monopoly of Big Data Corporations.

And basically it makes it difficult for people like us that have particular interest in computer

science and internet to make a living out of it while staying in the open source community.

So I guess it makes it very important to have funding, public funding and institutions that

stay strong and to still consider that the internet is a public service and should be

more of a public good.

For instance, just try to self-host your own mail server and you'll see the difficulty

that you can meet while doing so because basically big companies have got their hold on these

activities and self-hosting is not so easy.

Yeah, like Camille said, I think internet is very, very far from being actually a common

good or a public good and it doesn't belong to people.

It's actually controlled by a very small number of very powerful entities, commercial entities

usually that have their own agenda and that sometimes have political bias in this agenda

to continue making profit.

I think one of the key issues today is the issue of centralization, bringing back the

power to the people, also involving empowering people to own their little share of the internet

and not be controlled by a few companies that control search results, social medias, basically

ideology bubbles, everything.

How does your project contribute to addressing those issues?

I think our projects come from the idea that both Camille and I wanted to have some piece

of control over what software we use on the daily basis.

We started building some kind of very small home lab micro data center to host our own

services and we found that we were facing some difficulties in some areas and one of

them was having an easy way to have full disk encryption on our servers without it being

too much of a hassle, like when you have to reboot the server with full disk encryption,

you physically have to do an action on the server remotely but you have to be there and

do it.

It was a blocker for us and that led to our first part of the project on Clevis.

We found out that using NixOS we were able to much more efficiently maintain our services,

our self-hosted services, but with the virtualization technology that we were using, Proxmox, we

also found out that we had to go away from NixOS just for our hypervisors and that led

to the second part of the project.

And basically, our project contributes in two ways.

First on the macro level, it's a project that is part of the alternative approach to the

centralization of the internet that we discussed earlier because it is a project that is fully

open source and funded by public institutions and also on a more individual centered level,

it enables people like us without a big structure and just like people that have passion for

computer science also to maintain their own infrastructure without the human power necessary

to have quite a big infrastructure actually because with Proxmox or with Clevis you can

really have a fleet of servers or many services and just because of the, I'd say the cleverness

of these open source projects and the ideas behind them, you can just be like two or three

people and have a really big infrastructure.

Yeah, I can agree on that.

We are doing the same for the NLnet network and yeah, it works great.

Other people use other automation services for that and we do it all with the Nixos.

What would be the use cases for remote unattended disk encryption?

Well, disk decryption, I must say.

I think the use case is that this decryption is a necessary security measure that everyone

should adopt today but on laptops it's becoming easier and easier to do so with a lot of distributions

that have an easy toggle to do it but on servers you always have the problem that when the

server boots you need to input your password to continue the boot process.

It can be done usually physically, you have to be there and type the password with your

keyboard for example but you also can do it remotely but it doesn't remove the fact that

when the server reboots you have to be there and have a manual intervention.

The idea is that with Clevis you have a framework that allows for automatic decryption of the

disk using some kind of secret and the secret can be either locally on your device with the TPM

or it can be also stored on other machines and so you have the idea that as long as you have one of your

servers that is booted and up and running it can help the other server boot unattended.

Yes I understand that and I think that's very helpful but isn't it true that after the key exchange

the key will be in memory right?

Yeah it's true.

So for a VM with a hosting provider that hosting provider might also have access to your memory?

Yeah that's true. So that's the whole question around trusted computing and the new security extensions

and processors that will help us go further into building trust within an untrusted hypervisor.

That's not exactly the use case here that we tackle mostly this is useful for booting bare metal

and this was also the reason why when we were in a situation where the hypervisor was not running

NixOS but we were running all our VMs on NixOS we were having this frustration of having our

Clevis contribution not be that useful to us and that's why also we wanted to port the hypervisor

under NixOS because now we can have this the hypervisor use this encryption and we have a cluster

of several servers and they can help each other boot and decrypt the disk at boot using Clevis.

What were the challenges to get Proxmox running on the NixOS?

I guess the code base is pretty huge on Proxmox there's many packages that had to be incorporated in the same project

and yes because of the way that NixOS function there was a lot of patching to do in the Perl code base of Proxmox

and also the Rust ecosystem because they're currently moving from a Perl based project to Rust slowly

and so that was quite time consuming I'd say.

The Proxmox ecosystem is of course free and open source but it's not really designed to be run under different conditions

than where it's been developed and it's been developed to run under Debian and so you have a lot of I would say

things that are assumed to be there or paths that are assumed to be existing or the Rust ecosystem relies a lot on the Debian APT system

so it's kind of we had to strip the software of its original home and make it a new home under NixOS.

Was it hard to get your changes, your patches into the Proxmox code base? Was it hard to get them accepted?

We didn't get any patch accepted or submitted to the Proxmox code base. We have some patches that are really NixOS

relevant only for NixOS and then we have stored these patches locally.

We have a few that we feel would be interesting to upstream especially Camille adapted a small part of the code base

to be able to build under ARM or AH64 and so we feel even though the upstream project doesn't really build NixOS or packages for ARM

we were able to do it without too much hassle just changing a sub part of the code and we are planning to submit this patch soon.

And going back to Clevis, you said it's good if at least one of your services is running.

Does that mean that your challenge can be accepted by any of a running service or must it be one specific server that answers the security challenge?

Actually, if I understood the question correctly, I guess the Tong server can be...

You can actually describe your own security protocol if you want either one server and one TPM.

You can basically, using Shamir's secret sharing, describe your own security scheme, either any of the servers or all of them.

Okay, clear. But what were the challenges there to get Clevis working?

Yes. Well, this was actually very difficult because coming back to Proxmox, it was mostly about pioneering in the direction that people never actually ported Proxmox out of Debian.

But for Clevis, the difficulty was very different because it was mostly about understanding the boot process and working in the first stages of the boot process in its RD.

And since on NixOS, it works quite differently from the other distributions.

And also, we spent a lot of time working on a test that would be able to prove that Clevis works efficiently and correctly when installing a new NixOS machine, configuring some Clevis keys and a Tong server, and then rebooting and checking that the partition is deciphered correctly.

Yeah, I think as Camille said, we spent a lot of time designing a NixOS test, which is an integration test for a feature, which is a very nice part of the NixOS ecosystem.

So you can have tests that spawn NixOS VM, run some commands on them, and then check what have some kind of assertion on the state of the VM.

And we wanted to do a very, very extensive test where we could install a brand new NixOS machine, then provision, like do the partitioning, then test all.

We have three kinds of disk encryption, one with zfs, one with bcachefs, and one with luks.

And we tried these three different kinds of this encryption.

And then we provisioned the machine so that it uses our Clevis module to decipher the disk at boot.

And the tests were able to show that both that our module was working to decipher the disk when booting, but also if there was any kind of struggle or problem, it would roll back to a simple passphrase handling like usual.

So we were able to prove during our inside our pull requests that we were not going to destroy anyone's setup with this change.

And that's where that got us a long way into getting it accepted.

So that is nice added information.

This means that even if the tank server cannot be reached, your disk is not lost, but you fall back to passphrase on the console, which you of course have to physically access.

Yeah, exactly.

So with NixOS, you could always, if you do something wrong, you could always roll back to an earlier generation and have your disk be safe like this.

But we also wanted to make sure that in any case, the worst case scenario is you fall back to the original behavior without Clevis.

Yeah, yeah, sure.

So I'm hearing NixOS a lot now.

What is so special about NixOS?

Why do you like it so much?

Well, I guess from a quite outside point of view, because my main domain of expertise is not actually computer science, because I'm a PhD student in statistics.

And so I actually use NixOS on a daily basis for my work, because basically, as I said, it makes me able to maintain quite a big infrastructure on my own.

And to be sure, like when I do something, I don't have to do it again if my server crashes.

For me, it's really a concrete way to make use of what the open source community has to offer and to speed up my work in statistics and to have a reprehensible environment.

And me personally, when I discovered NixOS, I was already a Linux user for some time.

And then a friend of mine showed me what NixOS is.

And I started because we are both academics and we are both I am a theoretical computer scientist from my academic background.

So I started reading the Eelco's, the creator of NixOS, PhD thesis and also other kind of resources I could find.

And my conclusion, why this is how we should have done computer science forever.

And I started to find it very elegant.

And that was basically what pushed me to learn it, because it's also quite a difficult journey to learn Nix and NixOS where you need motivation not to say, oh, I don't really understand how I should do this, this and there is no documentation online.

So I would just fall back to something else.

So I continued in this process and I learned it and I started using it for my my different computing devices.

So especially my servers, because I had a few servers, I was already self hosting stuff.

And then it became interesting to me that it felt way less work to maintain all these services that I was using.

And before that, I was mainly using container solutions.

And they were breaking quite often on me and that was some work to keep them up to date and working.

And when using NixOS, it felt that this was working just for free.

It was really something that I learned over the year of using it, that it was way less taxing on me to maintain my services using NixOS.

And to me, this is like you have the technology, Nix and NixOS are interesting technology,

but you also have a very high value in the set of people that are packaging things into Nix package and maintaining these things,

because they pour all their expertise into several into some topic, into packaging modules and software into NixOS

so that people then can use them without needing all the knowledge to operate them.

And that's what we did with Clevis. Clevis is not a very complicated technology.

It's also not a very simple technology to set up if you don't know anything about it on other distribution.

But on NixOS, you don't really need the domain specific knowledge that we had to acquire to write the module to operate it as a user.

And that's something that is really interesting in the NixOS distribution, according to me.

Yeah, here you have it. Both ways you can arrive to NixOS.

The hard way like Julien did, like trying for many years many different things, containers,

and having to try every technology possible and to reboot each machine after it has crashed and restarted from the beginning.

And I, when I arrived and that Julien has already discovered NixOS,

I actually only ever installed an Arch Linux, one Arch Linux machine in my life and used it for like something like 30 minutes before Julien told me,

yes, you should switch to NixOS. It's a little more difficult just the first on the first days.

But actually, I guess I kind of dodged the bullet and just started right away on the easy path.

Yeah, it's a learning curve, but it's well worth it.

Especially when you, like you said, when you want to maintain several services.

You mentioned community. So, so do you already have a community behind Clevis or are you planning to build a community around it?

What does that mean for long term sustainability of your of your projects?

So according to me for Clevis, as we upstream our change into Nix package, there is no real notion of community.

We gave it to the community. We plan to maintain it over time.

But it's kind of now including it included into a bigger project.

But for Proxmox NixOS, we have it as a standalone project for NixOS.

We have it as a standalone project for now, planning to upstreaming into at some point, but not right now.

And we have already so that there are quite a few people that are interested into the technology and participate to the project.

And as it's a it's quite a big piece of software, it's kind of a relief that other people want to help and have their own idea and their own extensions or project into this inside this big projects so that they can do the project also belongs to other people and will evolve.

Not just by our fact.

Yeah. And I'll add that for Clevis.

The point was kind of to make an initial proposal like here is how you can use it.

Some examples. But the thing is, it's quite personal and how you would use Clevis because your each setup is quite different.

So the idea was really to use the power of NixOS and Nix packages to like propose your use case and people can rely on it and then propose their own.

And of course, for instance, people might want another kind of partition scheme or format.

And maybe this is we didn't we only use the zfs and Luks and bcachefs.

And if people want to add something else, then they contribute.

And that's how it works, I guess.

That also leads me to the question, because you started this project with the NGI Zero funding as Clevis.

But then you morphed also into Proxmox.

But both these projects already sound separately rather big.

How did you manage to do so much in only one project, which are actually two projects?

Long of hard night's work, I guess.

No, no, but actually they were for us quite like they were our big project on a personal level.

Like that's what we needed for our infrastructure.

I guess that there was there were both the next logical steps because we had already some services deployed.

We had already an supervisor with Proxmox on Debian and we needed to move to full disk encryption.

And so, well, we took some time because we've been actually thinking about it for a long time.

And so we had some ideas that already had some were already a little more mature than maybe like starting the project right away.

And just going in the open and we yes, we knew what we wanted to do precisely, I guess.

And also that's maybe more of response, but we did the Clevis part and I reached out to Michiel and he said to me,

Oh, you want to do this? Let's just add it to your first project. It's easier.

OK. Are there any next steps that you are thinking about?

For the Proxmox project, we are we have quite a few people that reached out with suggestions for extensions.

And that's basically what we we think we will do whenever we get a bit of time to do it.

But the two extensions that we think about currently is implementing some kind of

option for declarative configuration of the virtual machines on the Proxmox host.

So so far, we just have a Proxmox instance running on NixOS.

But there is minimal configuration that you can do through the NixOS module system.

But when you want to create a new virtual machine, you have to use the web interface as people do when they use Proxmox.

But because we are NixOS users, we also consider having another way to configure this virtual machine through the NixOS configuration itself,

which some people have expressed interest in.

And we also want to consider other storage layers for the virtual machine.

So you have typically in Proxmox setup, you can have a safe cluster that acts as a layer for the storage of your virtual machine

so that you can move some virtual machine from one physical machine to another super easily.

And it helps you achieve what we call high availability.

Like you get one node of this cluster that for some reason has a problem and shut down the virtual machines that are actually running on this node.

They can migrate to another node so that you don't you don't get service interruption.

So there is safe that is already included in NixOS.

And we had suggestion for implementation of something called LinStore, which is another layer.

And we are we are also considering this.

Yeah. And also, I guess one concrete big step will be to also move our little homemade data center fully to our Proxmox NixOS.

And it would be quite an interesting proof of a concept and of robustness of our implementation.

And and yeah, usually we will really use the fact that safe allows like to disconnect one node, then may reinstall this node on Proxmox NixOS.

Hope that that we actually don't lose any of our our infrastructure.

But yes, that would be once this is done, we kind of achieved what we we wanted to do for a long time.

So far, we've we've developed on a development cluster and we had like we had no problem so far.

We think everything works correctly.

But the next step is committing with our own hardware and move like migrate our production cluster, which has a lot of edge case and weird configuration.

So we know that if there is something not working, we will find it at this point.

So Proxmox cannot cannot only be used to build physical machines, but also for containers.

I was just thinking, doesn't it make it easier to build NixOS based containers when there is a NixOS layer under it?

Well, honestly, this is not this is not our use case.

We don't we don't run containers.

So we didn't really go too much in this direction.

But we hope we hope that people that have different use case than ours can do their own experimentation and either report if something is not working as they as they think it should or contribute just to make it also work for their own use cases.

There is quite a lot of stuff that you can make with Proxmox.

And for sure, not everything is working today.

Only things that we have tested because we know that we need it and that most people need it.

But over time, we will get this this proportion of things working over the total bigger and bigger because other people we come and say, oh, this thing is not working.

That's like I think it should and they will fix it or we will fix it.

Yeah, and clearly, I think there's a lot of wonderful ideas yet to be yet to be had on on based on this Proxmox on well this hypervisor based on on NixOS and and then all of your containers and VMs that are also on NixOS.

And I guess that's a big direction in towards smashing the state and be able to have like a fully fully reproducible.

Well, fleet of VMs and containers.

Yeah.

Maybe coming back to the community question again.

I mean, for for Proxmox, Proxmox, there is already a big community and I can only think that that that there are also people that are in the NixOS community that are also in the Proxmox community.

So I can only imagine that that that there is a community waiting there to to support you.

Yeah, that was actually quite a big surprise because the day that we released the project, we were really happy to see that many people were already like waiting for it and contacting reaching out to us to say, yeah, that's that's super useful.

And so that like that's that makes us happy to have contributed to this project.

Yeah, we were we were not expecting this kind of hype, but we had a lot of people reach out.

I guess 300 stars on the repository in a few days.

So I think there is, of course, a community of people that are really interested in that.

I don't know how much people have actually taken the step to use it in their own production environment.

But the idea is that everyone that is committed to to to use something that is not commercial because our project is not commercial and will not do any kind of support like the Proxmox project does.

We will help them move to it and we will accept contribution and issues.

So we would be very happy having this kind of little community around the project that make it grow.

Yeah, and it was nice to to to see the impact of the of this project because mostly because we're in Ph.D.

both in Ph.D. are some of our projects were like research oriented like packaging,

like the Lattek editor or Zotero, which is for bibliography management.

And while the hype was really bigger than on this very niche little projects and so very, very, very happy to to see that.

Yeah, interesting. And how did NGI or NGI Zero help your project?

I guess by by giving us the chance to have time to dedicate on this.

We it's difficult when you when you're a Ph.D. student or to to to be able to dedicate some of your time to to open source and free software like you have to to prioritize things.

And often like for me, open source is a priority.

But being able to dedicate some time being financially supported is is luck that that we've been able to have thanks to NGI Zero.

And probably we would have done this kind of project anyway at some point.

But having this financial support gave us legitimacy and to allocate some time of some part of our time to do it and do it as soon as possible.

Yes, clearly the impulse, they gave us the impulse to to work on it and to make something that is... yes to go,

I'd say to the last step of the project, and not just like start it and then abandon it for and that was really nice to to to achieve something.

And and also I say that's in a way quite I'd say financially interesting because in a way it's something that we we do on our free time.

And and if it was if it were our main work, maybe we would get in a way paid more.

But we maybe would do it with less passion because here it's really something that we use for our needs.

So we really wanted to make it the best we could.

And and so the motivation and the financial support is really a good I guess a good way to to support projects and maybe more efficient that what you would meet in a private company or something.

And if you had to give any advice to other people who are considering to apply for funding, what what would you say to them?

Yeah, I'd say do it apply.

It's you you never know what what will happen.

And so do try to apply the application process is not to taxing actually.

It's not something that's going to take days and days, probably just a few hours.

So you can if you can afford taking these few hours, writing your ideas down and trying something that's that may become a very nice experience.

It's always worth it to do it.

And I want to mention that when I applied first for for Clevis, I had some very insightful feedback from while being rejected.

I had some very insightful feedback from from the NLnet team.

And that was really interesting to give me a new angle and vision on what I wanted to do.

So at the beginning, the project was based on another technology called Mondos and it was rejected.

But the feedback helped me have a better overview on the ecosystem that were existing, was existing on this kind of software and to do actually a new a new application that went through.

And it's really interesting to see that we have teams, a team of very, very informed, well informed experts that can give you feedback that will actually make a difference for your project.

Yeah. And I'd just yes, do it.

Do consider applying for an NGI Zero funding and and do it with the right idea that you're contributing to something that is funded by a public institution.

And so you're you're contributing to to making Internet a common good.

And so, yes, that's your project should be oriented in that direction.

Yeah. And maybe then some advice from our side that would be the feedback is indeed very helpful.

And maybe sometimes you don't get the feedback proactively.

So always ask for the feedback because, yeah, we or they another grant grantor, not grantee.

Yeah, they they they need to give you proper feedback about why you are why you were rejected and you can learn from it.

Yeah. And really, at some point, we thought that we might not do do it and get not get funded because we we were first rejected.

But that's where like thinking like really taking into account the feedback and like modifying the project in the way that Julien did.

Well, it was really relevant because in the end we we we got accepted and and and and yeah, the project was indeed better.

That's using the other technology. And now we do use a clevis that we even didn't know before.

Well, when we first started the project. So I would like to add a little disclaimer for people who are listening because currently, at least the people who are assessing all the applications, they are very, very busy.

And so getting feedback might be a little less likely now.

Maybe it will again, you know, in the future be be be possible again.

But I think at this very moment it will be a bit hard to to get for them to make time for it.

Although everybody agrees that it is super important, but it's a bit busy at the moment.

Yes, that's why we need more money in that in that in that domain.

We need more more more people helping.

And yes, that's really an important step to to get rid of this issue of centralized Internet.

And yeah, yes, I agree.

It would be really good if because, of course, NGI Zero is financially supported by the European Commission.

And that's great. But also maybe nation states.

Germany is now also putting some money or a lot of money into open source.

It should be more normal for public institutions to support the public Internet.

We cannot leave it to companies to build it, obviously, because it becomes a dystopia.

It's also very sounds like investment strategy for for the public sector.

Like you keep we are we are draining so much money, at least in France, being,

I'd say, dependent on software that is proprietary and very expensive and keeps you locked into one kind of ecosystem.

And and just putting money into building blocks for software or for the Internet that is a common good,

that is free software that is built by passionate people that do not have in mind capitalistic goals or is just for me a very sound political strategy.

And in a way, we save so much money in comparison when we like paying a very, very expensive well,

a big, big money to the to a tech developer that would maybe just work on its specific use case for its specific company with a close source and code.

And then and then no one can rely on it to to make well advances in society.

Cool. Maybe coming back to your projects, how can the listeners contribute to bringing the projects you are working on further?

Basically, I'd say do try Proxmox, do try Clavis.

And we've already had some some feedback and we want more and like also contribution because, of course, we're still doing Ph.Ds.

And yes, we need people to actually be involved in it.

So because we cannot like think of all the use cases that people would imagine.

And so, yes, we already have some warm messages and we want others and more.

And yeah, yeah. And knowing that both of you are doing Ph.Ds, the grant only gives you money, not time.

Right. Exactly. Only 24 hours in a day.

And we spend some days, maybe a lot of these 24 hours working on on these on these on these projects and really happy to have done it because now we can move to next steps.

We don't really know where for right now.

But yeah, we still a lot of it's quite infinite, actually, because it's both a passion and work.

So, yeah. Is there anything that that that that we that we missed in our conversation that that you need to add?

I just would like to add that all this journey from Camille and I was fueled by the idea that we should be less dependent on software.

We don't have control on software that is managed by companies that can access, analyze and sell our private data.

And it was it was a journey to own our software again.

And I would advise anyone listening to ask themselves, I might have, what kind of software do I rely on on a daily basis?

And if you're using commercial software, are you at ease that the software might be that the data you're giving to the software might be

harvested, analyzed, sold for purposes that are in no way aligned with your own purpose?

Are you are at ease that software that you might using abusing on a daily basis is trying to control the way you think by showing you targeted contents?

And you have no way to control this. Are you at ease that, and that was, I'm saying this because I was, I had this personal reflection that am I at ease that software media that I'm using is specifically designed to make me addicted to it.

And at some point, if you're, if one of these answers is no, maybe do read about self hosting and free software.

And also, it's more easy than you think to migrate some of your users, well, use cases to open source and self hosting like really what we, what I learned I was from I was starting from nothing.

Basically, I was just enjoying computer science like from afar and in only a few few months I already had my media server my services.

My services. As I said, only the mail server is quite difficult. But one day maybe we are in a society where it's possible to self host your basic computer site, well, internet needs and yes to have kind of a freedom and and basically yes.

So keep on smashing the state.

And that's the whole reason we are working on NixOS is because we believe NixOS can make it so that you don't need to be an expert in computer science or infrastructure or DevOps to be able to have your first feet into this.

Your first foot into this new world of self hosting and owning your own software.

Yeah, we couldn't say it better. I mean, this is what NGI stands for the next generation internet with which is more private, more secure, and more sustainable.

And more user control.

Exactly.

Sovereignty.

Yes, thanks a lot for that passionate call for a better internet and by extension a better world.

Thank you very much Julien and Camille for this conversation. It was really enlightening.

Thank you so much.

Thank you.