Equitable Internet Access for the Under-served - LibreQoS

Welcome to the NGI Zero Podcast where we talk to the people who are building the next generation

internet.

Hi, Ronny Lam.

And I'm Tessel Renzenbrink.

We're both from NLnet, a foundation which supports people who are working on a free

and open internet.

For season two of this podcast, we will be focusing on digital sovereignty through free

and open source technologies.

Our guests today are the team behind LibreQoS, a Quality of Experience & Smart Queue Management

system designed for ISP networks.

We have three guests and that's Robert Chacón, he's the chief executive officer, Herbert

Wolferson, chief product officer, and Frank Borsik, who is the chief operating officer.

Welcome all and thank you for joining us in this podcast.

Thank you for having us.

Yeah, thank you.

Hi, welcome all.

LibreQoS started during the COVID pandemic, right?

Can you tell us the origin story?

Sure.

So, I operate a small ISP here in West Texas and Southern New Mexico here in the States.

And we primarily service rural folks, low income families, families that otherwise wouldn't

have connectivity.

So, we noticed a trend during the COVID pandemic that usage went through the roof.

A lot of access points and choke points along the network were experiencing intermittent

packet loss, which is a phenomenon known as bufferbloat.

And it was causing some issues in terms of people's ability to do online learning, people's

ability to do work from home.

And we realized that we needed to do something and it wasn't practical to swap every router

on the network and put FQ-CoDel on it.

So, instead, we started working on a script that just basically did queuing across the

network and it implemented FQ-CoDel, the algorithm that was co-authored by Dave Täht, one of

our team members.

And it was greatly helpful in terms of reducing the jitter and latency and the buffer bloat

on the network.

So, we noticed that gamers stopped having issues, people who work from home are having

more stable experience.

It was helping with VoIP calling and things like that.

So, we decided to kind of expand on that script a little bit and then we open sourced it.

So, we uploaded it to GitHub and then the whole community has formed around it since

that time.

Herbert has joined us, Frank's joined us, Dave, one of the original authors of the algorithm,

has now joined us.

And it's been a really rewarding experience to see it help ISPs around the world.

But you see, you say that you use it for people in their homes.

So, where do you exactly place this?

Is this in the router in the home or is this in the router at the ISP?

That's exactly right.

It's sandwiched between two routers at the ISP level.

So, typically, an ISP or a smaller ISP might have, for example, a border router, something

that does BGP, NAT, firewall, things like that.

And they might have a core distribution router that then sends the traffic to the different

sites, whether those are towers, OLTs, things like that, that distribute across a fiber

access network or across fixed wireless.

So, it's basically an x86 appliance.

It could be any server or desktop computer that has an x86 CPU.

And it just sits as a sort of transparent bridge between the border gateway and the

core distribution router.

And you had to do this, of course, at every core distribution router.

Yes, I mean, typically for most smaller ISPs, there will only be one.

So, there's only like in the case of my network, there's only one LibreQoS box.

But it's able to shape traffic for up to tens of thousands of users per box.

So, we see some networks now that are deploying LibreQoS at up to 100 gigabits per second.

Other networks are much smaller and might only need a few hundred megabits.

But it scales up to that point.

Amazing. And the box is still there?

Yes, the same one since we deployed it has been running smooth.

It's given us an incentive to really optimize the code on our end so that, you know, as the

network's grown, we've been able to keep the same equipment.

But a lot of smaller operators will, for example, use an old server they have lying around.

There's some in the African continent that have been extremely resourceful and found

equipment that we didn't expect to work, but it did.

For example, CTN networks in Malawi has had great success with it.

But it's been really, really encouraging to see people use hardware of all types to deploy it.

Yeah, exactly. I was thinking about the African continent a friend of mine.

He's delivering internet there for years now, and they have to do it with whatever network they can

get access to to connect to the internet.

And solutions like this really help to share that internet with everybody, I think.

Absolutely, especially in Africa.

I mean, the connectivity costs there are just astronomical compared to what we're used to in

North America or in much of Europe.

Many folks there will pay, I think it's in the ballpark of, you know, two, three thousand

dollars USD per month for 200 megabits of connectivity, something that just wouldn't

make sense in other markets.

But there, you know, maybe it does make sense.

So they need to really fairly distribute that traffic.

If you only have 200 megabits among thousands of users, it is viable, but you have to have a

queuing mechanism in place to make sure that traffic is fairly distributed and that one user

doesn't hog all of the traffic.

Say a video game or something, right?

So that's that's what LibreQoS offers.

So those networks, I think, are able to be not just more profitable and to scale further, but

also to make sure that everyone has an equitable access to technology and especially folks that

depend on it for things like not just work from home, but also telemedicine.

We have a number of clients on my own network that are permanently disabled and cannot leave

their homes very easily.

And they heavily depend on telecommunications for their ability to see

their doctor for routine

checkups.

So I'm sure that that applies to many ISPs around the world that have those kind of clients.

So we hope it's helping to improve equity around the world.

We would be happy to connect with your friend, Ronny.

Yeah, sure.

Let's do it afterwards.

Yeah.

And that was really what brought me on board was I saw Robert's script being talked about.

And the ISP I've been working with in central Missouri is kind of unusual that we also are very

community-focused.

We specialize in trying to get wireless service out to the parts of Missouri that nobody's going

to run fiber to, for example, in the middle of a swamp.

We're never going to be particularly profitable because there's frequently one house per three

miles.

But somebody's got to help them.

And a lot of them were really struggling, especially in the pandemic, because suddenly

kids were supposed to do homeschooling.

And I kid you not, they still have party lines, the old phone system where you pick up a phone

and can hear neighbors talking in some of these neighborhoods.

So getting any sort of wireless out there was a game changer to begin with.

But then LibreQoS, I already knew about FQ-CoDel.

I'd used it in some other projects.

I was super happy to see that Robert was getting Cake, which is an improved version of the same

thing, in.

So one day we put it in place, tried it out, and discovered that the number of people calling to

complain went down by a massive amount.

And that was enough.

I'm one of those weird obsessive programmers that once I've got a good idea, I love to keep

writing code.

So I ended up porting a bunch of it to Rust, got in touch with Robert, merged it, and we

combined our efforts to turn it into something a little more friendly than a giant script.

I've just been blown away seeing how much this is helping around the world rather than just in

my little neck of the woods.

Maybe this is something that you don't want to share, but how many instances are running now?

We can definitely share it.

The number is huge, and it's north of 500 ISPs so far.

And it can be even more, but we don't know for sure.

Because prior to the current versions, ISPs were able to not share any information with us.

So there were installations in the past.

So, but it's growing.

On the market, there are basically five other competing solutions.

We are the only one that's open source.

We are probably the second biggest right now, as for the number of customers.

We're in what, 44 countries?

No, no, no.

It's actually 58 countries and 28 US states on top of that.

It's even more right now because these are the numbers from December of last year.

Wow.

Didn't you also claim to be on seven continents?

It's close.

So there is no LibreQoS instance at Antarctica yet.

Not yet.

We are getting there.

Because in January 2026, there will be a first Antarctic Network Operator Group meeting.

So we hope to go there.

That would be amazing.

It would, yeah.

I would like to go back to what you said, Herbert, about small communities,

underserved communities that have no internet or less good internet access.

Can you, if I understand it correctly, LibreQoS really focuses on that demographic.

Can you explain one of you why you find this so important?

Yeah.

Oddly enough, we started with that as a goal and discovered it worked well for big networks too.

So we have ended up helping a bunch of big networks as well, but we,

you know, our hearts are very much with the small underserved.

People who need the help.

The main problem is you have a limited supply of internet out to the demand that can be either

geographically very spread out, in which case you're using wireless links that may not be the

most reliable because they get less reliable over distance.

Supply can be kind of limited sometimes because none of the big companies really want to spend

what it would cost to run fiber out to be local to these remote areas.

And, you know, when you talk to some of the providers in Africa,

they have really long hops to get out to an internet exchange to talk to the rest of the world.

But even in Missouri, which is far more developed, you are remarkably long way from a good IX

in physical terms when there's something like a swamp and a mountain between you and where the

fiber is because you either have to find a way to run the fiber all the way around or do what

one provider did here, which was run fiber through the swamp and replace it every year

until they went bankrupt, which didn't work out.

So the problem is you've sort of got these automatic choke points that are in place

and demand on, you know, pretty much every ISP oversubscribes.

And if you always had exactly as much capacity as all your customers could use if they all went

full speed at the same time, you'd be bankrupt because you'd be buying huge amounts of capacity

and never using most of it. But oversubscription gets kind of nasty when the bottleneck is how much

you can physically acquire as opposed to how much you can afford to acquire.

So if you've got a limited amount of bandwidth to divide out, what Cake really shines at is

optimizing the experience for the people who are using it, giving them what they can.

And it focuses heavily on identifying interactive sessions. So for example, your Zoom call,

your Voice over IP phone call gets significant better treatment than little Jimmy in the basement

deciding that it's time to download the latest game from Steam. And because flows get

independently tracked within each client and then tracked relative to one another, you get this

wonderful setup that, you know, Jimmy in the basement doesn't ruin your business meeting

by starting the download because relative to Jimmy, your interactive meeting flows get prioritized.

But also in the grand scheme of things, if you're overprovisioned, you're trying to use

more total bandwidth than you really have available, it penalizes the big bulk flows

far before it penalizes the interactive flows where you notice. So your videos don't skip

your meetings are good. You can still watch YouTube, Netflix, whatever streaming service.

But people's downloads do slow down very slightly, but you don't really notice if a 10 gig download

is going to take five minutes longer. Whereas you absolutely notice if the video meeting you're

having is jerking, Zoom is popping up the famous your internet connection is having troubles

message. That's what generates calls. That's what makes people unhappy. And that's what makes things

unusable in these underserved areas. Does that make sense? It really does. Yeah. And this is also

an advantage for gamers, I think. Absolutely. We have some folks on our own network that actually

make their living off of Twitch streaming and video gaming. For example, there's one person that

has a testimony on my ISP's website. They do a 24 hour live stream of a Turtle Tank. And that is

their primary income because it's a very popular stream. But they also do video gaming on Twitch.

And they're able to use their connection to its fullest extent, even when they're completely

maxing it out with multiple streams on top of each other, because of the Cake algorithm,

because we're able to use that. It really provides like Herbert's explaining the

fair priority in terms of latency to the applications that need it while kind of

setting aside the ones that maybe don't bulk flows, things like a torrent download or something like

if you're downloading maybe a Linux ISO or something that doesn't need to be

prioritized in terms of its latency. But it can have a huge impact in terms of someone's income

or if they're doing telemedicine and anything like that, it's a really big impact.

Yeah, exactly. Wow. So that's for the smaller ISPs. We are currently onboarding the biggest one so far.

And they have 800,000 plus subscribers. And they are covering something like eight US states.

So this will be the latest addition. That is good. Yeah.

Rumor is that there is Klingon inside LibreQoS.

That's actually Dave's fault, but I implemented it for him. It's not in the most recent version

of the user interface because we got fed up of so many people calling us and asking us what the

heck that was. But sharing screenshots with all your ISP customer data on them is really painful.

So we added a little thing that redacts customer information so that you can take screenshots.

And version one of that translated everybody's name to a hash table and then pulled Klingon

phrases out saying things like your brother has a smooth head.

But it's not there anymore. It's not. Although I've had enough requests to bring it back

that it'll probably reappear pretty soon. But I also like that people are calling

what is this on my screen? Maybe you should put it in there for that.

Yes. Yeah, those people have to be educated.

I mean, yeah, we were trying to explain it on social media, but not everybody read it.

Yeah, I guess I mistakenly thought that everybody who wanted to use LibreQoS was

going to be on the Star Trek. Disappointment. Disappointment.

I was wondering, could you paint a picture of the fields that you are operating in? And can you tell

us something about your competitors? Sure. So there are a number of, as they're called,

Quality of Experience software vendors that produce software somewhat similar to LibreQoS.

Many of them use a lot of the same tools that LibreQoS is based on. So LibreQoS is based on

the Linux kernel. It's based on additional features such as Linux traffic control system,

eBPF, XDP, things like the HTB, the hierarchical token bucket. So many of the other solutions

leverage a lot of these same open source tools, but are proprietary and closed source. So a big

difference with LibreQoS is that we are contributing back upstream. So we're contributing

back our contributions to the Linux kernel, to the XDP, eBPF projects, and other projects in order

to make sure that the benefits of not just fair queuing, but the ability to move traffic around

the Linux kernel become available to other projects. So we're really making sure that that's

available to others. But a lot of the computing solutions include, I believe, Backwind, Preseem,

Paraqum, and there are a few others out there that do similar work. But we're the only open

source solution at this time. We're the only pronounceable one too.

If you don't do it in Klingon, but yeah. Why did you choose to open source LibreQoS?

My dream when we open sourced it was, I mean, it was maybe ambitious, but my hope was one day

someone in a remote part of the world is going to download this and it's going to have the same

impact for my ISP or for their ISP that they have for my ISP. It's going to have some impact that

allows them to grow and to compete against these huge monopolies that exist in their markets.

I didn't know that that would ever happen, but I thought, let's take a chance. There might be one

person out there who will run this script. Instead, it's been a completely different story than what we

expected. So we're not just seeing one or two ISPs deploying this. It's in the hundreds. Most

recently we found in the Philippines, especially there are hundreds of ISPs that are very small,

one to five operator ISPs that are deploying the software and doing workshops. We actually found

out after the fact. So we saw some photos posted online of these workshops with 20 plus people all

wearing LibreQoS shirts and we're like, what's going on here? So it was just a very welcome thing

to see how this is helping their ISPs. They're spreading the word. They're doing these workshops

on their own initiative and helping these other ISPs in their same country to benefit from it.

So I think it's been great to see that kind of coming to fruition in a way we couldn't anticipate.

Wow. That's an amazing story. So because LibreQoS is also a business, so it wasn't planned to be

a business, but it got so popular that you made it into a business or how did that work?

Yeah, I think originally, I mean, the idea was just make this available to everyone. And then

over time, you realize that there are some inherent support needs and you need some full-time

developers on this project to really do what it warrants. So we decided to try to find some way to

thread things to where we could both support the open source side of it and then also support the

ISPs that are larger in scale that want to use this. So we've added an optional feature. It's

the LibreQoS Long-Term Stats or LTS. And that's basically an optional add-on that allows you to

track statistics long-term in a cloud-hosted way or in the next few months, it'll be available on

premises as well. But it's completely optional feature. The core fundamental technology that

we provide, the queuing, the traffic shaping, the accounting, all of that is fundamentally free and

open source in GPLv2. So we found that that was a compromise we could make that would allow us to

continue to support these efforts around the world and have a nice little extra feature for

the larger ISPs that might want to make use of it. And in the end, there will be more competitors

that will arrive to the market because it's really an lucrative market. There will be a lot of money

to be earned. And we understand that we need to help so many people and money will help basically

to do it. So Robert, for example, can make a choice to give the paid product that we have

for free to someone that really needs it. So we have this kind of control and we can do

these kind of things in order to help people in some countries.

Yeah, we've also talked about trying to keep the pricing low enough that we're effectively keeping

the other players in the market honest. Not necessarily a nice thing to do in some ways,

but I firmly believe that the whole market, much of the internet needs this. It's a good idea to

make sure that it's widely accessible even if you don't choose us. So there's something to be said

for that. My wife told me to stop feeling guilty about needing to eat. So that brought me on board

with the idea that not everything can be open source and I can't give everything away.

So my take on it is when we first tried talking to big rich ISPs about a pay it forward,

subsidize the people who don't have anything type model, that got nowhere. Nobody liked that and we

were almost ashamed that we were getting more donations from Africa than we were from the

United States at one point. So we're trying to work around that and make sure that we can at

least survive in a way that we're comfortable and still give everything away to the people who

really need it. And the core is absolutely going to remain free. Even if we ended up rewriting all

of it, I'm pretty sure core version two would be free because it needs to be free if people need it.

Do we have any other plans for the future? Well, right now we're working on version two of what

we're calling LTS. It's going to be called Insight and it's a very exciting product. I think people

will be excited to see it, but it leverages all of the queuing and the traffic analytics that we're

doing in LibreQos to provide network operators with a lot more insight in terms of what's happening

at a high level in their network. We think it's something that's going to improve the,

not just the usability of the software, but reduce the time it takes to troubleshoot a customer

issue. So for example, if a customer calls in, you basically press a button and it tells you right

away what's going on for that customer's connection. So those tools are meant to help operators as they

scale to be able to save time and to really make sure their customers have a great experience.

And again, the core product stays free and open source, but we have a lot of plans in terms of

improving the product. We don't want to give too much away, but Frank and Herb, are there any

particular parts you want to add to that? What I'm kind of proud of is we've been deliberately

skewing the stats we gather towards what's requested by some of the much smarter than me

doctoral people who work on the core of Cake and future queuing standards to try and give them the

data they need to advance the field as a whole. And quite a lot of them are tied into Linux Kernel.

So I'm hoping that by gathering this giant amount of data, part of what we can do is make life

better for everyone by giving, I'm not sure if I'm NDA'd, so I'm trying not to say their names,

by sending the data in that direction so that they can work on it and do research. And I find

that exciting. I'm not smart enough to do the research. My focus is very much implementing

the cool stuff they come up with, but if they don't have the data, I don't think they could do that.

And it's just exciting to me that we're potentially improving everything.

Yeah, exactly. And it would be very cool to have insights into what LibreQoS is actually

doing in your network and how it is improving the network connections of the end users.

Yeah, I mean, it makes a huge difference in terms of just the day to day operation of an ISP. I think

when I first started out, we had a huge difficulty with customer calls and has a problem. Okay, what

do I do? A trace route? Do I do just ping test, ICMP? It really was not very intuitive to

troubleshoot any issues. I mean, at this point, if you run LibreQoS, just the open source component

of it, you're able to click in on a customer's profile and see all their traffic flowing by you

in real time. You're able to see, okay, well, they're connected to this ASN. The round trip time to

this ASN is higher or lower than the average. Maybe this entire class of traffic is having trouble.

It can really make a difference in terms of the subscriber knowing that you can see what's

going on in a way that helps them. You can troubleshoot in a lot more proactive ways.

We think it really is helping, especially in parts of the world where maybe latency is

varying quite a bit. For example, the Asian continent and many parts of Africa, there's a

tremendous difference in latency between endpoints just because of constricted back calls and fiber

links. We think it helps a lot there, especially in South Africa. We've seen a large benefit to

people being able to see those insights, to be able to see that maybe their connection to

cloud fliers taking the wrong path, that kind of thing. Those insights can really help to improve

the customer experience. Oh, goodness. Do you want to share the story about the

fiber cuts to Africa and the sheer number of messages we suddenly had?

Yeah. It thankfully kept the ISPs that already had it deployed very happy. The ones that were

seeing that all of the bandwidth across the board was dropping to really abysmal levels were able to

continue to run. Even on their 200 meg, 500 meg links, they were equitably distributing the

traffic among all clients, which is something that even if you had more bandwidth, wasn't

necessarily happening when that fiber cut affected you. The fair queuing, I think, really saved the

day for a lot of those ISPs and we got a lot more emails from folks that wanted to try it out.

If there is one continent that's not performing well, that's Europe.

We don't have so many ISPs from Europe and it's really sad for me because they need help.

We see so many speed tests. They are really terrible. We can see high latencies and people

think that when they are deploying fiber, then everything is solved, but that's not true.

We see fiber ISPs in need to use LibreQoS or any other Quality of Experience solutions out there.

We need to get more Europeans on board. That's kind of interesting too. If you look at our

approximate map for where we've had anonymous stats reported, we've got a pretty solid wall

down Eastern Europe. A few in the periphery. It just seems like the middle doesn't like us.

That's because we don't provide native translations. Nobody's heard of us. I don't

know. For some reason, we're very much in a circle around Europe, even including small

islands off Scotland, but not so much in the middle. Basically, the biggest number in Europe

is in Poland, then Czech Republic, but not many in Germany. In Poland, you have so many fiber

and they still need help. It's really interesting. This leads me to the question because Robert and

Herbert, you both run an ISP. Can you explain something about the field of ISPs? What is it

like? Are they monopolies? Are they playing fair? Are they being nice to their customers?

I would say that's the whole reason I started my ISP. I was a student at the local university and

I always wanted to do something to compete against these big monopolies. At the time,

in my city, there was effectively one operator that controlled 90-95 plus percent of the market.

A lot of elderly folks, for example, would be approached by a sales representative and told,

okay, well, you can have this internet service for $50 per month. Sounds great. They weren't

told the asterisk part, which is that after, I think, 12 months, their rate would go up,

not just for the internet service, but for everything. It would go up by not just a few

percentage points. It was going from, for example, $50 per month to up to $270 per month.

Those folks are typically, maybe in my region, not fully English speaking. They maybe are

transplants from other countries. These folks are very much taken advantage of.

I think seeing that was really disheartening to me. I knew that a lot of elderly folks,

especially who were on fixed incomes, were going to be affected by that. I knew some personally.

We decided to start an ISP that would compete against those approaches to business. I think

we've succeeded in our market because of that. I think we've shown, built trust with the community

and shown that you can offer internet at fair rates. Often, just because a region has connectivity

doesn't mean that it's accessible or affordable. I think that's a really big dimension of things.

Just like Frank was saying with having interchanges and exchanges across Europe,

it's important that we have those across the US and across the entire world because they improve

the competitive potential of people in those markets. If there's no IX, there's really no

opportunity for someone to start an ISP like we did. We think that it's important not just to

have a third or fourth option, but also because many of the first three options that people have

are not necessarily within their financial means. I'm going to try really hard not to get myself

deported here. Missouri is a little strange. I live in Columbia, Missouri, which is a city

slap bang in the middle of the state. My city was one of the first to build a fiber municipal network.

We actually built out fiber to every house within city limits. A week before it went live,

the state government prohibited municipal networks and the whole thing has been rotting,

never turned on ever since. They waited until it was built too, so they've spent the money.

The fiber is still there. It just doesn't go anywhere. The state is pretty much divided between

every area that has one cable, has one of two cable providers and one of two phone companies.

They very carefully demarcate where their zones are, so when you move into a house you can pick

one of those two. There's plenty of small ISPs filling in the gaps and they're absolutely vital

because the telcos have been extremely selective. They won't build any of the good stuff out to

rural areas where density is below a certain amount, but they're also very selective in towns.

In Columbia here you can get two and a half gig fiber from one of the phone companies on one street

and not the street adjacent to it because one street everybody's middle class, affluent.

The other street is public housing and doesn't have access to this kind of thing. So my ISP,

guess which of those two streets we serve? It's not the one with great internet. Somebody's got

to do it though because they've got kids, those kids are in school, those kids need to learn.

So I like to say that ISPs, especially the small ones, are the salt of the earth because they are

really delivering something like, you know, internet today is something like water or electricity like

50 years ago or 70 years ago. Yeah, it's really interesting to hear that, well,

both these stories that you tell are quite gruesome.

Yeah, to some extent even some countries in the global south can have it better than the

developed countries. People can't believe that, but I was in Africa with LibreQoS for the first

time last August and it was visible. So even though they don't have so much, they can work

with it and provide better service. One of our customers can have better service with

really small internet packages than the fiber ISPs in the US and it's crazy.

Yeah, you mean because they actually try to provide good internet rather than just make money.

Yeah, and they are trying to leverage the tools, right? For example, LibreQoS.

But what can regular people do in their homes, in their offices to get rid of

bufferbloat, latency, jitter? There are a few options. For example, many folks will run OpenWRT.

Dave's a big proponent of it and OpenWRT is firmware that you can load onto many recent and

older model routers. It allows you to run the algorithms such as Cake and FQ-CoDel that we use

for LibreQoS and it can make a big impact. It's not enough on its own in some cases. There can be

congestion just on the network level for your ISP, but it can make a big improvement for many folks.

So we definitely recommend that folks flash their router firmware with OpenWRT if it's something

that's possible. And there are also non-open source solutions that can really help. For example,

simple thing is to have a MikroTik router and you can turn on SQM, which is Smart Queue Management,

and you will have this solved. And there are other open WRT-based companies that are selling

routers. So you can really get quite a lot of solutions these days. And like as of late,

there is a US company Alta Labs and they implemented Cake. So that's another

good router company that people can get if they don't feel like working with OpenWRT on their own.

Cool. Yeah. That's good information, I think. And also let's try to get Europe more educated

on this for you. Yeah. So we tried at FOSDEM this year, thanks to NLnet. And I was surprised to see

many, many people coming to me to speak about us. And they said they are following LibreQoS online

and they are trying to talk with their ISPs and everything. So we see it growing also in Europe.

Oh, cool. So people are asking their own ISPs like, can you install LibreQoS? Oh, that's the best way

of getting them to know you. From the bottom up? Yeah, exactly. So I was also wondering,

because LibreQoS received NGI Zero funding, how did it help you to improve it?

I should probably answer that because I think I received a fair chunk of it. Where it helped me

was it allowed me to set aside a few weeks at a time to do a deep dive into a very necessary

piece of software improvement, as opposed to trying to fit it in between day job and playing

with my wife and three-year-old. I could actually set aside whole days, work uninterrupted, and know

that I was still going to be able to pay the bills. That makes just such an enormous difference.

All the proposals have been very targeted that whatever we request funding for is going into

the open source section. It'll never touch the other part. It'll be given away free and it's

broad enough that it's going to help a lot of people. For example, in the next funding round,

we have a pretty ambitious target to create hitless rate changes. What that means is,

through the API or the interface, you can change details about somebody's connection

without causing their connection to burp at all. Other than potentially some discards if the speed

limit goes down. I'm not a magician, but I can make it a lot better. It's a hugely requested

feature, partly because some of the big ISPs we've talked to, like one in Poland,

was complaining that sometimes it can take three seconds to change everything on his system.

During that three seconds, his friends would call him. His friends obviously dial faster than I do,

but I'll believe him. Implementing that is going to be a couple of weeks of hard work.

We're not listing the planning stuff. We've got it pretty much planned out how to do it. It's just

physically a matter of having the time to do it and knowing that I'm still putting yummy food

on the table and can relax and not stress out about trying to fit it in between other things.

I'm hugely grateful. It's been an absolute game changer. It's really helped LibreQoS because

suddenly we have these nice targeted items. We have some funding for it. We can go out and do it,

make it happen. Otherwise, we'd probably still be tugging along trying to get them in the spare time

and probably two, three years behind on where we are now.

Yeah. Thank you because this is literally you're making the internet better on a very low level.

It's really useful what you're working on. Thanks a lot for that.

Absolutely.

Yeah. Also, I was always thinking and maybe that's the thinking throughout the whole of Europe

is that bandwidth is free. If you have problems, throw more bandwidth at it.

Yeah. It's a misunderstanding of I think that the access technology, even if you have a fiber

to the X location type access network, it doesn't necessarily mean that you don't have choke points.

If you have an OLT that's only able to provide GPON type speeds, you're going to hit that gigabit

limit. Even if you're providing gigabit plans in theory, most of your subscribers are not

actually subscribed to that. They're typically subscribed to the 100 meg, 300 meg, 500 meg plans.

They can easily saturate that link with a single download of a video game or something in the

background or a Linux torrent, whatever it happens to be. That's something that's not really

considered. It's thought, okay, well, that's too high of a limit. They'll never hit that limit, but

we actually can saturate those limits. Things are well enough interconnected that you can.

Those choke points exist on any network, whether it's a small fixed wireless network in

Malawi or a larger fiber optic network in Western Europe. It's something that we need to address,

whether it's through LibreQoS or other means.

I think those are all the questions we had from our side. Is there anything that you would still

like to share with us? Well, something I wanted to say was to thank NLnet for all of its

contributions. Like Herbert has explained, we have many large projects that have to be done,

especially on the open source side, that are not really viable in the timescale that we would

otherwise fit them into. For example, the rewrite of the UI, which has really greatly improved

accessibility, especially for ISPs in the Philippines and in the developing world,

all of that was made possible by NLnet's contributions. We're extremely grateful for

that. It's been interesting to see how those things that we thought might have been

interesting or helpful have ended up being really critical. For example, the ability for people to

change their settings through the web UI has made it much more accessible to deploy this software,

even if you're not a seasoned Linux administrator. That's something that really opens it up to many

small ISPs around the world. Those are things that maybe NLnet saw before we did in terms of the

impact it would make around the world. We're grateful to see that coming to fruition and

helping all these small ISPs. We thank NLnet for all of its contribution.

Besides actual ISPs, LibreQoS found a way to improve internet also in other locations. For

example, we have helped a company that's giving connectivity to cruise ships and other vessels

already. That's a really interesting vertical that we hope to get in and help even more.

Also, there was a deployment in mining already in Finland. That was also really nice.

There are other use cases. We are told that people are using LibreQoS at LAN parties.

You know, LAN is coming back. People want to meet and play games. This is really cool.

Also, there are university campuses, for example. That's a really nice use case.

If your ISP is not cooperating, you need to fix the connectivity for a few thousand people

that are living and breathing and doing hard work at the campus. That's also something else.

We are always looking for new use cases that will be really helpful to the people.

I just wanted to mention you guys have the coolest stickers. Frank mailed me over a bunch of the

stickers you guys put together. I've been giving them out. My daughter thinks they're the best.

I have to admit a fair number of them have ended up splattered over my daughter, her stuff,

and our fridge. They are really cool, so thank you.

You're so welcome.

Yes. This year, we got two colors, which was a really nice touch from NLnet.

That's really great. Thank you so much for all these nice things you say. It's nice to hear that.

Thank you even much more for building LibreQoS and continuing to do that and helping people

to get better internet. I really like this story, so thanks a lot.

Yeah, thank you.

Absolutely.

Thank you.

Thank you all.

Thank you all much, guys.

Okay.

The end.

You make me think, Robert. Have you said thank you once?

Well, we are appreciative. What Frank was mentioning, this is not just helping the ISPs

like for the mining operation. They were finding ways that before you had to be following these

huge mining machines, miles underground or whatever, with a fiber optic cable and a little

control panel. If the machinery crushed something or it crushed the support and it collapsed,

you were at grave danger. Now, they're actually deploying it so that LibreQoS is keeping the

latency low enough that they say, okay, fine, we now trust that you can operate this machine from

miles above on the surface. All the risk is gone for a lot of these operators. It's potentially

saving some people from unnecessary danger. There's all kinds of use cases like this around

the world. That's why we're so appreciative of NLnet. It's not just going to help ISPs or people

who are gaming or something. It's going to help my client, the one that's paraplegic who depends on

telemedicine for 90% of her doctor's visits. It's making an impact. We just appreciate everything

that you all do to help move forward this mission and to make sure that these proprietary solutions

aren't the only option for people. Regardless of your ISP size, you can help people with this.

The other thing, basically, NLnet was one of the reasons that we were not forced to take

VC money, for example, because we were approached quite early. We were approached by VC funds

because they see that there is a chance to earn a lot of money in this space.

We can stay private. And honest. NLnet's allowing us to stay honest, right?

And honest, yes. If you get VCs in, the honesty changes.

It's a lot of pressure. We've seen companies like, I think,

ZeroTier. There's been companies that have just privatized their own open source

solutions that were developed by the community and we don't think that's the right thing to do.

You see these other solutions that we compete against, like Preseem and Paraqum. They're all

based on fundamental technology that was open source and none of those contributions are going

back upstream. So, NLnet's making sure that all this goes back upstream, like with Toke's work and

all the folks at Karlstad University that are working on the kernel. We're just really

grateful that this ecosystem can be fostered and it's thanks to you all.

Great, thank you.