---
date: '2025-11-18'
title: 'Joining Canonical to work on RISC-V support for Ubuntu'
tags:
  - riscv
  - ubuntu
---

Two months ago, on September 1st 2025, I started a new job at Canonical, the company behind Ubuntu.
My role involves working on RISC-V architecture support for the operating system.
It allowed me to learn a lot about Debian and Ubuntu, packaging for and maintaining an operating system,
as well as how the RISC-V architecture and ecosystem work and evolve.

<!-- more -->

## Joining Canonical

### The interview process

As you may have heard, the interview process at Canonical is tough.
It takes a **long** time, includes some unique steps that might seem unusual, and their overall approach is
very distinctive, which can be confusing.

However, I didn't find the experience *unpleasant*. Of course, any intensive process is challenging, but
given that I was still in university (completing my master's internship) and then on vacation, I wasn't overwhelmed.

After submitting my resume, I went through a **written interview**, followed by a **technical assignment** completed at home, and then a series of **technical interviews**. Later stages included an HR interview and meetings with potential future managers and colleagues.

After many rounds and nearly six months, I received an offer and happily (but nervously) accepted.

Overall, **yes**, their process is unique and demanding, but I believe it makes sense for their situation:
- They hire globally for fully remote positions
- They are a world-famous company (thanks to Ubuntu) but remain relatively small
- They have a very opinionated, data-driven approach to talent acquisition

### Onboarding

The onboarding process started even before my first day.

Since much of Ubuntu's development happens on [Launchpad](https://launchpad.net), I had to create a [Launchpad account](https://launchpad.net/~vhaudiquet) and register it through Canonical's internal systems. There was also the standard paperwork for any new job.

During my first week, I discovered the company. After conversations with my colleagues, I worked through
various administrative tasks. On the technical side, my team explained the context of
Ubuntu and RISC-V. Basically, RISC-V had a new ratified CPU profile, RVA23, which mandates RISC-V extensions 
like vector (SIMD), hypervisor, and others. On Ubuntu, starting from 25.10 Questing Quokka, RVA23 will become the
baseline, and older hardware will no longer be compatible (only supported by 24.04 Noble Numbat and below). The twist
is that RVA23 hardware is not yet available, so we do all our work in virtual machines. Supporting RVA23 early is crucial for bridging the gap between RISC-V and other architectures. Our main task is to rebuild the packages in the archive using the RVA23 compiler profile, and fix the issues that might arise from those rebuilds.

## Initial work: fixing bugs

During my first week, I was (also) tasked with dealing with **FTBFS bugs**.
An FTBFS, for **f**ailed **t**o **b**uild **f**rom **s**ource, is when a source package fails to build on the archive.
Ubuntu builds its *main* archive of packages from source code, and we need these packages to always be *buildable*.
This ensures we can patch and rebuild packages to fix issues.
To ensure that, scheduled tests rebuilds of the archive are run regularly.
When a package fails to build, it is treated as a **bug**, and a report is filed on Launchpad.
If a package FTBFS's only on the `riscv64` architecture, it becomes my (team's) responsibility to fix it.

Here is what I worked on in my second week:
- Fixed [LP: #2121938](https://bugs.launchpad.net/ubuntu/+source/mozc/+bug/2121938) on the **mozc** package.
  That bug was caused by a library expecting `-std=c++20`, but the actual compile flag was `-std=c++17`.
  This was a straightforward fix that helped me learn how to patch a Debian package and forward the change.
- Fixed [LP: #2121937](https://launchpad.net/bugs/2121937) on **libcupsfilters** package.
  That was a `dpkg-gensymbols` error. The compilation process for this library performs strict checks on exposed symbols, to
  ensure compatibility. However, the expected symbol file contained
  earlier bugs, causing the build to fail when newer RISC-V vector symbols were added.
- Fixed [LP: #2122479](https://launchpad.net/bugs/2122479) on **protobuf**.
  protobuf required a recursive function to be optimized with a tail-call, but GCC was not able to do that on
  `riscv64`. The bug was already patched upstream, making the tail-call requirement architecture-specific.
  I simply backported that patch to the Debian package.

Later, I also worked on **autopkgtest** bugs. After a package is built, it is made available on the 
`proposed` pocket, which is a specific repository of the archive that users typically don't have enabled on their systems
(unlike `release`, `updates`, or even `backports`). To migrate from `proposed` to `release`, a package must pass all automatic tests (*autopkgtests*), along with tests for its reverse dependencies.
Again, if these tests fail only on `riscv64`, it's our job to fix them.

Here is what I did in my third week:
- Fixed [LP: #2122492](https://bugs.launchpad.net/ubuntu/+source/pam/+bug/2122492) on **pam**.
  That test was failing for `arm64` and not `riscv64`, but it was triggered by a rebuild that my colleague had done for `riscv64`, so I was involved in the fixing process. The solution was to rebuild all the dependencies of that package, as well as the package itself, to enable a new ARM security feature (GCS) that made the linker complain.
- Fixed [LP: #2122271](https://bugs.launchpad.net/ubuntu/+source/inetutils/+bug/2122271) on **inetutils**.
  Again, that test was failing for all architectures, but was triggered due to a `riscv64`-needed rebuild.
  The issue was a faulty test, which needed to be removed.

I worked on more *FTBFS* and *autopkgtest* bugs after that. While the process always starts with checking a log file and ends
with sending a patch, the steps in between can vary wildly, which is really stimulating.
While I've described my personal work, it's important to note that every bug fix was a collaborative effort.
I consistently received help from different colleagues, which also gave me the chance to connect with many people on the way.
However, fixing bugs all day can become monotonous after a couple of weeks, so I began looking into new features that were needed for Ubuntu on `riscv64`.

## Beyond the bugs: features!

After that initial bug-solving phase, I was tasked with investigating feature support.
The RISC-V community had introduced new extensions, `Zicfiss` and `Zicfilp`, to provide RISC-V with *control flow integrity* (CFI)
features.

### Control Flow Integrity?

Control Flow Integrity is the collection of techniques preventing malware attacks from redirecting the flow of execution
(control flow) of a program. In this section, I will try to give you an overview of what CFI is, and why it is important.

#### Backward-edge CFI

##### The `return` mechanism
::BlogImageText{image="/img/joining-canonical-riscv/callret.png"}
*The following section assumes some architectural knowledge, e.g. how function calls work at assembly level.*

On most architectures, the `return` mechanism uses the **stack** to store return addresses during function calls (at least for non-leaf functions, where a register used to do that would need to be *spilled*, saved on the stack).
::

##### Stack smashing
::BlogImageText{image="/img/joining-canonical-riscv/smashed.png" imageRight=true }
A basic attack called **stack smashing** exploiting this mechanism involves using a memory corruption vulnerability in the target to overwrite the return address on the stack, as well as copying a **payload** on it. 

This can be easily avoided by disabling execution on the stack memory.
::

##### Return-oriented programming
::BlogImageText{image="/img/joining-canonical-riscv/rop.png"}
A more advanced attack, **return-oriented programming** (ROP), only overwrites multiple return addresses on the stack to
direct control flow to specific execution sequences, called **gadgets**, to fulfill the same role as a **payload**.
::

##### A solution: shadow stack
::BlogImageText{image="/img/joining-canonical-riscv/shadow_stack.png"}
To prevent this kind of attack, one solution is to use a **shadow stack**: a second stack that only stores return addresses.
It's immune to buffer overflows since it holds no buffer, and each `return` operation compares the address from the regular stack with the one on the shadow stack.

This can be implemented in several ways:
- Compiler support, modifying function prologues/epilogues
- Dynamic binary rewriting
- **Hardware support** (the best)
::

##### RISC-V Hardware support: Zicfiss

`Zicfiss` is a RISC-V ISA extension that provides hardware support for a shadow stack.

The shadow stack is implemented using a dedicated `ssp` (shadow stack pointer) register.
It is possible to push an address to the shadow stack using `SSPUSH ra`, 
and to pop (while checking that the address matches the one in `ra`) using `SSPOPCHK ra`.

To set up the stack, `SSRDP rd` reads `ssp` into register `rd`, and `SSAMOSWAP.D rs1 rs2` allows switching stacks (`ssp`).
To protect that stack, the MMU contains a new page type, `SS` page, which allows only the previous instructions to access it.

These instructions are designed so that an RVA23 RISC-V processor without CFI support can execute them as NOOPs. This means we can enable the feature broadly; processors with hardware support will benefit, while others remain unaffected.

Implementing this requires support from the compiler (`gcc`), linker (`binutils`), kernel (`linux`) and dynamic linker (`glibc`). Dynamic linker support is crucial because it allows dynamically turning on/off the feature depending on whether or not every dependency of the running program supports it (if a library expects a shadow stack populated, but another library calls it without shadow stack support, everything will go wrong).

#### Forward-edge CFI

A similar attack is possible for forward-edge flow (i.e. indirect jumps). I won't go into details here,
but you can research "jump-oriented programming". The corresponding RISC-V extension is called `Zicfilp`,
and implements a solution called "landing pads".

### RISC-V CFI support on Ubuntu

That was a lot of information!
The important thing: there are security hardware extensions (`Zicfiss` and `Zicfilp`) that we want to support on Ubuntu. However, they need software support from the entire toolchain, before trying to rebuild
packages with that extension enabled. For context, other major architectures (amd64 / x86_64, arm64) have CFI hardware support, with enabled software support in Ubuntu ; so investigating CFI support on RISC-V is another step towards bridging the gap.

First, I needed to check the status of upstream projects (`binutils`, `gcc`, `glibc`, `linux`) support.
At that time, `gcc` and `binutils` had upstream support (`binutils` was missing one patch), but `glibc` and `linux` support
was not merged (though patches were available on the mailing list).

The next step was to try and build those packages with the necessary patches, which was not easy. In fact, some of those packages are still not built at the time I write this article. This is the role of the operating system: to unify all
of those patches and make them work together (and it can sometimes be challenging).

The good thing is that with those patches, it is possible to build C programs with CFI features enabled. Once the latest versions of all patches are built and working together, it *should* be possible to test such an example program that executes a CFI-illegal instruction (easy to write in assembly) and verifies that it is killed by
the kernel. Afterwards, we'll need extensive testing before enabling CFI for the packages in the archive, but we're not there yet.

This investigation is still ongoing at the time of writing, though it's not the highest priority. 
While I was investigating, time flew by, and suddenly it was release week!

## The Questing Quokka

::BlogImageText{image="/img/joining-canonical-riscv/questing-quokka-bg.svg"}

Ubuntu 25.10 Questing Quokka was released on October 09, 2025. This was my first release week. 
Even though we had been testing daily images before, I expected it to be more complicated. 
As it turned out, everything went smoothly, the testing and bugfixing we had done previously prepared us perfectly for the release.
For riscv64, the image took a long time to build, but that wasn't a major issue, I just tested one image a bit later than the others.
::

<!-- The main thing that  -->


## RISC-V Summit North America 2025

On October 21-23, the RISC-V Summit North America 2025 took place in Santa Clara, California, USA.
Of course, given that it is the most important event for the RISC-V community, Canonical had to be there;
and given that I'm on the RISC-V team, I was there as well. I was even tasked to prepare some demos
beforehand, which was really fun. One of the demos I enjoyed preparing was getting **Minecraft** to run, which was easy
thanks to the [HMCL launcher](https://github.com/HMCL-dev/HMCL). It ships with native libraries already patched and
built for RISC-V. It was almost plug and play, with the only adjustments made being the use of performance
mods (Sodium, Lithium).

::BlogImage{image="/img/joining-canonical-riscv/minecraft_riscv.png"}
*Minecraft running on RISC-V hardware (SiFive HiFive Premier P550) on Ubuntu 24.04*
::

## Induction and engineering sprints in Gothenburg

After the summit, there was no time to rest, as it was already time for the induction and engineering sprints
in Gothenburg, Sweden. The first week was **induction**, filled with presentations of different
areas of Canonical. We learned about various Canonical products like Ubuntu, but also [juju](https://github.com/juju/juju), [MAAS](https://canonical.com/maas), [microcloud](https://canonical.com/microcloud), and more. It was interesting
to meet all the new joiners, and I also got to meet some of my colleagues.

The following week was the **engineering sprint**. I finally got to meet my team in person. It is fascinating to see in 
real life the people I'd been working with for two months, and get to know them personally. It was also refreshing
and motivating to be able to talk with each other directly, which is what we did mostly (with a little bit of direct 
work in the middle). Meeting other teams allowed us to brainstorm solutions for some of our shared
problems and plan our work for the following release (26.04 LTS Resolute Raccoon).

All of that was really motivating, first the summit discovering the future of RISC-V and then the sprints where we could
plan the future of Ubuntu for RISC-V. It made me eager to come back home and start implementing all of what we talked about.

## Towards the future

I am now back home and have been working again for one week.
With all that motivation during the sprint, my TODO list became enormous.
Fortunately, there is plenty of time before the end of the cycle, so let's get started!

If you want to follow my work, I try to publish weekly updates on [Ubuntu Discourse](https://discourse.ubuntu.com/c/project/foundations/25); look for "Foundations Team Updates" topics.
For now, it's been mainly continuing the CFI investigation, fixing bugs again, and keeping up the effort towards supporting an Ubuntu RISC-V desktop image. That 'desktop' work went a bit far, as I'm currently trying to add RISC-V support to Flutter.
We will see how that goes, and I hope that we will be able to bridge the gap between RISC-V and other architectures!
