I'm placing here 2 applications at once, instructed by a senior DMB member that it would be acceptable.

MOTUApplication

I, Rafael David Tinoco, hereby apply for per package upload rights to:

1. pcs
2. crmsh

to work on them from the server-team's point of view. Both packages don't look like they have dedicated uploaders/maintainers, and, this will be part of my current job of maintaining Ubuntu HA related packages, including, but not limited by, the following packages:

1. booth
2. cluster-agents
3. cluster-glue
4. corosync
5. corosync-qdevice
6. crmsh
7. csync2
8. dlm
9. drbd-doc
10. drbd-utils
11. drbd8
12. fence-agents
13. gfs2-utils
14. heartbeat
15. kronosnet
16. libqb
17. ocfs2-tools
18. openais
19. pacemaker
20. pcs
21. redhat-cluster
22. resource-agents
23. sbd
24. keepalived
25. pcs
26. crmsh

thus the reason why I'm applying to both roles simultaneously. Furthermore, Ubuntu documentation shall concentrate efforts in having only "pcs" tool used to configure HA pacemaker clusters, since its the current preferred tool for upstream, but, as crmsh still is heavily used, its important that it is updated and fixed regularly. PCS might face a MIR soon so at least one cluster configuration tool is well supported and maintained in "core" pocket.

If board does not think I'm ready yet for those roles, the alternative will be to continue doing the merge requests and asking for not only reviews but also sponsored uploads until I'm considered good for the roles.

UbuntuServerApplication

Who I am

• I'm a Software Engineer, member of Canonical, in the Server Team since May 2019. I'm also former Sustaining Engineer at SEG team, where I worked for 4 years. In the Sustaining Engineering Team I was also the tech lead for all "userland" related bugs (and UA cases), not only working with bugs in specific areas, but also guiding new engineers in the next steps in their bugs/cases. I have also worked in several kernel related bugs regarding KVM, scheduling, memory management, block devices and SCSI protocol & transport. Nowadays I'm focused in starting good HA related software maintainership, helping out engineers in charge of virtualization AND debugging/fixing bugs for all useful packages for Ubuntu Server.

In formal terms:

• Rafael is a Linux Developer. He has worked for the past 20 years in companies such as: Linaro (Kernel Validation Engineer and HPC Tech Lead), Canonical (Linux Sustaining Engineer and Tech Lead), IBM (s390x Lab Engineer and Tech Lead, Residency with s390x performance team), Red Hat (Solution Architect), Locaweb (Linux/Email devops), Sun Microsystems (Consultant, Systems Engineer and Solaris Ambassador). Rafael has carved his career from field engineering positions, in constant contact with customers, into pure engineering ones, focused in Linux internals, userland, kernel and general OS & performance debugging and bug solving.

Curiosities:

• I have started learning Linux since I was 12 years old (I'm 37 nowadays). A BBS sysop sent me Minix floppy disks and also told me to start studying Linux at that time. I literally spent the next years reading Richard Stevens and playing with Linux. I can pretty much say Richard Stevens thought me not only Programming, TCP/IP, but also ENGLISH. My mom used to become pretty upset about me not sleeping at night, reading books, and falling asleep in the classroom in the next day (when my teachers would call her to complain). I joined "business life" very young (around 16 years old). Sun Microsystems was basically a school for me - together with Physics University and "special" Optics Interferometry classes - where I used to visit customers to deploy all "enterprise" related software (clusters, backup softwares, tape library managers, Solaris, oracle, etc). After some years I moved into the HPC world so I could work more with Linux (as Solaris 10 was still dominating Sparc High-End/Mid-End servers, my daily job back then). Lot of time passed, I graduated in a Business Administration school, and I found myself playing with Mainframes and their "new" technologies (yes, s390x has NEWS features every year, despite still being called Mainframe). IBM was also another school, where I was able to enter specifics on s390x instructions, architecture, hardware and hypervisors (note: don't call z/VM a hypervisor to old guys, as CP/CMS is a full operating system, like they usually say). At that time I had an awesome opportunity to visit Germany, make new friends, and get to know s390x performance engineering team, which basically raised my ill for an "engineering only" position. That is what motivated me to start working in the sustaining engineering team @ Canonical, back then, together with the remote work: something that could help me taking care of my son.

Personal life:

• I currently live in Curitiba, Brazil - yes I love cold places - with my wife, whom I love very much, and my 2 kids: a 12 years old boy and a 3 years old daughter. We have 1 dog, an English Staffordshire Bull Terrier - or just Staff Bull - called Trevi (Because of Fontana de Trevi in Rome) and a very young cat called "Luna", whom we rescued from bellow our neighbor's car after she traveled for more than 200 kms with Luna under her car). I guess Luna really wanted a new life.

My Ubuntu story

• I used to use Slackware in the beginning, but the fact that I had to rely on an external tool to deal with .tgz packages (slackpkg) that had almost no relationship among them took me to Debian really quickly. After Ubuntu arrived, moving to Ubuntu, because of its better Desktop experience initially, was quite automatic for me. Of course we all had the Gentoo experience, but after nights and nights recompiling X11 because of different compilation flags, or just because of new dependencies versions, make us re-think this "let me compile everything approach". That also goes for OpenBSD ports, and the incredible attempt of making NetBSD pkgsrc to behave like a Linux distro packaging system (I really tried to stay with compiled only packages in NetBSD). My experience with RHEL and CentOS was mainly targeted to enterprise customers using it, and not for fun and/or development purposes. Because of the "enterprise" type jobs, I was never much into the contributing-to-public-projects before working @ Canonical (SEG team), something I regret. All the work I can give as a reference starts at that time, but I think it is enough to show how I have already contributed to Ubuntu project and how I can, even more, help the project in the future.

My involvement

Examples of my work / Things I'm proud of

• Literally designing all z13 mainframe I/O programming from the ground: Defining all possible s390x sub channel-systems and their LPARs based on very little information (at that time) in order to maximize Canonical Mainframe usage and help out Foundations and Server Team to start porting Ubuntu to s390x architecture. Christian and I did all the mainframe enablement in very short time to help out a fast pass porting plan (and go to market).
• Helping out Canonical customers in the Ubuntu Advantage program. It was quite nice to help big companies to use Ubuntu Linux in their environment. Sometimes I felt like a "taylor-made" specialized engineer in "very specific problems" related to virtualization and performance.
• Having worked as a test engineer for official Linux stable trees - from Greg - when I was working @ Linaro. Running almost 30k functional tests - focused in armhf and arm64 architecture - for every single branch, of every supported kernel version, providing feedback for all regressions that happened in between the releases, with root cause and involving "who-to-blame" (and possibly fixing them) was quite an adventure.
• Having contributed to several upstream projects regarding bug fixes, including: Linux (kernel), QEMU, libvirt, Apache, Linux Test Project, LVM, Systemd, and several others.
• Having contributed to deploy some Infiniband-only HPC clusters - one of them was TOP-65 in Linpack Benchmark in ~2009 for Petrobras (Pre-Salt Project) inside Brazilian Universities, in a project called Grade-BR. I loved spending huge amount of time at University of Sao Paulo TPN inside Engineering Campus.
• Having learned s390 architecture nowadays. Specially fighting z/OS engineers on how good Linux was doing what they did not know Linux was capable of (and listening that Mainframes already did that since 70s).
• Having installed the first Linux-only - alright, on top of z/VM, but no z/OS - z196 Mainframe in Brazil, consolidating more than 300 servers in a single Enterprise box, for a telco company in Brazil.

Areas of work

Recent work, trying to show different skills (merging, debugging, backporting)

A complete example going upstream and backporting the fix

• LP: #1840958 - defragfs.ocfs2 hangs (or takes too long) on arm64, ppc64el

  1. LP: #1840958

  2. LP: #1837089

• Provided a Merge request to be reviewed with a PPA:

  1. https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/ocfs2-tools/+git/ocfs2-tools/+ref/lp1840958-eoan

  2. https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1840958

• I've provided explanations, about the bug, upstream:

  1. https://github.com/markfasheh/ocfs2-tools/issues/42

  2. https://github.com/markfasheh/ocfs2-tools/pull/43

• Suggested a merge to Debian considering upstream fix already:

  1. https://salsa.debian.org/rafaeldtinoco-guest/ocfs2-tools

  2. https://salsa.debian.org/ha-team/ocfs2-tools/merge_requests/1

An example of complex debugging for QEMU

• LP: #1805256 - qemu-img hangs on rcu_call_ready_event logic in Aarch64 when converting images

  1. Upstream work fixing QEMU Async I/O for Aarch64: Somehow ARM64 (D06 Huawei Server) existing MEM barriers aren’t enough for this machine, causing Async I/O to hang (this is also true for QEMU core parts, not only for qemu-img).

  2. https://lists.nongnu.org/archive/html/qemu-devel/2019-09/msg02346.html

  3. I have added a mutex around affected parts and it was enough to make it work for ARM64 but upstream might want to continue with primitives so I might have to re-check which barriers weren’t enough for the mem ordering and fix them.

And one example backporting new features (security related) to QEMU

• LP: #1828495 - QEMU HW mitigations support (ARCH_CAPABILITIES): New QEMU version for Ubuntu Disco, supporting CascadeLake/IceLake, IA32_ARCH_CAPABILITIES MSR, thus the following HW mitigations:

  1. IBRS_ALL (enhanced IBRS support)
  2. SKIP_L1DFL_VMENTRY (L1D flush is needed on VMENTRY)
  3. RDCL_NO (HW is vulnerable to Rogue Data Cache Load)
  4. Foreshadow-NG (OS) vuln. (L1 terminal fault, OS)
  5. Foreshadow-NG (VMM) vuln. (L1 terminal fault, VMM)

And some of my work helping out @rbasak during this last weeks for Eoan MySQL 8 upgrade: @rbasak has done a huge amount of work to upgrade MySQL to 8 in Ubuntu Eoan

1. dbconfig-common (Upstream Debian Merge Request): Because of cacti (2) I had to create a debconf variable to set MySQL authentication plugin to be used by dbconfig-common consumers. Because of MySQL 8 changes, ALTER SQL commands had to be altered also.
2. cacti (Upstream Debian Merge Request and Upstream Merge Request): MySQL 8 does not allow ALTER to create a USER by default anymore AND its default authentication plugin is caching_sha2_password, not supported by some PHP packages.

Some public visibility of my Sustaining Engineer work at SEG

• Kernel – codes/bugs/debugs/upstream

  1. https://goo.gl/LNhYGC - megaraid/sas/scsi

  2. https://goo.gl/R8kwca - linux-scsi (iscsi)

  3. https://goo.gl/QTh765 - linux-scsi

  4. https://goo.gl/AB7bA4 - numa/scheduler

  5. https://goo.gl/3kgCLF - perf/scheduler

  6. https://goo.gl/jZLHUN - hpproliant/ubuntu

  7. https://goo.gl/rPi1fV - hp proliant/watchdog

  8. https://goo.gl/amK5UV - vm/scheduler

  9. https://goo.gl/KDLRtq - network/namespaces

  10. https://goo.gl/EsCFDm - apparmor

  11. https://goo.gl/zbqy3k - infiniband/iscsi/iser

  Userland – codes/bugs/debugs/upstream

  1. https://goo.gl/25Z8h9 - qemu

  2. https://goo.gl/AK3Xwr - qemu

  3. https://goo.gl/frVDeJ - qemu

  4. https://goo.gl/xta33w - qemu

  5. https://goo.gl/DZZtHQ - libvirt

  6. https://goo.gl/eJwzWc - nfs-utils / systemd

  7. https://goo.gl/ai3eUw - sudo

  8. https://goo.gl/dErX1e - isc-dhcp

  9. https://goo.gl/UvsyJF - isc-dhcp

  10. https://goo.gl/FAk5Y6 - libvirt

  11. https://goo.gl/RXH6kz - crmsh

  12. https://goo.gl/8GeF37 - pcs

  13. https://goo.gl/XDRtZ3 - pacemaker

Things I could do better

• Be less prolix in application wiki pages.
• Sometimes I have to force myself to revisit things: SRU verification. Keeping the dev environment is usually good for verifying things faster, I should do that more often.
• Changelogs and patch descriptions are some of the things caught in upload reviews. Sometimes, despite being "ONLY" a "format change", it is important that formats and best practices are always followed. I tend to think more of the fix, backport/cherry-pick and functionality then the formats, I have to change a bit my mindset (even after all the SRUs done).
• Merges: I have done some merges (even using git-ubuntu flow) but I should do more. Now that I'll be taking care of HA related packages, I tend to think this will be a natural consequence. I was able to unblock all blockers from Eoan excuses before Eoan freeze dates, and after Eoan is GA it will be the time for all HA related SW merges and syncs.

• Review more merge requests: I thought I was "too new" in the team to review the merges, but, naturally, this sensation is going away. Reviews aren't there because of the experience of the reviewer, sometimes its there so code has another pair of eyes before landing in the repo <- That is something I'm improving but my reviews will definitely get better from now on.

Plans for the future

General

• I'd like to create a regression test scheme for HA related software: configuring multiple environments with the most used HA scenarios and resources and testing to check if current Ubuntu development release is causing any regression to those supported scenarios.
• Still regarding HA, I have to create good HA documentation for 20.04, with the scenarios I mentioned in the first item, with pictures an examples of those scenarios and how to get them installed (using pcs, thus the MOTU request altogether).
• I'd like also to continue debugging all kinds of server related bugs. As part of my daily work, but also as part of professional development AS doing that guarantees that we become "deep generalists", with eyes in several different technologies. If I may choose: I prefer those hard-to-debug cases, with dumps and reverse-engineering needs.
• Apply for core-dev. I sense that, for that to happen, I have to do future merges, and, 20.04 will be the perfect version for me to do so.

What I like least in Ubuntu

• Lack of a centralized place for public documentation. I quote here archlinux wiki pages, Red Hat administration and development manuals, SUSE HA documentation, and some others. Usually we find documentation through launchpad bugs, which is great, but not necessarily the best for all situations. If you go after the bugs I've been involved, I tend to do really prolix comments because of that reason, but I would be willing in providing those comments as "tips" in a wiki of some sort (and, yes, I'm aware discourse.ubuntu.com might be there for that, unsure if it will look like something as good as archlinux wiki, for example).

Comments

If you'd like to comment, but are not a sponsor, please do it here. Don't forget to sign with @SIG@.

Endorsements

If you'd like to endorse me, please do it here. Don't forget to sign with @SIG@.

TEMPLATE

== <SPONSORS NAME> ==
=== General feedback ===
## Please fill us in on your shared experience. (How many packages did you sponsor? How would you judge the quality? How would you describe the improvements? Do you trust the applicant?)
=== Specific Experiences of working together ===
## Please add good examples of your work together, but also cases that could have handled better.
=== Areas of Improvement ===
## Add here areas where you think applicant could improve.

CategoryUbuntuServerDevApplication