CoreDev

CoreDevApplication

After applying for MOTU and ServerDev, and exploring a CoreDev possibility, I have received feedback about needed remaining steps in order to be a CoreDev. Usually, this kind of feedback focus in things you haven't much experience with AND are needed for the role.

In my case, I was told I was missing: seed changes, full package migrations AND turning deltas into sync. Since I have helped with mysql-server-8 migration (dependencies, merge back to debian, etc) we have focused in seed management, specifically MIRs, and syncs.

Since then, I have been working in the following MIRs to Focal:

  1. ndctl - https://bugs.launchpad.net/bugs/1853506

  2. mysql-router - https://bugs.launchpad.net/bugs/1852367

  3. libslirp - https://bugs.launchpad.net/bugs/1854404

And I have continued doing new merges, upstreaming delta and doing syncpackages:

  1. drbd-utils

  2. heartbeat - https://bugs.launchpad.net/bugs/1853467 (upstreamed delta from xnox, syncpackage)

  3. crmsh - https://bugs.launchpad.net/bugs/1853450 (syncpackage)

  4. dbconfig-common - https://bugs.launchpad.net/bugs/1852788 (upstreamed delta, syncpackage)

  5. sg3-utils

So, after addressing DMBs concerns, and with the good recommendations I received in my previous application, many of them mentioning that I should go for CoreDev instead - at that time - I'd like to be considered for the Ubuntu CoreDev role, with the promise that I'll keep working hard trying to continue making Ubuntu the best Linux Distribution out there.

(I have copied my previous application here, to use as a reference, removing the comments and endorsements that didn't explicitly mention core-dev as a path for me. Please feel free to add and/or remove comments or endorsements as you judge appropriate)

I'm placing here 2 applications at once, instructed by a senior DMB member that it would be acceptable.

MOTUApplication

I, Rafael David Tinoco, hereby apply for the following roles:

  1. MOTU
  2. Ubuntu Server Developer
  3. Ubuntu Core Dev (TO BE ADDED IN A NEAR FUTURE)

with the intent to maintain and develop Server, Universe and Multiverse related packages. My initial focus will be maintaining Ubuntu HA related packages, including, but not limited by, the following packages:

  1. pcs
  2. crmsh
  3. booth
  4. cluster-agents
  5. cluster-glue
  6. corosync
  7. corosync-qdevice
  8. crmsh
  9. csync2
  10. dlm
  11. drbd-doc
  12. drbd-utils
  13. drbd8
  14. fence-agents
  15. gfs2-utils
  16. heartbeat
  17. kronosnet
  18. libqb
  19. ocfs2-tools
  20. openais
  21. pacemaker
  22. pcs
  23. redhat-cluster
  24. resource-agents
  25. sbd
  26. keepalived
  27. pcs
  28. crmsh

thus the reason why I'm applying to both roles simultaneously (note: I'm placing here 2 applications at once, instructed by a senior DMB member that it would be acceptable).

Furthermore, future Ubuntu Server documentation, in my opinion, should concentrate efforts in having only "pcs" tool used to configure HA pacemaker clusters, since its the current preferred tool for upstream, but, as crmsh still is heavily used, its important that it is updated and fixed regularly. PCS might face a MIR soon so at least one cluster configuration tool is well supported and maintained in "core" pocket.

If board does not think I'm ready yet for those roles, the alternative will be to continue doing the merge requests and asking for not only reviews but also sponsored uploads until I'm considered good for the roles.

UbuntuServerApplication

Name

Rafael David Tinoco

Launchpad Page

https://launchpad.net/~rafaeldtinoco

Wiki Page

n/a

Who I am

  • I'm a Software Engineer, member of Canonical, in the Server Team since May 2019. I'm also former Sustaining Engineer at SEG team, where I worked for 4 years. In the Sustaining Engineering Team I was also the tech lead for all "userland" related bugs (and UA cases), not only working with bugs in specific areas, but also guiding new engineers in the next steps in their bugs/cases. I have also worked in several kernel related bugs regarding KVM, scheduling, memory management, block devices and SCSI protocol & transport. Nowadays I'm focused in starting good HA related software maintainership, helping out engineers in charge of virtualization AND debugging/fixing bugs for all useful packages for Ubuntu Server.

In formal terms:

  • Rafael is a Linux Developer. He has worked for the past 20 years in companies such as: Linaro (Kernel Validation Engineer and HPC Tech Lead), Canonical (Linux Sustaining Engineer and Tech Lead), IBM (s390x Lab Engineer and Tech Lead, Residency with s390x performance team), Red Hat (Solution Architect), Locaweb (Linux/Email devops), Sun Microsystems (Consultant, Systems Engineer and Solaris Ambassador). Rafael has carved his career from field engineering positions, in constant contact with customers, into pure engineering ones, focused in Linux internals, userland, kernel and general OS & performance debugging and bug solving.

Curiosities:

  • I have started learning Linux since I was 12 years old (I'm 37 nowadays). A BBS sysop sent me Minix floppy disks and also told me to start studying Linux at that time. I literally spent the next years reading Richard Stevens and playing with Linux. I can pretty much say Richard Stevens thought me not only Programming, TCP/IP, but also ENGLISH. My mom used to become pretty upset about me not sleeping at night, reading books, and falling asleep in the classroom in the next day (when my teachers would call her to complain). I joined "business life" very young (around 16 years old). Sun Microsystems was basically a school for me - together with Physics University and "special" Optics Interferometry classes - where I used to visit customers to deploy all "enterprise" related software (clusters, backup softwares, tape library managers, Solaris, oracle, etc). After some years I moved into the HPC world so I could work more with Linux (as Solaris 10 was still dominating Sparc High-End/Mid-End servers, my daily job back then). Lot of time passed, I graduated in a Business Administration school, and I found myself playing with Mainframes and their "new" technologies (yes, s390x has NEWS features every year, despite still being called Mainframe). IBM was also another school, where I was able to enter specifics on s390x instructions, architecture, hardware and hypervisors (note: don't call z/VM a hypervisor to old guys, as CP/CMS is a full operating system, like they usually say). At that time I had an awesome opportunity to visit Germany, make new friends, and get to know s390x performance engineering team, which basically raised my ill for an "engineering only" position. That is what motivated me to start working in the sustaining engineering team @ Canonical, back then, together with the remote work: something that could help me taking care of my son.

Personal life:

  • I currently live in Curitiba, Brazil - yes I love cold places - with my wife, whom I love very much, and my 2 kids: a 12 years old boy and a 3 years old daughter. We have 1 dog, an English Staffordshire Bull Terrier - or just Staff Bull - called Trevi (Because of Fontana de Trevi in Rome) and a very young cat called "Luna", whom we rescued from bellow our neighbor's car after she traveled for more than 200 kms with Luna under her car). I guess Luna really wanted a new life.

My Ubuntu story

  • I used to use Slackware in the beginning, but the fact that I had to rely on an external tool to deal with .tgz packages (slackpkg) that had almost no relationship among them took me to Debian really quickly. After Ubuntu arrived, moving to Ubuntu, because of its better Desktop experience initially, was quite automatic for me. Of course we all had the Gentoo experience, but after nights and nights recompiling X11 because of different compilation flags, or just because of new dependencies versions, make us re-think this "let me compile everything approach". That also goes for OpenBSD ports, and the incredible attempt of making NetBSD pkgsrc to behave like a Linux distro packaging system (I really tried to stay with compiled only packages in NetBSD). My experience with RHEL and CentOS was mainly targeted to enterprise customers using it, and not for fun and/or development purposes. Because of the "enterprise" type jobs, I was never much into the contributing-to-public-projects before working @ Canonical (SEG team), something I regret. All the work I can give as a reference starts at that time, but I think it is enough to show how I have already contributed to Ubuntu project and how I can, even more, help the project in the future.

My involvement

Examples of my work / Things I'm proud of

  • Literally designing all z13 mainframe I/O programming from the ground: Defining all possible s390x sub channel-systems and their LPARs based on very little information (at that time) in order to maximize Canonical Mainframe usage and help out Foundations and Server Team to start porting Ubuntu to s390x architecture. Christian and I did all the mainframe enablement in very short time to help out a fast pass porting plan (and go to market).
  • Helping out Canonical customers in the Ubuntu Advantage program. It was quite nice to help big companies to use Ubuntu Linux in their environment. Sometimes I felt like a "taylor-made" specialized engineer in "very specific problems" related to virtualization and performance.

  • Having worked as a test engineer for official Linux stable trees - from Greg - when I was working @ Linaro. Running almost 30k functional tests - focused in armhf and arm64 architecture - for every single branch, of every supported kernel version, providing feedback for all regressions that happened in between the releases, with root cause and involving "who-to-blame" (and possibly fixing them) was quite an adventure. Some of the contribution log: contriblog.

  • Having contributed to several upstream projects regarding bug fixes, including: Linux (kernel), QEMU, libvirt, Apache, Linux Test Project, LVM, Systemd, and several others.
  • Having contributed to deploy some Infiniband-only HPC clusters - one of them was TOP-65 in Linpack Benchmark in ~2009 for Petrobras (Pre-Salt Project) inside Brazilian Universities, in a project called Grade-BR. I loved spending huge amount of time at University of Sao Paulo TPN inside Engineering Campus.
  • Having learned s390 architecture nowadays. Specially fighting z/OS engineers on how good Linux was doing what they did not know Linux was capable of (and listening that Mainframes already did that since 70s).
  • Having installed the first Linux-only - alright, on top of z/VM, but no z/OS - z196 Mainframe in Brazil, consolidating more than 300 servers in a single Enterprise box, for a telco company in Brazil.

Areas of work

Recent work, trying to show different skills (merging, debugging, backporting)

A complete example of a merge and then another quick one on top

This is an example of a merge https://launchpad.net/ubuntu/+source/bind9/1:9.11.5.P4+dfsg-5ubuntu1 for bind9 during Eoan development and, right after, another quick merge with Debian as soon as they've delivered the _same_ fix for the CVE in question, bind9/1:9.11.5.P4+dfsg-5.1ubuntu1, as it was better to keep up with upstream code instead of the delta.

All of my upstream work is done with latest Ubuntu Devel package sources debian/ directory being placed in upstream tree (with proper .gitignore). This way I usually do merges whenever working with upstream, even when not uploading them. The work directory I use is described at https://github.com/rafaeldtinoco/work and has been presented in a conference also: video

A complete example going upstream and backporting the fix

An example of complex debugging for QEMU

  • LP: #1805256 - qemu-img hangs on rcu_call_ready_event logic in Aarch64 when converting images

    1. Upstream work fixing QEMU Async I/O for Aarch64: Somehow ARM64 (D06 Huawei Server) existing MEM barriers aren’t enough for this machine, causing Async I/O to hang (this is also true for QEMU core parts, not only for qemu-img).

    2. https://lists.nongnu.org/archive/html/qemu-devel/2019-09/msg02346.html

    3. I have added a mutex around affected parts and it was enough to make it work for ARM64 but upstream might want to continue with primitives so I might have to re-check which barriers weren’t enough for the mem ordering and fix them.

And one example backporting new features (security related) to QEMU

  • LP: #1828495 - QEMU HW mitigations support (ARCH_CAPABILITIES): New QEMU version for Ubuntu Disco, supporting CascadeLake/IceLake, IA32_ARCH_CAPABILITIES MSR, thus the following HW mitigations:

    1. IBRS_ALL (enhanced IBRS support)
    2. SKIP_L1DFL_VMENTRY (L1D flush is needed on VMENTRY)
    3. RDCL_NO (HW is vulnerable to Rogue Data Cache Load)
    4. Foreshadow-NG (OS) vuln. (L1 terminal fault, OS)
    5. Foreshadow-NG (VMM) vuln. (L1 terminal fault, VMM)

And some of my work helping out @rbasak during this last weeks for Eoan MySQL 8 upgrade: @rbasak has done a huge amount of work to upgrade MySQL to 8 in Ubuntu Eoan

  1. dbconfig-common (Upstream Debian Merge Request): Because of cacti (2) I had to create a debconf variable to set MySQL authentication plugin to be used by dbconfig-common consumers. Because of MySQL 8 changes, ALTER SQL commands had to be altered also.
  2. cacti (Upstream Debian Merge Request and Upstream Merge Request): MySQL 8 does not allow ALTER to create a USER by default anymore AND its default authentication plugin is caching_sha2_password, not supported by some PHP packages.

Some public visibility of my Sustaining Engineer work at SEG

Things I could do better

  • Be less prolix in application wiki pages.
  • Sometimes I have to force myself to revisit things: SRU verification. Keeping the dev environment is usually good for verifying things faster, I should do that more often.
  • Changelogs and patch descriptions are some of the things caught in upload reviews. Sometimes, despite being "ONLY" a "format change", it is important that formats and best practices are always followed. I tend to think more of the fix, backport/cherry-pick and functionality then the formats, I have to change a bit my mindset (even after all the SRUs done).
  • Merges: I have done some merges (even using git-ubuntu flow) but I should do more. Now that I'll be taking care of HA related packages, I tend to think this will be a natural consequence. I was able to unblock all blockers from Eoan excuses before Eoan freeze dates, and after Eoan is GA it will be the time for all HA related SW merges and syncs.

  • Review more merge requests: I thought I was "too new" in the team to review the merges, but, naturally, this sensation is going away. Reviews aren't there because of the experience of the reviewer, sometimes its there so code has another pair of eyes before landing in the repo <- That is something I'm improving but my reviews will definitely get better from now on.

Plans for the future

General

  • I'd like to create a regression test scheme for HA related software: configuring multiple environments with the most used HA scenarios and resources and testing to check if current Ubuntu development release is causing any regression to those supported scenarios.
  • Still regarding HA, I have to create good HA documentation for 20.04, with the scenarios I mentioned in the first item, with pictures an examples of those scenarios and how to get them installed (using pcs, thus the MOTU request altogether).
  • I'd like also to continue debugging all kinds of server related bugs. As part of my daily work, but also as part of professional development AS doing that guarantees that we become "deep generalists", with eyes in several different technologies. If I may choose: I prefer those hard-to-debug cases, with dumps and reverse-engineering needs.
  • Apply for core-dev. I sense that, for that to happen, I have to do future merges, and, 20.04 will be the perfect version for me to do so.

What I like least in Ubuntu

  • Lack of a centralized place for public documentation. I quote here archlinux wiki pages, Red Hat administration and development manuals, SUSE HA documentation, and some others. Usually we find documentation through launchpad bugs, which is great, but not necessarily the best for all situations. If you go after the bugs I've been involved, I tend to do really prolix comments because of that reason, but I would be willing in providing those comments as "tips" in a wiki of some sort (and, yes, I'm aware discourse.ubuntu.com might be there for that, unsure if it will look like something as good as archlinux wiki, for example).

Comments

If you'd like to comment, but are not a sponsor, please do it here. Don't forget to sign with @SIG@.

I've worked with Rafael in the same team. He's been my mentor and tech lead, and while being technically skilled in userspace and kernel, he's proven to be a hard working, committed and reliable engineer. Being the proficient Ubuntu contributor he is, his natural step would be to be a full coredev, but he would still be a great asset as a MOTU :).

-- vtapia 2019-10-15 13:16:32

Rafael has been a expert reference for us (SEG) in the Linux internals (in both fronts; userland and kernel), virtualization, storage and HPC areas since he joined Canonical.

His debugging and analytical skills have produced reliable and consistent bug fixes and outstanding bug reports over the years that have benefited both the Ubuntu users and the many customers that Canonical supports.

In my opinion, based on the amount and quality of his contributions, it should be pretty clear that having him as part of the coredev or MOTU teams would be a ideal scenario that would benefit the whole Ubuntu ecosystem.

-- niedbalski

Rafael used to be my team member and a tech lead. He was always a invaluable source of knowledge and ideas for approaching difficult technical problems. I have always values his ability to create efficient software and workflow setups optimizing performance.

I strongly believe he has everything it takes to be a great MOTU and eventually a coredev.

-- dgadomski 2019-10-15 14:23:48

Rafael helped out on a project (especially setting things up) before I joined Canonical. Since joining Canonical I am now working in that same project and still benefit from his structured work and farsightedness - and this is now years ago! During my time at Canonical I got in touch with Rafael every now and then, mainly in his role at SEG, and I always enjoyed his deep expertise, commitment and patience. I am convinced that he has everything that's required, not only to be a Masters of the Universe, but definitely also to become a coredev!

-- frank-heimes 2019-10-18 15:23:00

Endorsements

If you'd like to endorse me, please do it here. Don't forget to sign with @SIG@.

Dan Streetman

General feedback

I've worked with tinoco for many years; he was a co-worker when I first started at Canonical and shortly after rose to Team Lead. He's extremely knowledgeable and sharply focused, and I have always had 100% faith he will be capable to handle any task he takes on. He has not proven me wrong yet. I have no reservations about endorsing him for MOTU or listed per-package rights.

Specific experiences of working together

I've only one example of actually sponsoring his work, however I've worked with him on debugging/fixing many other packages.

Areas of improvement

I'm not sure why he is applying only for MOTU, instead of full coredev, but hopefully the coredev application won't be far behind. Smile :)

Update (Dec 3, 2019)

For the same reasons I endorsed him for MOTU, I highly endorse tinoco for his current coredev application. He has additionally done more work towards coredev activities since his MOTU application, and I have full confidence he will do excellent work as a coredev.

-- ddstreet 2019-12-03 18:39:49

-- ddstreet 2019-10-14 20:42:23

Eric Desrochers

General feedback

Rafael and I worked in the same team (Canonical STS for at least 2 years). During that period of time, I had the chance to work with him closely and learn from him. Rafael used to be my mentor back in the days.

Specific Experiences of working together

Over the years I've sponsored a few package for Rafael such as multipath-tools, kexec-tools, systemd and apache2. The packages I've sponsored and other updates I've seen were high quality, and showed a good attention to detail.

Areas of Improvement

I have the same question as ddstreet who also point it out in his endorsement ... I wonder why Rafael doesn't go straight for coredev instead but hopefully, that would be the next step for him.

-- slashd 2019-10-15 07:04:12

Bryce Harrington

General feedback

I work close with Rafael on the server team, and follow his work on a day-to-day basis. I also had the opportunity to get to know him personally as roomies at the Paris Ubuntu Engineering Sprint. I agree with others Ubuntu would be improved having him as Core Dev, and see attainment of MOTU as a solid step towards that objective.

I once heard that a good sign of someone being ready for MOTU/CoreDev is when people react, "What?? He's not that already??" and this definitely applies to Rafael. Smile :-)

Specific Experiences of working together

Many of the packages he tackles (e.g. qemu/libvirt) are in areas I don't feel suitably qualified to judge, but usually by the time he needs formal review the changes have been thoroughly studied and discussed so the reviews have been straightforward. Even when I have little to give in review but nit picks, he's been welcoming of critiques. Rafael is comfortable with critiquing his own work, and not shy about pointing out flaws, which gives me strong confidence he is worthy of trust.

In addition to reviewing his HA packages, he and I collaborated on some of the MySQL transition tasks, and I was very impressed by both his teammanship and his ability to dig deep into difficult problems even in unfamiliar areas. He is a good communicator and verbose documenter, and I found it easy to hand off things I was stuck on, and easy to pick up and continue from his work in turn.

As a verbose person myself, I certainly can't fault anyone for lack of brevity, but actually I have more than once appreciated Rafael's detailed comments as saving me time coming up to speed on the issue. I say keep it up!

Areas of Improvement

I feel Rafael is already more than qualified for MOTU, and probably CoreDev as well. For proving the latter, I'd encourage generalizing by working on a wider variety of packages, including non-Server types of software, and to look especially for opportunities to do review/sponsorship.

Update (Dec 3, 2019)

I'm still a strong +1 for Rafael receiving CoreDev. I'm doing merge coordination for the server team for focal, and as such I've been working closely with Rafael. I find I can trust that his work will be solid, well thought-out and well tested. Among other things, he has taken ownership over the high-availability stack which involves a variety of different packaging activities (MIRs, syncs, upstreaming, merging, etc.) He plans this work with care to ensure packages get updated in concert, and things have moved quite smoothly. I trust his work and feel he would be an asset to add to the CoreDev team.

-- bryce 2019-10-15 19:37:23

Andreas Hasenack

General feedback

Rafael produces high quality work and leaves "no stone unturned" when chasing down a bug. He also comes up with test cases, which I always appreciate, and good instructions on how to do things, like setting up a testing rig.

I think Rafael will be great asset to Ubuntu as a MOTU, and Server Dev, and he should be able to become CoreDev very quickly.

Specific Experiences of working together

I sponsored 7 unique packages for Rafael: list

Of these, the most complex one was samba, with many changes, and that was also his first. There was a lot of back and forth, but I'm happy to say that the subsequent uploads were straightforward.

Rafael picked up the Debian merge workflow very quickly, and I think just a few more merges, so some edge cases can present themselves, will be enough.

Rafael quickly volunteers to fix hard bugs, and a recent example as bug #1838525 which affected the subiquity server installer. I troubleshooted that a short while with him, and it was clear he was heading in the right direction.

Areas of Improvement

I think in the upcoming LTS cycle Rafael can ramp up his Debian merges, and that will give him a lot more packaging experience and should make obtaining CoreDev the next natural step. I'd also like to see him get more experience in seed management and chasing down dependency trees in Ubuntu, which is something CoreDevs have to do sometimes.

-- ahasenack 2019-10-18 14:47:26

TEMPLATE

== <SPONSORS NAME> ==
=== General feedback ===
## Please fill us in on your shared experience. (How many packages did you sponsor? How would you judge the quality? How would you describe the improvements? Do you trust the applicant?)
=== Specific Experiences of working together ===
## Please add good examples of your work together, but also cases that could have handled better.
=== Areas of Improvement ===
## Add here areas where you think applicant could improve.

CategoryUbuntuServerDevApplication

rafaeldtinoco/CoreDev (last edited 2019-12-03 18:39:49 by ddstreet)