I'm placing here 2 applications at once, instructed by a senior DMB member that it would be acceptable.
I, Rafael David Tinoco, hereby apply for the following roles:
- Ubuntu Server Developer
- Ubuntu Core Dev (TO BE ADDED IN A NEAR FUTURE)
with the intent to maintain and develop Server, Universe and Multiverse related packages. My initial focus will be maintaining Ubuntu HA related packages, including, but not limited by, the following packages:
thus the reason why I'm applying to both roles simultaneously (note: I'm placing here 2 applications at once, instructed by a senior DMB member that it would be acceptable).
Furthermore, future Ubuntu Server documentation, in my opinion, should concentrate efforts in having only "pcs" tool used to configure HA pacemaker clusters, since its the current preferred tool for upstream, but, as crmsh still is heavily used, its important that it is updated and fixed regularly. PCS might face a MIR soon so at least one cluster configuration tool is well supported and maintained in "core" pocket.
If board does not think I'm ready yet for those roles, the alternative will be to continue doing the merge requests and asking for not only reviews but also sponsored uploads until I'm considered good for the roles.
Rafael David Tinoco
Who I am
I'm a Software Engineer, member of Canonical, in the Server Team since May 2019. I'm also former Sustaining Engineer at SEG team, where I worked for 4 years. In the Sustaining Engineering Team I was also the tech lead for all "userland" related bugs (and UA cases), not only working with bugs in specific areas, but also guiding new engineers in the next steps in their bugs/cases. I have also worked in several kernel related bugs regarding KVM, scheduling, memory management, block devices and SCSI protocol & transport. Nowadays I'm focused in starting good HA related software maintainership, helping out engineers in charge of virtualization AND debugging/fixing bugs for all useful packages for Ubuntu Server.
In formal terms:
Rafael is a Linux Developer. He has worked for the past 20 years in companies such as: Linaro (Kernel Validation Engineer and HPC Tech Lead), Canonical (Linux Sustaining Engineer and Tech Lead), IBM (s390x Lab Engineer and Tech Lead, Residency with s390x performance team), Red Hat (Solution Architect), Locaweb (Linux/Email devops), Sun Microsystems (Consultant, Systems Engineer and Solaris Ambassador). Rafael has carved his career from field engineering positions, in constant contact with customers, into pure engineering ones, focused in Linux internals, userland, kernel and general OS & performance debugging and bug solving.
- I have started learning Linux since I was 12 years old (I'm 37 nowadays). A BBS sysop sent me Minix floppy disks and also told me to start studying Linux at that time. I literally spent the next years reading Richard Stevens and playing with Linux. I can pretty much say Richard Stevens thought me not only Programming, TCP/IP, but also ENGLISH. My mom used to become pretty upset about me not sleeping at night, reading books, and falling asleep in the classroom in the next day (when my teachers would call her to complain). I joined "business life" very young (around 16 years old). Sun Microsystems was basically a school for me - together with Physics University and "special" Optics Interferometry classes - where I used to visit customers to deploy all "enterprise" related software (clusters, backup softwares, tape library managers, Solaris, oracle, etc). After some years I moved into the HPC world so I could work more with Linux (as Solaris 10 was still dominating Sparc High-End/Mid-End servers, my daily job back then). Lot of time passed, I graduated in a Business Administration school, and I found myself playing with Mainframes and their "new" technologies (yes, s390x has NEWS features every year, despite still being called Mainframe). IBM was also another school, where I was able to enter specifics on s390x instructions, architecture, hardware and hypervisors (note: don't call z/VM a hypervisor to old guys, as CP/CMS is a full operating system, like they usually say). At that time I had an awesome opportunity to visit Germany, make new friends, and get to know s390x performance engineering team, which basically raised my ill for an "engineering only" position. That is what motivated me to start working in the sustaining engineering team @ Canonical, back then, together with the remote work: something that could help me taking care of my son.
- I currently live in Curitiba, Brazil - yes I love cold places - with my wife, whom I love very much, and my 2 kids: a 12 years old boy and a 3 years old daughter. We have 1 dog, an English Staffordshire Bull Terrier - or just Staff Bull - called Trevi (Because of Fontana de Trevi in Rome) and a very young cat called "Luna", whom we rescued from bellow our neighbor's car after she traveled for more than 200 kms with Luna under her car). I guess Luna really wanted a new life.
My Ubuntu story
- I used to use Slackware in the beginning, but the fact that I had to rely on an external tool to deal with .tgz packages (slackpkg) that had almost no relationship among them took me to Debian really quickly. After Ubuntu arrived, moving to Ubuntu, because of its better Desktop experience initially, was quite automatic for me. Of course we all had the Gentoo experience, but after nights and nights recompiling X11 because of different compilation flags, or just because of new dependencies versions, make us re-think this "let me compile everything approach". That also goes for OpenBSD ports, and the incredible attempt of making NetBSD pkgsrc to behave like a Linux distro packaging system (I really tried to stay with compiled only packages in NetBSD). My experience with RHEL and CentOS was mainly targeted to enterprise customers using it, and not for fun and/or development purposes. Because of the "enterprise" type jobs, I was never much into the contributing-to-public-projects before working @ Canonical (SEG team), something I regret. All the work I can give as a reference starts at that time, but I think it is enough to show how I have already contributed to Ubuntu project and how I can, even more, help the project in the future.
Examples of my work / Things I'm proud of
- Literally designing all z13 mainframe I/O programming from the ground: Defining all possible s390x sub channel-systems and their LPARs based on very little information (at that time) in order to maximize Canonical Mainframe usage and help out Foundations and Server Team to start porting Ubuntu to s390x architecture. Christian and I did all the mainframe enablement in very short time to help out a fast pass porting plan (and go to market).
- Helping out Canonical customers in the Ubuntu Advantage program. It was quite nice to help big companies to use Ubuntu Linux in their environment. Sometimes I felt like a "taylor-made" specialized engineer in "very specific problems" related to virtualization and performance.
Having worked as a test engineer for official Linux stable trees - from Greg - when I was working @ Linaro. Running almost 30k functional tests - focused in armhf and arm64 architecture - for every single branch, of every supported kernel version, providing feedback for all regressions that happened in between the releases, with root cause and involving "who-to-blame" (and possibly fixing them) was quite an adventure. Some of the contribution log: contriblog.
- Having contributed to several upstream projects regarding bug fixes, including: Linux (kernel), QEMU, libvirt, Apache, Linux Test Project, LVM, Systemd, and several others.
- Having contributed to deploy some Infiniband-only HPC clusters - one of them was TOP-65 in Linpack Benchmark in ~2009 for Petrobras (Pre-Salt Project) inside Brazilian Universities, in a project called Grade-BR. I loved spending huge amount of time at University of Sao Paulo TPN inside Engineering Campus.
- Having learned s390 architecture nowadays. Specially fighting z/OS engineers on how good Linux was doing what they did not know Linux was capable of (and listening that Mainframes already did that since 70s).
- Having installed the first Linux-only - alright, on top of z/VM, but no z/OS - z196 Mainframe in Brazil, consolidating more than 300 servers in a single Enterprise box, for a telco company in Brazil.
Areas of work
Recent work, trying to show different skills (merging, debugging, backporting)
A complete example of a merge and then another quick one on top
This is an example of a merge https://launchpad.net/ubuntu/+source/bind9/1:9.11.5.P4+dfsg-5ubuntu1 for bind9 during Eoan development and, right after, another quick merge with Debian as soon as they've delivered the _same_ fix for the CVE in question, bind9/1:9.11.5.P4+dfsg-5.1ubuntu1, as it was better to keep up with upstream code instead of the delta.
All of my upstream work is done with latest Ubuntu Devel package sources debian/ directory being placed in upstream tree (with proper .gitignore). This way I usually do merges whenever working with upstream, even when not uploading them. The work directory I use is described at https://github.com/rafaeldtinoco/work and has been presented in a conference also: video
A complete example going upstream and backporting the fix
LP: #1840958 - defragfs.ocfs2 hangs (or takes too long) on arm64, ppc64el
- Provided a Merge request to be reviewed with a PPA:
- I've provided explanations, about the bug, upstream:
- Suggested a merge to Debian considering upstream fix already:
An example of complex debugging for QEMU
LP: #1805256 - qemu-img hangs on rcu_call_ready_event logic in Aarch64 when converting images
- Upstream work fixing QEMU Async I/O for Aarch64: Somehow ARM64 (D06 Huawei Server) existing MEM barriers aren’t enough for this machine, causing Async I/O to hang (this is also true for QEMU core parts, not only for qemu-img).
- I have added a mutex around affected parts and it was enough to make it work for ARM64 but upstream might want to continue with primitives so I might have to re-check which barriers weren’t enough for the mem ordering and fix them.
And one example backporting new features (security related) to QEMU
- IBRS_ALL (enhanced IBRS support)
- SKIP_L1DFL_VMENTRY (L1D flush is needed on VMENTRY)
- RDCL_NO (HW is vulnerable to Rogue Data Cache Load)
- Foreshadow-NG (OS) vuln. (L1 terminal fault, OS)
- Foreshadow-NG (VMM) vuln. (L1 terminal fault, VMM)
And some of my work helping out @rbasak during this last weeks for Eoan MySQL 8 upgrade: @rbasak has done a huge amount of work to upgrade MySQL to 8 in Ubuntu Eoan
- dbconfig-common (Upstream Debian Merge Request): Because of cacti (2) I had to create a debconf variable to set MySQL authentication plugin to be used by dbconfig-common consumers. Because of MySQL 8 changes, ALTER SQL commands had to be altered also.
- cacti (Upstream Debian Merge Request and Upstream Merge Request): MySQL 8 does not allow ALTER to create a USER by default anymore AND its default authentication plugin is caching_sha2_password, not supported by some PHP packages.
Some public visibility of my Sustaining Engineer work at SEG
Kernel – codes/bugs/debugs/upstream
https://goo.gl/LNhYGC - megaraid/sas/scsi
https://goo.gl/R8kwca - linux-scsi (iscsi)
https://goo.gl/QTh765 - linux-scsi
https://goo.gl/AB7bA4 - numa/scheduler
https://goo.gl/3kgCLF - perf/scheduler
https://goo.gl/jZLHUN - hpproliant/ubuntu
https://goo.gl/rPi1fV - hp proliant/watchdog
https://goo.gl/amK5UV - vm/scheduler
https://goo.gl/KDLRtq - network/namespaces
https://goo.gl/EsCFDm - apparmor
https://goo.gl/zbqy3k - infiniband/iscsi/iser
Userland – codes/bugs/debugs/upstream
https://goo.gl/25Z8h9 - qemu
https://goo.gl/AK3Xwr - qemu
https://goo.gl/frVDeJ - qemu
https://goo.gl/xta33w - qemu
https://goo.gl/DZZtHQ - libvirt
https://goo.gl/eJwzWc - nfs-utils / systemd
https://goo.gl/ai3eUw - sudo
https://goo.gl/dErX1e - isc-dhcp
https://goo.gl/UvsyJF - isc-dhcp
https://goo.gl/FAk5Y6 - libvirt
https://goo.gl/RXH6kz - crmsh
https://goo.gl/8GeF37 - pcs
https://goo.gl/XDRtZ3 - pacemaker
Things I could do better
- Be less prolix in application wiki pages.
- Sometimes I have to force myself to revisit things: SRU verification. Keeping the dev environment is usually good for verifying things faster, I should do that more often.
- Changelogs and patch descriptions are some of the things caught in upload reviews. Sometimes, despite being "ONLY" a "format change", it is important that formats and best practices are always followed. I tend to think more of the fix, backport/cherry-pick and functionality then the formats, I have to change a bit my mindset (even after all the SRUs done).
- Merges: I have done some merges (even using git-ubuntu flow) but I should do more. Now that I'll be taking care of HA related packages, I tend to think this will be a natural consequence. I was able to unblock all blockers from Eoan excuses before Eoan freeze dates, and after Eoan is GA it will be the time for all HA related SW merges and syncs.
Review more merge requests: I thought I was "too new" in the team to review the merges, but, naturally, this sensation is going away. Reviews aren't there because of the experience of the reviewer, sometimes its there so code has another pair of eyes before landing in the repo <- That is something I'm improving but my reviews will definitely get better from now on.
Plans for the future
- I'd like to create a regression test scheme for HA related software: configuring multiple environments with the most used HA scenarios and resources and testing to check if current Ubuntu development release is causing any regression to those supported scenarios.
- Still regarding HA, I have to create good HA documentation for 20.04, with the scenarios I mentioned in the first item, with pictures an examples of those scenarios and how to get them installed (using pcs, thus the MOTU request altogether).
- I'd like also to continue debugging all kinds of server related bugs. As part of my daily work, but also as part of professional development AS doing that guarantees that we become "deep generalists", with eyes in several different technologies. If I may choose: I prefer those hard-to-debug cases, with dumps and reverse-engineering needs.
- Apply for core-dev. I sense that, for that to happen, I have to do future merges, and, 20.04 will be the perfect version for me to do so.
What I like least in Ubuntu
- Lack of a centralized place for public documentation. I quote here archlinux wiki pages, Red Hat administration and development manuals, SUSE HA documentation, and some others. Usually we find documentation through launchpad bugs, which is great, but not necessarily the best for all situations. If you go after the bugs I've been involved, I tend to do really prolix comments because of that reason, but I would be willing in providing those comments as "tips" in a wiki of some sort (and, yes, I'm aware discourse.ubuntu.com might be there for that, unsure if it will look like something as good as archlinux wiki, for example).
If you'd like to comment, but are not a sponsor, please do it here. Don't forget to sign with @SIG@.
I've worked with Rafael in the same team. He's been my mentor and tech lead, and while being technically skilled in userspace and kernel, he's proven to be a hard working, committed and reliable engineer. Being the proficient Ubuntu contributor he is, his natural step would be to be a full coredev, but he would still be a great asset as a MOTU :).
-- vtapia 2019-10-15 13:16:32
Rafael has been a expert reference for us (SEG) in the Linux internals (in both fronts; userland and kernel), virtualization, storage and HPC areas since he joined Canonical.
His debugging and analytical skills have produced reliable and consistent bug fixes and outstanding bug reports over the years that have benefited both the Ubuntu users and the many customers that Canonical supports.
In my opinion, based on the amount and quality of his contributions, it should be pretty clear that having him as part of the coredev or MOTU teams would be a ideal scenario that would benefit the whole Ubuntu ecosystem.
Rafael used to be my team member and a tech lead. He was always a invaluable source of knowledge and ideas for approaching difficult technical problems. I have always values his ability to create efficient software and workflow setups optimizing performance.
I strongly believe he has everything it takes to be a great MOTU and eventually a coredev.
-- dgadomski 2019-10-15 14:23:48
We worked in the same team for almost 4 years, he was always open to help others and when he was asked about things outside his expertise domain he would catch up very fast and give guidance. I remember how he helped me to understand corosync's totem protocol works and how Pacemaker makes use of it, this knowledge allowed us to analyze the behavior of resources agents and come up with improvements to deployed OpenStack clouds.
-- freyes 2019-10-15 20:21:01
Rafael helped out on a project (especially setting things up) before I joined Canonical. Since joining Canonical I am now working in that same project and still benefit from his structured work and farsightedness - and this is now years ago! During my time at Canonical I got in touch with Rafael every now and then, mainly in his role at SEG, and I always enjoyed his deep expertise, commitment and patience. I am convinced that he has everything that's required, not only to be a Masters of the Universe, but definitely also to become a coredev!
-- frank-heimes 2019-10-18 15:23:00
If you'd like to endorse me, please do it here. Don't forget to sign with @SIG@.
I've worked with tinoco for many years; he was a co-worker when I first started at Canonical and shortly after rose to Team Lead. He's extremely knowledgeable and sharply focused, and I have always had 100% faith he will be capable to handle any task he takes on. He has not proven me wrong yet. I have no reservations about endorsing him for MOTU or listed per-package rights.
Specific experiences of working together
I've only one example of actually sponsoring his work, however I've worked with him on debugging/fixing many other packages.
Areas of improvement
I'm not sure why he is applying only for MOTU, instead of full coredev, but hopefully the coredev application won't be far behind.
-- ddstreet 2019-10-14 20:42:23
I was working with Rafael not only in the Canonical Server team for a few months now. Before that we had a great relation when was in SEG/STS fighting virtualization and Mellanox related, but actually all kind of tasks that were thrown at him. And even before that we worked together at IBM on Linux for s390x. In all those years I've seen him very dedicated at work and loving to drill deeper and deeper into an issue until it can be fixed. He has a very deep understanding and toolset for all sorts of debugging tasks which sometimes is quite impressive.
I might have sponsored 10-20 packages in the recent months. But across the years, departments and companies I have "reviewed and sponsored" much more. Due to that I have learned to trust the applicants code. But maybe more importantly whenever I found something on review he wanted to understand the details, quickly adapted and rarely fell into the same traps again.
Specific Experiences of working together
As I said we share quite some history together, so let me round this up with a very old and a very recent story.
Far past: We were benchmarking and needed very accurate (nano seconds) timing throughout a stack of hipervisors. There was a kernel module that we used but it was ugly code and often caused issues with different kernel versions. Then there was an HR action that send us over this yet unknown guy from Brazil for two months. He was from the Field, but wanted to get deeper into the tech side and for that learn more from the Engineering team. It turned out to be Rafael and I rarely have seen someone hitting the ground running so fast. After a few weeks there as a userspace program which accessed a new Kernel interface for the timing data and to my knowledge it is still used today. Lessons learned: he'll be focused, adapting quickly and oriented to get the things solved that you need to be solved (It seems that hasn't changed since then).
Recent: I was totally drowned in other cases and then came a very complex case for qemu/libvirt that we need to work on. I was asking him to take over backporting, testing and communicating for that case and it worked great. The code was backported well, packaging was done fine - and as I said above the few things I found on review he only did wrong once which is all you can ask for.
Areas of Improvement
I sometimes challenge a behavior that you might call "lost in details". E.g. you get a great description of a case but it is three pages long and it can be hard to quickly pick up what needs to be done next. On the other hand: a) I tend to do the same often enough which might be why I'm more sensible to it b) It can be used as documentation or to transition a case as most state is in those comments (which is good)
Sometimes adding a TL;DR section might help, but IMHO that clearly doesn't block those upload right request that are discussed here.
-- paelzer 2019-10-15 08:24:12
Rafael and I worked in the same team (Canonical STS for at least 2 years). During that period of time, I had the chance to work with him closely and learn from him. Rafael used to be my mentor back in the days.
Specific Experiences of working together
Over the years I've sponsored a few package for Rafael such as multipath-tools, kexec-tools, systemd and apache2. The packages I've sponsored and other updates I've seen were high quality, and showed a good attention to detail.
Areas of Improvement
I have the same question as ddstreet who also point it out in his endorsement ... I wonder why Rafael doesn't go straight for coredev instead but hopefully, that would be the next step for him.
-- slashd 2019-10-15 07:04:12
I work close with Rafael on the server team, and follow his work on a day-to-day basis. I also had the opportunity to get to know him personally as roomies at the Paris Ubuntu Engineering Sprint. I agree with others Ubuntu would be improved having him as Core Dev, and see attainment of MOTU as a solid step towards that objective.
I once heard that a good sign of someone being ready for MOTU/CoreDev is when people react, "What?? He's not that already??" and this definitely applies to Rafael.
Specific Experiences of working together
Many of the packages he tackles (e.g. qemu/libvirt) are in areas I don't feel suitably qualified to judge, but usually by the time he needs formal review the changes have been thoroughly studied and discussed so the reviews have been straightforward. Even when I have little to give in review but nit picks, he's been welcoming of critiques. Rafael is comfortable with critiquing his own work, and not shy about pointing out flaws, which gives me strong confidence he is worthy of trust.
In addition to reviewing his HA packages, he and I collaborated on some of the MySQL transition tasks, and I was very impressed by both his teammanship and his ability to dig deep into difficult problems even in unfamiliar areas. He is a good communicator and verbose documenter, and I found it easy to hand off things I was stuck on, and easy to pick up and continue from his work in turn.
As a verbose person myself, I certainly can't fault anyone for lack of brevity, but actually I have more than once appreciated Rafael's detailed comments as saving me time coming up to speed on the issue. I say keep it up!
Areas of Improvement
I feel Rafael is already more than qualified for MOTU, and probably CoreDev as well. For proving the latter, I'd encourage generalizing by working on a wider variety of packages, including non-Server types of software, and to look especially for opportunities to do review/sponsorship.
-- bryce 2019-10-15 19:37:23
Rafael produces high quality work and leaves "no stone unturned" when chasing down a bug. He also comes up with test cases, which I always appreciate, and good instructions on how to do things, like setting up a testing rig.
I think Rafael will be great asset to Ubuntu as a MOTU, and Server Dev, and he should be able to become CoreDev very quickly.
Specific Experiences of working together
I sponsored 7 unique packages for Rafael: list
Of these, the most complex one was samba, with many changes, and that was also his first. There was a lot of back and forth, but I'm happy to say that the subsequent uploads were straightforward.
Rafael picked up the Debian merge workflow very quickly, and I think just a few more merges, so some edge cases can present themselves, will be enough.
Rafael quickly volunteers to fix hard bugs, and a recent example as bug #1838525 which affected the subiquity server installer. I troubleshooted that a short while with him, and it was clear he was heading in the right direction.
Areas of Improvement
I think in the upcoming LTS cycle Rafael can ramp up his Debian merges, and that will give him a lot more packaging experience and should make obtaining CoreDev the next natural step. I'd also like to see him get more experience in seed management and chasing down dependency trees in Ubuntu, which is something CoreDevs have to do sometimes.
-- ahasenack 2019-10-18 14:47:26
Rafael joined us on the Canonical Server Team about five months ago, but many of us have interacted with him in the past when he worked in a different department escalating fixes for customers to us to land in Ubuntu. I have sponsored a handful of packages for Rafael over the years in both of these two roles. From my memory, his work is generally correct with appropriate attention to edge cases. I don't recall any review that showed any fundamental misunderstandings or gaps in knowledge; the only review tweaks I suggest for Rafael are minor and subjective.
A couple of things I think Rafael does outstandingly well, compared to average. He asks for advice, and really takes adopting feedback to heart. I'm always confident that Rafael has understood feedback, adopts it and never forgets it.
Specific Experiences of working together
Recently Rafael helped me with the MySQL 8.0 transition. Specifically I asked him to take care of a complex issue with cacti and dbconfig-common. This was one of those more complex tasks where APIs and assumptions across the stack needed adjusting rather than a simple bug fix being sufficient. It was certainly not a task suitable for beginners. Rafael handled it very well, demonstrating an ability to understand the complex interactions involved, complex packaging concerns (debconf and the indirection through dbconfig-common), the Ubuntu delta and the trade-offs in maintaining it, and the appropriate communication and coordination with Debian maintainers. I can't think of anything on this issue that I would have preferred Rafael to have handled differently.
Given that experience, I am confident that Rafael will make an excellent and responsible uploader.
Areas of Improvement
I think that Rafael is particularly goal orientated, possibly influenced by his previous role championing the landing of fixes that customers need. This is a good thing! The caveat is that sometimes it might be appropriate for us to avoid landing a fix in Ubuntu. The importance of the issue might not justify the maintenance burden and the risk that a fix that filters down from upstream will find itself in conflict with our patches when it eventually arrives. I think this came up once, but as above I don't think Rafael has forgotten it and I don't recall an instance where this has come up since.
== <SPONSORS NAME> == === General feedback === ## Please fill us in on your shared experience. (How many packages did you sponsor? How would you judge the quality? How would you describe the improvements? Do you trust the applicant?) === Specific Experiences of working together === ## Please add good examples of your work together, but also cases that could have handled better. === Areas of Improvement === ## Add here areas where you think applicant could improve.