Maintaining Your Love For Passion Projects - Josh Wolfe

When you can't find the right tool, build your own and make it the best.

Our very last speaker is Josh Wolfie, a Zig Software Foundation board member and core team member advisor. When asked if there was anything else the audience should know about him, he humorously responded, "I am just kind of some guy, that's me."

After a quick tech check confirming the microphone was working, Josh began his presentation. He introduced a roller coaster of a love story about a music player project he worked on with Andrew Kelly called GR Basin. In 2014, they aimed to upload zip files of music into their music library, which ran on a Node.js server. This required reading zip files in JavaScript. Josh surveyed existing options on npm and found them all problematic. Issues included locking up the main thread during synchronous IO, buffering the entire file in RAM, or using a streaming API to read the zip file from start to finish, which had its own set of problems.

To address these issues, Josh created his own library called Yael (Yet Another Unzip Library). After a couple of months of development, he decided to seriously support it and aimed to make it the go-to zip file library on npm. He added all the features expected in a high-quality open-source project, including API documentation and tests. He also engaged with users on platforms like Stack Overflow, helping them debug their issues even when the problems were not directly related to his software.

Yael became very successful and was eventually picked up by Browserify, significantly boosting its popularity. Today, it is depended on by frameworks such as Electron and Cypress, boasting 14 million weekly downloads. This success is partly due to npm's dependency structure, but it also speaks to the library's reliability and utility.

One of Josh's favorite moments was receiving a simple "thanks" in Yael issue 86, which the author closed immediately. This highlighted the appreciation users had for his work. Despite its success, Yael faced challenges. Initially, it supported Node 0.6, which used a buffer constructor that was later deemed insecure by Node maintainers. Although Josh's implementation was secure, security scanning bots flagged his software, forcing him to work around this issue. Additionally, there were problems with other zip file implementations, such as The NET Framework, which used backslashes in filenames, violating zip file standards.

Despite these hurdles, Josh remained committed to making Yael a well-supported software package, reflecting his dedication to the open-source community and the success of his project.

=> 00:05:48

Dealing with other people's design flaws and unstable tools is a constant headache for developers.

There is a security problem with the buffer constructor, but I was careful to initialize every byte, so my software didn't have this problem. However, the security scanning bots flagged my software as being insecure because it was using the buffer constructor. Someone else's design flaw means that I now have to work around this in my code. I'm responsible for solving other people's problems, which is frustrating. In addition to general npm instability, there were problems with other zip file implementations in the wild. For instance, The NET Framework had versions where they would put backslashes in file names, which is not allowed in a zip file, but nonetheless, there were backslashes in file names and zip files in the wild. I had to work around that by putting in special handling for normalizing backslashes, even though that was against the spec.

The WinRAR utility, which can create zip files, does Unicode in a really weird way where they lie about the file name and then in extended data, they say, "actually the file name is this." So, the file name is in there twice. The Apple archive utility built into Mac OS also had significant issues. To explain these problems, we need to take a technical interlude and understand how the zip file format works.

An archive format is where you take lots of files and stick them all together in one big file. The purple chunks represent the files accumulated together, and the gray slices are the metadata that tells you things like the file name, permission bits, and where in the archive the file data starts and ends. Surprisingly, the central index of where everything is located is at the end of the file. You start reading zip files at the end, and then it points backward to where everything is. This is why reading a zip file from start to finish doesn't work; you have to start at the end.

The other gray rectangles are the local file headers, where almost all the metadata is duplicated. The local file header also gives you the file name, which means for WinRAR, it's in there four times. It also tells you the size of the purple file data, so you can start at the beginning and read through to the end. However, the central directory is necessary because, in the official specification, you might not know the file size as you compress the file data. This data might be missing from the local file header but is present in the central directory.

The main mistake the library made was expecting the local file header to be complete when it's not guaranteed to be. To fix this, right after the purple data, there is something called the data descriptor, which has a magic number (optional), a CRC 32, and the file sizes pointing back to the start of the file. This allows you to find the end of the file by locating this magic number. However, if the file data is not compressed, this Sentinel can be trivially spoofed, creating ambiguity.

=> 00:09:50

Apple's archive utility creates zip files with corrupted metadata, making them nearly impossible to read accurately.

The issue at hand is that the local file header was expected to be complete, but it is not guaranteed to be. The zip file specification author proposed a solution: right after the purple data, a data descriptor should be added, which includes a magic number (optional), a CRC 32, and the file sizes pointing back to the start of the file. This allows one to identify the end of the file by finding this magic number. However, if the file data is not compressed, the Sentinel can be easily spoofed, creating an ambiguous zip file and a security hazard.

Even if the spoofing is not intentional, packaging a zip file inside another zip file can cause issues. The inner zip file might have the magic number in the middle of the purple data, leading to incorrect checksums (cheim). This raises questions: is the cheim supposed to detect single-bit corruption, or should it indicate that the data descriptor is incorrect? The file sizes, which should be reliable, are either 32 bits or 64 bits, based on information found only in the central directory. This entire approach is flawed and misleading, potentially creating security hazards for naive implementers.

As a zip file reader library, the focus is on how Apple Archive Utility writes archives. File sizes in zip files can be 32 bits or 64 bits. Originally, 32 bits were sufficient, but an extension was added to the zip file specification to accommodate oversized integers. If files exceed 4 GB or there are more than 65,000 files, the extended metadata (zip 64 extension sections) must be used. However, Apple Archive Utility does not support this extended metadata, yet it still creates oversized zip files. Instead of using a proprietary extension, they let their integers overflow and write truncated bits, resulting in local file headers that point to garbage.

This practice by Apple Archive Utility is worse than trash; it is Omega trash. For instance, if an archive contains 66,000 files, it might incorrectly indicate only 464 files. This can be reproduced with tools like unzip DT, which shows that the archive utility lies about the file count. This makes it extremely difficult to support. Yazel, a security feature to prevent zip bomb attacks, ensures accurate file sizes before extraction. To support reading these zip files correctly, Yazel would need an advanced lookup method to account for potential offsets by multiples of 4 billion.

A complex solution was devised on paper, requiring significant effort to implement. After working on it intermittently for over a year without progress, a test case was created that failed without any code to solve the problem. This led to a complete loss of enthusiasm for the project, halting any further progress.

=> 00:14:50

Burnout isn't just about the work; it's about understanding and empathizing with your own feelings.

To support reading these zip files, Yao would need to do a really advanced lookup where it skips around in the archive to see if maybe everything's off by a multiple of 4 billion. I had a plan for doing this and came up with a really complicated solution on paper. It was going to be a lot of work to implement, and I was working on it off and on for over a year without making progress on anything else in this project. Eventually, I came up with a test case that failed without any code to actually solve the problem. That's when my enthusiasm for this project completely died, and I couldn't bring myself to make any more progress on it.

This was a project that I had cared so much about. I was willing to help people with their homework, adding features, and working around other people's bugs. But now, I just didn't want to think about it anymore. Even on the issues, people would ask simple questions like this one, which would have taken me 10 seconds to respond to with "not supported, see the readme, closed." But I just left it open. I couldn't even bear to think about this project. I was so burnt out. At one point, someone opened an issue that just said, "this project is unmaintained, use my fork," and I just left it open.

For reasons related to software engineering, I was getting actual therapy. I was trying to answer some really important questions like this one: how do I really feel? This is a concept in therapy called empathizing with yourself. If I had heard this phrase when I was in high school, I would have thought it was stupid. Empathizing with myself? Well, I am myself. I'm always empathizing with myself. How could I ever not know how I feel? Stupid. Okay, let's come back to that.

At the time when I was burnt out on Yazel, I thought I knew why. I thought I knew what was bothering me so much, and I would tell people confidently, whoever wanted to ask about this, that I knew the reason why I burnt out was the buffer constructor. I was sick of solving other people's bugs like the buffer constructor, and that just burnt me out on the project. I knew that the archive utility corruption thing was hard to work around, but I just saw it as a challenge, not as a drain on my enthusiasm. But something's wrong with this buffer constructor hypothesis, which is that it's really easy to work around. You just change one expression of code to another, and then it's solved. This is not hard.

So before we get into what really killed my enthusiasm, it's worth focusing on this for a moment. I was wrong about how I felt. I didn't understand my own feelings. If you're out there and you think that empathizing with yourself is a trivial concept, that you can obviously empathize with yourself, you might be bad at this too. When I went to therapy, I was able to learn some techniques for identifying how I felt. I discovered, and again, I was going to therapy for reasons unrelated to software, that a lot of problems in my life had been caused by my inability to empathize with myself.

There were relationships with my family that felt really strained and weird and really uncomfortable for me, and I didn't really know why until I realized that I had feelings that I wasn't aware of. I had friendships where I thought I was looking forward to hanging out with them until it came to the point to leave the house, and then I was dreading going because I didn't know how I really felt about that friend. I can't even tell you how many romantic relationships were sabotaged by myself because I didn't know how I felt about my partner.

After learning this skill and going back to this project, Yazel, and applying it, I eventually figured out what had drained my enthusiasm. The answer is that I was allowing Apple to emotionally abuse me. This may sound silly—Apple is a giant corporation, they don't care about you, they don't even know you exist. They just made some...

=> 00:19:54

I learned to accept myself and my work without needing Apple's approval, and it reignited my passion and productivity.

I was looking forward to hanging out with them until it came to the point to leave the house, and then I was dreading going because I didn't know how I really felt about that friend. I can't even tell you how many romantic relationships were sabotaged by myself because I didn't know how I felt about my partner. So, after learning this skill and going back to this project to Yasel and applying it, I eventually figured out what had drained my enthusiasm. The answer is that I was allowing Apple to emotionally abuse me.

Yes, this may sound silly—Apple's a giant corporation, they don't care about you, they don't even know you exist. They just made some buggy piece of software and then did nothing. Apple didn't do anything here, but I allowed Apple to emotionally abuse me because of the relationship that I created with them. Let's look at this bug: this archive utility corruption bug. Archive utility is closed source, there's no public bug tracker, I couldn't tell if anyone had reported this before or if the maintainers were interested in fixing it. So, I can't work with them; they're not interested in working with me on this. If I want to test my changes, I don't own any Apple technology, so I would have to pay them to work around their bug. This whole thing is me doing a lot of work for them and them doing nothing for me.

To make this even worse, I thought I had to support this. I had supported theet bug, I had supported the WinRAR bug, and archive utility is even more popular than these other two. Of course, I have to support this if I want to be a real zip file implementation. I know what you're thinking: the solution is obvious, but it's really difficult to let go of this issue. It's really difficult to allow myself to be proud of my software with a flaw like this. But in therapy, I learned about accepting myself. If this phrase also sounds bizarre to you, you might benefit from going to therapy. I learned to accept myself, and I learned that my software was valuable. There were 14 million downloads a week; people were using my software even without this feature. I was using my software; it was working for Groove Basin. I don't need Apple's approval for my software to be good enough.

If you model this, if you model an enthusiasm-driven project economically, enthusiasm is the funding, and the cost of interfacing with archive utility is just too high. So, eventually, I finally managed to convince myself not to support it. I closed the Yasel issue, which was the most active issue on the bug tracker by far. I apologized, said sorry, I'm not going to support this, it's just too hard. Just like that, my enthusiasm returned for the project. I was able to make several new releases after six years of silence. I implemented all sorts of features and helped out all sorts of people who had issues, but now I was doing things that I enjoyed and that I thought were fun. I was only able to make that discernment because I knew how I felt.

Also, no more helping people with their homework. There was one issue somebody opened, and I appreciate the honesty. It said, "I haven't read the code or looked at any of the issues, but you should be using Bigin." And this was my response: "Nah, issue closed."

Alright, that's it. [Applause] Thank you very much, Josh. So, take a moment to sip some water, pull up your mic; I think it's falling out a little bit. Yeah, there you go. Okay, so I think there's going to be some questions from the audience. Man, there's been a couple of very relatable talks, huh? At the end of this event, I did not plan for this. I expected FS to be good, but I'm also being surprised that it was great. You know what, let's have questions, and then maybe I'll add a comment or two afterward. So, let's see, raise your hands if anybody has questions.

Okay, I don't see hands. You know what, I'll break the ice then. So, what are your thoughts about, for example, the Log4Shell situation? Because you remember, right, when people discovered that there was this logging library used everywhere in the Java ecosystem, and then the outdoors...

=> 00:25:18

Passion projects thrive on enthusiasm, not money.

MC: I think it's falling out a little bit. Yeah, there you go. Okay, so I think there's going to be some questions from the audience. But man, there have been a couple of very relatable talks, huh? At the end of this event, I did not plan for this. Like, I had expected FS to be good, but I'm also surprised that it was great. You know what, let's have questions, and then maybe I'll add a comment or two afterwards. So, let's see, raise your hands if anybody has questions.

MC: Okay, I don't see hands. You know what, I'll break the ice then. So, what are your thoughts about, for example, the Log4Shell situation? Because you remember, right, when people discovered that there was this login library used everywhere in the Java ecosystem, and then the authors got super pressed into delivering a fix very quickly. Everybody started shouting about, "Oh my God, open source needs more funding," because of the situation. Did you have any thoughts? Open source needs more funding is the idea.

Speaker: Well, I have to say I'm not an expert in funding open source projects. The project that I was doing was all volunteer work. I will say that the bug seemed pretty bad, that Log4J vulnerability. So, like, don't screw up Log4J. I'm trying to think of something more interesting to say in response to this. Okay, let me ask you then a related question: would more funding have helped you with your project?

MC: No, this project is not motivated by money. I don't really want to accept any money for this. I'm pretty defensive about the motivation that I have for this project. I want it to be motivated by making quality software and supporting actual people who would use it. The enthusiasm of the community and my own enthusiasm are the funding. If I accepted money, then it would become like a business venture. Whenever I make this evaluation about whether I should bother to do something or not, I can't just rely on enthusiasm; I have to also weigh in the money. Putting a monetary value on how much pain I'm willing to go through is a situation I really don't want to be in. So, I much prefer to do this project driven only by enthusiasm. But I respect everybody who is not willing to work for free; it's not really something that you can expect someone to do. But that is how I preferred to do this project.

MC: Absolutely, and by the way, the people who were talking about funding, as far as I'm aware, weren't even the authors. They were like, you know, the peanut gallery people looking at open source and the kind of discourse that you have in the wild. I do feel like, and again, I can't speak from experience because I haven't been there personally, but I do feel like the healthiest choice for you was at some point to say, "Allow me this, I'm not going to support this stupid thing, I'm going to drop it." That's where I decide my project goes; this is where I decide to stop. I feel like also the people from the login library could have been right. They had the right to say, "Whatever, you want to fix this bug? I'm busy doing something else. You do it. I gave you the software with the usual warranty where there isn't any warranty." But I don't think that's what happened in that case. I don't remember all the details, but I feel like people felt like being more involved, which probably had bad implications for their mental health.

MC: Let's see, any questions in the meantime? Oh yeah, we got one up there and a couple more. Oh, and one here, sorry, let's go.

Audience Member: Yeah, so the idea of the conclusion really resonated with having something that you are really invested in and not having a way to solve it for a very long time, and then being able to finally put it down and recognize your own issues with it. Is there a name for the condition of not being able to recognize how you feel about something or what your relationship is with, whether that be a person or whatever project that you're working on, that interacts with your personality?

Speaker: So, I'm not a psychologist. I'm not an expert. I actually suck at emotional intelligence. That's why I'm going to therapy. But maybe an answer for you is just the concept of emotional intelligence. Being able to empathize with yourself is extremely valuable in passion projects and all over your life.

=> 00:29:59

Empathizing with yourself is key to emotional intelligence and can transform your passion projects and life.

Having something that you are really invested in and not having a way to solve it for a very long time, and then finally being able to put it down and recognize your own issues with it, is a significant experience. Is there a name for the condition of not being able to recognize how you feel about something or your relationship with it, whether that be a person or a project that interacts with your personality? I am not a psychologist or an expert; I actually struggle with emotional intelligence, which is why I am going to therapy. Emotional intelligence might be the concept you are looking for. Being able to empathize with yourself is extremely valuable in passion projects and all aspects of life.

Earlier today, it was mentioned that many people in this room probably think of themselves as very logical. However, if you think you are logical and not emotional, you are probably just unaware of your emotions. Every human is emotional, and if you are not aware of it, you might want to get therapy. I cannot give you a detailed answer for what this is called, as I was not given a diagnosis, but empathizing with myself was the most helpful thing for my continued contribution.

Yael: Hey, I have a question. First of all, I really liked the talk; this was my favorite talk of the whole conference. My question is, it sounds like your whole driving force behind working on this project is wanting it to be one of the best projects and wanting other people using it to be successful. How do you reconcile that with not wanting to help people with their trivial homework at the end?

One of the key points about not wanting to help people with their homework is that I am being more defensive about how much of my own enthusiasm I am willing to put into it relative to how much effort the other person is putting into it. There was an issue opened that said, "invalid comment expected length 16 found length 60,000," or something like that, and that was the entire text of the issue. I am not going to ask this person for more information; I am just going to say I cannot help you without more information. That was a change I made because this person did not seem interested in not wasting my time. This is someone who just posted a very lazy comment and was hoping someone else would solve their problem for them. Not helping people with their homework means I am not going to put myself out there unless I think it is going to be worth it for me.

Audience Member: Thanks, Josh. Great talk, I really enjoyed it. How do you think it would have played out if, in the middle of that six-year period, Apple had been more responsive and maybe asked you for more information or given you what you needed to fix the bug or situation? Or even if someone had just dropped in with a perfect PR that fixed it, how would that have affected your feelings about the project?

That is a great question. Unfortunately, because I struggle with emotional intelligence, I cannot answer it for sure. It is a bit of a hypothetical, but if Apple had been more responsive and willing to work with me, that would have helped a lot. The main problem was the lopsided relationship I had with Apple, where I was putting in a lot of work, and they were doing nothing. If they had been willing to put in the work, it would have helped me emotionally. For example, with the net bug with the backslashes, before I was even aware of it, they had already fixed it. That was technically easy for me to support and indicated a good relationship, meaning I was not alone in solving the problem. With the archive utility corruption bug, which was a really nasty technical bug to solve, I am not exactly sure where I would have landed, but it would have helped emotionally if they had been willing to work with me.

Audience Member: Hi, I would like to make an appeal to the audience. If you are a user of a project and have some experience with it, you can help the maintainers not do the homework for other people by just going...

=> 00:34:35

Collaborate with project maintainers on GitHub to keep the community thriving and help solve issues together.

For example, the net bug with the backslashes was fixed before I was even aware of it. They had already put in a fix for that, making it technically easy for me to support. This demonstrated a good relationship, indicating that I was not alone in solving this problem. However, with the archive utility corruption bug, it was a really nasty technical issue to solve. I'm not exactly sure where I would have landed on that, but it would have helped me emotionally if they had been willing to work with me.

Hi, I'd like to make an appeal to the audience. If you're a user of a project and have some experience with it, you can help the maintainers by not doing the homework for other people. Just go onto the issues on GitHub for a project and help with the trivial things. It takes absolutely no time but can be kind of annoying.

I also have a question. Almost all GitHub repositories become unmaintained in a short amount of time, yet they might still have some value. If someone is starting a new project, how do you prepare for it becoming unmaintained eventually while still hopefully bringing value to somebody?

If you're worried that your own project will become unmaintained, it shouldn't be a worry; it's a reality. If you want to support a project, you can make your intention known through your actions or an explicit statement. If you consider the work to be complete and are not interested in working on it anymore, you can just say that at the top of the README. GitHub even has a feature where you can archive a repository. There should be no shame in documenting what you're not willing to do. If someone says they don't want to maintain it anymore, that's perfectly fine. It signals to the rest of the community that it's time to look for an alternate version. Does that answer your question?

Yes, I think it does. Thank you.

Oops, I want to respond to this question about accessing emotions or feelings. There's a scientific term, Alexithymia, for not feeling emotions. Another recommendation is the modality of focusing and the concept of the felt sense. That would be my recommendation if someone's interested in this area. Thank you.

Hi, great talk, very relatable. One thing I got from how you described it, and I may be wrong, is that it was more or less self-induced frustration with the whole situation. I have found abuse online in comments and bug reports. Have you had the same experience, and how did you deal with it?

People sometimes write really aggressive comments like, "You have to solve this, otherwise you're an a-hole." Have you had those kinds of comments on this specific issue or while working on this software? What did you do if you got them?

I believe I have been fortunate enough not to receive that kind of aggression online. I think my eagerness to help probably mitigated those kinds of responses. During the years with no updates, there were people posting questions and not getting answers, but other people responded and helped with trivial questions. It never escalated to any kind of significant aggression.

=> 00:39:19

Recognize when your goals drain your enthusiasm and take action to keep your passion alive.

Years actually. So, have you had that kind of comments on this specific issue or on working on this software? Because you haven't updated it for six years, what did you do with it if you got it? I believe that I have been fortunate enough not to receive that kind of aggression online. I can speculate on why that is. I think that my very eagerness to help attitude probably set things off in a direction that sort of mitigated that kind of responses.

During the years with no update, there were people who were posting questions and not getting an answer. During that time, actually, there were other people that did respond and helped people with trivial questions. It never came to any kind of aggression against me, which I am very grateful for. So, I can't really help with that situation.

Let's do a couple more questions about the Apple thing. Would you think that also like the kind of thing where you can't let go of a certain feature but that also applies to self-set goals? So, like something that's not imposed by a third party where you can control it, but it just becomes sort of an obsession more than anything?

Yeah, if you find yourself in an obsession to accomplish a goal and the pursuit of that goal is so draining that it's robbing your ability to do anything else, I would recommend emotional awareness to recognize what's going on. If you've got a reservoir of enthusiasm and you need that bucket to be full to make progress, and there's a hole in the bucket, something's draining your enthusiasm out of the bucket, you got to know what that is. It could be that there's a self-imposed goal that's doing this; it could be externally imposed. But for a volunteer project, it's always self-imposed. Nobody can tell you what to do in a volunteer project. I mean, they can, but ultimately, you get to decide what you want to do for free and you get to decide what you want to support. Ideally, you document what you want to support and what you don't want to support.

Now, this all gets a lot more complicated if this is in your job. If your employment, if your well-being depends on doing something that's a huge emotional drain to you, I don't have an easy solution to that. It's easy enough to say just get a new job, but that's not always an option for everybody. So, that's a lot more tricky if the external factor that's draining your enthusiasm really can't be avoided by just giving up on it. I don't have an easy answer for that, but that is a much more tricky situation.

One last question. Thanks for the talk. You talk about three archivers that we can say are so far you cannot love, like the WinRAR, the Apple one, and the Net. Are there any that you will say are archivers that you can love and that reach high quality and are up to spec and whatnot?

The info zip implementation, which is the one on standard Linux, like the command line unzip program, that one seems fine. I've read through the source code to try to get insight into what I should be doing, and there's some questionable decisions in there. I wish it were a little bit better documented their rationale for doing it, but I've had no problems with info zip. That one seems great.

Okay, thank you very much, Josh. I think that was a beautiful way of closing the conference. We started by talking about business and how hard cold money does make a difference, and then we ended with a different problem where money isn't really the problem at all. But at the same time, it is still a little bit a matter of economics. It's just a different type of economics, right? It's a matter of enthusiasm, and you have enthusiasm vampires and situations that are like black holes for enthusiasm.

I've talked also in the past about how that also can happen when building communities themselves. It's a real thing; it's a real facet of social economics in communities. I hope that everybody got something out of this entire set of talks. Please give it up again for Josh.