I wish there was an actual thriving business model like this -- just fixing most annoying bugs, for a price, of commonly used desktop software. Why proprietary software companies cannot or do not want to provide this service is over me. Perhaps I'm too much used to consulting.
Given that “fixing this issue required weeks of intensive work from multiple people”, the price would have to be prohibitively high.
More generally, software is really, really expensive to produce and maintain. The economics only work at scale, in particular for B2C. (Maybe AI will change that, if it becomes more reliable.)
For many large companies or even teams, there exists a class of bugs / issues / features where dropping 5-10k on a bounty is extremely cost efficient compared to working around the issue or internal development. That might not fund development outright, but at worst it would point out the features people want and serve to inform what to work on next. I think there are a couple reasons why that is not prevalent. Most important one is that highly compensated enterprise teams that would benefit the most from placing bounties tend to avoid software that is lacking features or has bugs. Secondary is not implemented here ego and general disconnect between people in the trenches that know what needs to be done and people controlling ability to place bounties.
Imagine FAANG assigning $500 per engineer per year to allocate to feature / bug bounties.
Bounties for security holes make sense because you don’t need to submit the patch, just find the hole.
And bounties for open source (like in this case) also make sense because you have everything you need to submit a patch.
But for everything else (like big tech, startups, and so on) bounties can’t fix bugs because even if I find a bug, how am I going to patch it without access to the source code? How can someone submit a patch to Netflix or whatever?
IME your average SV startup has a long list of bugs they are aware of, but just haven’t gotten around to fixing because other priorities are in the way. But people can’t help patch unless you have an open development process.
You can fix bugs without source lots of ways, although many are arcane and finicky. An example of a healthy and productive ecosystem for this is in game modding. Sometimes this relies on vendor supplied tools (like a modkit, e.g. Elder Scrolls games), messing with bytecode directly (Minecraft until recently), or some cooperation from the vendor (Dwarf Fortress).
In all of those cases users/players were able to fix bugs and add desired functionality (mostly) independently, on a closed-source program.
For industrial software you don't see as much, even though arguably cracks (to skip license check) qualify here.
That seems different to me: a user can download and run a mod, but the fix isn’t then a part of the game itself and available by default to all users. Unless of course the real developers back port it to the game, but that’s just the kind of development effort the parent’s comment seems to be seeking to avoid.
The parent seems to be talking about the companies using bug bounties as a way to fix bugs in their software and the fixes becoming part of that software (not a separate mod run on top).
> even if I find a bug, how am I going to patch it without access to the source code?
That's how. Bethesda put a mod manager in Skyrim and works with some of the developers, they distribute fixes as game patches, you can distribute yours as "mods" or let them repackage it into an official patch or the next update.
I guess maybe it could apply to some niche cases of locally run software like photoshop, though I’d be be shocked if the marginal gains of a bug bounty program could justify the massive cost of implementing a mod system like this for photoshop.
But the fact is that most software in the world doesn’t work like Skyrim. Large parts of most software runs on servers or on locked down mobile operating systems where modding systems are not possible.
What you are proposing kind of already exists for web frontends in the form of browser extensions, but having worked on several apps for which an ecosystem of browser extensions sprung up, my experience is that there is no simple way to port these features to the main product. For security and QA, every line of code needs to be vetted anyway, and then “translated” into a form appropriate for the existing code base. At most, they just validate demand for a feature or bug fix.
Most larger companies would probably find it way easier and more sensible to contract with some outside consultancy to work on these issues than just posting a random bounty, even if the latter might potentially be cheaper. See Google Summer of Code projects for a very practical example of how "just pay randos to work on issue X for cheap" can quite often end up in failure.
Yes, when my org needed a very specific feature from an open source project the company reached out to the authors. I don’t know the terms, but they dropped a chunk of cash. No strings either on the new feature and everyone benefited in the end.
> See Google Summer of Code projects for a very practical example of how "just pay randos to work on issue X for cheap" can quite often end up in failure.
That potential for failure is there for any "subcontractors". I wonder if anyone has any stats on this.
While you are completely correct about the bounty price, sometimes there are people who work deeply in the field and can solve those things relatively fast because they have already done similar things in the past.
Especially if you’re talking about a business who takes on these types of bounties routinely. I imagine you’d be able to build up a body of historical knowledge about fixing common issues. You could see how that could be a viable business model.
Eh, I think you're underestimating some people perseverance.
You generally only need multiple people for timely action, and it usually even slows you down (from the perspective of total hours spent)
Like 2k bug bounty? I guarantee you some people would be willing to spend a lot of time for that. But yeah, people which are gainfully employed and have a decent salary - likely not.
People will have fun spending their free time on such projects. But it’s virtually impossible to turn it into “an actual thriving business model” that people can make a living on.
This $1900 bug bounty is quite an outlier, you generally won’t find one per month. An additional challenge is that it’s hard to predict how much work something will take, or whether there are any showstoppers. Also, if you don’t live in the same country as the client, it will be more difficult to get legal assurance that you’ll receive your money (or for the client that they won’t lose their money).
For small stuff, the cost is just going to be too much for people to want to pay it. This bug had a $1900 bounty attached. Let's put the cost of one software engineer (salary plus overheads) at $200,000 a year, which I think is an underestimate. That's $3850 a week, so unless your bug can definitely be fixed (including getting any necessary hardware, investigation, fixing, code review overhead, etc) in two or three days it doesn't pay. And if it could obviously be done in two days then it's likely somebody would have already done that.
The above back of envelope maths ignores the overheads of interacting with the people who posted the bounties to get them to agree to pay up, and of the cost overruns on the class of bugs that look like two day fixes but take two weeks.
I assumed the commonly cited 2x markup, so that would be a $100k salary, which is less than various websites say is the average US software dev salary. You could probably find cheaper elsewhere in the world, but even if you cut the salary in half that's still "bug must be doable in a week", which isn't going to cover many of the bugs people will care about.
Paying for software developers is really weird. State governments for example struggle to pay for a FTE that makes $140k. But they can pay me over $200/hour for consulting services for multiple years. The technical FTE employees that they have generally aren't qualified to evaluate their consulting needs so you get multi-million dollar contracts with very little actual oversight. I was really impressed with the folks I was working with at this particular state government and looked into what it would look like if I joined them full time as a FTE technology leader. I would have to take almost a 50% pay cut. The top senior IT position that oversees all of the state resources makes 70% of what I do. It's crazy. Unless you're working in medicine or sports, government pay sucks.
I've seen similar but less extreme examples play out in the private sector. 16 year senior architect making less than freshly hired software dev that was just an intern within the same company. Software developer pay is largely based on what you're demanding. In a lot of companies, there is a wide range of pay for folks doing literally the same job. They will hire a dev at $180k because that dev wouldn't go lower and turn around and push back to get another dev at $120k for the same level of unproven experience.
200k is a fairly high salaried software eng in expensive markets, a bounty program like this would be open worldwide and many people would be willing to work for a fraction of that, quality control is another concern but take a look at prices on sites like upwork and bids for this type of work and realize 200k is nowhere near the lower baseline.
$200k in cost to the company is a lot different than $200k in salary. It probably relates to someone making like $140k, depending on the various tax rates.
$200k is on the extreme high-end of software engineers. For example in eastern europe $30k is normal. And that's not even the floor. You can go to india or africa to get even cheaper. The problem with this bug bounty though is that it requires pretty rare expertise. It's not a "throw any developer at it" type of thing.
Did you realize that you didn't include 'open source' in your statement? This is exactly what the desktop OS makers -Microsoft and Apple- do every single day. Their prices are mostly B2B and therefore hidden, but there is a steady income for each person involved in making the fix.
it's the management structure that's broken. Plenty of decent engineers around microsoft who could fix it, plenty of customer and enterprises willing to pay, but they are not allowed to work on it because of prioritization bullshit, allegedly they could get more money elsewhere
That's literally the issue, management by KPI frameworks
I think it has more to do with bundling reducing the need to compete to zero. Change that and the economics of competition would take over and the changes would get prioritized but nobody at Teams needs to sell a single license, so the priorities become the bs like internal status and visibility and not product success.
How many companies have Teams for basically free with their 365 license but still pay for Slack? The marginal value of Teams is nearly zero.
There is also a matter of selective effort by staff senior enough to make their own choices. Many SDE3 (or whatever MS equivalent is) wouldn’t want to be associated with a dumpster fire product like Teams.
The economics for something like MS teams is not what you'd expect.
It has to be good enough that other options are not worth the hassle to switch over to, for enterprise customers. The quality doesn't matter in the slightest, because making it 5-10% better would cost double or triple.
Where quality does matter for these customers, backward compatibility, Microsoft does pretty well.
If I try to play a video in a chat, and then press the three dots to go full screen or change quality or scroll through the video, it just jumps to the top of the chat. Messages do not sync between devices. Microphone or video sometimes do not work and requires a restart. Sending attachments often just results in a failed message. Note these happen on all the OS I have triend on high end devices and on extremely high end internet 10-gigabit.
There is also just feature jank like for instance you cannot have two instances open at the same time if you have two organisations that you work for, you have to switch constantly. This is a disaster for any consultants or contractors who are placed in-org on teams.
The calendar space for an all day event takes up a sliver of space on the calendar, meaning people will often instead schedule an event for 9-5 or worst case 12-12 hours so it's not missable on the calendar easily etc...
It asks me a quality survey after every call even though I have surveys off.
It shows unread messages for a chat that has the focus and the “unread” message is visible.
When using the keyboard shortcut to create a new message (Command-N) it drops the first character of the recipient unless I introduce a noticeable delay between shortcut and recipient.
I’m sure I have more, but these are just from memory.
I just click the X rather than leave and I don’t get the survey popup. I mute most of the group chats. But it still shows unread so I know to go back and look at them
Most recently I had it put meetings on a different day because something was broken with it's outlook integration w.r.t starting the week on a Sunday vs Monday.
"Lots" is a relative term, but the overwhelming majority of kernel developers are employed and usually do kernel work as part of their job (usually at least ~80% but it could be argued as high as 97% depending on how you interpret the breakdown done by LWN of each release[1]).
And I would guess that most of the kernel devs who are "working for free" are doing the stuff they personally enjoy and find satisfaction in working on, because it's a hobby -- so many of them are probably not interested in fixing random bugs for cash either.
I wish there was regulation that you have to sell and maintain a working product, so that open source devs don't have to waste their time fixing proprietary products.
It looks like these laptops are usually sold with Windows; are you saying that every manufacturer should be obligated to develop drivers for every software which is theoretically compatible with it? Or are you just saying that we need even more caveats in the interminable EULAs we all just click through?
Maybe the obligation should be to provide adequate information about the hardware, so anyone could make a driver for their own software if they so desire.
Since you are talking about proprietary software, I assume you mean fixing bugs by the corpo devs themselves.
Well, this would imply broken software. You already payed for the software, now you are required to pay to get bugs fixed? Bad optics, although not beyond contemporary sentiments... Inherently shady incentives: https://en.wikipedia.org/wiki/Perverse_incentive
This kinda only works best for FOSS, incentivizing external devs IMO.
Sort of unrelated, but I've been thinking a lot of founding a non-profit that fund raises just to undercut the usual shitty consultancy companies that build government websites and apps just to build them properly.
Yeah, you'd want some sort of micro-kickstarting website where users can pool money that goes into paying for some fix or feature if the committed money crosses a threshold.
People spam the most minimal viable patch to collect the bounty and move on. And these days they are sending an AI slop solution. It doesn’t promote good code like actually hiring someone.
Out of all bugs and feature requests, this one is an outlier in that it requires specific hardware to work on and has an obvious success condition. This means that every man and his dog is not going to be throwing an LLM at this to see if their particular slop wins the prize. People get weird when money is on the line and managing a bounty is a job for which I would never volunteer.
I'd gladly pay a couple hundred to have Swift-like optionals in Godot's GDScript, among other things that are just a pain to convince all the random idiots on their official spaces of, but GitHub doesn't have a way to offer that :(
That thread is a fun (though frustrating for them!) conversation to read through.
After about a hundred back-and-forths getting the guy with the actual hardware to try different commands, I was thinking to myself man, maybe he should just give him remote access to work on the target PC, this is torture for both of them. And then I see him comment:
> Honestly I'm thinking of this and maybe something insane like organizing ssh access or something to quit torturing Nadim with building and rebooting all the time
And Nadim replies:
> Haha, sorry, but there's no way I'm giving you SSH access!
> I’m fine with continuing with tests!
Which is fair enough! But was funny to see right when I was thinking the same thing. Great perseverance from both of them.
Was slightly disappointing they they moved off GitHub to Discord eventually so after all that, we miss the moment of them actually getting it working!
This has been an old problem with Legion laptops. All this will be available free of cost to everyone! Mad respect for people who are pledging their own money and the person who fixed it.
Also, Lenovo Legion Pro 7* are not cheap (not that this would have been justified for cheap laptops).
Shame on Lenovo/<big company> who should have fixed this years ago.
Intel HDA was supposed to be a better standard than the AC'97 it was meant to replace. IMHO the blame lies solely on the codec makers for not working with the default settings (they can add additional functionality, but the base audio I/O should work with a generic HDA driver.)
Wow... one solution is of course to deactivate these. Which is what I did for my Legion 5 Gen 10. Speakers don't seem to be much in use these days anyway.
Still, I didn't expect this amount of custom configuration for my new laptop. Most importantly Bluetooth sound and getting Nvidia driver support. For Bluetooth I ended up writing my own tiny daemon. While driver support exists there seems to be a race condition somewhere between Pipewire, systemd and the bluetooth drivers. And for Nvidia drivers I ended up using the CUDA driver repository which is curiously only available for Debian 12.
There is a common problem with Realtek ALC3306 on Linux (Kernel Bug 213159). This affects many Lenovo laptop models. For example, my fairly old Legion S7 15IMH5 laptop also does not work.
I'm not willing to pay $1000 for a fix (it's easier for me to buy a new laptop that will work with Linux), but $100 is probably okay. :)
The sound only does not work from the laptop speakers; wired headphones work perfectly. Sometimes you want your laptop speakers to produce sound. So an external sound card does not solve the problem.
It's funny, but for as long as I can remember Linux (20+ years), there have always been some problems with sound.
> Approximately 95% of the engineering work was done by Lyapsus. Lyapsus improved an incomplete kernel driver, wrote new kernel codecs and side-codecs, and contributed much more. I want to emphasize his incredible kindness and dedication to solving this issue. He is the primary force behind this fix, and without him, it would never have been possible.
> I (Nadim Kobeissi) conducted the initial investigation that identified the missing components needed for audio to work on the 16IAX10H on Linux. Building on what I learned from Lyapsus's work, I helped debug and clean up his kernel code, tested it, and made minor improvements. I also contributed the solution to the volume control issue documented in Step 8, and wrote this guide.
If there's some Lenovo EU rep with enough budget for their department, this person should really be contracted to fix the audio from their laptops one by one. This seemed to take 2 weeks with a dedicated person and remote access but debugging with access to the device could be much faster!
Those two weeks probably would've been two days had the main developer been able to reboot and test the device directly.
Outsourcing this work to outside developers on the regular probably would make the media claim something like "Lenovo is too lazy to pay their people to make their speakers work so they make strangers on the internet do it instead" which isn't half false.
It'd probably be better and cheaper if they'd just hire someone to make their speakers work on Linux.
I heard from a frontier coder at deepmind that Gemini, whilst not great at novel frontier coding, is actually pretty good at debugging. Wonder if someone is going to automate a bug bounty pipeline with AI tools.
If this ai coding things are so good why nobody is capitalizing on it? Why are they selling this studf to developers instead of taking contracts and making billions?
IF $1900 is the bounty, it means it doesn't hurt enough.
I remember going for the highest paying bounty in the Ethereum VM several years ago (I think it was ~$400 DAI/SAI). I did it because I wanted to force myself to learn the internals and to see for myself if the bounty system works. I think I spent a few weeks debugging and ended up splitting the bounty.
As long as the user-facing issues are disconnected from the technical issues, it's going to be hard to get the true value.
Many of these firmware blobs are non redistributable, if you require this blob, you should grab it and back it up. I know a lot of firwmare can't be legally distributable in linux-firmware.
Lenovo may not be as friendly as IBM to its opensource.
I have a couple old-ish Samsung Galaxy Book x86 tablets that have a similar issue, that I have never quite goaded myself into trying to reverse engineer. I'd love some better material on trying to reverse engineer windows drivers: presumably maybe running windows in qemu with some kind of intercepting pass through?
Actually, the mercenary garbage that Lenovo started doing was absolutely hateful. I'd been tearing my hair out forever trying to figure out why nothing I could do would make my bluetooth work right on an old T430, even when I upgraded the chip. Assumed it was a Linux bug.
Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.) I have no idea why you would do that except out of spite; I don't remember them selling bluetooth upgrades or anything. They were just keeping their options open? This is aside from having to hack the bios in order to upgrade wireless or use generic batteries.
I would be awesome if the people that sold me products weren't awful people. They don't have to feel bad about it, but they should.
I don't care that you're angrier at Apple than at Lenovo. I'm angrier at the electric company, but I don't bring it up to defend my local alderman. I also don't care that FOSS hasn't solved all your problems for free like they apparently promised you they would.
> Turns out that Lenovo put awful bluetooth in the laptop, and made it ignore any other bluetooth chip you installed (you can get around this in Linux by force ignoring what the system reports.)
I used to Hackintosh Lenovos -- I thought this was at the bios level, so even if you did DSDT patching (linux or mac wise) it wouldn't work?
I use a T530, which is the larger brother of the T430. My overall impression is that it's a fine machine. (A lot of that opinion is motivated by the fact that when I bought it, it represented the very last of the plain black business-ey 15" PC laptops that wasn't cursed by the inclusion of a numeric keypad. Nowadays, Framework might fill that niche.)
Anyway: IIRC, a lot of the reason for locking down wireless hardware support is the FCC. AFAIK, the machine is tested and certified as a whole: The entire combination of its chassis and antennas and shielding and transmitting radios forms the item-under-test.
It's not just the sum of its parts; the whole thing gets tested. Deviation by end-users can result in having a non-compliant device, and the BIOS seeks to restrict that deviation.
And no, I don't like that aspect either.
But it only cares about the radios[1]. The rest of the guts are pretty freely exchanged: Upgrade to a different optical drive? SSD? Hard drive? RAM? CPU? Hack in a different keyboard? It's not picky at all about those things; have at it. (They could have locked down the entire system so that only parts matching a secret sauce would work, but they didn't.)
Anyway, Bluetooth module is garbage even under Windows. It's a bad design.
I bought my T530 from some random seller on eBay. It was obviously rebuilt by an outfit where they have a pile of variously-destroyed computers and take the cleanest chassis and put it with the fanciest screen and the nicest motherboard and sell it with some manner of RAM and an SSD for as much as they can get, even if that particular combination of stuff was never sold by Lenovo.
And that's fine, except: The Bluetooth module didn't work. I became convinced that it didn't even have one, even though it was advertised as including Bluetooth. So I bought a Bluetooth module (after validating the correct Lenovo FRU from the service manual, of course) and tore the thing apart to install it.
And once I was in there, I discovered that there was already a Bluetooth module present. It just wasn't installed properly.
IIRC there's one screw, one alignment pin, and one board-to-board connector for that part. The screw was tight, the pin was lined up fine, but the connector wasn't quite seated. It looked OK at a glance, but the whole module, in-situ, was very slightly bent by doing all 3 of those things concurrently.
That was annoying because only a bad design would have allowed for this to happen in the first place.
But I put my "new" non-bent module in and... it worked. (I haven't used it in Linux yet, though. Day job requires Windows software that talks to external hardware and I don't like dual-boot systems, so I'm kind of stuck.)
In terms of upgrading that part: It looks like the next wireless card is almost certain to also include Bluetooth by default, so after I hack the BIOS (I have a flashy-tool to poke at it with and an entire spare motherboard) to get around the trickery and plug a different wifi card in, I'll also have a newer Bluetooth radio.
[1]: I deliberately didn't mention batteries. My T530 still has the official Lenovo-minted BIOS; I've even updated it myself at one point using a Lenovo download. But it came to me with an aftermarket battery that worked great for a couple of years, and it now has a different aftermarket battery that also works great. I've heard the storied ruminations about aftermarket battery woes but simply have not experienced them myself. Indeed, the positive reviews on the Amazon listing I bought from suggests that it wasn't an issue for any of those folks, either -- it was like a sea of people who were just blissfully unaware of the issue.
I'm guessing it's the fact that linux has in-tree drivers, so you necessarily need to "patch the kernel" in order to write/fix a driver for a non-standard compliant device?
I can’t see any blocker to publishing this as a prebuilt kernel module honestly.
For driver developers the above where you rebuild the kernel is a necessary step in developing the driver but now the above is done someone should make the trivial next step to make this into a prebuilt kernel module which are trivial to install for end users with no rebuild/reboot required. (I have built kernel modules before but I don’t have this laptop myself, sorry!).
That's how it works in Linux land for two reasons. One, drivers live in the kernel (roughly). Two, Linux is aftermarket for many hardware, in which cases there's hardware first, then the support.
I agree. I can’t see any reason this couldn’t be packaged as a prebuilt kernel
module so end users can trivially install it. The instructions and code here can be used to build the kernel module.
I don’t have this laptop but have built kernel modules in the past to give context. It’s a tiny step to publish this as a kernel module so end users can trivially install this (this reduces the instructions to downloading one file, running one command, with no reboot or rebuild needed) so it’s quite reasonable to call this out and ask someone to do it.
It’s a bit like publishing a windows driver as raw source code. Great work but there’s no reason not to ship the prebuilt driver right?
> I can’t see any reason this couldn’t be packaged as a prebuilt kernel module so end users can trivially install it.
I don't think you have to be the original developer to create packages one can distribute. Go for it!
From TFA:
> This guide is currently for Linux kernel version 6.17.8. It will be updated for future kernel versions as they are released, until the fix is fully integrated into the kernel.
So it sounds like the plan is to get it into the mainline kernel, at which point it will get to all the distributions. So I sent see the problem here.
Some device classes can be supported in userspace because no matter how an adversarial driver might get the device to misbehave, it cannot possibly break the kernel's security model. This might even apply to some audio devices, depending on how exactly they're hooked up to the rest of your system. But the more typical devices, especially those in your average SoC and those connected to a PCIe bus or the like, have full privileges within the system and will need kernel-level support for the foreseeable future.
Kernel modules absolutely run in kernel space though.
I’ve literally written kernel modules for high speed networking devices that have full access to the memory bus and enumerate pci devices. There’s no userspace or kernel space question here. It’s merely a matter of someone turning this into an easily installable kernel module
Kernel modules are not going to be "easily installable" anyway because their whole purpose is to poke at kernel-internal structures that will change all the time as the kernel evolves. With source code, you'll hopefully get notified if there is breakage - the module fails to build and you need to forward-port it to the current kernel.
They have great stability between kernels by design. Better than Windows dll based drivers IMHO.
As someone who actually writes drivers I'm a little frustrated at this whole thread with people claiming Linux drivers have to be distributed this way.
Kernel modules exist for a reason, literally to allow end users as easy and as forwards compatible of a way to install drivers as windows dll based drivers. This whole thread has a lot of know nothings chiming in if I'm blunt.
reply