Excellent write-up. I love these kinds of "I had this problem, these things on hand and an idea for how to solve it" followed by the story of every barrier and how it was overcome. It's becoming more and more rare with people preferring to go the YouTube route. I'm not knocking video-versions of these things, they have a lot of value (among them being "less friction to publish easily" for some folks[0]) but convenience of text is vastly superior for consumption as far as I'm concerned.
It takes a lot of time and energy to put these sorts of things together but they have value beyond the original topic (e.g. perhaps one of those barriers you encountered designing this is a barrier to other problems[1]).
[0] Sometimes that's perception, sometimes it's "I don't type very quickly/use a phone exclusively for my computing needs"
[1] Unexpectedly, your site produced a TLS error in Firefox on one of my laptops, so it's causing me to investigate a problem I didn't know I had :)
Thanks for being so kind. It indeed takes a lot of time and energy, but it is incredibly rewarding when you finally share it with the world and people find value in it (far more than I expected)
There are numerous times when I encountered a hurdle trying to fix something or set something up and was lucky enough to find a blog of someone else who encountered the very same hurdle and managed to solve it.
Without the efforts of these people I would have continued to waste time trying to figure something that was important to me at the time, but out of my knowledge sphere.
In a way you are giving people life when you share your knowledge.
On a more serious note, this is one of the main points that drove me to start such a blog. I've been the one learning from other people's posts for a long time. Now that I've reached a point where I can give back to the community, I'm really proud of the positive feedback I've got.
Moreover, while writing this took some time, I actually enjoyed going back over what I did and putting it in a way that I can share with the world. I can't stop thinking of ideas for future articles
Thanks for putting in the time to write this up. As someone who is writing a lot of procedural instructions for colleagues I can appreciate the time and effort it takes to write something down using full sentences, structure and flow.
Not for me. Videos are the second-worst way for me to learn things (just audio is the worst). At 2x speed, I absorb nothing. Reading is the best way for me.
Use AI to dynamically time warp the video relative to the information density in the video. Fast through the boilerplate, easy to understand parts, slow through the hard parts.
> I love these kinds of "I had this problem, these things on hand and an idea for how to solve it" followed by the story of every barrier and how it was overcome.
Agreed. Building within constraints (even strict, self-imposed constraints) is more fun and interesting.
This is nice, but has too many moving parts and ffmpeg-like approaches are _not_ the way to go for low-end desktop streaming--I tried doing that in unicast and multicast scenarios quite a few times, and have a few notes on the results:
Even that far back, it worked _extremely_ well on a single-core Pi, and I suspect it would fly on a Pi 3. I used one recently to build a GPU-accelerated thin client:
I think DirectFB already supports RemoteFX/GFX AVC:444 and all the other niceties that you get with xorgxrdp-glamor on the server side, and am willing to bet the whole thing would run a lot _faster_ and more efficiently than this solution all around, power and bandwidth-wise.
(this is being written in one such RDP client against a Fedora server, and I have YouTube open with audio streaming to the client... Which also works OK via Wi-Fi)
A long time ago when I was looking for a low latency solution for streaming _from_ the Pi (should also have a similar performance in the other direction), gstreamer[1] was the only usable option.
Hey there ! Thank you for sharing my blog post. I tried to share it myself earlier, but my post was hidden (probably because my account is new).
This is my first technical blog post. I intended for it to be easy to follow along, and tried my best to write in a pleasant style. However, as a non-native speaker, I know that my writing sometimes feels awkward.
Please tell me everything that is wrong with this post, whether it's from a technical or a linguistic standpoint. Don't worry about sounding harsh, this will only help me improve :)
You state multiple times that the Raspberry Pi 3 does not support hardware accelerated h264 decoding, but that's simply not true.
All Pis (except Pico series) support both accelerated encoding and decoding, and have done so since 2012. The Broadcom VideoCore IV in RPi 3 Model B does hw decode at 1080p30.
Yeah you're right. Someone mentioned it on reddit and the issue was indeed not the lack of hardware decoding, but the fact that using it disabled my optimizations regarding latency. You can read more in this thread :https://reddit.com/r/programming/comments/11r9osx/using_a_ra...
Wanted to write a top level comment to suggest Sunshine/Moonlight too. I've been using it to stream 3.5k@60fps from my midrange gaming PC (Ryzen 5, 3060 Ti) to my Macbook Pro.
I set up a virtual display with my Macbook's native resolution (minus some height because I don't want to overlap with the notch), and it's just perfect, with no perceptible input lag. Sometimes I need to restart Sunshine, but I could just move the PC to the basement and operate it completely remotely (Moonlight supports WoL).
Thanks for that link! I've been meaning to write the EDID to the dummy dongle I have to enable resolutions/refresh using CRU. But I think I'll dive into this first.
I too have a workstation/gaming pc presently in the basement, for about a year it lived ~15km away. I used Parsec at first but migrated to Sunshine when I started using Linux clients due to the more recent Parsec clients dropping GPU acceleration.
I just got a refund on my Lenovo X1 Nano laptops because they were non functioning after years of service troubleshooting. (Nightmare experience, ThinkPads are not what they used to be!) Took my money and got a MacBook Pro, Streaming to this screen is amazing. Looking forward to getting 120Hz working.
So an important piece of info might be that Sunshine automatically switches to the virtual display if you disconnect any physical monitor. I didn’t spend much time on it but couldn’t manage to get it to use the virtual display with the physical one still attached.
(I wouldn't try this on a Linux Host) But I believe the shortcut for the Windows Moonlight client was Ctrl+Alt+Shift+F1/F12 to switch between the displays.
Instead of this, I happen to setup the Host with Win+P to use the second display only and it works great.
If you want to limit Sunshine to stream the virtual/second display, using the control panel:
Sunshine Configuration > Audio/Video > Output Name
The virtual adapter I just setup with this driver mentioned above, was assigned: '\\.\DISPLAY25' but go ahead and run that 'tools/dxgi-info.exe' to find out.
Also, option.txt using:
3024, 1890, 120
Was about the easiest thing ever.
Great ideas!
About the monitor name - that’s the thing, the virtual monitor didn’t seem to have a name when using the tool you mentioned. But maybe that’s because it hadn’t been active at that point. I‘ll try DISPLAY25 :)
HDR: I didn’t look into that, unfortunately. That might then be a case for a programmable HDMI dummy, right?
You totally hit the mark on being easy to follow, and pleasant to read, I love blog posts like this which don't just dump a solution and then leave it at that but also walk through the process of getting to that point.
Can I suggest adding an RSS feed to your blog? I'd love to follow it, but realistically unless I can add it to a feed reader I'm going to forget it exists!
Congrats on being on the front page of HN ! Are you the Pierre Couy I met in Centrale Lyon ? We did a project together using D3.js for a pharmaceutical company. Valentin
I like this post. As someone who is deeply afraid of their Raspberry Pi, this could be the silver bullet.
An RPI project to me is 20% surface level solution and 80% troubleshooting. Your standard RPI forum thread to do one thing quickly spans to a mile long, because "I tried what you recommended but what to do now that I have a new problem". You can really feel that experiencd in this thread. Most RPI experience in my opinion is similiar to mine. You can not solve "a" problem with "a" solution. You always need a series of solutions with progressive tweaking. This post captures that essence, solving a problem and coming to a goal with an RPI that is written like a travel story.
> As someone who is deeply afraid of their Raspberry Pi, this could be the silver bullet.
To me this is much the same as writing "As someone who is deeply afraid of their computer, this could be the silver bullet." A Pi is just a credit card sized ARM computer - there is no magic, it can run many of the same OSes as any other desktop or laptop, it can run the exact same apps and software in many cases. Thinking of it as somehow different is already a bad way to begin.
Well, not just a "ARM computer", a Arm 32-bit computer.
If you're stuck with the "Raspberry Pi 1", "Raspberry Pi 2" and "Raspberry Pi Zero", you are really stuck with a 32 bit computer.
Even assuming you have a arm64 enabled Raspberry Pi, you need to use the 64-bit version of Raspberry Pi OS, or you're stuck in a 32-bit userland.
Even assuming you have Raspberry Pi OS (64-bit), you now are using a experimental version of Raspberry Pi OS, so expect issues.
> there is no magic, it can run many of the same OSes as any other desktop or laptop
There is actually a lot of magic involved, specifically, GPU blobs required for booting, and a patched (and old) kernel. Just try loading any other OS that doesn't specifically bundle those patches and a newer kernel and the rough spots get you pretty fast.
> it can run the exact same apps and software in many cases
The same apps and software, possibly.
Recent versions of said apps and software - unlikely.
It defaults to safe and old Debian versions of apps and software.
> It defaults to safe and old Debian versions of apps and software.
It doesn't default to anything - you can load whatever OS you like that will run, including latest major Linux distros, and plenty to choose from. You can obtain your software from anywhere you want, not just the distro's default package manager. Exactly like a plain ole computer.
> Recent versions of said apps and software - unlikely.
I have the latest Ubuntu LTS with Firefox, VSCode, Chromium etc all running on a Pi right now, the latest versions much the same as any other computer I own. It runs the latest versions of virtually all programming runtimes. It can have the latest version of Docker... Getting recent releases of Linux software appropriate for a credit sized hobbyist arm computer is rarely a problem at all on the Pi 4, which has been out now since ~2019. Its still not all that much harder on a Pi 2, 3 or zero.
> I have the latest Ubuntu LTS (...) all on the Pi 4, which has been out now since ~2019.
Well, "out since 2019", but since 2021 new Raspberry Pis are basically unobtainable. And most "Pi 4"s are new.
> Its still not all that much harder on a Pi 2, 3 or zero.
Said Ubuntu Desktop 22.04.2 LTS for Raspberry Pi only "supports" the "Raspberry Pi 400" and "Raspberry Pi 4/Raspberry Pi CM4" +2GB ram configurations. Even Canonical expects pain if you don't use those.
"Raspberry Pi 3" might be useable if you have tamed expectations.
"Raspberry Pi 2" has issues because of it's 32-bit distribution. Using anything "modern" on it probably means compiling stuff (very slowly if on-device) and fixing issues yourself.
"Raspberry Pi Zero" isn't even listed as supported anywhere because attempting to running anything non-purpose built for it is painful.
Well, at least we are coming around to the fact its just a computer! And for what its worth, the Pi 4, 3, 2 and zero have crazy levels of support out there, your description does not match the reality I've lived. You might be surprised just how much runs on a Pi Zero.
I think it depends on the use case. My experience with my RPi used as a 3D printing tool has been plug and play with OctoPi and haven’t needed to tweak it much at all
I stopped using raspberry pis for this reason. A small form factor refurb PC suits what I am doing way better and I don't have to try to find ARM binaries for everything.
I've used mine as mostly always-on servers running stock raspbian with services that I need. They have been very reliable and power+space efficient compared to the low-end surplus pcs I previously used for these purposes. I say all this to encourage you to dip your toes into it with something super basic - once you see that it just works, maybe the projects that follow will seem less daunting.
Anyone who uses the term "deeply afraid" to describe their feelings towards what amounts to just a smaller version of their laptop... I just don't know what to say really.
What exactly is the silver bullet here unless you were planning on doing this exact project? Seeing someone else solve a problem with the Pi? There must be thousands of instances of people solving problems with a Pi on the internet before this post. And it sounds like you're familiar with them, since you cite statistics on the kinds of problems people run into when using the Pi.
> An RPI project to me is 20% surface level solution and 80% troubleshooting.
Don't be afraid, I'll give you a few examples to cheer you up and motivate you... (just make sure to backup your working Pi setup: I just dd the entire microSD card once it's working).
I've got one of my Pi who literally reached more than one year uptime. It's gently sitting, since years, in another country, constantly and relentlessly establishing reverse-SSH tunnels. Zero issue.
I ran a RasPBX on a Raspberry Pi 1 (yup, 1, a decade ago or so) for years, hooked to a SIP trunk, to provide VoIP to four Cisco VoIP phones. Zero issue. Once it was setup, I did "dd" its microSD card and kept the backup safe and that was it. I never needed them. My wife's SME's employees used that daily, for years on. Zero problem.
Regarding TFA:
> Since I was not willing to spend more money on my setup, I used a Raspberry Pi 3 which was sitting unused in one of my drawers.
That's the spirit!
Recently I decided to use one of my idle Raspberry Pi 3 as a "firewall+router" between two private LANs. It's surprisingly capable at that even if it is definitely the bottleneck: I've got fiber to the home and the Pi 3 cannot do anywhere near Gigabit ethernet, especially seen that it's only got one ethernet port and I needed two, so I had to use a USB-to-ethernet adapter... But the Pi 3 only does USB 2.0. A few firewalling rules and it's happily firewalling and routing traffic.
After reading a few comments here I also decided to configure another Raspberry Pi as a DNS, running unbound (while my main "workstation" runs dnsmasq as a cache, but the DNS is unbound on the Pi).
Someone also mentioned using a Pi Zero to transform a non-networked printer into a networked one: I'm planning to do just that for my trusty B&W laser Brother printer (haven't done it yet).
I've got a Raspberry Pi with a Pi2JAMMA adapter in my vintage arcade cab. Zero issue. (Once again: I've got a "dd" of the full microSD card should some issue show up). Every single time I turn it on, it just works.
One of my Pi is my "airline / airgapped / booting and remounting the system read-only", with X all working in RAM.
Sure, these Pis aren't advanced servers full of redundancy and ECC RAM but they cost next to nothing and they're trusty little beasts. And on that subject of these Pis not being that powerful: there's no way I'd use a Pi as my main desktop, I've got a beefy desktop and I wouldn't want to replace it with a Pi.
P.S: I'll probably be only buying Beaglebones from now on because the Pis now seems to be unobtainium.
> had not noticed that the laptop I chose was missing an essential feature : it did not have Display Port over USB C. Not being able to use my second external monitor on this new laptop felt like a huge downgrade from my previous one
Any DisplayLink hub will solve this problem. (That's what I did with the M1 Macbook Air) Probably not as fun as hacking on a Raspberry Pi, but a lot easier to find than a Raspberry Pi :-)
Note that DisplayLink is awful and should not be encouraged. It works on Linux with proprietary drivers, but badly and not natively (at least on Ubuntu) and will eat your CPU. It has issues on Macs that DRMed media in the browser won't display (cos the OS sees it as a screen recorder), and it will probably eat your CPU.
I don't recall having those issues (I've since moved on to an M1 Pro and then an M2 Max that support multi-monitors), but I'm not necessarily surprised.
How about using socket activated service on the pi to receive data over a port? That's could automatically handle turning on/off the display whenever the service starts. It's a while ago and I don't use it any more but I did this, to send audio to my pi:
That would be one nice way make the whole thing cleaner on top of turning it into a deb package. For now, it's really hacky but has proved to fulfill my needs.
I may or may not take the time to polish this up in the future. In the meantime, I've licensed the repo under GPL so if someone feels like doing it, they're more than welcome to.
I thought about that in different angle - does it possible to use a second linux laptop as external monitor for a mac. I've tried few similar streaming solutions without investing much in tuning and creating proper wire connection and as result my initial impression was that is too much latency issues (as well with some additional issues related to creating a virtual screen), but probably with your approach I'll try again. Thanks.
Barrier will let you use your mac’s keyboard and mouse on the second linux laptop. Depending on your use case that might be good enough. I actually prefer this as I can avoid opening YouTube etc on my work laptop.
it actually very good setup for some cases and I've worked in similar mode for a while using by usb switch device with single button - but I found that is little bit too painful because in may cases to move something from one machine to another is required unexpected 'extra' step which in lot of cases breaking usual productivity flow in worst moment.
With barrier you just move your cursor between screens (or use a hotkey). It also shares the clipboard so it’s perfect for opening up a reference webpage on the second system.
I'm using a capture card to use my Macbook monitor as a screen for my PS4 when I travel. You could use one to turn your Linux laptop into an external monitor.
I’ve done this the opposite way round, using a cheap USB C capture dongle from Amazon to capture output from an RPi and then using my a Mac as a virtual screen for the RPi. I do this occasionally when I need to setup something but don’t want to drag out an external monitor.
The capture card/dongle are designed for using in Twitch game streaming where you are using it to capture the output from a game and feed it into something like OBS. The one I have was pretty cheap, but only supports an output resolution of 1080p (input up to 4K). I don’t remember the exact brand, but the above should give you enough keywords to get started.
I know many Windows laptops support Miracast (both as source and as sink) for wireless display transfer.
I think Apple has developed its own Airplay protocol, though. There are Airplay sinks for Linux, so if you can find a way to share a screen over Airplay the setup shouldn't be too bad, especially wired.
Did you ever try a smaller GOP after you patched it with the 300 frame interval?
-g 1 would give a keyframe every frame so I'm wondering if the -g 100 holdover might be counterproductive once you'd done your other tweaks. Right now you have an I frame followed by 99 P-frames which could lead to a flake out for 3s if you ever lost an I-frame. The P-frames are more efficient than I-frames, but at 50mbit you don't need to set your compression too aggressively.
I think I'd try (and I know your project is done) some flags along the lines of
On one hand, this is a GREAT example of screws looking like nails when all you have is a hammer. On the other hand, this was a VERY impressive way to screw in the screws using a hammer!
I don't think that would allow me to seamlessly move workspaces and windows between monitors. From the top of my head, your suggestion would imply the Pi running it's own desktop environments and having its own input devices
I thought the point of an Xserver is that it has no input devices, it simply receives instructions about cursors and rectangles and text, no? Perhaps I do not understand X.
What i meant when mentioning input devices is that the Pi would have it's own cursor, and I would have needed to handle the forwarding of inputs from the laptop's X server to the Pi's. This would have been a lot more difficult as I would have had to forward the inputs to the Pi only when moving the cursor out to the Pi-controlled monitor.
Keep in mind that I'm writing this from the top of my head, without googling anything or experimenting with the setup we're talking about. It's totally possible that I'm wrong.
Anyway, the hard part in this project was optimizing everything. The first proof of concept stream was pretty straightforward to get. Getting a slow PoC of what you described should be kind of easy as well. I encourage you to experiment yourself with the setup you suggested, you'll learn a lot more than you could possibly learn by reading a writeup. If you ever try it out, I'd love to hear about how it went.
I ran it in console directly rendering to framebuffer and it has hardware-accelerated video decoding. On Raspberry pi 3, hooked to lan and my macbook on wifi the lag is ok. It's about the same as ssh over slow cellular. I mainly run my terminals watching the logs/builds etc. on that display.
Just to let people know how to get the latency down with uxplay:
Upgrade the debian to bookworm (uxplay is in the repos for that one)
apt install uxplay
in in /boot/config.txt
change doverlay to:
dtoverlay=vc4-kms-v3d
force hdmi: (i had some problems with kms)
hdmi_force_hotplug=1
add gpu mem:
gpu_mem=256
then run: (-a disables audio)
uxplay -vs kmssink -a
and now you have and airplay monitor that can mirror or extend your desktop.
The hard part for me was getting the latency low enough that I can comfortably type and move the mouse. But I may look into is as a 4th screen for logs would be interesting
Very interesting!
The choice with the subnetting is a bit odd to me, it's surprising that you got the hosts talking to each other with that addressing.
I would have chosen a proper /30 between the two computers, that would be 10.0.0.1/30 and 10.0.0.2/30.
Yeah, someone else mentioned this on reddit. When taking the time to think about it, I as surprised as you are that it worked. I'll update the post when I have more time to take care of all the things people suggested
I don't clearly remember, but I think ffplay detects there is no X server and automatically switches to using Direct Rendering Manager. For mpv, I added a specific option to use DRM output
Apple Firewire to Thunderbolt adapter plugged into Apple Thunderbolt 2 to Thunderbolt 3 adapter cable is more or less the only way to do it. You have to have Thunderbolt PCIe on the USB-C port to support the hardware requirements of firewire.
Such a solution has nothing at all to do with virtual monitors or display protocols though. Firewire has a standard for transporting digital video, but it's not useful for much of anything except speaking with niche and legacy AV equipment.
I'd love to get your feedback (in Github discussions or issues, depending on your results), especially regarding the installation scripts. I somewhat tested them but did not take the time to start fresh and check if they actually take care of everything
Although it looks like they haven't really refreshed that product line since 2016. Seems like they're trying to pivot to wireless now that Alt Mode USB ports are mainstream.
For me, the main issue was the proprietary software I needed to run un the host computer. I also read somewhere it doesn't play well with Linux.
To be honest, I didn't even consider it before rolling my own solution because it was not immediately clear to me that it was something different from display port over USB
Why would you need to do this? USB to HDMI adapters exist and now that USB-C is mainstream there are adapters for that too and significantly cheaper than a raspberry pi. One use I would like to see is a pi convert an HDMI signal to ethernet and convert it on the other side with a second pi.
Lol basically any DisplayLink adapter would work. Way cheaper.
Since you mention Thinkpad, here is a personal pet peeve.
A company called StarTech makes ExpressCard devices and USB devices. They make:
* An ExpressCard HDMI capture device (no longer manufactured) that sits flush with the body of the Thinkpad. [0]
* ExpressCard USB3 peripherals that adds one or two USB ports to the Thinkpad and sits flush with the body of the Thinkpad. [1]
* USB3 to HDMI output devices that leave an unsightly dongle protruding from the Thinkpad. [2]
So, what StarTech does not make (and my pet peeve):
An ExpressCard device that adds an HDMI output port to a laptop and sits flush with the laptop's body.
Note that such a product would seem to be a trivial combination of their existing products. I would also expect that such a device would be cheaper to manufacture than the equivalent capture device.
I bought one of those USB-C to HDMI adapters just to find out that the USB-C ports of my desktop computer, my laptop and my phone do not support HDMI over USB-C.
Some devices only have DisplayPort alternate mode output, so may need a cable (or adapter) that does the DP->HDMI conversion. Assuming your devices support DP Alt. mode. These are active adapters.
It takes a lot of time and energy to put these sorts of things together but they have value beyond the original topic (e.g. perhaps one of those barriers you encountered designing this is a barrier to other problems[1]).
[0] Sometimes that's perception, sometimes it's "I don't type very quickly/use a phone exclusively for my computing needs"
[1] Unexpectedly, your site produced a TLS error in Firefox on one of my laptops, so it's causing me to investigate a problem I didn't know I had :)