We rarely, if ever, make responses to blog posts or articles. In fact, there is only one instance that I can think of when we did so. So then, somewhat uncharacteristically, this is a response post to Martijn Braam’s blog in which he explains why he left the PINE64 community. Let’s get one thing out of the way first: Martijn has done a lot for mobile Linux and PINE64 – he is a valued contributor and a colleague with a good insight into how PINE64 and the Pine Store Ltd operate. I should add that his opinions are welcome just as they have always been. Finally, there is no denying that his leaving is a significant loss to the project and, on a private level, a sad state of affairs for us in the community. If it wasn’t clear, we really like Martijn. But this isn’t what this blog post is about. Instead, this is a response to the points and concerns Martijn raises.
A short summary first: Martijn’s blog entry alleges that following PinePhone community editions, and after settling on Manjaro with KDE’s plasma mobile as the default OS, PINE64 and the Pine Store Ltd have sidelined developers from other mobile Linux projects. The argument is made that this has hurt development. The example given in the blog post supposes that the community team and Pine Store employees were firmly intent on removing SPI on the PinePhone Pro and coerced not to ship Tow-Boot. He concludes by saying that we no longer listen to the development community.
But here is the thing, SPI has been included on the PinePhone Pro due to the input from developers and against our initial intent. The talks concerning SPI were tense, as Martijn mentions in the article, in part because we did approach them with a presumption that we’ll inform developers of our plan to drop SPI. I may add that the decision was made among ourselves, without the input of any third parties or partners. The reasoning behind not including SPI on the PCB wasn’t motivated by any single project or outside source – it was based on the fact that for years SPI was largely unused on PINE64 devices. In instances where it was used, it sometimes caused issues, which in turn led to a few support nightmares. I should also mention that SPI has skyrocketed in price at the time and became harder to source; all this, collectively, had us initially pretty firmly determined to drop SPI altogether. And yet, despite this, we agreed to include it on the PinePhone Pro because developers from multiple projects – postmarketOS being one of them – were adamant that it was an absolute necessity.
We were also convinced to flash the SPI with Tow-Boot on the most recent batch of PinePhone Pro, which was the preferred bootloader of the majority of members in the private discussion group. Moreover, I should add that SPI is present on the Pinebook Pro and there is no need to solder one on. Pinebook Pro’s SPI doesn’t come with a pre-loaded bootloader, but the option to flash it is there for anyone who wishes to do so. As for the reason why Pinebook Pro doesn’t ship with a bootloader on SPI – it isn’t caused by favoritism for a particular group of developers or disregard for good ideas, but rather something more trivial. Namerly, a functional Tow-Boot build for the Pinebook Pro wasn’t available at the time of manufacture and shipping.
We created a space for development talks to be held (as we always had), there was a lengthy exchange, there was a difference of opinion, we listened to all developers, and the outcome of the discussion changed our mind on the subject of SPI inclusion and Tow-Boot. And to be clear, we listened to different people. Differences of opinion and unwillingness to listen are two different things and cannot be equated.
All this shouldn’t exactly be a surprise to anyone who has ever interacted with us since, on a weekly basis, we talk to people affiliated with different projects (and to those not affiliated with any projects). Over the past month alone we talked with developers from a handful of major Linux projects regarding support for existing and future devices. We are also in constant contact with community contributors working on non-Linux devices – PineTime, PineBuds and the Pinecil V2 being examples of the latter. And, of course, we talk to our core partners too. This, however, has no impact on whether a good idea from a third party gets implemented or not. We always have been open to suggestions and we will keep on listening to input from the community.
Now, surely there is more that we could have done in the past in the way of supporting development, and undoubtedly there is more we should do in the future. We haven’t delivered on a variety of ideas concerning supporting our development community – the DevZone being a prime example of a crucial system that still needs to be fully implemented (and it will). The DevZone will, among others, be a place where bounties will be offered to all contributors regardless of affiliation. Mea culpa, an argument can be made that in this regard we haven’t done enough in the past year. At the same time, I believe that our track-record of working with the community is such that listing examples of successful ongoing cooperations and of continuous support is unnecessary.
To sum up this response post; Martijn has his reasons for leaving and we should all respect those. We are also thankful for all the work he, and the entire postmarketOS has, poured into the Linux smartphone space and the PinePhone in particular. That said, we don’t want to leave the claim that we do not listen to our development community unanswered – because it is unfair and demonstrably false. We are not mindless marionettes unable to make up our own minds about what course to pursue either. As a project, PINE64 is not afraid of taking responsibility or admitting to a fault, and surely there are many mistakes that the management of the project has made in the past, but blind favoritism or unwillingness to listen to our community are not on the list.
30 responses to “A response to Martijn’s blog”
Well put.
Cheers
Question from a non technical person interested in the project. In a hypothetical world where this SPI thingy was dropped, how would this affect/change the process of changing or replacing the OS?
In case of PinePhone Pro and Pinebook Pro (so all devices which are using RK3399 SoC), as was stated in Martijn’s blog, all OS developers will need to agree on specific structure on the eMMC, so when user will be changing the OS, the bootloader (u-boot or tow-boot), will not be replaced, or corrupted. So for example, if any OS will try to install their own bootloader (replacing the old one), and something bad will happen during installation, it might result on non-bootable device, and it will require to re-flash bootloader via SD Card (with hardware eMMC boot bypass).
so developer leaves and makes drama whats new?
The important news is in the blog: https://blog.brixit.nl/
“As a project, PINE64 is not afraid of taking responsibility or admitting to a fault”
Coming from a company that knowingly shipped a broken keyboard attachment for months without acknowledging it. Get out of here. After the keyboard shipped immediately people noted issues with it not working and people needing to shim it themselves. The next blog post simply said that the launch of the keyboard wasn’t as smooth as you’d have hoped. You didn’t say the keyboard won’t work for lots of people or that if you purchase a keyboard then you will have to be ready to physically modify just for it to work. You side stepped responsibility of a broken product then refused refunds for products that were shipped broken. That was the last straw for me. I wish the community the best of luck, but I have no faith in Pine64 as a manufacturer anymore and will not purchase any products of yours ever again.
Sadly I feel the same. Love the ideas, projects and community but when I got the Pine Phone on the first batch of Postmarket OS, I ordered the upgraded version and was sent the lower grade version. I raised the issue with Pine64 only to get a ‘will get back to you.’ Never heard anymore. I was willing to send it back for a replacement but I got nothing. I know it was inexpensive but it’s not the point. Not the level of customer service I was expecting.
Well, that’s easier to swallow than buying two Pine Phone Plus devices that came with faulty software and needed a workaround to even get them charged… I left the plastic on them and returned them but they kept 30% for a restocking fee of each phone (roughly $100 with shipping per phone.) Clearly they knew the device was not in any state of even being close to useable and sadly nearly 6 months later nothing has changed. 🙁
I’m disappointed…
Their products are full of hidden, undocumented hardware flaws. PinePhone Beta didn’t have a working usb-c video output when it came out, because of a hardware issue. It took the community devs months to find the cause and fix it in software. The microphone killswitch on this phone doesn’t even kill the mic, it just makes the volume very very very quiet. I wasted money buying PinePower, because it turned out it was ungrounded and didn’t work with Pinecil at all. Those are just a few issues that I know of.
If you want to buy anything from them, you have to do a ton of research first, because you never know if it works at all. There are some reviewers who review their products, but they are usually biased and don’t really talk about the flaws. Fortunately I decided to wait before buying the keyboard and now I’m glad I didn’t buy that piece of trash. Their store page still doesn’t mention that there is something wrong with it, of course.
Then they opened their EU store with ridiculously high prices. They even have their bullshit excuses to try to explain the huge price increase. At the moment it’s probably cheaper to order from US instead. I can’t imagine buying anything from them again and I think if there existed unbiased and competent reviewers for those type of products, they wouldn’t get away with any of this. The lack of competition also helps them and I hope that will change soon.
The response completely ignores the concerns that the Manjaro project has too much say and is too much of Pine’s focus.
The slow progress on PPP software is a valid example showing something is amiss.
Making bootloader dependent on the operating system is a terrible idea. Surprised that Manjaro folks don’t agree with this TBH.
I commend Pine64 for addressing the DevZone situation (I have been eagerly waiting for a means of putting my money where my mouth is to help get developers paid and issues fixed. This is undoubtedly how we get Pine64 devices into a better state.)
That said, the article goes from mentioning Manjaro and the potential harm that partnership has caused directly into the SPI issue. I don’t think the concerns around Manjaro were adequately (or at all) addressed. I also think this: https://drewdevault.com/2022/08/18/PINE64-let-us-down.html is a valid criticism of the situation at hand. Would Marek or anybody else at Pine64 be willing to comment on these criticisms?
Why should they after having made their benefit?
I posted a comment but I’m not sure it went on-line. If this is a duplicate, please delete.
I find a number of things in this response concerning as someone who has spent a fair amount of money on PINE64 products (I even have a shipment in the mail right now).
> A short summary first: Martijn’s blog entry alleges that following PinePhone community editions, and after settling on Manjaro with KDE’s plasma mobile as the default OS, PINE64 and the Pine Store Ltd have sidelined developers from other mobile Linux projects.
This to me seems like a bit of an oversimplification, though it’s not wrong. While it is also troubling that PINE64 is supporting a single operating system for the PinePhone, which could harm softwware diversity, the issue is that Manjaro in particular is a bad choice because they’ve done little to actually support the PinePhone. Manjaro uses a telephony stack written by Mobian developers and a camera stack written to help postmarketOS. Manjaro developers meanwhile have shipped alpha versions of software causing users to flock to software support channels for builds they shouldn’t be running in the first place.
> But here is the thing, SPI has been included on the PinePhone Pro due to the input from developers and against our initial intent.
None of this paragraph actually disagrees with Martijn’s post except perhaps that “the decision was made among ourselves, without the input of any third parties or partners”. Martijn only said that PINE64 is /incentivized/ to listen to the Manjaro project in particular because Manjaro is the operating system that ships with PinePhones from the factory.
> Moreover, I should add that SPI is present on the Pinebook Pro and there is no need to solder one on.
Martijn’s blog: > PINE64 representatives went so far as to say, quote, “people who want [an SPI chip] can just solder one on”.
Martijn didn’t say it was necessary to solder an SPI chip onto the Pinebook, just that it’s unreasonable to suggest users do so when soldering is outside of the skillset of many developers (and most users).
I would have appreciated a shorter, more straightforward PINE64 response rather than a day-after fixup. A lot of time was spent focusing on Martijn’s one SPI example when his complaint was much more focused on the lack of /listening to developers/! A bounty program is nice and I’m glad PINE64 is working on that for the PinePhone. I’m also glad Martijn will be spending more time on postmarketOS, which is a great Linux software distribution I’m excited to see made even better. But I didn’t buy a PineNote because I wasn’t convinced PINE64 could drum up the developers to work on it (it looks like my worries were justified) and I’m hesitant to buy the PineBuds when they come out even though the product concept is very attractive because who’s to say developers will actually work on that?
It seems the developers of tow-boot didn’t find out about the incompatibilities with the new Pinebook Pro until it was reported to them BY A USER. Is this true? How come pine64 didn’t reach out to them, even if it was too late for it to be included in the production batch?
Well, I’ve read Martijns post and tend to agree. I think this kind of monoculture thinking will only seed more and more bad decisions that will be based on politics and not users or developers, just look at Ubuntu/canonicals list of bad decisions.
I’m personally a fan of manjaro. I run a cluster of ten 8gb raspberrypi’s I’m hoping to pickup a pine pro soon. Personally all drama associated with development is normal. To whom it concerns keep up the great work. The pinephone project is something this world needs and badly.
Just out of curiosity, what do you use the cluster for? I’m always interested in creative uses for this hardware, but it seems all ideas I have always reflect their portability and gpio access rather than pure compute capability. I wonder what one could use a cluster of 8 for?
My daily driver.
Own up to this shit, guys. Manjaro ain’t it and that’s all there is to it. The largest complaint in the blog post you failed to mention even once. Jesus Christ.
I’m a retired engineer, once responsible for an 83 acre Navy scientific station and all its equipment from diesel generators to atomic clocks, early GPS development work, a 20 meter radiotelescope as well as suppooirting and even running long term experiments from allied nations. I have been a runner, far more than a developer, administering three small ainframes, a handful of mixed breed Unix and PC systems. In 1988, I established our internet URL (1988) as wsell as selecting, installing and running its first firewall appliance.
For more than ten years I have been looking for a good way out of the Intel/AMD world, both desktop and mobile. I currently own a Pinephone,m a RockPro64 and two Pinebook Pros. The big problem that I have with your operation is the total lack of any basic support. One of my Pinebook Pros hasa frozen case hinnge I have tried unsuccessfully to get a replacement for more than a year. Look at your replacement parts pages! Almost everything that might be needed for repairs and modification is almost perpetually out of stock! Firmware-wise, there is no official, reliable method of loading an operating system reliably on the Pinebook or the RockPro because it appears to me that you are more like children running wild in a toy store than serious developers, who unfortunately, apparently have never thought of supporting the people who are PAYING MONEY to be your beta testers.
I’ve discarded the Pinephone as not worth the trouble, replacing it with a decent flip phone which satisfies ll my needs. BUT… The absolutely superb Pinebook Pro’s apalling lack of 1. An available stock of spare parts, 2. A functional, even if imperfect, method of loading and running other operating systems,with known and reliable “hooks” for the o/s porter to refer to, is inexcusable.
Pine64 is onto a lot of really good if not great things,and I wish you tons of good fortune and my loyal support, within my abilities, in order to aid your success. But first, you’ve got to provide some organisation so those of us who are ponying up time and bucks have at least minimal support.
Like other commenters on this “dissatisfaction thread,” I have several Pine64 devices (pinephone + keyboard, pinecil, 2 pinebook pros). I was attracted to Pine64 by 3 addumptions I made about the Pinebook Pro:
1. Open ARM-based hardware
2. Inexpensive
3. Small and light
4. Repairable
I’m currently not using any of these devices, even though I used the Pinebook Pro as my only laptop in nursing school for a year.
4. After my daughterboard broke when the computer was dropped, the $7 part has never been in stock, for almost 2 years.
4. My mainboard charged undependably, and at times flashed LEDs but would not power on. I successfully troubleshot these intermittent problems. Finally it showed similar behavior, but nothing I did could power it on. I wish I had a better understanding of the problem, but community hardware documentation I had hoped for was never written.
1. Being stuck with Manjaro isn’t very different from being stuck with Windows or Darwin. Manjaro-i3 doesn’t recognize PBP’s hardware keys by default, and I’ve been down some rabbit holes unsuccessfully. One day I’ll find the time and risk-tolerance to try to install OpenBSD on the still-functioning PBP. However, the SPI chip problem indicates I can’t expect the straightforward experience of swapping OSs on an old ThinkPad or Dell.
In the meantime I’ve watched Pine64 light on half a dozen new projects, buying cheap closed-ish hardware and selling it to community developers to open it.
I’m glad PBPs are for sale again. I wish I could buy parts and financially incentivize development aimed at user documentation and ease-of-OS installation. I don’t want disposable machines.
Please listen to and take care of developers.
I agree “his [Martijn] leaving is a significant loss” and you’re right this is a sad state of affairs. Besides Martijn work on the PinePhone, his youtube videos did a lot to promote PinePhone. It feels very much like an own goal. Is it too late for change or reconciliation?
This response is extremely disappointing. You have significantly misrepresented Martijn’s points were you did not simply ignore them. Your ‘short summary’ is almost entirely wrong: the only correct part is ‘we no longer listen to the development community.’ You appear to have demonstrated this by failing to give a fair account of Martijn’s post.
I would very strongly recommend anyone reading this compare Martijn’s actual points with the ‘rebuttals’ here.
As an example:
‘But here is the thing, SPI has been included on the PinePhone Pro due to the input from developers and against our initial intent.’
If you had actually read Martijn’s post in good faith, you would know he makes no claim to the contrary, and your response here is aimed at a fantasy argument no one has made.
Martijn’s point is that the process for this was a ‘nightmare’, and that the only technical fact requiring it at all was to escape the locked-in Manjuro-only boot option you had put in place.
Why do you respond to a nonsense argument Martijn did not make, and not this one, which he did?
Extremely disappointing. If this is how you ‘listen to developers’…
In principle, I understand why it is better to focus on supporting one operating system than trying to support them all. Pine64’s goal absolutely has to be to bring one OS to a level of maturity that you can confidently let it loose on non-geeks. You have limited resources, so focusing those resources on achieving that goal makes sense. What I cannot fathom is why you would choose to focus on Manjaro. I have been struggling with annoying bugs in Manjaro/Phosh for about two years, and some of them make the phone unusable in a professional setting. For example, when I’m talking on the phone with someone (for example a client) and a second person calls, the first call is dropped without warning, but the second call isn’t accepted either. This was still happening about two months ago. For this reason, I decided to try some other operating systems. I have been watching the PinePhone on UBPorts for a long time, but regrettably it didn’t seem to be making any progress, so with a heavy heart I ordered a Google Pixel 3a XL and installed Ubuntu Touch on it. Guess what guys? It just works. Yes, yes, UT is also still a little rough around the edges in some areas, but there is absolutely no comparison with Manjaro/Phosh. The community is big, motivated and active. The operating system and apps are usable. I don’t know what other factors might influence your decision to “settle on Manjaro”, but if I were the CEO of Pine64, I would give Manjaro the boot on the spot and focus on UT. As the old native American saying goes, “When your horse is dead, get off of it.”
I think the has a point though. By favoring one distro as default, you deprive the developers from all other distros to some extent of their recognition no matter how much you communicate otherwise. I mean you can say they should grow the hell up, but the truth is many of us are some weirdos in their basement, that found recognition in some online world and open-source projects, because we were not getting it in sports class for example. Many of those have become excellent developers and if you make them feel left out, they will lose motivation. All the symptoms Martijn describes you can find online as symptoms of employees who feel like they are not paid fairly or are deprived of recognition: stress, bad sleep and thus increased irritability.
I like Manjaro. I also know it is always better to learn how to give recognition to yourself (struggling there myself at times) and that running a business is not always easy. Neither is being favored, although it comes with great powers and responsibilities. I don’t want to say anyone is wrong or right here. Just that all parties have something to learn from this situation.
I am really looking forward to the day that mobile Linux will be as functional as the other operating systems. You can make it happen guys!
At least for the PinePhone Pro, because of its higher price and more premium materials a dedicated SPI chip for the bootloader certainly seems like a reasonable ask to have included by default.
I guess I’d like to record the fact that I installed a bootloader to the SPI flash of my Pinebook Pro as soon as one was available. It has a transformative effect on the hackability of the device but more importantly it means I will “drop off the radar” if I don’t post how much I love having good firmware here. Once there is a good quality SPI firmware *and* kernel upstreaming then the whole (rather antiquated) idea of per-device images goes away… I can simply boot a *real* Debian ISO and install it myself.
That’s what happens when you get firmware right! When the firmware is right then making “magic” Pinebook Pro images (or custom Debian installers like are not longer needed. You can take the OS direct from the upstream provider.
I won’t comment on the competing blog posts or how much Manjaro distorts the ecosystem but, from a technical point of view, Martijn is 100% right: shipping good quality firmware by default and in separate media (or at least in an EBBR compliant manner) is very important!
Despite the comment that you like him, I think you have made an error. I like Manjaro, but if no one is developing software for your platform, that is going to catch up with you, sooner or later.
Remember, these people were working for free. You had it good.
By the way, I just bought the Pinephone Pro. While I like Manjaro, will they be writing camera drivers for your device? Will they be improving power management on it?
While you have every right to make business decisions, the people working for free have every right to do the same.
I hope you have plans on how you are going to make this product a daily driver product.
I know i’m a little late to the comment section, but here is a thought I had.
Perhaps Pine64 should gather all active developing communities together to develope a ground up OS from scratch. All devs would be put into large chat rooms with sub sections for each item needing development. This would allow everyone to coordinate solutions across every community. Pine64 could put all its dev funds into this one big community and perhapd could integrate the dev zone to work with it. This way for example, if I put 30 dollars in pinephonepros camera development, all those who directly worked on the camera will receive my money once its completed to a general use status. Atleast for pine related products we could quickly create a end user software and set our sails to a new and even better pinebook and pinephone and start going head to head with the big folks. Just a suggestion