I also got this email, but professionally and personally.
Personally, I voluntarily built and run open source apps for 16 different cities for their transit system. This gave me two weeks to update 16 apps, for no benefit of anyone. My app is a PWA, and the Android version just uses cordova + a few plugins to add a few native options. Unfortunately, updating cordova to support the new target android api broke some of the plugins, which haven't been updated yet, so it ended up being a full weekend of work and testing.
Truthfully, I'd prefer to get rid of the app and just have users go to the website and install the PWA, but the average user still doesn't know how to do this. And the Play Store is still the first place users go to find apps. If google would just allow submitting a PWA directly to the app store, that'd be nice... I am not looking forward to doing this yearly.
Professionally, we are also scrambling. We have a legacy app that some supported customers are still using until the end of the year. The app is a fairly complex application, and basic testing has already shown that just changing the target api version has broken quite a few things. We have gotten the extension, but we know this will take 1-2 weeks of developer time + 1-2 weeks of QA's time, for an update that does nothing but appease Google. All for an app we are going to officially remove from the store at the end of the year, once all customers transition to the new app is complete.
> Truthfully, I'd prefer to get rid of the app and just have users go to the website and install the PWA, but the average user still doesn't know how to do this. And the Play Store is still the first place users go to find apps. If google would just allow submitting a PWA directly to the app store, that'd be nice... I am not looking forward to doing this yearly.
This is what Trusted Web Activities (TWA) are for, you can use it to list PWAs in the Play Store without frameworks like Cordova
My TWA app still got hit with this mandatory update requirement. I'll have to rebuild the wrapper app for the first time in years. I don't even remember how I did it last time but I do remember that Android makes it nowhere near as trivial as it ought to be. I can practically guarantee that something will break if I try to do all this busywork. I'm probably not going to bother, it's just a free app that I made as a side project and a large majority of users visit the website in a browser.
If Google actually cared about getting PWAs into the store, the process would be as simple as submitting the URL of your PWA manifest to the Play Console.
> Truthfully, I'd prefer to get rid of the app and just have users go to the website and install the PWA, but the average user still doesn't know how to do this.
Let's be clear: that's not because users don't know how to do it. It's because Google and Apple haven't made it as easy as installing an app from their app store. That's a choice, and it's a deliberate one.
I disagree, at least on the Android side of things (Apple has long been hostile to PWAs). Installing a PWA from a website is trivially easy on Android, it's just that most users really have separated in their minds (not surprising due to history) that apps come from app stores, and the browser is used for websites.
Also, Google has made in much easier in recent years to submit plain PWAs to the Play Store: https://youtu.be/ddbHp8tGBwQ
On Android you have to pop open a menu and find the install option. That's not inherently discoverable as you need to know it's even possible, and most people don't.
It would be trivial to present the user with a more proactive notification that a site can be installed as an app, or even include such a notice in their search results on Google, but they choose not to do so.
Actually nowadays it's not that bad anymore. Android browser itself offers installable PWA and there is an event called beforeinstallprompt event (https://developer.mozilla.org/en-US/docs/Web/API/Window/befo...), which can be used to perform PWA installation on user interaction. Of course it's not supported in every browser.
iOS is more difficult since user needs to understand that "saving to home screen" is same as installing "app" and there's no way to trigger it programmatically or help user in any other way than with visual illustrations.
The share menu is used for everything on ios, I had to get used to it at my first apple device’s case.
One notably stupid usage of the Share option was (I believe it is no longer how it’s done) adding an image to a hidden folder — that’s something you definitely don’t want to share, yet quite easy to accidentally send to someone during this process.
Thankfully this insanity is changing in the upcoming release due in September. A button for a 3 dot/hamburger menu appears in the bottom right when you select photos.
Because it doesn't make sense that it's part of the web share functionality to begin with.
The native share dialog is about "I have this piece of content, now share it with one of these other apps", e.g. saving a document locally, or sending it with AirDrop, or saving to Google Drive, or sending as an email attachment, etc.
Installing a site as an app on your homescreen has absolutely nothing to do with sharing content to begin with.
I have some PWAs in the Play Store via that method. I also got this same Android API update email and had to jump through various hoops to update them all. I wish I could list an actual PWA (like a URL to a website) in the Play Store and then never have to worry about Android API version updates, since I'm literally not using any Android-specific features.
> it's just that most users really have separated in their minds (not surprising due to history) that apps come from app stores, and the browser is used for websites.
Right, because for a very long time (and maybe still) PWAs have been much closer to terrible websites than good apps. They generally don't have the same UX properties as real apps.
Remember when the iPhone first came out and web apps were the only option and they sucked balls? That never really changed. People still find mobile sites with maps that are impossible to use. They don't expect that from app store apps.
And if it's just a web site, why do you need to "install" it? A link is surely sufficient?
But nowadays you can't really see the difference between good webapps and native apps. We also migrated our native apps to full SPA apps. And really it makes development so much easier. The apps we have are relatively simple without fancy stuff. But the css render engine is fast. Even on Android.
And we reduced some of our apps from 30mbs of java code to 150kb of java/typescript.
Plus as a bonus we can have a website and serve ios as well.
For us there is really no reason to go back.
Sidenote: some parts of the app are still native. Ads, Auth and analytics
Thanks for posting, I think this is a great point. Web tech has advanced to the point that a large swath of apps can be implemented as PWAs with no loss of experience (though last I checked iOS was still holding things back).
This isn't true for all apps, but with the notable exception of games, I'd say it applies to most: banking/finance apps, social media apps, travel/airline/booking apps, etc.
If you seriously think analytics makes for a worse user experience, then you're speaking out of ignorance.
Analytics tell developers exactly where bugs and crashes occur.
And on which devices or versions of the OS the problem is.
Without analytics it would take weeks/months to figure out exactly what line of code is causing the issue. Heck, the developers might NEVER KNOW that the software has an issue.
Apps would just keep crashing on users for years. And developers would have no idea why users were abandoning their product.
Nobody that has any idea what they're talking about would say that analytics makes for a worse user experience.
> And if it's just a web site, why do you need to "install" it? A link is surely sufficient?
The point is that there are apps that can pretty much be built entirely using web technologies as PWAs, but in doing so they are no longer "just web sites" and they need functionality of installed apps, like notifications. For example, most banking apps on Android could be entirely rewritten as PWAs, but they'd need to make use of things like notifications (I don't want a random website sending me notifications, but I DO want notifications of activity on my bank account) and camera APIs (e.g. for mobile check deposit).
I find bank apps to be the least flexible about notifications; there's no way to get them to use app notifications for a lot of things they insist on using email and/or SMS for.
The last thing being the worst as SMS 2FA is so insecure, but SMS of your bank deposits isn't much better.
Well, obviously all the features that are available to PWAs are available in browsers - a browser is the thing that executes a PWA to begin with.
It basically is just a difference in use case and how most people thing about "apps" vs. "websites". If I have a long term relationship with a business, and I access its functionality frequently, I'd rather have it as an app on my homescreen.
Just looking through the apps I currently have open… Bank app, chat app, maps, Tile, YouTube and a weather app. Only one of them is actually doing anything that wouldn’t fit a PWA. So why are they apps, not a collection of links?
I would say only the weather and bank apps would be really equal as a PWA.
Maps require complex gestures and advanced graphics and UIs that would never work well as a PWA. Try maps.google.com. Its nothing like the Google Maps app. Plus Android Auto integration.
Tile probably needs pretty deep Bluetooth integration and background processing that the web doesn't provide.
YouTube can do things like PiP that you can't do on the web.
Yes, "Add to home screen" is exactly how you install a PWA to your phone =)
It's a bit confusing, isn't it? "Add to home screen" makes it seem like you're just adding a link, but it's installing the PWA, possibly enabling notifications, and etc.
1. For people like you that don't want to install them, they're just a normal website.
2. For people that do want to install them for the added functionality (things like notifications), then it is easy to install, and furthermore cheaper for developers to build and maintain (one codebase instead of multiple).
You say "you have no desire to have an app", but I think for most people that's really dependent on the site/application. Yeah, for any site I just have a short term or infrequent transaction, I don't want an app either. But many/most people use apps for businesses they have long term relationships with (namely financial institutions).
That's not always because of anything intrinsic to the Web. We know that ordering a pizza isn't any more complicated than what can be represented on a paper form. Forms can also be capably represented in a browser. The problem is that app developers are by and large not capable of distilling the requirements down to what is necessary and sufficient for the task at hand. Instead, they're foolishly preoccupied with what they consider to be the mandate to deliver experiences.
I find it much easier to install a web app from the website than having to find anything on the App Store. There’s nothing easier, it’s only 2 taps when browsing the website, at least on Safari. The process is exactly the same as in iPhoneOS 1.0, when this was the only official way to get applications.
The problem is not that it’s difficult, it’s that the share sheet got bloated and should be completely rethought.
> I'd prefer to get rid of the app and just have users go to the website and install the PWA, but the average user still doesn't know how to do this
I think you misunderstood users, it's not just ignorance. I want apps to go back in the direction of real(not cordova) apps, not some low effort web thing. Basically zero web apps match the experience of a well crafted actual app.
I agree with you in many cases. At my professional company, we've made the decision to write native apps for Android, iOS, Windows, and the web, because we have pretty deep integration into each platform and want the best native experience for our users.
But, for my personal apps (which is what I am talking about in this thread), I can't support that. Writing it once and maintaining it takes up enough of my free time. This app also doesn't have any integration with the system, and has a minimal interface, so it works quite well as a web app.
The app my accountant provide me - a white-label finance app with the firm logo on it - used to be native. The developers had issues with updating the app to support a change in the camera API and I couldn't scan papers. They sent me get web app instead. Now whenever I use phone's native back button/gesture the app quits instead of going back within the app, there's a functional in-app back button, but my instincts will never disappear selectivity when I'm using their app. I have quit their app million times half-way of filling forms. Terrible UX.
As keeps getting pointed out, every. Time. This. Gets. Raised: evidently most SPA’s are not written properly, because “back button not working properly” is easily one of the most common complaints.
That's not the only point. Web apps can have far lower performance or battery utilization than real native apps for one thing. If you care about the environment or your battery life use a compiled app, not the web.
I see now what the parent poster meant. He said a web app can have lower performance and battery utilization. He meant lower performance and higher battery utilization. From the context it was clear and I misinterpreted it.
You do understand that not every developer is backed by billions of dollars of venture capital, right?
That there are people who have to target multiple operating systems, and don't have 100+ people working on their team, right?
I mean...it doesn't take a genius to figure out that there's a benefit to being able to write code once and deploy to ALL users/customers without having to dedicate entire teams of developers with expertise in various platforms.
I'm an Android developer with 10+ years of experience. I take pride in my native apps that I code for the company I work for. They are far superior to any web app or "multiplatform framework solution".
But if I had to create my own personal app, there's no way in hell I'd spend years learning everything it would take to create a native iOS, Windows, Linux, MacOS version.
I'd be an absolute idiot if I didn't just choose a solution, like a web app, or framework that was able to output for more systems, etc.
We love to discuss technology on here and I have my own opinions about that, but to be honest, as a user I have found absolutely no correlation between the underlying technologies and how well an app works for me.
It often hinges on seemingly small design decisions that make some frequent task either a breeze or a constant annoyance. There aren't many cases where making the right design decision depends on whether or not the code is "native".
I believe the most important distinction is whether the motivation for making an app is economically aligned with my motivation for using it and sustainably so.
I used to think that, and then I started using Phanpy and Voyager. PWAs can be surprisingly good if they're done well. The problem is the mobile platforms have done nothing to optimize the experience or encourage their development and so the good examples are few and far between.
>I think you misunderstood users, it's not just ignorance. I want apps to go back in the direction of real(not cordova) apps, not some low effort web thing. Basically zero web apps match the experience of a well crafted actual app.
For me, it also depends on how integral internet access is to the application's function. When I'm on my computer and want to check the weather, I don't want to launch a separate application. I just google "weather". I agree that games probably belong as standalone desktop/mobile applications and similarly for other programs requiring deep integration with OS APIs. But if an app mostly wraps content that's already available via a website (e.g. starbucks.com, subway.com, or news sites), I'd rather just use a full-blooded web browser.
Usually I would prefer no app. Better than wasting my time downloading an app, being disappointed that it was noticeably "web", and ditching it. Low effort implementations also make it harder for the grass of better native apps to grow amidst the web based weeds.
I have compassion with you, some of my private apps were also affected.
But we were being told this deadline for many months now.
It was clear that at some point they would show it more in your face.
I also disliked the way it was formulated, also, even if everything was fine in production, it complained when testing versions didn't comply (doesn't make sense).
Also, it was always only about pushing new updates (it's ever year like that). You could still keep the app live for some time.
My understanding from earlier notifications was that updates would not be accepted unless you targeted a new api version. That is fine, since I haven't updated the Google Play "app" since I first released it. This is because the real app is a PWA that I update through the web, and the Play Store app is just a shell around that.
The first time I was aware that the app would be delisted in the Play Store for new devices was in the August 18th email.
> Apps with a target level of Android 11 (API level 30)* or lower will not be available to new users running the Android OS higher than apps’ target API after August 31, 2023.
>Apps with a target level of Android 10 (API level 29) or lower have not been available to new users running the Android OS higher than apps’ target API after November 1, 2022, or May 1, 2023, if your app had an extension.
In my case, I was hired to work on an update to a legacy app ~8 months ago. So this is my first round of bs. Been writing software for 20 years and I’ve never seen this before… I wouldn’t consider it playing dumb, I’ve just never had cause to care or receive an email like this.
And you have also never had a security vulnerability in one of your dependencies causing you to update your software? A new database version? A new version of whatever runtime you were using?
I guess that’s the difference between a free marketplace and whatever this is; because this IS NOT a free market.
In a free market, you set the deadlines. If your publisher reaches out to you about a security issue you don’t have to fix it and they aren’t obligated to pull your product. In fact, you might make it into a feature or simply reconfigure a WAF.
Probably the closest thing to this would be in the early/late 00’s when some idiot would buy an ad in the paper/magazines. You’d have a fire under your ass for that. Even then, it was self-inflicted.
So the answer is no. I’ve never dealt with a publisher that had you by the balls, imposing random deadlines. There’s always been a human on the other end who has their own agenda, but also respected ours.
Not sure if it helps, but if I was in your situation I’d provide a few app updates with a screen to train users on how to install the PWA version and gracefully run away from these problems. Maybe also provide a some sort of a form to get some feedback over the difficulties encountered by users to get there.
Good luck!
The problem is that many of my users are temporary. For example, I have an app for the public transit system for a resort town in Colorado. The town has a decent, albeit small bus system. They technically have an app from their vendor, although it is not very good and is difficult to find. If you search for "$town_name transit app", it won't show up anywhere, where as my app does. And I think my app is much more user friendly. I wrote it because I visit this area a lot and hated the vendor app.
My users are visiting this town for a few days, and are most likely going to open the app/play store and search for an app, use it for a few days, then leave the town and forget about it. The least amount of friction I can provide the better. My only goal is to support public transit and make it a smoother experience.
Do people really search for entirely temporary/short-term/single-use use apps, like for a resort town's transit or a restaurant? For me it's a last resort thing, if there's no website or it's unusable.
Yes. I spent a week in Rome last month (first time visit) and ended up downloading four different apps for public transport and city guides, all
of which were useful. This is on top of Google Maps, Trip Advisor and everything else.
> Or do you think places are making apps that no one uses?
Of course, because apps are "modern". You've never seen an app that should have been a website? I know of multiple places that had shitty apps built for extremely narrow use cases that had close to zero use outside of the team that ordered it (while it was meant for a wider audience).
And yes, I'm genuinely baffled people will bother downloading an app for a very limited use, like the transit of a place they'll visit once for a few days at most (especially considering there's Google/Apple Maps, Citymapper Transit; unless you can buy tickets through the app it's a waste on top of a waste). Has it been ingrained to such an extent that phone == app? Or is that an iOS thing, or maybe an American thing?
What makes you think that companies continue to build and support apps that no one uses? Maybe they have more insight about their usage then a random person on HN?
All the apps mentioned in that page are ones that are used regularly. You can surely appreciate Tinder and Spotify are wildly different than a random resort town's transit times app? In the same way that there are useless apps nobody uses, there are many that are used for hours daily.
I got Citymapper, and that was the end of me looking for transit apps when going abroad. But true, these days the built-in map apps in all platforms are also decent at dealing with public transport options.
I love cordova for my personal apps, but once every year or two when I’m forced to update Android and iOS they become such a nightmare. I spend days or weeks getting unblocked because I don’t have unlimited time to maintain these apps alongside everything else in my life.
Your customers have been using websites for much longer than they have been using apps. I simply don't buy that they can't understand how to use a website. Stop underestimating your users.
It's amusing that you are certain that you know his customers better than he does. I believe that you are utterly wrong, and if not most than certainly many people who use public transport and smartphones have never or nearly never been using websites on their phones, many of them haven't been using websites on a computer two - this is very typical of the non-technical people, especially those who are members of the younger and the older generations.
That's not what I said, I live half my life within browsers. But big chunk of the population that you are probably not thinking about is not like that. My mother in law is using browser to read on her computer, and will never do that on her phone - the screen is too small and the whole experience not something that fits someone at her age. On the other hand, my children use apps since they are two years old, and at the age of seven they still rarely if ever used browser.
>On the other hand, my children use apps since they are two years old, and at the age of seven they still rarely if ever used browser.
Most of the people writing on HN were first exposed to the internet through web browsers. On the other hand, children these days who grow up with smartphones first interact with the internet through phone apps. Do they have any difficulty adapting to the unfiltered web when they grow older?
I sure hope not. I don't have definitive first hand experience though as my kids are not old enough to have to use a computer and they haven't grow much interest in that direction. When my dad's first PC arrived home it was for me the most exiting thing ever, but it didn't have to compete with a smartphone.
The screen isn't too small to use web browser presenting site that looks just like the app, obviously. It is too small for comfortably reading or using the browser in general, so using the phone's browser for searching stuff on google is something that many older people simply don't do (in my anecdotal experience, I have no hard data to back this claim up). Searching for the train times in city z, going into a web site and finding that it is actually identical to the app and just as useful is something that is very unlikely to happen to my mother-in-law.
There are a lot of websites that don't scale well with small screens.
Just the simple trick of zooming in on a website with a "2 finger pinch/pull" is something a big part of the population doesn't even know how to do. I know my mom would probably just give up.
Buttons on many websites are way too small for some people to use.
Those are never an issue on native apps. The UI on apps scale with the size of a phone's screen. No matter if you have the smallest iPhone or the largest Android.
This alone is enough for many people to not use the web browser unless absolutely necessary.
If his customers are under 25 or 25 probably not. There are also plenty of older people in the US who never had a computer. But they now have smart phones.
While I concur with the author on the challenges of Android development, the author made two major mistakes.
One, he didn't test his app on the latest version of Android. This a very major mistake and the reason why we keep 11 virtual machines around with all the versions of Android that our app is supported on. Two, when you release an app on Google Play, you never, never, never deploy the app to 100% of your user base, at least not initially. Staged rollouts are the norm and we never initially deploy an app to more than 10% of the user base after a release is published. Additionally, when we finally feel confident with a release, we deploy the app to 99% of our user base and not 100%. The reason for this is, if we need to halt a roll out for whatever reason, we are easily able to do so, so long as the release is not deployed to 100%.
For better or worse, both of these tactics are well-known by more experienced Android developers.
I agree that things could have been done better in many ways. However, as explained, it is a legacy application, which does not see any active development nowadays and it would just not make sense to build such a robust QA. This particular app have been written long time ago by another company and there's not even simple unit tests. That's the hard truth. But still, even as complex setup as you have, there's still going to happen mistakes and the real problem is that there is no way to pull-back/cancel/rollback release.
How would staged roll-out help in this situation for all customers? When end-user gets the faulty version of the app, does he/she have a way of getting the non-faulty version somehow?
Unfortunately, as you noted, the game is rigged and you have to play within the sandbox provided. With that, and as you also noted, your testing simply needs to be better, especially since this is a customer-facing app and not an app that's solely used internally. As for dealing with the rollout when a serious bug is now in production, if you didn't roll out your app to 100% of the user base you could halt the roll out so it wouldn't affect any more customers. Then, while far from ideal, you could ask affected customers to uninstall the app and then reinstall it, and the newly installed version would reflect the previous version of your app.
A-ha - good tip about roll-out, uninstall & install. Was not aware of that possibility. But yeah, it's still a hack and there should be a better way - just let me delete/cancel latest release even if roll-out is 100% for whatever reasons.
About better testing - there's always room to improve testing, but no way it's going to happen with a legacy application where no active team is assigned. Only these irregular updates mainly forced by Google are done. Unfortunately.
building a business on any big vendor (google, apple, amazon, microsoft, etc) without at least making sure these risks are shared with the client seems like a fast way to run into these costly overtime problems.
Rigged seems a bit extreme. They had to make a few changes to keep things up to date. That they tripped over this small hurdle and went tumbling down the stairs doesnt change that it's a small hurdle.
Just to add, even if you do not change anything at all, you will spend many hours just getting the build environment up to snuff if it hasn't been touched in over a year. Gradle, the plugins (android, crashlytics, etc), the build.gradle DSL, dependencies ... its quick bit rot.
Android as an OS does not support rollbacks. It can only upgrade apps in monotonically increasing `versionCode`s. The simple reason is that client side data may have been upgraded by a newer release to a format that is now incompatible with the old version. Sqlite databases follow the same policy.
This is well-known to anyone who has been doing Android development for a while.
Totally understand that a roll-back is more complex, but it doesn't explain why pulling back/pausing/deleting/yanking a release is not possible either - customers who got the newest faulty release can uninstall & install to get the previous version back (no need to worry about incompatibilities here) and the ones who did not get the updated version yet can stay using the old one until a new fixed version will be released.
This is exactly what staged rollouts are for. The author ignored that best practice. You can halt any release as long as it is in progress. Once you mark it complete (defined as rolled out to 100%), you can’t roll it back.
Each track in Google Play (alpha, beta, canary, prod) can hold up to one release at a time. It would get really confusing if it allowed more than one. And with the other rollout safeguards provided decelopers, it’s quite possible and very easy to do exactly what you’re asking for.
It's true that in the future I would do some more thorough testing, but never-ever for this legacy application can I build a bullet-proof automated testing solution - there will not be a budget for that for sure. However, even with a fancy solution mistakes will happen and you still can't stop release propagating. It's just a matter of time when it happens.
"A bullet-proof automated testing solution" isn't needed for the type of problem you describe. Anything that logs in (including a human on a real device) would catch it even if it does nothing else.
Protip: that field takes decimals too. We always go for 99.99999999% so no users get left out accidentally. But it allows you to cancel a rollout like you say
What you're saying is totally true and the only pragmatic way to update/release. But it is a shame that if the api update compiles you can't expect things will just work the same way.
This is why you set a version number for the API you want. No type system in wide production use is able to encode the semantics of APIs sufficiently. Many of the best APIs out there have version numbers that allow you to pin the API structure and semantics – GitHub and Stripe come to mind with this.
There’s this company out of I think Washington state who builds an operating system for x86 and x64 systems. I know for a fact that GUI applications from 15 years ago still work on the modern version. It’s called Microsoft Windows. You should check it out. The win32 api has changed but it’s backwards compatible.
Windows has "Compatibility Mode" to support these things. Additionally there are layers and layers of Windows API – an app built against Windows XP may still work but it's not using the same tech as one built for the current version of Windows. This is part of the reason that users always had to install different versions of .NET, or part of the reason why Windows is a huge OS (API surface area, disk usage) in a way that isn't suitable for mobile devices.
Yes MS are better at backwards compatibility, but everything is a tradeoff and they receive plenty of criticism for the direction they take.
Desktop computing has different constraints and is moving much more slowly than mobile computing, so it's more feasible to maintain that compatibility.
This is my same sentiment, author has been too dramatic with the blog title. Updating from 30 to 33 SDK is just a disaster waiting to happen. In the mobile world that is just too long.
But this is also kind of a problem created by Google - why force from 30 to 33 and not force from 30 to 31, then 31 to 32 etc? Gradual API level upgrades would make more sense in my world and probably would be less risky for every party.
I don't get the comments tearing into OP. Sure, he could have been more careful. He could have tested the login on the latest version of Android. But what if it wasn't a login crash? What if it worked on the latest version but not others? At what point do you draw the line?
At some point, you just have to say "OK this is a platform used by literally millions of apps and millions of developers, and mistakes will be made, and it should be easy to fix them by stopping your own rollout (without having to know tricks like doing a staged release) or immediately making an older, already approved version live again". It's such a basic design principle to make things revertible/recoverable, especially for something like an app store.
I'm the OP and thank you for thinking along with me here. As stated in numerous replies already then I totally agree that I could have done better in terms of testing things out - of course, there's always room for improvement in that regard. There was a deadline set by Google (again, first time I heard about it was at 18th of August, not before), change seemed trivial at time and since app worked on an old Android version as it was before then I didn't expect it to fail so miserably. Again, I'm not a seasoned Android dev, but have 15+ years of experience in software development in general so I have some expectations how things will work or not and what to expect and be afraid of. I really didn't know that "best practice" is to do a staged roll-out of "99.99999999%" to have a way of partial "yank" possibility of the latest release. To find out that there's no way to cancel/delete a latest release to fall back to previous working version was just something I did not expect in my wildest dreams (I guess this is something you only learn during situations like these). Yes, everyone can blame me for not testing every functionality with every Android version and I do the same, but please open your eyes and understand that the way releases are currently handled by Play Store is not a sane person would do outside of Play Store. Everyone will have a problem like this at one point and I do hope that this article will and thread in here will lower the number of developers experiencing situation similar to this.
but bugs slip testing all the time, that why even “the pros” have to use staged deployment trick, which takes us to the root problem: releasing on android is like running through a minefield.
there are a million “shoulds”. just because you know most of them or can afford to account for them on every release doesn’t excuse google for the maintenance nightmare that the android ecosystem is.
I mean yes there are a million “shoulds”. But logging into your app to see if login works is kind of basic. Not doing so is the definition of amateur hour.
> And having a phase roll out with a rollback is also not a new concept.
Having a quick revert is also "not a new concept". Most modern devops practices focus not just on Mean Time to Failure, but Mean Time to Recovery, and a quick revert is a critical part of that. Even Google's own SRE best practices encourage that.
> It's time to move back to open (web) standards and take control back into our own hands!
Wait till you discover that the web is also half controlled by the company that is causing your problem now (Google). And Apple who owns Safari, the only browser on iOS in the real sense, isn't really your friend either.
Yeah, that's also true, but at least this kind of problems can be handled by a roll-back or whatnot. Situation we're currently are in does not allow to do anything while we know that bad build is in production and phones are updating to it automatically. That's the worst situation to be in.
Sure, you own the servers so you can rollback. You don't own a user's device. If you want to rollback after a full rollout, all you do is build your new/rollback release from your previously working commit and update your version number.
Right, but in this case, changing the target API is what broke the app. Since Google won't allow you to release an update with an old target API, you can't just revert the change and increase the build number.
Yeah, I was talking about general rollback for Android. In this specific case, they chose to rush and they didn't test (boo-hoo). It's like me complaining that the tools shouldn't have let my bug go to PRD even though I didn't test (and apparently didn't know much about Android as evidenced by not having a physical Android device).
Same thing if your site cert expires, or browsers start blocking specific functionality/code/tags. Seems like they just want to complain about the thing they aren't familiar with.
Typical "legacy application situation where no team is assigned to" here. At least I've never seen in my 15+ year career a legacy application, which has a very good maintenance/testing model in place. It will just not work because maintenance/testing needs also resources and updates, but if priorities are in different places then it's not possible. Of course as stated already multiple times - things could have handled in better ways, but this doesn't mean that Google should not allow to stop release at least. Anyway, lessons learned.
MS expends an inordinate amount of effort on back compatibility, and much kudos to them. But it vastly increases their attack surface.
Likewise many of the worst things about the unfairly maligned C++ come from a hardcore position on back compatibility: as much as possible, old code, and even old C code, should continue to compile and work as expected, even to the point of linking old binaries to which you’ve lost the source code.
Most people don’t go to that effort and just invalidate old stuff in the name of maintenance, reliability, and security.
Whichever branch cut you take you’re going to cause a problem for somebody. If not, nobody is using your code.
Yes, that's definitely one side of the problem and I'm not chasing too much backwards-compatibility. My biggest concern in this particular situation is that there is no way (with Android, at least) to pull-back/cancel/rollback release and everything is blocked behind Google's review process. Why isn't it just possible to "yank" problematic release and continue showing previous release as the latest version. That would solve most of the issues within context of this problem.
Rollbacks allow malicious actors to /simply-easily/ circumvent device security and user preference. To allow rollbacks is to /significantly/ increase the attack surface of a device.
What do you mean by that? Are you effectively trying to say that allowing upgrades does not have any risk of attack surface? I'm pretty sure that updating things have also a pretty high risk on introducing new previously non-existing security issues into your code-base/product.
Not necessarily, Google has access to the developer's private key they use for signing their APKs so they could just make a fake release that has a bigger version number than the current version whenever a rollback is needed. No change is needed on Android itself, it's a Google Play issue.
The middle ground that I think helps everyone is, change your api and break compilation. Let devs fix the compile errors and use the api correctly, etc. But don't change the behavior of existing functions with the same signatures. As an api provider, do your best to make compilation mean something.
This is exactly how Android works. But because API changes bring privacy and security improvements, scummy software used old compilation targets to abuse backwards compatibility to avoid complying with privacy and security practices.
This is why Play slowly enforces apps to raise their compilation target and implement safer APIs. It's lagging for YEARS after API changes, so there's plenty of time to fix apps.
The OP just decided to be lazy and wait for last two weeks.
I'm the OP and wanted to clarify in case you missed some points - it is a legacy application which does not have any active dev teams on it and needs only developers attention when Google says so and as mentioned by multiple other commenters here the first time I got that e-mail from Google, was at 18th of August. I would not agree that I have been lazy, but instead trying to solve this problem in the time-constraints set by Google and failing to do so because of the inability to put a fix to production and/or pull back current release version. Of course I admit that there's always ways to improve quality assurance.
There has been zero communication towards me from Google until two weeks until deadline. Yes, maybe if I would have logged into Play Console then there might have been some notifications, but there have been no reason to do so until that e-mail (I'm usually not involved with Android projects, otherwise I might have noticed similar warnings via other projects early on).
Double-checked - no e-mails prior 18th of August from "Google Play Console". Also, double-checked "Google Play Console" (even under "Archived messages") - first message about deprecation I see is from 17th of August (a day before e-mail). Even if there were something a year ago, then 3 month, 1 month followup would make sense instead of 2-3 week reminder.
Maybe it's possible that different roles in "Google Play Console" will receive different type of messages at different time? I'm not an admin for that app and it's possible that someone else has gotten prior warnings, but might have ignored these, because usually they are not with technical background.
> But because API changes bring privacy and security improvements, scummy software used old compilation targets to abuse backwards compatibility to avoid complying with privacy and security practices.
Though unfortunately from time to time they also do break power user use cases…
There is no best win option, but you can also publish your app separately - these are Play Store requirements, not Android requirements.
It's far from trivial in many cases, and it'll never get anywhere near as much use as the Play Store version... but it is nice to have an escape hatch.
I got the impression the author was mainly complaining about Google's role as arbiter of what software users may run. The problems from deprecating APIs are just what brought the author's attention to how vulnerable we've become to Google's whims.
The majority of applications deployed to android are targeting android's bytecode. They aren't natively compiled applications.
The reason C++ presents insurmountable security problems is it's low level nature and the fact that once you have a native binary, you're done.
But a bytecode for a language with memory safety? How would it be possible to not backport security fixes. The very nature of running such code is one where you are constantly recompiling the bytecode.
Just to paint how absurd this position is for google. The JVM can still, today, execute and use classes targeting Java 1.0 (released in 1996).
This isn't a security issue, this is a "google doesn't want to support the platform" issue.
I'm more amenable to google clamping down on artifacts containing natively compiled code. However, a blanked "Your app was built targeting an old version of android, we won't support that anymore" is just ridiculous. Seems like a way for google to prune old apps from the store more than anything else.
Especially since a policy like this is by it's nature one that shifts a large maintenance burden on android developers. After all, if you want the widest support for your application, you target the oldest version of android possible. Very few people target the latest version of android for fear it will lock out too many of their customers.
If google was serious about security and making the latest android tech widely accessible, then they'd work towards decoupling the android runtime from the operating system version. There's no reason ART and the Dalvik runtime couldn't be distributed via google play like the rest of the android ecosystem. Removing the silly "you need android 17 to do this... opps your hardware manufacture isn't updating their hardware drivers."
But then, that would cut into new device sales and we can't have that.
> This isn't a security issue, this is a "google doesn't want to support the platform" issue.
Well, it is a platform security issue - sometimes a privacy issue.
Google essentially is improving security over time by fixing broken APIs which are deprecated and removed over time.
There are plenty of security and privacy related changes where APIs have been fixed in backwards incompatible ways.
Essentially, this has been solved by adding more permissions - where either Google Play, or the user needs to consent.
In order to not break backwards compatibility this has been enforced based on the "target API level", and in order to prevent malware from simply targeting old API levels, they enforce this in Google Play by forcing apps to target current API levels.
In most cases, the changes required are rather small, sometimes code changes, sometimes compliance/documentation changes - or a combination.
It was always obvious that Play Stores and their captive developer "communities" were a trap. Forced API upgrades are just one aspect of that.
There are certainly tasks that are best done by a phone or mobile app; usually, these are things that involve moving around, such as navigation, or depend on phone sensors, such as working out which way is "up". But nearly everything else can be done by a website.
I'd hate to be a one-man developer trying to maintain a cross-platform mobile app. I sympathise. But strictly from the sidelines; I long ago decided that I wasn't interested in playing games with proprietary platforms and gatekeepers.
> There are certainly tasks that are best done by a phone or mobile app; usually, these are things that involve moving around, such as navigation, or depend on phone sensors, such as working out which way is "up". But nearly everything else can be done by a website.
Actually, both of those things can easily be done in a browser app.
About the only thing that can't be done easily on mobile right now is GPU compute shaders. But that will also fall when WebGPU merges from desktop browsers.
The only other thing I can think of is generic, system-wide file management. That probably won't ever be coming. Though the File System API does allow users to grant access to specific directories, I don't think the permission persists past a page reload.
> It was always obvious that Play Stores and their captive developer "communities" were a trap. Forced API upgrades are just one aspect of that.
I was convinced the app stores would be a flop because I didn’t think there would be a critical mass of developers that were willing to give up the guarantee they could actually deploy their apps.
Funny reading this on a site that regularly tears into Google for not adequately gatekeeping software on Android and advertises iPhones as better because they police more aggressively.
I see cognitive dissonance on this site every day. Android or rather Chrome supports web apps, and Android allows apps not from the play store, unlike Apple, and yet people complain about both sides.
They'd also complain if Android allowed a loophole to violate privacy by targeting old insecure apis.
But you don't want to be ideological about user sovereignty over their own devices, do you? Be smart and pragmatic instead, and walk the path of least resistance into a cage.
"At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second. A request for a rollback is not interpreted as an attack on the releasing team, or even the person who wrote the code containing the bug; rather, it is understood as The Right Thing To Do to make the system as reliable as possible for the user. No-one will ask “why did you roll back this change?” as long as the rollback changelist describes the problem that was seen."
As of last year, due to a mix of Google Play policies and Android updates, if you're distributing an app on the Play Store, the only way to rollback is an uninstall + wipe user data.
----
* You cannot downgrade to a lower `versionCode` & `versionCode` is defined by the developer
* Android 11 prohibits writing to 'permanent' storage of the phone, with the exception of media, or using the `DocumentFile` API, which treats a local folder as cloud storage. `DocumentFile` is unusable for many use cases
* A developer can set `hasFragileUserData`, which should ask a user if they want to wipe their data on uninstall. There are Google/Android bugs which mean this dialog may not be shown
* If an app is uninstalled and the user doesn't delete data, Android blocks a downgrade to a lower `versionCode`
* You can use a regular java.io.File if you request `MANAGE_EXTERNAL_STRORAGE`
* Google Play only grants `MANAGE_EXTERNAL_STRORAGE` in exceptional circumstances
I had the same thing happening to a bunch of apps based upon a framework I built. The newer API version had problems with existing dependencies, really a shit ton of work to get back to exactly the same place I was already.
I really respect Microsoft a lot more, where stuff from the 90's has less issues running on the latest Windows version that mobile apps I wrote four years ago on Android.
Why? Should Android be bloated with compatibility layers for every buggy framework ever existed? Including third part stuff such as Cordova and Ionic that have more or less stopped development?
And it's not like Google is randomly breaking things. And even when there are breaking changes they provide support libraries and explain very clearly what has changed and why it needed to be changed.
Still, no sympathy for app shops that work according to fire and forget.
"Compatibility layers": it was a setting that is still a valid setting but you need special permissions for it nowadays. The code was compiling perfectly.
IMHO, there are too many drawbacks to being forced to distribute an app through only one "self-authorised" app store and very few advantages.
Pros (ironically):
1. the app store manager is supposed to check the app for malware and viruses before publishing it, ironically all the apps full of Google ads, Google and Facebook trackers
of any kind are welcome ...
2. make it very easy for the user to search, install and update the apps, if he can find what they are looking for through billions of game apps and very similar apps that are only published to send advertisements to the user or to collect his personal data.
Disadvantages:
1. you are tied to the whims of the app store manager
2. You have no control over the app publishing process:
You cannot decide if you can publish your app or not, only the App Store Manager has the power to decide when and if it is convenient for him.
I think only the user should have this right.
I'd prefer to install the native commercial native apps (for example, the home banking app) on my phone by downloading them from the developer's website, in a private area that I have to access with my credentials, where the app is GPG signed and I can check the integrity of the package. An auto-update feature is doable.
For open source apps, there is the F-Droid store where you can add your own repository (a bit technical and not for everyone, I admit : pre installing F-Droid on the new phones could help a lot, but I don't think Google will ever allow this).
Another viable option is PWA. In my opinion, there are very few apps that need the native features, for all the others, almost everything is now doable with PWA technology, and you do not have to bother sending updates to users or asking them to upgrade.
The monopolistic and commercial app stores, with the excuse of making life easier for users and in a supposedly "better security", are a "wonderful" way for Apple and Google to make billions on the work of independent developers and, in the case of Google, to collect and sell personal data about users and to sell ads of any kind in any way.
You forgot another pro: it keeps the Boston Strangler[0] out of your house. By tying users' hands behind your back you can ensure that you dictate the terms by which they use your work. No piracy, no adblock, etc.
This is why game developers largely fell in line with the "Licensed by Nintendo" model back in the 80s and 90s, and mobile app developers did the same thing with the iOS App Store in the late 2000s. They don't want to work directly with users to circumvent the platform owner, they want the platform owner to tie the users' hands, and they're willing to have their hands tied in the process.
In the specific case of, say, your banking app; they want secure remote attestation so they can ban users that are running credential stuffing attacks against their app, prevent malware from opening your banking app and clicking the "pay fraudster" button, prevent users from extracting various tap-and-pay related encryption secrets, and also prevent them from injecting stolen secrets into their phone app. The app cannot, on its own, validate that the user's hands are tied; it needs a trustworthy (to them, not you) third party that lives in the boot chain and/or EL3 to validate that. So they don't want to just give you an app and a signature. They want Google to do it so that Google can tie your hands.
It's important to note that all the user-facing benefits of app stores are backreasoning. The goal from the beginning was to tie users down, because to them, users are little thieving mosquitoes. This is the "quiet part" that they don't say out loud.
Let's hope to dear god that Google's WEI proposal doesn't happen and PWAs never have access to attestation. Google already has the whole "we don't let you login on unknown browsers" nonsense and we don't need that cancer spreading.
F-Droid is great. Google's shenanigans with Android need to be shut down. Hell, the EU was able to do that but the US needs to also do that and have it apply internationally.
[0] MPAA code word for noncommercial small-scale copying, i.e. someone with a VCR taping shows off TV. Jack Valenti really was a piece of shit, wasn't he?
> I personally have been against developing mobile apps for years now for the exact same reasons described in here and other similar articles — as soon as you decide to develop mobile apps then you give control of your product/service away
to a third party
Hard to disagree with that.
When "support" is hoping that HN or Twitter attracts helpful attention, we're going down the wrong path.
There's plenty of reasons to complain about the things Google does, but this isn't one. This failure is purely on the author.
1. Google had been mentioning this change for a while
2. Target SDK update is a big deal, especially if you don't know what legacy stuff the app was built on, and surprise, can impact OS versions differently.
3. The emulator is not a good gauge of reality. I get this for a constrained team, but if it didn't cross your mind to even think of if Samsung or some other manufacturer has issues, you're showing you've done little Android Dev.
4. Straight 100% rollout. WTF. The other three, I can see some very isolated reasons for not knowing, but you manually have to change the rollout from 20% to 100% when you release. You said nah, I'm 100% sure of this code I don't know and pushed it.
5. The issue was realized after the customer reported the issue, and was almost ignored. Author released the app and didn't bother to look at crash reporting in the console which would have a strong indicator if any fresh crashes. If you gave half a care, you'd have been all over this the day of a release.
I get it if you're a fresh web dev or something experimenting with Android, there will be surprises. And we can complain about Android backward compatibility and play store practices all day (I often do), but this isn't that. This was the mental equivalent of wanting to find out what lives in a hole in the forest by putting your hand in it first. Little to no thought of consequences or what a professional would do.
I don't care if you don't like mobile, you're telling someone you know mobile enough to maintain their apps. You don't. This is negligence to a degree I'd be worried about getting fired, if not also sued over.
I'm sorry, but this is on the developer / maintainer.
Google has been putting out these warning messages for a while, and any decent Android developer should know about target SDK versions.
Didn't test the app on the platform they were updating to?
Didn't function test the app on a physical device that surely had on hand?
Somehow this is Google's fault?
I dislike the stronghold Apple and Google have as much as anyone else, but this is just shoddy maintenance and you've effectively advertised that your "agile software company" can't update an app without proper process, lack the ability to keep on top of well-defined changes within Android app development, and to top it off have the gall to claim Google is being unprofessional?
The point being made by the author is that the app didn't actually need maintenance as far as its developers or users are concerned, which is a fair and reasonable complaint.
In theory this kind of "controlled" backwards compatibility break is good, it lets old apps work whilst encouraging developers to keep up with platform changes. In practice it has the nasty side effect of preventing the scaling of the software industry, and with it, industrial society. Therefore Google should drop this policy and allow old programs to be distributed forever.
With near perfect backwards compatibility, the amount of software that society can consume is in effect unlimited. The number of software developers is finite, but the amount of software they can create isn't: only the growth rate is finite. Thus society can benefit from an ever-expanding library of programs which increases overall wealth. Indeed, tool creation is the only way to increase wealth in countries with a flat or falling population (productivity * population = gdp, more or less). In this mode, software is like knowledge. It accumulates and compounds.
With imperfect backwards compatibility you are forced to engage in continuous maintenance of all existing software. Suddenly there is now a fixed limit on how much software society can have, it's a function of how many maintenance developers there are. You can literally reach a limit where things can slide backwards. Problems can become un-automated. In this mode, software is more like oil. It can run out.
Google want to force continuous maintenance because the Android team justify their existence with constant change, and if a platform is full of unmaintained apps then it will feel old and tired compared to a platform full of apps that have the freshest new looks and features. But that often doesn't matter, especially for non-consumer or specialized software used only by a small number of people (but for high leverage impact). Because Google and Apple are unapologetically consumer focused cultures, they care far more about things like how apps look and stuff that's irrelevant in business contexts (consumer privacy).
Google is preventing new users from installing the app for the first time, on a device that is substantially newer than the app they are installing.
Anyone who has already "purchased" (may be free) the app will still be able to access it, and anyone on an older device can still access it.
I the simplification of this that you specified is ok as a summary, but the devil is in the details, and the details here do make this policy much less impactful than your summary suggests.
Yes, exactly. It is a "controlled" break. The OS is capable of being backwards incompatible but store policies create an equivalent of it being not backwards compatible.
Kudos to that user reporting the issue, sometimes when I report issues I am 90% no one read those.
> There's nothing we as developers can do to speed up the reviewal process nor contact Google support in any way. There are no possible workarounds and we just have to wait. Wait until we're excused to put our fixes to production.
A couple years ago, and out of self-learning process, I decided to develop an Android app, never did any smartphone app development, Flutter was new so got excited to try it out, long story short, I submitted the app to play store, and it remained “under review” for 40 days! Just like OP, Next day I was checking the dashboard, silly me thinking the process is smooth, after 3 days I stopped checking and later forgot about it until they sent an email after 40 days, I pulled the app later and decided never to touch anything again with smartphones app development, and that was play store, supposedly the easier one after reading some horror stories of the App store.
> I'm not even sure why are we, as developers, allowing this to happen — there's usually not any good reason to develop mobile applications at all anymore. It's time to move back to open (web) standards and take control back into our own hands!
In general yes, but if I look at the apps on my phone I have
A mail client, K9, old UI. This clearly can't be replaced by a web app because I'm using it to look at my mail in a few POP3 servers and then I'll download those messages on my laptop (Thunderbird)
OSMAnd, offline maps and trop recording. Can't be web based.
Password manager, with a local dB synced with Syncthing.
Syncthing.
Epub reader, from files stored on my phone.
Photo gallery, for files on my phone. I backup to my laptop with Syncthing.
Banking apps, that I must use for 2FA in the web sites of those banks.
A network scanner, useful to debug networking issues.
Etc.
Of all the apps that are currently installed on my phone the ones that could be web apps are:
Yeah, but if the phone/browser decides to delete some of the PWA's code to save space, you won't be able to use any maps at all until you connect back to the internet.
Before you say "just download the code to your phone as a file", I'm going to assert that's exactly what an app is ;)
Interesting, Apple allows for expedited reviews in case of a critical bug or something time sensitive. IIRC you can demand that twice a year. I assumed Google would have the same.
To add more context, the [1] policy has been around since last year.
> Today, as part of Google Play’s latest policy updates, we are taking additional steps to protect users from installing apps that may not have the latest privacy and security features by expanding our target level API requirements.
Starting on November 1, 2022, existing apps that don’t target an API level within two years of the latest major Android release version will not be available for discovery or installation for new users with devices running Android OS versions higher than apps’ target API level. As new Android OS versions launch in the future, the requirement window will adjust accordingly.
These minimum SDK requirements have been known for a very long time. It's correct that the email only got sent recently, but the requirements are usually announced 2 years in advance in this manner:
After 0 days: New Android version comes out
After 1 year: App Updates need to target the latest Android version
After 2 years: Apps can't be downloaded on devices with the "new" Android version anymore unless they target the "new" Android version
So normally there should be plebty of time to prepare. However, if you're an indie developer or not actively maintaining the app, then it definitely is annoying, since usually the apps would work perfectly fine on the new Android version without updating the targetSdk version.
Many companies, institutions, public bodies hire external contractors to do a one-time development of a limited-scope app. I bet a bunch of them have their Google Play Console dev account email read by people in some engineering infrastructure function who know nothing about Android development.
Updating their apps now means an internal scramble and unexpected project costs. Not saying its wrong or that we shouldn't move forward. Just saying many were complacent and contractors may have good opportunties in this space.
This process has been in place for literal years and every professional Android developer part of such commissioning would know about it. Just like every iOS developer would know that apps need to be maintained for new iOS releases.
Its actually much worse than what the author wrote about - he/she just messed up execution of roll-out.
If they decided, the app was really not worth it, they could have unpublished it. However, even unpublished apps can jeopardize your account's standing by running afoul of a policy, which Google may have concocted long after the app was written.
Unpublished apps in this state can get your account permanently banned.
You have to support your apps for the rest of your life (if you care about your account).
Same experience here - forced to update a stable Android app due to minimum API requirements. Uploaded to the store - app does not crash but does not work as it does in the emulator. Customers pissed.
I mean, this is pretty much the paradigm for everything now. Angular versions go out of LTS in about 18 months, Android apps need an update in about the same timeframe, etc. Makes COBOL look like a dream - build a functional app that just works for 20 years.
It's a similar situation with Chrome extensions. Extension reviews are generally pretty quick (an hour or so), but every once in a while you get hit with a longer one (days+).
In this type of environment, you need to ensure every release is as rock-solid as possible. For our extension, we have beta extension with a sub-group of opted-in users that we test on for a week or so before doing a production release. Then we roll out the extension to production incrementally starting with 1% of users and slowly ramping that up to 100% (it seems Android has a similar staged rollout feature).
Good point, but as I also wrote in the article then one and only reason for doing anything was an API level deprecation e-mail from Google less than 3 weeks from its deadline and it states that review could take a week (never happened before with me though). If I would have noticed the "extend to November" immediately then it would have left more time. But yeah, I'm not saying that this situation could not have handled better, I'm saying that as soon as you make a mistake then there's nothing you can do. And mistakes will happen, even when testing thoroughly with every Android version.
Oh yeah all my old apps got taken down by Google because apparently, they don't like that I don't update my apps for no reason whatsoever, even though they were offline games...
Hackers and scammers get more advanced. Vulnerabilities are discovered with time.
Users expect more security today than 10 years ago.
For Google & device manufacturers to be able to guarantee that level of security, they need to keep their store updated. And the apps on there shouldn't be using outdated insecure APIs.
If you're not able to keep up with that, then you shouldn't have any expectation of getting new downloads on their platform.
Agreed about conflicting message. And even better - e-mail was pretty vague and had to Google a separate link (https://support.google.com/googleplay/android-developer/answ...) for more info. If I could go back in time then I would press that button, which asks for time until November, because as happened to us - it was not possible to go back to previous version in any way.
Yeah you just saved me; was working late last night to get all updated. But still some issues; and was worried about testing. So thank you. Sorry you didn't see it on timme
If you target SDK < 31 then you’ll see reduced availability on the store. That means users with phones running Android higher than your maximum targeted version won’t be able to newly acquire it. Existing users are unaffected.
If you target SDK < 33 then you need to update to SDK 33 because you won’t be able to make releases with the old target SDK anymore. (Existing bundles are fine). If you miss the deadline it doesn’t introduce further enforcement than that but it might mean if you had an urgent update you’d need to update the target SDK at that time. So sooner is better.
I shudder to think what happens in another 5 years or so when the min target is bumped. We have an app built with about 6-8 Jetpack Compose libraries that is already flaky (even after using BOM) and only a magic combination of library versions ensures the app runs across SDK versions.
If you think developer libraries provided by Google for Android are stable, battle-tested and you feel this is akin to first-party "platform code", please be advised that it can be a hellscape with you fighting Google's build tools and the IDE/dev env on one side and zero help on SO to debug device-specific issues with lengthy stacktraces on the other. You end up landing on an Android issue-tracker where some kind soul mentioned the device name, only to find that issue in limbo asking for a working reproduction.
This is a problem with software culture in general. 99% of us refuse to declare an application "done". We're always working on the next iteration, over and over forever. Cramming unwanted features, changing the UI over and over, updating dependency 2.8.1 to 2.8.2 and re-building. Most of it is change for the sake of change. The app stores obviously believe in this endless treadmill, too, and enforce it: Since everyone endlessly changes their software, you, too must endlessly change your software.
If you don't believe this is a deeply ingrained culture problem, try proposing to your manager "Hey, this software is already really good and customers love it. Let's make this version the last one we release. We can work on something new." See what they say.
> If you don't believe this is a deeply ingrained culture problem, try proposing to your manager "Hey, this software is already really good and customers love it. Let's make this version the last one we release. We can work on something new." See what they say.
This description is rather an argument that this problem is deeply ingrained in management culture, and not in software culture.
> 99% of us refuse to declare an application "done".
The article is literally about how they are not allowed to declare the app "done" because google is forcing them to upgrade to a new version. I'm sure the creators of this app wish they could just never touch it again.
Based on the crash behaviour, it sounds like Google never bothered trying to log in to the application if it passed review.
This also makes me wonder, was the app already crashing on Android 13 before updating the target SDK version? I suppose the backwards compatibility layers must've kept the app alive on modern phones?
Personally, as a user, I like that Google forces developers to update their apps, because the old API designs were terrible for privacy, but they won't be enforced unless devs update their targetSdk. Breaking installed apps is not an option, but forcing the issue in Google Play works well. I'd also say their testing times were reasonable, if it weren't for the fact that they don't see obvious problems such as "app was never even tested on Android 13".
As a user, I’d rather that Google make stable and coherent APIs the priority.
If the strategy is:
1. Make breaking API changes, but gate them behind the targetSdkVersion.
2. Force app devs to lift the targetSdkVersion to stay up to date by gating device access in the store.
It has a few effects:
- Developers can be slow to move. There will be zero million new addressable endpoints initially, and customers may similarly build an expectation of new devices being problematic/best-avoided.
- QA burden increases for split behavior *forever*.
- The play store becomes an integral element of the privacy and security picture of the device, something that should be the OS’s job.
What would I rather? I’d rather an API evolution plan that doesn’t rely on targetSdkVersion as a means of controlling behavior of APIs. It’s an attempt to have one’s cake and eat it, too, and it clearly falls apart, anyway.
I’m wary of any plan that basically amounts to needing people from other companies to do something in order to succeed.
The stable and coherent APIs Google wanted to make all the way back in Android 4.4 was met with strong resistance by power users and data collection companies.
Many API changes are either extremely minor ("set this flag if you want to keep the worse, legacy handling behaviour") or extremely important ("actually ask the user before you drain their battery tracking their location in the background"). The new handling of notifications (requiring explicit permission) is also a godsend.
Google hosts and distributes apps effectively for free (let's be honest, how many Android users actually even pay for apps). The least you can do to keep the app running is to make it work with the modern API once every three years, with an announcement about the API bump requirement a year in advance.
Before Google did this, the Play Store was filled with crap from the Android 2 era that crashed on startup. Paying the one-time $25 developer fee and chucking an APK over the wall isn't exactly a good way to create a decent app store.
You don't have to comply with Google's desires if you don't want to, but you'll have to host the APKs some place else. Termux has had to do this, unfortunately.
Apps that don't get updated can't be easily installed on new devices. They'll still work and be easily installable on old devices (matching the targetSdk version) in case you rely on an Android 9 tablet embedded into your POS.
I think it's entirely fair to expect a yearly "is everything still okay" check from application developers.
> Google hosts and distributes apps effectively for free
Google rakes in money from all the in app sales, store sales, and subscriptions. Further, google is often the provider of in app ads. Not to mention all the money they get from advertising with google searches and adwords built into (practically) every android device.
And, let's be frank here, a 1, 20, 100mb APK isn't exactly a huge amount of data to host. Especially since there's almost always a linear correlation with the popularity of an app and the amount of revenue google makes from that app.
They aren't exactly operating a charity here. The least they could do with all this money flowing in is support backwards compatibility on their platform.
The cost of having a thorough inventory of apps that being in monetization is vending apps that have none. It's part and parcel of keeping users inside the walled garden.
Google takes a hit on some apps to sell and advertise with others. What's hurting Google in this case is the release of an app update that is bad for customers and bad for the business. While it's just one papercut, the "huh, my iPhone version works just fine" razzing will ring just a bit more true for customers who got bit by the update. Do it enough, and they'll get tired of the hassle and jump.
It's a rough balance, building something scalable that also holds high quality, and separate policy decisions can sometimes lead to truly Kafkaesque journeys for developers and users. It's at least instructive for others. Making and evolving APIs and ecosystems is hard. Think very carefully about the holistic impacts of decisions.
Google’s review process is weird; I’ve had application crash fully. And see in the logs it also happened to the Google review phones. Yet they still approved it.
It was working with Android 12+ before without problems. Upgrading API target level made some flags mandatory (for PendingIntent to be specific), which throw RuntimeException. But as I also wrote in the article itself then this is a legacy application, which does not seen any active development for more than a year and tasks like these need to prioritized instead of other active developments and since app was working after testing then it didn't seem to be that a big of a deal. However, should have been tested on newer Androids too. But even when the problem happened in production then I was not that worried since Google reviews usually take few hours max. The problem is that it has been taken so long for now and there's no way to cancel/pause/delete latest faulty release. That's the scariest part.
Not exactly a fan of Play Store policies, but for this situation, I'd say that Google gave ample time for developers to migrate to API 33, and all behavioral changes and API deprecations are documented (however annoying it is).
The one and only reason I switched to web dev is the google they suspended my 4 years old play console account with 3 apps total download around 1.5 million they say I was Associated with an another account and they didn't provide any details about the other account I worked as a freelancer for almost 20+ app products it's hard to left android development and get In to web dev but I did I don't want someone to destroy all my 4 years Of work one email without any reason
the current system is already like this if you want to do sideloading/jailbreak/root of your devices. But that was the idea behind these stores: 'don't be like Microsoft and get viruses, we will make sure nooo malware ever enters our stores", which ofc is such a BS.
As for the developer in the story, he could've just make this a .pkg to be sideloaded by their clients, circumventing Google altogether.
TLDR; Written an article about a real-life case-study about Android app deployment/development problem where production version has a critical problem and update has been "in review" for 72h+ and there's nothing else we can do.
If there's some (ex)-Googlers who could help to speed up update approval process then I would be really helpful, if not then let it just be as a warning for anyone else being involved with mobile app development.
Behavior change hidden behind targetSdkVersion is a bear trap that keeps on providing trauma. It’s just a massively dangerous way to evolve API’s.
That said, automated regression testing, target environment deployment tests, and beta application groups are your friends. Yes, they cost money/time, but escapes are the Jack in the box cost of not having them.
Real life: Assume you are critical infrastructure of the whole world (as in people are in risk short term and thousands literally die in the long term because of potential delays
... as part of your day to day operations without special external event) and you need to run apps with proprietary third party sensors on millions of end users devices. Right now you run it with provisioned Android devices but users obviously complain about the lack of byod and customers about the costs. There is no way to have a risk free way of providing this service on Apple or Google stores even with perfect own behavior.
App Stores and their procedures are a risk to public health. Period.
"I don't know about you, but that doesn't sound professional way to solve any problems."
By unprofessional, is he talking about his failure to test a build intended to run on the latest Android devices, on the latest Android version here?
If not, I don't understand the problem, one shouldn't expect others to prioritise reviewing their work especially as a build already had attention within 24 hours (initial review).
I hope he took this as a lesson learned to adequately test production code for an environment he really has no deployment control of or final QA sign off.
One has to wonder if these “review” mechanisms are not really engineered like a Skinner box, where the developer is the pigeon.
I honestly cannot decide which is worse: app development, or the web ecosystem as a whole. But at least on “the web” you can push fixes at a resonable speed.
Am I correct in understanding that if you have an Android 11 device, you will still be able to use existing apps that are updated but not new applications?
Unfortunate for devices like boox ultra or the c version which were launched this year but unlikely to ever be upgraded to 12+ probably given SOCs.
This occurrence happens more often that you think if you are working on Mobile Apps. You just have to be extra careful with testing and verification, since the mistakes can be costly.
Successfully updated my 10+ year old Android app to latest SDK. Of course, there are still loads of errors about deprecated function, but it seems to be working, as it is pretty simple application.
Interestingly, Google apps (1P) are subject to the same review process, and can also get hung up. I experienced 72 hour waits to roll out an APK update.
First of all, new versions means its covered by improved security & privacy functions. Everybody should always upgrade to highest possible version. Google has nagged devs about this for YEARS.
Also, if you never update your app and consider it "done", you are probably ignoring security erratas in your libraries
Finally, who maintains an Android app, makes a major upgrade but doesn't have an Android phone to test it before pushing the update??
But how does that help end-users? They would still need to know that they need to use different store etc. It's a pain. At least Android allows (relatively) easy side-loading APK's for the most critical situations, but for iOS that would be a disaster (not sure if Apple's App Store has a way to delete/cancel/pull-back faulty release though).
Honestly this is really not a solution for anyone making a living off apps, or working on apps in a professional setting. You simply reach much fewer people and certainly a different audience too.
I've started frequently getting reviewed (from submission to approval) within hours, sometimes even within 10-15 minutes of submitting to Apple App Store. But it still varies and can take a day.
I want to figure out a good cross plat strategy but would not want to ever deal with Play and Google.
Our average in the Play Store is about an hour. Our average on the Apple side is a bit longer, but we've been blocked by them on multiple occasions for over a week while we turned out to be in the right.
"First idea was to roll back to the older working version in the Google Play Store so that only users who were running latest Android and had the latest version of the app would be affected and then deal with that problem in a proper way at the next day. For my surprise I found out that this is not possible - there is no way using Android eco-system to pull back or cancel latest release."
Perhaps this is an example of "anti-rollback technology".
"I personally have been against developing mobile apps for years now for the exact same reasons described in here and other similar articles - as soon as you decide to develop mobile apps then you give control of your product/service away to a third party, which you can't replace when problems happen."
What is an "app store". Computer owner cannot install any software they want on their own computer. Computer owner must select from pre-approved list provided by third party. The word "store" is misleading. 95%+ of the programs in the "store" are free. If the figure 95%+ is incorrect I apologise; I can get the exact figure. This has been publicly disclosed in litigation.
These concepts can seem outrageous to anyone who has watched computer and internet use go from uncommon to common. There are HN commenters who want to pretend they are totally organic and perfectly fine. Hypernormalisation. Beyond question. Right.
Maybe for those who were born into a world where computers and the internet are being dominated by a handful of online advertising services companies calling themselves "tech" companies, ideas like
(a) "you cannot use a previous version of this software that you got for free or paid for; an advertising company will protect your privacy" and
(b) "you must select from the following software chosen by an advertising company to run on your computer"
are an easy sell.
Those born into this hypernormalised environment are actually a minority of the population in the USA. For example, 66% were born before 1999.
We can change this "anti-rollback" and "app store" BS. (All falsely justified in the name of "security" and/or, ironically, "convenience", two mutually exclusive concepts.)
The personal computer belongs to the person who bought it. They can use any software they like, any version they like, and they can write their own software to run on their own computer, without any payment to a third party. This was once where things were before the so-called "tech" companies threw a monyewrench into the wheels of progress.
I do not sympathise with people who build apps for these evil stores. Google and Apple can only deliver value in their app stores because of you tending their gardens. As developers and businesses we must stop tending their gardens. Build on open platforms.
> I took the approach of better safe than sorry and prioritized this task even though it would add no business value — it was only required to be completed because Google said so.
Lol, maintaining the app provides "no business value." More like "If it doesn't come from above, it doesn't exist."
You might want to contact the Google developer in charge of the Android Java as he was originally an outsider, i.e. he authored the legacy approach to handle legacy APIs and shamed Google to do a better job of handling legacy APIs with new android SDK features.
digital jail evolved: from msft/apple digital jail model to open source (SDK included) google/meta/etc digital jail.
The new digital jail model is based on open source grotesquely and absurdely massive and complex software (and more and more private protocols), SDK included (c++/java syntax).
Defense: simple and modular software(SDK included, write simple and plain C, not c++)/protocols, but able to do a good enough job, stable in time. Benchmark: not 47398437829 modules, and each modules could be coded by one normal developer in a reasonable amound of time.
Ideas: to move from one module to another, proper URIs: irc[s]://|ircs:,mailto://,http[s]://,etc. We miss a really simple video/audio conf protocol, I mean REALLY simple (TCP based). Crypto-based authentication and end-to-end crypto will add a lot to do from those client programs to make it easy to use.
We can imagine a One Desktop Application handling most/all of them, optionally deferring some protocol handling to external apps (Basically, what is doing current web engines the wrong way).
Personally, I voluntarily built and run open source apps for 16 different cities for their transit system. This gave me two weeks to update 16 apps, for no benefit of anyone. My app is a PWA, and the Android version just uses cordova + a few plugins to add a few native options. Unfortunately, updating cordova to support the new target android api broke some of the plugins, which haven't been updated yet, so it ended up being a full weekend of work and testing.
Truthfully, I'd prefer to get rid of the app and just have users go to the website and install the PWA, but the average user still doesn't know how to do this. And the Play Store is still the first place users go to find apps. If google would just allow submitting a PWA directly to the app store, that'd be nice... I am not looking forward to doing this yearly.
Professionally, we are also scrambling. We have a legacy app that some supported customers are still using until the end of the year. The app is a fairly complex application, and basic testing has already shown that just changing the target api version has broken quite a few things. We have gotten the extension, but we know this will take 1-2 weeks of developer time + 1-2 weeks of QA's time, for an update that does nothing but appease Google. All for an app we are going to officially remove from the store at the end of the year, once all customers transition to the new app is complete.