Support forums for free tiers are the absolute worst. With some exceptions, the general pattern seems to be power users hanging out and answering questions on the company's behalf. Who are these people?
Another support anti-pattern is an untrained offshore support team whose entire mission seems to be to close the case and move on. There is little in life more enraging than a support rep who refuses to engage logical brain and escalate based on clear evidence of a technology issue.
At my most successful startup, we had an extremely smart guy in the support seat; he was eager to learn from the dev team how to triage, and so we'd get bug reports complete with network tab screenshots, api paths which had caused an error, etc. He could easily have been a skilled programmer had he chosen to pursue it. When we were acquired, he was merged into a larger team that had much less autonomy, and his morale tanked over time.
The zendesk ("here are articles that might be appropriate") approach sucks too, as do chatbots.
Perhaps fear of hallucinations is holding people back, but it seems like an LLM based product could take over the "keep it cheap" support vertical. 1) Have people file a report in their own words; 2) synthesize an answer for them, but only if the LLM identifies with high confidence their report as being a known problem that has an answer; 3) cluster reports from users based on salient features and escalate new clusters to the tech team; 4) and if the company can afford it, route everything else to a human team who can categorize, click a few buttons, and send the LLM back with an answer, or escalate.
> Support forums for free tiers are the absolute worst. With some exceptions, the general pattern seems to be power users hanging out and answering questions on the company's behalf. Who are these people?
In my company, we have free support forums, email support for the customers who jump through some hoops (to make it slightly harder than searching the forums), and phone support for select clients. But the "power users" in the forum are just our internal customer support staff operating a handful of sock puppet accounts, and it's the same people who answer phones and email.
The forums are a great searchable historical record. The email support is versatile and scalable. And the phone support is a non-scalable value-add that makes our top tier clients feel very special.
>The forums are a great searchable historical record.
Wrote a whole blog post about this: https://www.mooreds.com/wordpress/archives/3451
Many people have a drive to solve their problems themselves, and if you make content available to google, they'll find it. It's a fantastic way to self-empower your users and customers.
We even have an internal slack channel where I'll post questions and answers from the other support channels (email, slack) with a view to making them generic and turning them into forum posts. I don't get to that task often enough, but it is sure fun to do when I have time.
I also know a "support guy" who absolutely kicks ass. Probably a better dev than most of us regular product engineers. Makes me think support is actually way harder than the actual app dev.
I also wonder if the incentives are a little bad with this setup. Product devs' crappy code turn into tricky puzzles for our support team. Legend has it, RAD Game Tools used to have just 1-2 devs on each tool, and you'd do the support for your tool. Feels like this would pretty strongly encourage you to make your stuff good. And you'd have everything in your head too. I wish I knew of more similar setups in the web world.
I think it could be possible to feed the LLM with lots of data about your product, potentially even give it source code with instructions not to share it. Then have it act as a highly knowledgeable support rep for customers, and also be able to file detailed issues for the dev team when it identifies that a customer finds a bug. The risk of hallucinations is still there, but it could probably be mitigated by feeding the LLM's output into a fact-checker model. I think the bigger risk is leaking internal data through prompt injections
We engaged a startup (kapa.ai; they're great) to train an LLM on:
* our public facing docs
* our public repos
* our youtube channel
* our forum
and spit out answers. It's not right all the time, but it is most of the time and, crucially, it shares not just an answer but a link to the docs that led it to that answer. That makes it easier to fact check the LLM.
We've found it very useful for scaling support and making that corpus of knowledge available to folks both outside and inside the company.
> power users hanging out and answering questions on the company's behalf. Who are these people?
It's genuinely strange. People will even do it for banks and phone network companies.
As a company, all you need to do is have XP tiers for posting and someone will be willing to dedicate their life to becoming a Finco Bank Guru and running the forum for free. Usually they have a rather "Boomer" energy, so I get the feeling it's retired people who think the parish council/HOA doesn't meet often enough.
I’m sure I’m an outlier, but I did this once for a particular network equipment vendor. It’s a smaller vendor that aims at a low price point and skimps on things like support, but their software had a particular feature that was going to be very important for my employer. To get good at troubleshooting their gear fast - we were going to deploy the equipment for scenarios where mean-time-to-resolution was going to matter a lot - I camped on their forums and helped everyone I came across as practice matters a lot for that. After about year I’d achieved that and stopped, but I maxed out their karma rankings in the process. They didn’t have training material or certification programs and the equipment didn’t break much for us in production so this was the only way I could think of for getting the hours in.
On one hand it’s unfortunate that people don’t get paid for this kind of help and perhaps act in a short sighted way (giving away too much control to the provider).
On the other, a lot of these are people willing to help someone else for free and make their knowledge practically public in the process. Seems aligned with a hacker/wikipedia/oss ethos in many ways.
Yeah, until you realize that they get all that XP posting some variation of "turn it off and on again" ten thousand times
A long time ago I supported hp scanners on windows. I exclusively told people to uninstall and reinstall the drivers, or windows. My success rate was 100% and I outperformed all my peers who did actual troubleshooting in time to resolution and number of resolutions.
Sometimes the right answer is just the right answer.
As long as their comments are useful, I don’t mind. Many open source projects are low quality or trivial to others - but still provide big help to some.
Is there some reason Boomer isn't yet seen as offensive as a racist word?
Your comment isn't the worst, but the word is usually used as a placeholder for some bigoted stereotype against someone retired. Disclosure: I'm not a boomer and the word is not used much in New Zealand.
It's definitely a lazy slur, in the same way that "problematic" is a lazy complaint.
Why reach for a vague zeitgeist term that everyone knows, but no one has a precise definition for, instead of taking the time to use a specific word?
I mean, one reason is that no one has ever shouted “boomer” in a hateful voice while lynching someone for looking their way wrong.
We could also talk about how it’s usually not just used as an ageist slur but to describe a state of mind - if you ask the kids there’s plenty of old people who aren’t boomers - but saying it’s as offensive as being racist is completely inappropriate.
To sort of quote the comedian John Mulaney about the n-word, if one of the words is one you can’t say and have to use “the n-word” to describe it, that’s the bad word.
> no one has ever shouted “boomer” in a hateful voice while lynching someone for looking their way wrong.
That "while lynching" part is doing all the work here.
...really?
Many of the tiered support offerings from companies also doesn't make sense from a customer perspective. My favorite example is Elatic Co. trying to sell us support. We want one thing, and one thing only: "When this fails, in a way where we can't fix it using documentation, experience or the support forums", we need to be able to call someone. Everything you're offering below that is useless to us. We just need the phone number in the exceedingly rare case that this break down completely, everything else is not something we'd be willing to pay for. But the "call someone, who knows something" is the final tier, and includes all the things below it and is priced accordingly.
Understandably being able to call a developer should be expensive, but when you tier it, then customers feel ripped of, because they have to buy something they don't need, to get that one thing they really do need and are willing to pay for. In our case we where willing to pay some fixed price per year for access to phone support, and then an hourly rate if we called, but that wasn't the pricing structure.
In fairness the support structure and pricing from Elastic has improved since then, but the issue is the same, if you only need one thing from the higher tiers, then the pricing doesn't make much sense to you as a customer.
Would you prefer they only offer the call option without all the other stuff for the same price?
Then you don’t have the problem of paying for things you think you don’t need.
Your problem is that you want the highest value (direct on-call team with actual skills is expensive AF) for the budget price - as someone who helped design support tiers this is super common but obviously not economically viable.
edit: in any case on that price level the price structure should be more flexible I agree
This is not exactly how it works, but as a custom you look at the price of the tier below what you want, and the price of the tier which contains what you want. The price difference must be what the feature you want costs.
So if the tier with which provides on-call is $100.000 and the tier below that is $75.000, then the cost appears as $25.000 to me as a customer. That not true, when you dig into it, some of the cost is covered by "unused" benefits of the tiers below, so that one benefit on it's own might be closer to $50.000. If you truly don't need the stuff below, then you'd want more flexibility in the price structure and be able to spend the $50.000, rather than the $100.000. Instead the provider now ends up with no sale.
> direct on-call team with actual skills is expensive AF
More than most realise. My issue is that there are occasions where you don't mind paying for that service, but you're not allowed to, because the tiers are designed so that it becomes more expensive than it should be. I've seen a number of support contracts, where they only tier that actual provides much value is the one where you can call an expert, but that's hidden behind "Call Us" or "Super Premium Platinum" priced such that only governments and VC funded companies can/will pay for it.
As a developer, doing one or two support cases per day has been one of the best things I have done.
It really helps me understand what they’re doing better. It short circuits a lot of bureaucracy. And it’s great when you can push out a change for someone an hour after they wanted it for simple things. So they know that you can make changes fast to address their problems.
And it’s just nice to get to know the users and be friendly. We get to chatting. And it just makes everyone more chill knowing that the developers care.
We make B2B software, and we have roughly the setup described in the article, where L3 are the devs.
I don't recognize what the article describes at all. There are tons of questions answered by L1 which are not even issues with our software. And us devs certainly feel responsible for what we produce, and we still talk to our customers when needed.
So I think it's more a culture and possibly management problem than simply layering.
Years ago I worked for a telco. As part of trying to identify recurring problems and design possible solutions, we would sit in on support calls for a few hours a day, for a week. While we did find a few things that could be improved by adding more software, or reworking solutions, that wasn't what was needed. 90 - 95% of support calls where related to technical issues, which required first level support to create a ticket for technical support, or escalate to them immediately. First level support was pretty much not needed. It would make more sense to reverse the call flow and have technical staff pick up support calls, then transfer to first level support if the issue turned out to be non-technical, which it would be in 5% of the cases.
To this day that is still my issue when call ISP or telco support. Why am I trying to have some 19 year old, with access to nothing but my billing information, trying to solve the issue of my router not connecting?
The type of first level support most technical companies offers could be solved by better monitoring, better self-service and well written documentation.
I don’t necessarily think tiered support is the wrong way to go.
But as someone developing a product, sometimes when you’re closer to your users you find that many users get told to RTFM on a particular feature and maybe the actual problem is that the feature is unintuitive.
But I get feedback from L1/L2 when some feature generates a lot of support requests, and I've many times sat down with support and even called up customers to improve such features.
That said, I agree with the sentiment that as devs one shouldn't be far removed from customers. For example I do take time every now and then just looking at support working if they've got a customer showing an issue or similar. Or just hang around to hear what they complain about.
This is of course much more difficult if support is in another building and not just down the hall.
There are ways to deal with that without putting devs on the front line though.
I don’t even get the point, are Facebook developers supposed to deal with 500k support emails per day because grandma forgot her login? That’s unrealistic for any product with more than maybe 1k users.
We've come to expect zero customer support from the likes of Facebook and Google, but why does that need to be the norm ?
I didn’t say that you should have zero customer support, but I don’t think the level system is the problem. It’s a problem when the levels don’t properly escalate stuff they don’t understand. But you just can’t have every support case go directly to the developers because 90% of support cases are stuff developers can’t help you with.
It’s like saying every person in a hospital should directly go to the operating theater. The 10% people needing urgent care would like it but it’s a very inefficient way to use the time of the staff. That’s why you have a reception (Level 1) that’s triaging cases and checking if you even need a doctor.
> It’s a problem when the levels don’t properly escalate stuff they don’t understand.
This is the real human psychology trap. By definition, L1 doesn't understand anything it can't handle.
Unfortunately, at many companies L1 is also yelled at for escalating too many cases.
Consequently, the internalized message for L1 to not get yelled at is "Juggle the ticket you don't understand convincingly, until the customer abandons interaction from frustration."
A few anti-patterns I've seen in most support shops:
1) Bad KPIs / no post-hoc sampling and review. Without regular, random, higher-level sampling and classification of how tickets were handled, there's no signal that support is technically botching a large % of incoming tickets.
2) Disincentives to escalation, even when it's a true positive escalation that should be. IMHO, support should be incentivized to escalate true positives.
3) Failure to upskill support agents and build mix-of-expertise within the support function.
4) To 3, failure to pay support for the value they're creating. In a sane world, a few of your highest skilled support people will be paid like developers, because they are.
The exception being that the triage is a queue-based system, and eventually it actually handles your case: you show up at the hospital, describe your problem to a human being, and you're assigned a priority in the queue pertaining to the specialist that needs to see you. If you're not dying you might have to wait a few hours, but you can rest assured that you will be looked at by an expert -- even if you're a hypochondriac.
If your case is an emergency, you can cut through all the queues and go straight to the operating table.
Whereas most L1 support systems are either non-existent; a FAQ loop that gets you nowhere; or a Dialogflow/GPT chatbot with 0 reasoning skills, no technical knowledge, and limited domain knowledge (really just a more convoluted version of the FAQ loop).
If you're lucky, after some digging you might find a hidden link that lets you talk with a real human being.
These systems are deliberately set up to massively increase the effort required to get to talk to a technician, as those are a limited resource and companies don't want to spend money on stuff that would actually improve their internal processes.
And if hospitals worked like this then the majority of people showing up for any problem whatsoever would die before receiving care. This is not how triaging works.
Both Facebook and Google provide customer support. I think people are just confused about who their actual customers are.
Hint: It's the advertisers.
Sounds like you have not bought ads. No, they do not provide support there either. Maybe if you buy tons of ads, but not for smaller advertisers.
Still a tiered support system: last year I started a side project and I wanted to advertise on Google. Created the AdWords account, with all the legal documentation and everything, it got banned in two days. All the support I could reach was bots. Luckily I had some friends at Google who could escalate internally....
I bet they still have a level system for the advertiser support, there’s no way an actual developer reads the first message. Maybe for very large customers they have a dedicated support line, but not for a small business advertising on google.
One thing is customer support and the other is user support. There is few to none of the latter.
Regarding the former, my company pays for google workspace and an agent happily answered our questions about the recent less secure apps password phasing out process.
I can only imagine the author works for a B2B software priced at least $5000/seat/year. Or a software that practically has no user. These are the only two scenarios where "let customers talk to developers directly" could ever work.
There is another pitfall in the layered support structure: how would you hire, train, or otherwise produce L2 support engineers?
You cannot hire someone directly as L2, as there is a learning curve for company- and product-specific issues, and a freshly hired "L2" would have to learn. However, they are shielded from the most frequent issues by a hoard of L1s and thus cannot learn about the typical problem areas or troubleshooting approaches.
So they have to either graduate from L1s or accept a downgrade from being a developer.
> You cannot hire someone directly as L2
We've managed to do that by recruiting from our customers. During support cases and such we'll notice someone is more technical, and then at some point approach them with an offer.
This way they already know our software well and can get fairly straight into a L2 position. Of course in the start they'll get some more L1 stuff and such, but it significantly reduces the learning curve.
Customers aren't too miffed as the alternative would almost always be that they'd go to a competitor.
There are usually a number of people in L1 roles looking to transition to more development. You can have them be L2 by having 1/3 of their time spent contributing to bug fixes and small features of the software so they get to know it better.
I dont think they would have to come from L1 from the same company / project though. Experience does allow one to learn faster, communicate more efficiently, etc.
I think it's also good practice to have devs rotates on some small shifts of L1/2 so they don't get out of touch with the reality of maintaining their stack.
Business Analysts can fit the L2 role. Where I work, they are the interface between the end users and the developers. They usually come from the either the business side or the support center. And if they came from the support center, they usually have prior experience as an end user. However, I can see how that wouldn't work out for all industries.
To me, it looks, like this is contingent on the kind of product they need to support. For instance, in storage business, L2 support are often paid more than developers (and are very hard to find!) They usually come from the development side of things, attracted sometimes by being hired in a different capacity (eg. being a consultant running their own consultancy business rather than being a direct hire), some like to go on business trips (this kind of work often involves being on the customer's site, physically plugging and unplugging equipment etc.)
In medical s/w development, L2 can be nurses or doctors, they might be attracted to this position due to relatively low load (compared to clinical medicine), flexible schedule etc.
I don't think there is / possible a single answer to how to find these kinds of specialists.
Isn't this tiered support a necessity when operating at scale? How can you help thousands of customers where faqs and getting starteds don't cut it? You need a filtering mechanism (L1) to help users directly without overloading the product team with a standard reply rtfm.
If you're operating at scale you hire at scale. How do you suppose Target staffs their "at scale" brick and mortar stores?
It's triaging the support so the people who are best-capable of answering specific things don't waste all their time answering the stuff a new hire can take care of after one week training.
Two layers is fine, so long as they're allowed to talk to each other. Adding a third causes a lot more problems.
Once you get beyond having a single dev team, a layer of support is necessary: you can't expect consumers to know which of your dev teams they ought to be talking to.
My employer has an amazing support team, who are effective champions for the user when interacting with development teams. They're fantastic.
That's an important requirement of an effective support team: they have to have a seat at the project management's table. They have to have the power to request features + changes from developers, at the same priority level as the rest of the product development team. IMHO it's important both for the support team morale, as well as ensuring a positive UX.
In my experience, the real problem with any support scheme is that the decision makers don't have enough skin in the game. It's easy to deprioritize tech debt and bug fixing when you aren't facing 3am pages.
The obvious solution to this is a distributed team. There can still be some holes (e.g. Jan 1 is, afaik, pretty universally celebrated), but having worked at companies that had a team in the US, Europe and Asia was...really nice with respect to this.
The incentives are frequently wrong as well. The support staff for many companies are rewarded on volume, call-queue length and "resolution time" (not actual resolution time, but how quickly you can get the customer off the phone).
The only valuable metrics, in my mind, are "Did we actually solve the issue for the customer" and "Have we effectively prevented this from being an issue in the future".
The need for devs to have a sense of what's happening on the user side, get support requests and work with the customer support is spot on. It really makes a world of difference.
The crucial issue though is to also have competent CS working with the dev team. It's not easy to have two separate teams share info and collaborate efficiently, it requires skills, and these skills need to be on both sides of the table.
That makes it harder to retrofit these setups in an existing org when these aspect where completely ignored at hiring time for instance.
The article is from the time when CI / CD was relatively new and many felt like it needs promotion / wanted to jump on the bandwagon.
Today, after some practical experience, it looks like some conclusions this article makes are unwarranted. For instance, the idea that the development team should be directly involved in addressing customer issues, if implemented, often results in development plans being ruined, QA doing a lot of busy work, release schedule going down the drain.
Ultimately, the problem is the lack of expertise necessary to triage and to schedule development necessary to address the problems the customer is facing. Low-tier customer support is usually a low-paid position that doesn't require expertise in the product being supported, or anything at all really. Is there a solution that doesn't involve rising the requirements for the support staff (and rising the pay)? -- I don't know. Doesn't look like it. It seems like asking for having your cake and eating it too.
TIL that ".me.uk" is the domain reserved for "personal names"
https://en.wikipedia.org/wiki/.uk#Active
Which is pretty cool, considering that when I registered a domain under ".us" in 1993, I used our street address as the leaf node, and I've seen a few random "StonesFamily.org" or similar; it's nice for UK persons to stake a claim here. How long has it been active? How do they resolve disputes? Do they permit subdomains for family groups?
> How long has it been active?
I think pretty much forever at this point, the only addition to .uk in recent memory is the ability to register .uk TLD itself (before it was all second-level only).
I think it might have been around the 2000's when they introduced me.uk.
> How do they resolve disputes?
AFAIK me.uk falls back to the general rule of "first-come first-serve". The only real requirement for me.uk is that it has to be registered in the name of a natural person and not an agent, trustee, proxy or representative. Any transfer or renewal has to continue to adhere to this rule.
There is no UK nationality or residency requirement for me.uk either AFAIK.
Disputes fall back to Nominet's standard Dispute Resolution Service.
> Do they permit subdomains for family groups?
Not AFAIK. Well, of course you can register the base me.uk and DIY DNS subdomains.
