• 0 Posts
  • 23 Comments
Joined 1 year ago
cake
Cake day: August 4th, 2024

help-circle




  • Unlike the dotcom bubble, Another big aspect of it is the unit cost to run the models.

    Traditional web applications scale really well. The incremental cost of adding a new user to your app is basically nothing. Fractions of a cent. With LLMs, scaling is linear. Each machine can only handle a few hundred users and they’re expensive to run:

    Big beefy GPUs are required for inference as well as training and they require a large amount of VRAM. Your typical home gaming GPU might have 16gb vram, 32 if you go high end and spend $2500 on it (just the GPU, not the whole pc). Frontier models need like 128gb VRAM to run and GPUs manufactured for data centre use cost a lot more. A state of the art Nvidia h200 costs $32k. The servers that can host one of these big frontier models cost, at best, $20 an hour to run and can only handle a handful of user requests so you need to scale linearly as your subscriber count increases. If you’re charging $20 a month for access to your model, you are burning a user’s monthly subscription every hour for each of these monster servers you have turned on. That’s generous and assumes they’re not paying the “on-demand” price of $60/hr.

    Sam Altman famously said OpenAI are losing money on their $200/mo subscriptions.

    If/when there is a market correction, a huge factor of the amount of continued interest (like with the internet after dotcom) is whether the quality of output from these models reflects the true, unsubsidized price of running them. I do think local models powered by things like llamacpp and ollama and which can run on high end gaming rigs and macbooks might be a possible direction for these models. Currently though you can’t get the same quality as state-of-the-art models from these small, local LLMs.


  • I think there are two things. There’s definitely a level of brainwashing where mediocre MBAs who have built a career on “failing upwards” project their own lack of scruples onto their workforce i.e. “if I worked from home I’d just play golf all day so I assume this is true for everyone”. They genuinely don’t understand management models beyond micromanagement because they have no frame of reference for “self-motivated” or “autonomous”.

    Then the other factor is that many of the c-level execs at these companies or their bosses (the board) have commercial real estate portfolios. Propping up the value of those units is contingent on companies renting office space. The bosses know which side their bread is buttered and even if they don’t have skin in the game directly will happily do favours for ‘friends’ who they want to impress to help them climb that next rung of the ladder.













  • In the UK we already have a law where isps block porn by default (blacklisting) the adult who took out the plan can contact the isp and ask them to opt out of these blocks. That’s been a thing for about 10 years. You can own a Pay-as-you-go sim as a minor but you have to send government id to prove you are over 18 to get the adult content filtering turned off.

    That’s one of the things that made it clear to me that the new law is an authoritarian data mining operation and blatant power grab. Like… We already have these tools in place. If you don’t want your kid accessing porn, don’t opt out of the filters provided by your isp.

    You could argue that putting the onus on the platform is more effective at “protecting kids” than having the isps maintain blacklists but there will always be small sites that don’t comply and enterprising kids who find a way around any block. Just like the law requires you to be 18 to buy alcohol or tobacco here but there are always dodgy shops who sell tobacco to underage kids. There are older siblings and relatives willing to buy cigarettes and alcohol for underage teens.

    This was never about protecting the children. That was the Trojan horse used to justify these laws to the technically uninformed.