The people calling it "OK" probably tried it for themselves. Whatever model is being demoed in that video is not the same as the 25MB model they released.
It doesn't sound so good. Excellent technical achievement and it may just improve more and more! But for now I can't use it for consumer facing applications.
Speech speed is always a tunable parameter and not something intrinsic to the model.
The comparison to make is expressiveness and correct intonation for long sentences vs something like espeak. It actually sounds amazing for the size. The closest thing is probably KokoroTTS at 82M params and ~300MB.
The voices sound artificial and a bit grating. The male voices especially are lacking, especially in depth: only the ultimate voice has any depth at all, while the others sound like teenagers who haven't finished puberty. None of the voices sound quite human, but they're all very annoying, and part of that is that they sound like they're acting.
The only real questions are which Chinese gacha game they ripped data from and whether they used Claude Code or Gemini CLI for Python code. I bet one can get a formant match from output this much overfit to whatever data. This isn't going to stay up for long.
Impressive technical achievement, but in terms of whether I'd use it: oof, that male voice is like one of these fake-excited newsreaders. Like they're always at the edge of their breath. The female one is better but still someone reading out an advertisement for a product they were told they must act extra excited for. I assume this is what the majority of training data was like and not an intentional setting for the demo. Unsure whether I could get used to that
I use TTS on my phone regularly and recently also tried this new project on F-Droid called SherpaTTS, which grabs some models from Huggingface. They're super heavy (the phone suspends other apps to disk while this runs) and sound good, but in the first news article there were already one or two mispronunciations because it's guessing how to say uncommon or new words and it's not based on logical rules anymore to turn text into speech
Google and Samsung have each a TTS engine pre-installed on my device and those sound and work fine. A tad monotonous but it seems to always pronounce things the same way so you can always work out what the text said
Espeak (or -ng) is the absolute worst, but after 30 seconds of listening closely you get used to it and can understand everything fine. I don't know if it's the best open source option (probably there are others that I should be trying) but it's at least the most reliable where you'll always get what is happening and you can install it on any device without licensing issues
anyone else wants to try sherpaOnnx you can try this.. https://github.com/willwade/tts-wrapper we recently added in the kokoro models which should sound a lot better. There are a LOT of models to choose from. I have a feeling the Droid app isnt handling cold starts very well.
Performance Results:
Initial Latency: ~315ms for short text
Audio Generation Speed (seconds of audio per second of processing):
- Short text (12 chars): 3.35x realtime
- Medium text (100 chars): 5.34x realtime
- Long text (225 chars): 5.46x realtime
- Very Long text (306 chars): 5.50x realtime
Findings:
- Model loads in ~710ms
- Generates audio at ~5x realtime speed (excluding initial latency)
- Performance is consistent across different voices (4.63x - 5.28x realtime)
Thanks for running the benchmarks. Currently the models are not optimized yet. We will optimize loading etc when we release an SDK meant for production :)
I hope this is the future. Offline, small ML models, running inference on ubiquitous, inexpensive hardware. Models that are easy to integrate into other things, into devices and apps, and even to drive from other models maybe.
This is what Apple is envisioning with their SLMs, like having a model specifically for managing calendar events. It doesn't need to have the full knowledge of all humanity in it - just what it needs to manage the calendar.
Dedicated single-purpose hardware with models would be even less energy-intensive. It's theoretically possible to design chips which run neural networks and alike using just resistors (rather than transistors).
Such hardware is not general-purpose, and upgrading the model would not be possible, but there's plenty of use-cases where this is reasonable.
But resistors are, even in theory, heat dissipating devices. Unlike transistors, which can in theory be perfectly on or off (in both cases not dissipating heat).
Hmm. A pay once (or not at all) model that can run on anything? Or a subscription model that locks you in, and requires hardware that only the richest megacorps can afford? I wonder which one will win out.
The headline feature isn’t the 25 MB footprint alone. It’s that KittenTTS is Apache-2.0. That combo means you can embed a fully offline voice in Pi Zero-class hardware or even battery-powered toys without worrying about GPUs, cloud calls, or restrictive licenses. In one stroke it turns voice everywhere from a hardware/licensing problem into a packaging problem. Quality tweaks can come later; unlocking that deployment tier is the real game-changer.
yeah, we are super excited to build tiny ai models that are super high quality. local voice interfaces are inevitable and we want to power those in the future. btw, this model is just a preview, and the full release next week will be of much higher quality, along w another ~80M model ;)
The issue is even bigger: phonemizer is using espeak-ng, which isn't very good at turning graphemes into phonemes. In other TTS which rely on phonemes (e.g. Zonos) it turned out to be one of the key issues which cause bad generations.
And it isn't something you can fix, because the model was trained on bad phonemes (everyone uses Whisper + then phonemizes the text transcript).
> IANAL, but AFAICS this leaves 2 options, switching the license or removing that dependency.
There is a third option: asking the project for an exception.
Though that is unlikely to be granted¹ leaving you back with just the other two options.
And of course a forth choice: just ignore the license. This is the option taken by companies like Onyx, whose products I might otherwise be interested in…
----
[1] Those of us who pick GPL3 or AGPL generally do so to keep things definite and an exception would muddy the waters, also it might not even be possible if the project has many maintainers as relicensing would require agreement from all who have provided code that is in the current release. Furthermore, if it has inherited the license from one of its dependencies, an exception is even less practical.
Ah, yes, good catch, I didn't look deeper into the dependency tree at all. I'll update my footnote to include that as one of the reasons an exception may be impossible (or at least highly impractical).
A fourth option would be a kind of dual-licensing: the project as-is is available under GPL-3.0, but the source code in this repository excluding any dependencies is also available under Apache 2.0
Any user would still effectively be bound by the GPL-3.0, but if someone can remove the GPL dependencies they could use the project under Apache
That is an option for the publisher of the library, not the consumer of it. If it isn't already done then asking for it to be done is the same as asking for an exception otherwise (option three).
The use of the library is four lines. Three set up the library (`phonemizer.backend.EspeakBackend(language="en-us", preserve_punctuation=True, with_stress=True)`), the other calls it (`phonemes_list = self.phonemizer.phonemize([text])`). Plus I guess the import statements. Even ignoring Google vs Oracle I don't think those lines by themselves meet any threshold of originality.
Obviously you can't run them (with the original library) without complying with the GPL. But I don't see why I couldn't independently of that also give you this text file under Apache 2.0 to do with as you want (which for the record still doesn't allow you to run them with the original library without complying with the GPL, but that'd be phoneme forcing you to do that, not this project)
You would have to be very specific about the dual-licensing to avoid confusion about what you are allowed to do under Apache conditions though. You can't just say "it's dual-licensed"
You could even extract out the parts that do not call the GPL library into an upstream project under the Apache 2.0 licence, and pull in both that and the GPL library in the downstream project, relying on Apache 2.0 -> GPL 3.0 compatibility instead of explicit dual licensing to allow the combined work to be distributed under GPLv3.
This would only apply if they were distributing the GPL licensed code alongside their own code.
If my MIT-licensed one-line Python library has this line of code…
run([“bash”, “-c”, “echo hello”])
…I’m not suddenly subject to bash’s licensing. For anyone wanting to run my stuff though, they’re going to need to make sure they themselves have bash installed.
(But, to argue against my own point, if an OS vendor ships my library alongside a copy of bash, do they have to now relicense my library as GPL?)
The FSF thinks it counts as a derivative work and you have to use the LGPL to allow linking.
However, this has never actually been proven in court, and there's many good arguments that linking doesn't count as a derivative work.
Old post by a lawyer someone else found (version 3 wouldn't affect this) [1]
For me personally I don't really understand how, if dynamic linking was viral, using linux to run code isn't viral. Surely at some level what linux does to run your code calls GPLed code.
It doesn't really matter though, since the FSF stance is enough to scare companies from not using it, and any individual is highly unlikely to be sued.
> For me personally I don't really understand how, if dynamic linking was viral, using linux to run code isn't viral. Surely at some level what linux does to run your code calls GPLed code.
The Linux kernel has an explicit exception for userspace software:
> NOTE! This copyright does not cover user programs that use kernel services by normal system calls
And the GPL also has an explicit exception for "system" software such as kernel, platform libraries etc.:
> The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
> The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work.
> This would only apply if they were distributing the GPL licensed code alongside their own code.
As far as I understand the FSF's interpretation of their license, that's not true. Even if you only dynamically link to GPL-licensed code, you create a combined work which has to be licensed, as a whole, under the GPL.
I don't believe that this extends to calling an external program via its CLI, but that's not what the code in question seems to be doing.
(This is not an endorsement, but merely my understanding on how the GPL is supposed to work.)
This is a false analogy. It's quite straightforward.
Running bash (via exec()/fork()/spawn()/etc) isn't the same as (statically or dynamically) linking with its codebase. If your MIT-licensed one-liner links to code that's GPL licensed, then it gets infected by the GPL license.
GPL is for boomers at this point. Floppy disks? Distribution? You can use a tool but you cant change it? A DLL call means you need to redistribute your code but forking doesn't?
Yes, but if you use open source libraries for your closed source SaaS - thats fine. People get their software _over_ the network delivered to them in a VM (your browser).
The result can only be distributed under the terms of the GPL-3. That's actually a crucial difference: there's nothing preventing Kitten TTS from being Apache licensed, soliciting technical contributions under that license, and parts of its code being re-used in other software under that license. Yes, for the time being, this limits what you can do with Kitten TTS if you want to use the software as a whole (e.g. by embedding it into your product), but the license itself is still Apache and that can have value.
Okay, what's stopping you from feeding the code into an LLM and re-write it and make it yours? You can even add extra steps like make it analyze the code block by block then supervise it as it is rewriting it. Bam. AI age IP freedom.
Morals may stop you but other than that? IMHO all open source code is public domain code if anyone is willing to spend some AI tokens.
One person reads the code and produces a detailed technical specification. Someone reviews it to ensure that there is nothing in there that could be classified as copyrighted material, then a third person (who has never seen the original code) implements the spec.
You could use an LLM at both stages, but you'd have to be able to prove that the LLM that does the implementation had no prior knowledge of the code in question... Which given how LLMs have been trained seems to me to be very dubious territory for now until that legal situation gets resolved.
AI is useful in Chinese walling code, but it’s not as easy as you make it sound. To stay out of legal trouble, you probably should refactor the code into a different language, then back into the target language. In the end, it turns into a process of being forced to understand the codebase and supervising its rewriting. I’ve translated libraries into another language using LLMs, I’d say that process was 1/2 the labor of writing it myself. So in the end, going 2 ways, you may as well rewrite the code yourself… but working with the LLM will make you familiar with the subject matter so you -could- rewrite the code, so I guess you could think of it as a sort of buggy tutorial process?
I am not sure even that is enough. You would really need to do a clean room reimplementation to be safe - for exactly the same reasons that people writing code write clean room reimplementations.
Yeah, the algorithms and program flow would have to be materially distinct to be really safe. Maybe switching language paradigms would get that for you in most cases? Js->haskell->js? Sounds like a nightmare lol.
Tell me you don't know how to use LLMs properly without telling me.
You don't give the whole codebase to an LLM and expect it to have one shot output. Instead, you break it down and and write the code block by block. Then the size if the codebase doesn't matter. You use the LLM as a tool, it is not supposed to replace you. You don't try to become George from Jetsons who is just pressing a button and doesn't touch anything, instead you are on top of it as the LLM does the coding. You test the code on every step to see if the implementation behaves as expected. Do enough of this and you have proper, full "bespoke" software.
A Festival's English model, festvox-kallpc16k, is about 6 MB, and it is a large model; festvox-kallpc8k is about 3.5 MB.
eSpeak NG's data files take about 12 MB (multi-lingual).
I guess this one may generate more natural-sounding speech, but older or lower-end computers were capable of decent speech synthesis previously as well.
What about the training data? Is everyone 100% confident that models are not a derived work of the training inputs now, even if they can reproduce input exactly?
I play around with a nvidia jetson orin nano super right now and its actually pretty usuable with gemma3:4b and quite fast - even image processing is done in like 10-20 seconds but this is with GPU support. When something is not working and ollama is not using the GPU this calls take ages because the cpu is just bad.
Does anybody find it funny that sci-fi movies have to heavily distort "robot voices" to make them sound "convincingly robotic"? A robotic, explicitly non-natural voice would be perfectly acceptable, and even desirable, in many situations. I don't expect a smart toaster to talk like a BBC host; it'd be enough is the speech if easy to recognize.
A robotic, explicitly non-natural voice would be perfectly acceptable, and even desirable, in many situations[...]it'd be enough is the speech if easy to recognize.
We've had formant synths for several decades, and they're perfectly understandable and require a tiny amount of computing power, but people tend not to want to listen to them:
The YouTube video [1] was published in 2019. The Blog spam posts range from Nov 2022 to July 2023.
Other than the video, the only relevant content is on the about page [2]. It says the voice is a collaboration between 5 different entities, including advocacy groups, marketing firms and a music producer.
The video is the only example of the voice in use. There is no API, weights, SDK, etc.
I suspect this was a one-off marketing stunt sponsored by Copenhagen pride before the pandemic. The initial reaction was strong enough that a couple years they were still getting a small but steady flow of traffic. One of the involved marketing firms decided to monetize the asset and defaced it with blog spam.
Huh. Sounds perfectly intelligible and definitively artificial. Feels weakly feminine to me, but only because I was primed to think about gender from the branding.
It’s a good choice for a robot voice. It’s easier to understand than the formant synths or deliberately distorted human voices. The genderless aspect is alien enough to avoid the uncanny valley. You intuitively know you’re dealing with something a little different.
In the Culture novels, Iain Banks imagines that we would become uncomfortable with the uncanny realism of transmitted voices / holograms, and intentionally include some level of distortion to indicate you're speaking to an image
Depends on the movie. Ash and Bishop in the Alien franchise sound human until there's a dramatic reason to sound more 'robotic'.
I agree with your wider point. I use Google TTS with Moon+Reader all the time (I tried audio books read by real humans but I prefer the consistency of TTS)
Slightly different there because it's important in both cases that Ripley (and we) can't tell they're androids until it's explicitly uncovered. The whole point is that they're not presented as artificial. Same in Blade Runner: "more human than human". You don't have a film without the ambiguity there.
I remember that the novelization of the fifth element describes that the cops are taught to speak as robotic as possible when using speakers for some reason. Always found the idea weird that someone would _want_ that
I tried to replicate their demo text but it doesn't sound as good for some reason.
If anyone else wants to try:
> Kitten TTS is an open-source series of tiny and expressive text-to-speech models for on-device applications. Our smallest model is less than 25 megabytes.
I got an error when I tried the demo with 6 sentences, but it worked great when I reduced the text to 3 sentences. Is the length limit due to the model or just a limitation for the demo?
"This first Book proposes, first in brief, the whole Subject, Mans disobedience, and the loss thereupon of Paradise wherein he was plac't: Then touches the prime cause of his fall, the Serpent, or rather Satan in the Serpent; who revolting from God, and drawing to his side many Legions of Angels, was by the command of God driven out of Heaven with all his Crew into the great Deep."
It takes a while until it starts generating sound on my i7 cores but it kind of works.
This also works:
"blah. bleh. blih. bloh. blyh. bluh."
So I don't think it's a limit on punctuation. Voice quality is quite bad though, not as far from the old school C64 SAM (https://discordier.github.io/sam/) of the eighties as I expected.
Thanks, I was looking for that. While the reddit demo sounds ok, even though on a level we reached a couple of years ago, all TTS samples I tried were barley understandable at all
> Error generating speech: failed to call OrtRun(). ERROR_CODE: 2, ERROR_MESSAGE: Non-zero status code returned while running Expand node. Name:'/bert/Expand' Status Message: invalid expand shape
yeah, this is just a preview model from an early checkpoint. the full model release will be next week which includes a 15M model and an 80M model, both of which will have much higher quality than this preview.
On PC it's a python dependency hell but someone managed to package it in self contained JS code that works offline once it loaded the model? How is that done?
ONNXRuntime makes it fairly easy, you just need to provide a path to the ONNX file, give it inputs in the correct format, and use the outputs. The ONNXRuntime library handles the rest. You can see this in the main.js file: https://github.com/clowerweb/kitten-tts-web-demo/blob/main/m...
Plus, Python software are dependency hell in general, while webpages have to be self-contained by their nature (thank god we no longer have Silverlight and Java applets...)
Not open source. "You will need internet connectivity to validate your AccessKey with Picovoice license servers ... If you wish to increase your limits, you can purchase a subscription plan." https://github.com/Picovoice/orca#accesskey
Going online is a dealbreaker but if you really need it you could use ghidra to fix that. I had tried to find a conversion of their model to onnx (making their proprietary pipeline useless) but failed.
Hopefully open source will render them irrelevant in the future.
Does an apk for Android exist for replacing its speech to text engine? I tried sherpa-onnx but it was too slow for real time usage it seemed, and especially so for audiobooks when sped up.
I can't test this out right now, is this just a demo or is it actually an apk for replacing the engine? Because those are two different things, the latter can be used any time you want to read something aloud on the page for example. This is the sherpa-onnx one I'm talking about.
If you have `uv` installed, you can try my merged ref that has all of these PRs (and #22, a fix for short generation being trimmed unnecessarily) with
uvx --from git+https://github.com/akx/KittenTTS.git@pr-21-22-24-25 kittentts --output output.wav --text "This high quality TTS model works without a GPU"
Thanks for the quick intro into UV, it looks like docker layers for python
I found the TTS a bit slow so I piped the output into ffplay with 1.2x speedup to make it sound a bit better
uvx --from git+https://github.com/akx/KittenTTS.git@pr-21-22-24-25 kittentts --text "I serve 12 different beers at my restaurant for over 1000000 customers" --voice expr-voice-3-m --output - | ffplay -af "atempo=1.2" -f wav -
I was commiserating with my brother over how difficult it is to set up an environment to run one LLM or diffusion model, let alone multiple or a combination. It's 5 percent CUDA/ROCm difficulties and 95% Python difficulties. We have a theory that Lanyone working with generative AI has to tolerate output that is only 90% right, and is totaly fine working with a language and environment that only 90% works.
Why is Python so bad at that? It's less kludgy than Bash scripts, but even those are easier to get working.
> JS/TS/npm is just as bad with probably more build tools/frameworks.
This is flat out wrong. NPM packages by default are local to a directory. And I haven't seen a package rely on a specific minor version of node in literally years. Node's back compat is also great, there was one hiccup 5 or 6 years ago where a super popular native package was deprecated ago but that's been about it.
I can take current LTS node and run just about any package from the NPM repo written within the last 4 or 5 years and it will just work. Meanwhile plenty of python packages somehow need specific point releases. What the unholy hell.
Node version manager does exist, and it can be setup to work per directory, which is super cool, but I haven't needed NVM in literal years.
Yeah, but it's easily solved, with directives, headers, or make files that specify which language standard it follows. Better yet, you can use different syntax with different language standards, so it's clear which to follow. If a compiler can automatically figure whether I'm compiling C or C++, why can't a Python interpreter figure out if I'm running version two or three, of the same language?
I think it can install Python itself too. Though I have had issues with that - especially with SSL certificate locations, which is one of Linux's other clusterfucks.
The project is like 80% there by having a pyproject file that should work with uv and poetry. The just aren't any package versions specified and the python version is incredibly lax, and no lock file is provided.
A tool that was only released, what, a year or two ago? It simply won't be present in nearly all OS/distros. Only modern or rolling will have it (maybe). It's funny when the recommended python dependency manager managers are just as hard to install and use as the script themselves. Very python.
PYTHON(1) General Commands Manual PYTHON(1)
NAME
python - an object-oriented programming language
SYNOPSIS
python [ -c command | script | - ] [ arguments ]
DESCRIPTION
Python is the standard programming language.
Computer scientists love Python, not just because whitespace comes first ASCIIbetically, but because it's the standard. Everyone else loves Python because it's PYTHON!
There are still people who use machine wide python installs instead of environments? Python dependency hell was already bad years ago, but today it's completely impractical to do it this way. Even on raspberries.
Yep. Python stopped being Python a decade ago. Now there are just innumberable Pythons. Perl... on the otherhand, you can still run any perl script from any time on any system perl interpreter and it works! Granted, perl is unpopular and not getting constant new features re: hardcore math/computation libs.
Anyway, I think I'll stick with Festival 1.96 for TTS. It's super fast even on my core2duo and I have exactly zero chance of getting this Python 3'ish script to run on any machine with an OS older than a handful of years.
It reminds me of the costs and benefits of RollerCoaster Tycoon being written in assembly language. Because it was so light on resources, it could run on any privately owned computer, or at least anything x86, which was pretty much everything at the time.
Now, RISC architectures are much more common, so instead of the rare 68K Apple/Amiga/etc computer that existed at the time, it's super common to want to run software on an ARM or occasionally RISC-V processor, so writing in x86 assembly language would require emulation, making for worse performance than a compiled language.
You're getting a lot of comments along the lines of "Why don't you just ____," which only shows how Stockholmed the entire Python community is.
With no other language are you expected to maintain several entirely different versions of the language, each of which is a relatively large installation. Can you imagine if we all had five different llvms or gccs just to compile five different modern C projects?
I'm going to get downvoted to oblivion, but it doesn't change the reality that Python in 2025 is unnecessarily fragile.
That’s exactly what I have. The C++ codebases I work on build against a specific pinned version of LLVM with many warnings (as errors) enabled, and building with a different version entails a nonzero amount of effort. Ubuntu will happily install several versions of LLVM side by side or compilation can be done in a Docker container with the correct compiler. Similarly, the TypeScript codebases I work with test against specific versions of node.js in CI and the engine field in package.json is specified. The different versions are managed via nvm. Python is the same via uv and pyproject.yaml.
I don't doubt it, but I don't think that situation is accepted as the default in C/C++ development. For the most part, I expect OSS to compile with my own clang.
Oof, those are poor examples. Most compilers using LLVM other than clang do ship with their own LLVM patches, and cross-compiling with GCC does require installing a toolchain for each target.
Cross-compiling is a totally different subject… I'm trying to make an apples-to-apples comparison. If you compile a lot of OSS C projects for the host architecture, you typically do not need multiple LLVMs or GCCs. Usually, the makefile detects various things about the platform and compiler and then fails with an inscrutable error. But that is a separate issue! haha
system python is for system applications that are known to work together. If you need a python install for something else, there's venv or conda and then pip install stuff.
I tried it. Not bad for the size (of the model) and speed. Once you install all the massive number of libraries and things needed we are a far cry away from 25MB though. Cool project nonetheless.
To make the setup easier and add a few features people are asking for here (like GPU support and long text handling), I built a self-hosted server for this model:
https://github.com/devnen/Kitten-TTS-Server
The goal was a setup that "just works" using a standard Python virtual environment to avoid dependency conflicts.
The setup is just the standard git clone, pip install in a venv, and python server.py.
The repository already runs an ONNX model. But the onnx model doesn't get English text as input, it gets tokenized phonemes. The prepocessing for that is where most of the dependencies come from.
Which is completely reasonable imho, but obviously comes with tradeoffs.
For space sensitive applications like embedded systems, could you shift the preprocessing to compile time?
You would need to constrain the vocabulary to see any benefits, but that could be reasonable. For example, you an enumeration of numbers, units and metric names could handle dynamic time, temperature and other dashboard items.
For something more complex like offline navigation, you already need to store a map. You could store street names as tokens instead of text. Add a few turn commands, and you have offline spoken directions without on device pre-processing.
assuming most answers will be more than a sentence, 2.25 seconds is already long enough if you factor the token generation in between... and imagine with reasoning!... We're not there yet.
Hmm that actually seems extremely slow, Piper can crank out a sentence almost instantly on a Pi 4 which is a like a sloth compared to that Ryzen and the speech quality seems about the same at first glance.
I suppose it would make sense if you want to include it on top of an LLM that's already occupying most of a GPU and this could run in the limited VRAM that's left.
While I think this is indeed impressive and has a specific use case (e.g. in the embedded sector), I'm not totally convinced that the quality is good enough to replace bigger models.
With fish-speech[1] and f5-tts[2] there are at least 2 open source models pushing the quality limits of offline text-to-speech. I tested F5-TTS with an old NVidia 1660 (6GB VRAM) and it worked ok-ish, so running it on a little more modern hardware will not cost you a fortune and produce MUCH higher quality with multi-language and zero-shot support.
For Android there is SherpaTTS[3], which plays pretty well with most TTS Applications.
Fish Speech says its weights are for non-commercial use.
Also, what are the two's VRAM requirents? This model has 15 million parameters which might run on low-power, sub-$100 computers with up-to-date software. Your hardware was an out-of-date 6GB GPU.
Hmm the quality is not so impressive. I'm looking for a really naturally sounding model. Not very happy with piper/kokoro, XTTS was a bit complex to set up.
For STT whisper is really amazing. But I miss a good TTS. And I don't mind throwing GPU power at it. But anyway. this isn't it either, this sounds worse than kokoro.
The best open one I've found so far is Dia - https://github.com/nari-labs/dia - it has some limitations, but i think it's really impressive and I can run it on my laptop.
Thanks I'll try! I like how it sounds, the quality is really good. But the limitations are really severe (shorter than 5 seconds is not ok, > 30 seconds is not ok, it will play a random voice every time, those make it pretty much unusable for an assistant to be honest).
But it might be worth setting it up and seeing if it improves over time.
> Hmm the quality is not so impressive. [...] And I don't mind throwing GPU power at it.
This isn't for you, then. You should evaluate quality here based on the fact you don't need a GPU.
Back in the pre-Tacotron2 days, I was running slim TTS and vocoder models like GlowTTS and MelGAN on Digital Ocean droplets. No GPU to speak of. It cost next to nothing to run.
Since then, the trend has been to scale up. We need more models to scale down.
In the future we'll see small models living on-device. Embedded within toys and tools that don't need or want a network connection. Deployed with Raspberry Pi.
Edge AI will be huge for robotics, toys and consumer products, and gaming (ie. world models).
> This isn't for you, then. You should evaluate quality here based on the fact you don't need a GPU.
I know but it was more of a general comment. A really good TTS just isn't around yes in the OSS sphere. I looked at some of the other suggestions here but they have too many quirks. Dia sounds great but messages must have certain lengths etc and it picks a random voice every time. I'd love to have something self hosted that's as good as openai.
Microsoft's and some of Google's TTS models make the simplest mistakes. For instance, they sometimes read "i.e." as "for example." This is a problem if you have low vision and use TTS for, say, proofreading your emails.
You probably mean "e.g." as "for example", not "i.e."?
This might be on purpose and part of the training data because "for example" just sounds much better than "e.g.". Presumably for most purposes, linguistic naturalness is more important than fidelity.
Sometimes I use “for example” and “e.g.” in consecutive sentences to not sound repetitive, or possibly even within the same sentence (e.g. in parentheses). In that case, speaking both as “for example” would degrade it linguistically.
In any case, I’d like TTS to not take that kind of artistic freedom.
Well, speech synthesizers are pretty much famous for speaking all sorts of things wrong. But what I find very concerning about LLM based TTS is that some of them cant really speak numbers greater then 100. They try, but fail a lot. At least tts-1-hd was pretty much doing this for almost every 3 or 4 digit number. Especially noticeable when it is supposed to read a year number.
Not entirely related but humans have the same problem.
For scriptwriting when doing voice overs we always explicitly write out everything. So instead of 1 000 000 we would write one million or a million. This is a trivial example but if the number was 1 548 736 you will almost never be able to just read that off. However one million, five hundred and forty eight thousand, seven hundred and thirty six can just be read without parsing.
Regarding humans, yes and no. If a human had constantly problems with 3 and 4 digit numbers like tts-1-hd does, I'd ask myself if they were neurodivergent in some way.
And yes, I added instructions along the lines of what you describe to my prompt. Its just sad that we have to. After all, LLM TTS has solved a bunch of real problems, like switching languages in a text, or foreign words. The pronounciation is better then anything we ever had. But it fails to read short numbers. I feel like that small issue could probably have been solved by doing some fine tuning. But I actually dont really understand the tech for it, so...
From the web demo this model is really good at numbers. It rushes through them, slurs them a bit together, but they are all correct, even 7 digit numbers (didn't test further).
Looks like they are sidestepping these kinds of issues by generating the phonemes with the preprocessing stage of traditional speech synthesizers, and using the LLM only to turn those phonemes into natural-ish sounding speech. That limits how natural the model can become, but it should be able to correctly pronounce anything the preprocessing can pronounce
would love to see how that turns out. the full model release next week will be more expressive and higher quality than this one so we're excited to see you try that out.
The samples featured elsewhere seem to be from a larger model?
After testing this locally, it still sounds quite mechanical, and fails catastrophically for simple phrases with numbers ("easy as 1-2-3"). If the 80M model can improve on this and keep the expressiveness seen in the reddit post, that looks promising.
Thanks! Yeah. It definitely isn’t the absolute best in quality but it trounces the default TTS options on macOS (as third party developers are locked out of the Siri voices). And for less than the size of many modern web pages…
Question for the experts here; What would be a SOTA TTS that can run on an average laptop (32GB RAM, 4GB VRAM). I just want to attach a TTS to my SLM output, and get the highest possible voice quality/ human resembleness.
Most of these comments were originally posted to a different thread (https://news.ycombinator.com/item?id=44806543). I've moved them hither because on HN we always prefer to give the project creators credit for their work.
(it does however explain how many of these comments are older than the thread they are now children of)
thanks, but keep in mind that this model is just a preview checkpoint that is only 10% trained. the full release next week will be of much higher quality and it will include a 15M model and an 80M model.
This feels different. This feels like a genuinely monumental release. Holy cow.
Very well done. The quality is excellent and the technical parameters are, simply, unbelievable. Makes me want to try to embed this on a board just to see if it's possible.
I'm curious why smallish TTS models have metallic voice quality.
The pronunciation sounds about right - i thought it's the hard part. And the model does it well. But voice timbre should be simpler to fix? Like, a simple FIR might improve it?
We change our tone based on personal style, emotion, context, and other factors. An accurate generator might need to encode all that information in the model. It will be larger than a model that doesn't do all of that.
Not bad for the size (with my very limited knowledge of this field) !
In a couple tests, the "Male 2" voice sounds reasonable, but I've found it has problem with some groups of words, specially when played with little context. I think it's small sentences.
For example, if you try to do just "Hey gang!", it will sound something like "Chay yang". But if you add an additional sentence after that, it will sound a bit different (but still weird).
Awesome work! Often times in the TTS space, human-similarity is given way too much emphasis at the expense of hurting user access. Frankly as long as a voice is clear and you listen to it for a while, the brain filters out most quirks you would perceive on the first pass. Hence why many blind folks still are perfectly fine using espeak-ng. The other properties like speed of generation and size make it worth it.
I've been using a custom AI audiobook generation program [0] with piper for quite a while now and am very excited to look at integrating kitten. Historically piper has been the only good option for a free CPU-only local model so I am super happy to see more competition in the space. Easy installation is a big deal, since piper historically has had issues with that. (Hence why I had to add auto installation support in [0])
What I am still looking for is a way to clone voice locally. I have OK hardware. For example I can use Mistral Small 3.1 or what it is called locally. Premade voices can be interesting too, but I am looking for custom voice. Perhaps by providing audio and the corresponding transcript to the model, training it, and then give it a new text and let it speak that.
If you're looking for other languages, Piper has been around in this scene for much longer and they have open-source training code and a lot of models (they're ~60MB instead of 25MB but whatever...) https://huggingface.co/rhasspy/piper-voices/tree/main
Actually I found it irritating that the readme does not mention the language at all. I think it is not good practice to deduce it from the language of the readme itself. I would not like to have German language tts models with only a German readme...
TTS is generally not multilingual. One might think a well-annotated phonetic descriptions of voices would suffice, but that's not quite how languages work nor how TTS work.
(but somehow LLMs handle multilingual input perfectly fine! that's a bit strange, if you think about that)
How does one build similar model, but for different languages? I was under impression that being open source, there would be some instructions how to build everything on your own.
A localized version of this, and I could finally build my tiny Amazon Echo replacement. I would love to see all speech synthesis performed on a local device.
I'm doing this now with Home Assistant voice. All the TTS, STT, and LLMs involved run locally on my network. It's absurdly superior to every other voice assistant product. (Would be nice if it was just a pure multi-modal model though)
Nothing to do with Apple Intelligence. The speech synthesiser manager (the term manager was used for OS components in Classic Mac OS) has been around since the mid 90s or so. The change you’re hearing is probably a new/modified default voice.
It looks like it's Python, so it might be possible to use via https://github.com/livebook-dev/pythonx ? But the parallel huggingface/bumblebee idea was also good, hadn't seen or thought of, that definitely works for a lot of other models, curious if you get working! Some chance I'll play with this myself in a few months, so feel free to report back here or DM me!
I just decided to try this quickly and hit some issues on my Mac FYI, it might work better on Linux but I hit a compilation issue with `curated-tokenizers`, possibly from a typo in setup.py or pyproject.toml in curated-tokenizers, spotted by AI:
-Wno-sign-compare-Wno-strict-prototypes
should be
-Wno-sign-compare -Wno-strict-prototypes
so could perhaps fix with a PR to curated-tokenizers or by forking it...
Might well be other issues behind that, and unclear if need any other dependencies that kitten doesn't rely on directly like torch or torchaudio? but... not 5 mins easy, but looks like issues might be able to be worked through...
For reference this is all I was trying basically:
Mix.install([:pythonx])
Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = ">=3.8"
dependencies = [
"kittentts @ https://github.com/KittenML/KittenTTS/releases/download/0.1/kittentts-0.1.0-py3-none-any.whl"
]
""")
Good TTS feels like it is something that should be natively built into every consumer device. So the user can decide if they want to read or listen to the text at hand.
I'm surprised that phone manufacturers do not include good TTS models in their browser APIs for example. So that websites can build good audio interfaces.
I for one would love to build a text editor that the user can use completely via audio. Text input might already be feasible via the "speak to type" feature, both Android and iOS offer.
But there seems to be no good way to output spoken text without doing round-trips to a server and generate the audio there.
The interface I would like would offer a way to talk to write and then commands like "Ok editor, read the last paragraph" or "Ok editor, delete the last sentence".
It could be cool to do writing this way while walking. Just with a headset connected to a phone that sits in one's pocket.
On Mac OS you can "speak" a text in almost every app, using built in voice (like the Siri voice or some older voices). All offline, and even from the terminal with "say".
I tried it a few months ago to narrate an epub in Apple Books and it was very broken in a weird way. It starts out decent but after a few pages, it starts slurring, skipping words, trailing off not finishing sentences and then goes silent.
(I've just tried it again without seeing that issue within a few pages)
> Siri voice or some older voices
You can choose "Enhanced" and "Premium" versions of voices which are larger and sound nice and modern to me. The "Serena Premium" voice I was using is over 200Mb and far better that this Show HN. It's very natural but kind of ruined by diabolical pronunciation of anything slightly non-standard which sadly seems to cover everything I read e.g. people/place names, technical/scientific terms or any neologisms in scifi/fantasy.
It's so wildly incomprehensible for e.g. Tibetan names in a mountaineering book, that you have to check the text. If the word being butchered is frequently repeated e.g. main character’s name, then it's just too painful to use.
TL;DR: If you are interested in TTS, you should explore alternatives
I tried to use it...
Its python venv has grown to 6 GBytes in size. The demo sentence
> "This high quality TTS model works without a GPU"
works, it takes 3s to render the audio.
Audio sounds like a voice in a tin can.
I tried to have a news article read aloud and failed with
> [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Expand node. Name:'/bert/Expand'
> Status Message: invalid expand shape
If you are interested in TTS, you should explore alternatives
Does your Eloquence installation include multiple languages? The one I have is only 1876 KB for US English only. And classic DECtalk is even smaller; I have here a version that's only 638 KB (again, US English only).
I'm so confused on how the model is actually made. It doesn't seem to be in the code or this stuff is way simpler than i thought. It seems to use a fancy library from japan, not sure how much it's just that
https://www.youtube.com/watch?v=60Dy3zKBGQg
It doesn't sound so good. Excellent technical achievement and it may just improve more and more! But for now I can't use it for consumer facing applications.
The comparison to make is expressiveness and correct intonation for long sentences vs something like espeak. It actually sounds amazing for the size. The closest thing is probably KokoroTTS at 82M params and ~300MB.
I use TTS on my phone regularly and recently also tried this new project on F-Droid called SherpaTTS, which grabs some models from Huggingface. They're super heavy (the phone suspends other apps to disk while this runs) and sound good, but in the first news article there were already one or two mispronunciations because it's guessing how to say uncommon or new words and it's not based on logical rules anymore to turn text into speech
Google and Samsung have each a TTS engine pre-installed on my device and those sound and work fine. A tad monotonous but it seems to always pronounce things the same way so you can always work out what the text said
Espeak (or -ng) is the absolute worst, but after 30 seconds of listening closely you get used to it and can understand everything fine. I don't know if it's the best open source option (probably there are others that I should be trying) but it's at least the most reliable where you'll always get what is happening and you can install it on any device without licensing issues
Ubuntu 24, Razer Blade 16, Intel Core i9-14900HX
The tech is still public and the research is available
Such hardware is not general-purpose, and upgrading the model would not be possible, but there's plenty of use-cases where this is reasonable.
Have you seen the code[1] in the repo? It uses phonemizer[2] which is GPL-3.0 licensed. In its current state, it's effectively GPL licensed.
[1]: https://github.com/KittenML/KittenTTS/blob/main/kittentts/on...
[2]: https://github.com/bootphon/phonemizer
Edit: It looks like I replied to an LLM generated comment.
And it isn't something you can fix, because the model was trained on bad phonemes (everyone uses Whisper + then phonemizes the text transcript).
There is a third option: asking the project for an exception.
Though that is unlikely to be granted¹ leaving you back with just the other two options.
And of course a forth choice: just ignore the license. This is the option taken by companies like Onyx, whose products I might otherwise be interested in…
----
[1] Those of us who pick GPL3 or AGPL generally do so to keep things definite and an exception would muddy the waters, also it might not even be possible if the project has many maintainers as relicensing would require agreement from all who have provided code that is in the current release. Furthermore, if it has inherited the license from one of its dependencies, an exception is even less practical.
IIUC, the project isn't at the liberty to grant such an exception because it inherits its GPL license from espeak-ng.
Any user would still effectively be bound by the GPL-3.0, but if someone can remove the GPL dependencies they could use the project under Apache
Obviously you can't run them (with the original library) without complying with the GPL. But I don't see why I couldn't independently of that also give you this text file under Apache 2.0 to do with as you want (which for the record still doesn't allow you to run them with the original library without complying with the GPL, but that'd be phoneme forcing you to do that, not this project)
You would have to be very specific about the dual-licensing to avoid confusion about what you are allowed to do under Apache conditions though. You can't just say "it's dual-licensed"
If my MIT-licensed one-line Python library has this line of code…
…I’m not suddenly subject to bash’s licensing. For anyone wanting to run my stuff though, they’re going to need to make sure they themselves have bash installed.(But, to argue against my own point, if an OS vendor ships my library alongside a copy of bash, do they have to now relicense my library as GPL?)
However, this has never actually been proven in court, and there's many good arguments that linking doesn't count as a derivative work.
Old post by a lawyer someone else found (version 3 wouldn't affect this) [1]
For me personally I don't really understand how, if dynamic linking was viral, using linux to run code isn't viral. Surely at some level what linux does to run your code calls GPLed code.
It doesn't really matter though, since the FSF stance is enough to scare companies from not using it, and any individual is highly unlikely to be sued.
[1] https://www.linuxjournal.com/article/6366
The Linux kernel has an explicit exception for userspace software:
> NOTE! This copyright does not cover user programs that use kernel services by normal system calls
> The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
> The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work.
As far as I understand the FSF's interpretation of their license, that's not true. Even if you only dynamically link to GPL-licensed code, you create a combined work which has to be licensed, as a whole, under the GPL.
I don't believe that this extends to calling an external program via its CLI, but that's not what the code in question seems to be doing.
(This is not an endorsement, but merely my understanding on how the GPL is supposed to work.)
Running bash (via exec()/fork()/spawn()/etc) isn't the same as (statically or dynamically) linking with its codebase. If your MIT-licensed one-liner links to code that's GPL licensed, then it gets infected by the GPL license.
I don't know if this has ever been tested in court.
Sillyness
[0]: https://www.gnu.org/licenses/license-list.html#apache2
The result can only be distributed under the terms of the GPL-3. That's actually a crucial difference: there's nothing preventing Kitten TTS from being Apache licensed, soliciting technical contributions under that license, and parts of its code being re-used in other software under that license. Yes, for the time being, this limits what you can do with Kitten TTS if you want to use the software as a whole (e.g. by embedding it into your product), but the license itself is still Apache and that can have value.
Morals may stop you but other than that? IMHO all open source code is public domain code if anyone is willing to spend some AI tokens.
There are standard ways to approach this called clean room engineering.
https://en.m.wikipedia.org/wiki/Clean-room_design
One person reads the code and produces a detailed technical specification. Someone reviews it to ensure that there is nothing in there that could be classified as copyrighted material, then a third person (who has never seen the original code) implements the spec.
You could use an LLM at both stages, but you'd have to be able to prove that the LLM that does the implementation had no prior knowledge of the code in question... Which given how LLMs have been trained seems to me to be very dubious territory for now until that legal situation gets resolved.
You don't give the whole codebase to an LLM and expect it to have one shot output. Instead, you break it down and and write the code block by block. Then the size if the codebase doesn't matter. You use the LLM as a tool, it is not supposed to replace you. You don't try to become George from Jetsons who is just pressing a button and doesn't touch anything, instead you are on top of it as the LLM does the coding. You test the code on every step to see if the implementation behaves as expected. Do enough of this and you have proper, full "bespoke" software.
https://github.com/espeak-ng/espeak-ng/blob/a4ca101c99de3534...
eSpeak NG's data files take about 12 MB (multi-lingual).
I guess this one may generate more natural-sounding speech, but older or lower-end computers were capable of decent speech synthesis previously as well.
$ ls -lh /usr/bin/flite
Listed as 27K last I checked.
I recall some Blind users were able to decode Gordon 8-bit dialogue at speeds most people found incomprehensible. =3
What about the training data? Is everyone 100% confident that models are not a derived work of the training inputs now, even if they can reproduce input exactly?
Iam curious how fast this is with CPU only.
It sounds ok, but impressive for the size.
We've had formant synths for several decades, and they're perfectly understandable and require a tiny amount of computing power, but people tend not to want to listen to them:
https://en.wikipedia.org/wiki/Software_Automatic_Mouth
https://simulationcorner.net/index.php?page=sam (try it yourself to hear what it sounds like)
DECtalk[1,2] would be a much better example, that's as formant as you get.
[1] https://en.wikipedia.org/wiki/DECtalk [2] https://webspeak.terminal.ink
https://discordier.github.io/sam/
Other than the video, the only relevant content is on the about page [2]. It says the voice is a collaboration between 5 different entities, including advocacy groups, marketing firms and a music producer.
The video is the only example of the voice in use. There is no API, weights, SDK, etc.
I suspect this was a one-off marketing stunt sponsored by Copenhagen pride before the pandemic. The initial reaction was strong enough that a couple years they were still getting a small but steady flow of traffic. One of the involved marketing firms decided to monetize the asset and defaced it with blog spam.
[1] https://www.youtube.com/watch?v=lvv6zYOQqm0
[2] https://genderlessvoice.com/about/
It’s a good choice for a robot voice. It’s easier to understand than the formant synths or deliberately distorted human voices. The genderless aspect is alien enough to avoid the uncanny valley. You intuitively know you’re dealing with something a little different.
Well sure, the BBC have already established that it's supposed to sound like a brit doing an impersonation of an American: https://www.youtube.com/watch?v=LRq_SAuQDec
I agree with your wider point. I use Google TTS with Moon+Reader all the time (I tried audio books read by real humans but I prefer the consistency of TTS)
If anyone else wants to try:
> Kitten TTS is an open-source series of tiny and expressive text-to-speech models for on-device applications. Our smallest model is less than 25 megabytes.
"This first Book proposes, first in brief, the whole Subject, Mans disobedience, and the loss thereupon of Paradise wherein he was plac't: Then touches the prime cause of his fall, the Serpent, or rather Satan in the Serpent; who revolting from God, and drawing to his side many Legions of Angels, was by the command of God driven out of Heaven with all his Crew into the great Deep."
It takes a while until it starts generating sound on my i7 cores but it kind of works.
This also works:
"blah. bleh. blih. bloh. blyh. bluh."
So I don't think it's a limit on punctuation. Voice quality is quite bad though, not as far from the old school C64 SAM (https://discordier.github.io/sam/) of the eighties as I expected.
https://clowerweb.github.io/node_modules/onnxruntime-web/dis...
(seems reverted now)
Doesn't seem to work with thai.
Plus, Python software are dependency hell in general, while webpages have to be self-contained by their nature (thank god we no longer have Silverlight and Java applets...)
Hopefully open source will render them irrelevant in the future.
https://k2-fsa.github.io/sherpa/onnx/tts/apk-engine.html
On another machie the python version is too new, and the package/dependencies don't want to install.
https://github.com/KittenML/KittenTTS/pull/21 https://github.com/KittenML/KittenTTS/pull/24 https://github.com/KittenML/KittenTTS/pull/25
If you have `uv` installed, you can try my merged ref that has all of these PRs (and #22, a fix for short generation being trimmed unnecessarily) with
I found the TTS a bit slow so I piped the output into ffplay with 1.2x speedup to make it sound a bit better
Nice one, thanks!
https://docs.astral.sh/uv/guides/tools/
uv installation:
https://docs.astral.sh/uv/getting-started/installation/
Why is Python so bad at that? It's less kludgy than Bash scripts, but even those are easier to get working.
JS/TS/npm is just as bad with probably more build tools/frameworks.
Rust is a mess.
Go, well.
Even perl was quite complicated.
This is flat out wrong. NPM packages by default are local to a directory. And I haven't seen a package rely on a specific minor version of node in literally years. Node's back compat is also great, there was one hiccup 5 or 6 years ago where a super popular native package was deprecated ago but that's been about it.
I can take current LTS node and run just about any package from the NPM repo written within the last 4 or 5 years and it will just work. Meanwhile plenty of python packages somehow need specific point releases. What the unholy hell.
Node version manager does exist, and it can be setup to work per directory, which is super cool, but I haven't needed NVM in literal years.
Pretty impressive but this seems to be a staple of most AI/ML projects.
"Works on my machine" or "just use docker", although here the later doesn't even seem to be an option.
Anyway, I think I'll stick with Festival 1.96 for TTS. It's super fast even on my core2duo and I have exactly zero chance of getting this Python 3'ish script to run on any machine with an OS older than a handful of years.
I send you a 500kb Windows .exe file and claim it runs literally everywhere.
Would it be ignorant to say anything against it because of its size?
Now, RISC architectures are much more common, so instead of the rare 68K Apple/Amiga/etc computer that existed at the time, it's super common to want to run software on an ARM or occasionally RISC-V processor, so writing in x86 assembly language would require emulation, making for worse performance than a compiled language.
This package is the epitome of dependency hell.
Seriously, stick with piper-tts.
Easy to install, 50MB gives you excellent results and 100MB gives you good results with hundreds of voices.
With no other language are you expected to maintain several entirely different versions of the language, each of which is a relatively large installation. Can you imagine if we all had five different llvms or gccs just to compile five different modern C projects?
I'm going to get downvoted to oblivion, but it doesn't change the reality that Python in 2025 is unnecessarily fragile.
> if we all had five different llvms or gccs
Oof, those are poor examples. Most compilers using LLVM other than clang do ship with their own LLVM patches, and cross-compiling with GCC does require installing a toolchain for each target.
Yes, because all I have to do is look at the real world.
To make the setup easier and add a few features people are asking for here (like GPU support and long text handling), I built a self-hosted server for this model: https://github.com/devnen/Kitten-TTS-Server
The goal was a setup that "just works" using a standard Python virtual environment to avoid dependency conflicts.
The setup is just the standard git clone, pip install in a venv, and python server.py.
ONNX runtime is a single library, with C#'s package being ~115MB compressed.
Not tiny, but usually only a few lines to actually run and only a single dependency.
Which is completely reasonable imho, but obviously comes with tradeoffs.
You would need to constrain the vocabulary to see any benefits, but that could be reasonable. For example, you an enumeration of numbers, units and metric names could handle dynamic time, temperature and other dashboard items.
For something more complex like offline navigation, you already need to store a map. You could store street names as tokens instead of text. Add a few turn commands, and you have offline spoken directions without on device pre-processing.
Aside: Are there any models for understanding voice to text, fully offline, without training?
I will be very impressed when we will be able to have a conversation with an AI at a natural rate and not "probe, space, response"
My mid-range AMD CPU is multiple times faster than realtime with parakeet.
Average duration per generation: 1.28 seconds
Characters processed per second: 30.35
--
"Um"
Average duration per generation: 0.22 seconds
Characters processed per second: 9.23
--
"The brown fox jumps over the lazy dog.. The brown fox jumps over the lazy dog.."
Average duration per generation: 2.25 seconds
Characters processed per second: 35.04
--
processor : 0
vendor_id : AuthenticAMD
cpu family : 25
model : 80
model name : AMD Ryzen 7 5800H with Radeon Graphics
stepping : 0
microcode : 0xa50000c
cpu MHz : 1397.397
cache size : 512 KB
I suppose it would make sense if you want to include it on top of an LLM that's already occupying most of a GPU and this could run in the limited VRAM that's left.
OpenAI's whisper is a few years old and pretty solid.
https://github.com/openai/whisper
[0]: https://github.com/openai/whisper/discussions/679 [1]: https://github.com/openai/whisper/discussions/928 [2]: https://github.com/openai/whisper/discussions/2608
While I think this is indeed impressive and has a specific use case (e.g. in the embedded sector), I'm not totally convinced that the quality is good enough to replace bigger models.
With fish-speech[1] and f5-tts[2] there are at least 2 open source models pushing the quality limits of offline text-to-speech. I tested F5-TTS with an old NVidia 1660 (6GB VRAM) and it worked ok-ish, so running it on a little more modern hardware will not cost you a fortune and produce MUCH higher quality with multi-language and zero-shot support.
For Android there is SherpaTTS[3], which plays pretty well with most TTS Applications.
1: https://github.com/fishaudio/fish-speech
2: https://github.com/SWivid/F5-TTS
3: https://github.com/woheller69/ttsengine
Also, what are the two's VRAM requirents? This model has 15 million parameters which might run on low-power, sub-$100 computers with up-to-date software. Your hardware was an out-of-date 6GB GPU.
For STT whisper is really amazing. But I miss a good TTS. And I don't mind throwing GPU power at it. But anyway. this isn't it either, this sounds worse than kokoro.
But it might be worth setting it up and seeing if it improves over time.
This isn't for you, then. You should evaluate quality here based on the fact you don't need a GPU.
Back in the pre-Tacotron2 days, I was running slim TTS and vocoder models like GlowTTS and MelGAN on Digital Ocean droplets. No GPU to speak of. It cost next to nothing to run.
Since then, the trend has been to scale up. We need more models to scale down.
In the future we'll see small models living on-device. Embedded within toys and tools that don't need or want a network connection. Deployed with Raspberry Pi.
Edge AI will be huge for robotics, toys and consumer products, and gaming (ie. world models).
I know but it was more of a general comment. A really good TTS just isn't around yes in the OSS sphere. I looked at some of the other suggestions here but they have too many quirks. Dia sounds great but messages must have certain lengths etc and it picks a random voice every time. I'd love to have something self hosted that's as good as openai.
Why does it happen? I'm genuinely curious.
This might be on purpose and part of the training data because "for example" just sounds much better than "e.g.". Presumably for most purposes, linguistic naturalness is more important than fidelity.
In any case, I’d like TTS to not take that kind of artistic freedom.
For scriptwriting when doing voice overs we always explicitly write out everything. So instead of 1 000 000 we would write one million or a million. This is a trivial example but if the number was 1 548 736 you will almost never be able to just read that off. However one million, five hundred and forty eight thousand, seven hundred and thirty six can just be read without parsing.
Same with urls, W W W dot Google dot com.
And yes, I added instructions along the lines of what you describe to my prompt. Its just sad that we have to. After all, LLM TTS has solved a bunch of real problems, like switching languages in a text, or foreign words. The pronounciation is better then anything we ever had. But it fails to read short numbers. I feel like that small issue could probably have been solved by doing some fine tuning. But I actually dont really understand the tech for it, so...
Looks like they are sidestepping these kinds of issues by generating the phonemes with the preprocessing stage of traditional speech synthesizers, and using the LLM only to turn those phonemes into natural-ish sounding speech. That limits how natural the model can become, but it should be able to correctly pronounce anything the preprocessing can pronounce
Foundational tools like this open up the possiblity of one-time payment or even free tools.
After testing this locally, it still sounds quite mechanical, and fails catastrophically for simple phrases with numbers ("easy as 1-2-3"). If the 80M model can improve on this and keep the expressiveness seen in the reddit post, that looks promising.
[0] https://old.reddit.com/r/LocalLLaMA/comments/1mhyzp7/kitten_...
(it does however explain how many of these comments are older than the thread they are now children of)
It would be great if the training data were released too!
For instance, try adding `np.random.shuffle(ref_s[0])` after the line `ref_s = self.voices[voice]`...
EDIT: be careful with your system volume settings if you do this.
https://github.com/ggml-org/whisper.cpp
https://github.com/primaprashant/hns
Very well done. The quality is excellent and the technical parameters are, simply, unbelievable. Makes me want to try to embed this on a board just to see if it's possible.
The pronunciation sounds about right - i thought it's the hard part. And the model does it well. But voice timbre should be simpler to fix? Like, a simple FIR might improve it?
In a couple tests, the "Male 2" voice sounds reasonable, but I've found it has problem with some groups of words, specially when played with little context. I think it's small sentences.
For example, if you try to do just "Hey gang!", it will sound something like "Chay yang". But if you add an additional sentence after that, it will sound a bit different (but still weird).
I've been using a custom AI audiobook generation program [0] with piper for quite a while now and am very excited to look at integrating kitten. Historically piper has been the only good option for a free CPU-only local model so I am super happy to see more competition in the space. Easy installation is a big deal, since piper historically has had issues with that. (Hence why I had to add auto installation support in [0])
[0] https://github.com/C-Loftus/QuickPiperAudiobook
https://codepen.io/logicalmadboy/pen/RwpqMRV
But yeah, if it's like any of the others we'll likely see a different "model" per language down the line based on the same techniques
(but somehow LLMs handle multilingual input perfectly fine! that's a bit strange, if you think about that)
I'm curious, but right now I don't want to install the package and run some code.
That being said, the ‘classical’ (pre-AI) speech synthesisers are much smaller than kitten, so you’re not wrong per se, just for the wrong reason.
https://project64.c64.org/Software/SAM10.TXT
Obviously it's not fair to compare these with ML models.
Running `man say` reveals that "this tool uses the Speech Synthesis manager", so I'm guessing the Apple Intelligence stuff is kicking in.
Might well be other issues behind that, and unclear if need any other dependencies that kitten doesn't rely on directly like torch or torchaudio? but... not 5 mins easy, but looks like issues might be able to be worked through...
For reference this is all I was trying basically:
to get the above error.[1] https://github.com/elixir-nx/bumblebee/issues/209
BEAT THIS!
I'm surprised that phone manufacturers do not include good TTS models in their browser APIs for example. So that websites can build good audio interfaces.
I for one would love to build a text editor that the user can use completely via audio. Text input might already be feasible via the "speak to type" feature, both Android and iOS offer.
But there seems to be no good way to output spoken text without doing round-trips to a server and generate the audio there.
The interface I would like would offer a way to talk to write and then commands like "Ok editor, read the last paragraph" or "Ok editor, delete the last sentence".
It could be cool to do writing this way while walking. Just with a headset connected to a phone that sits in one's pocket.
(I've just tried it again without seeing that issue within a few pages)
> Siri voice or some older voices
You can choose "Enhanced" and "Premium" versions of voices which are larger and sound nice and modern to me. The "Serena Premium" voice I was using is over 200Mb and far better that this Show HN. It's very natural but kind of ruined by diabolical pronunciation of anything slightly non-standard which sadly seems to cover everything I read e.g. people/place names, technical/scientific terms or any neologisms in scifi/fantasy.
It's so wildly incomprehensible for e.g. Tibetan names in a mountaineering book, that you have to check the text. If the word being butchered is frequently repeated e.g. main character’s name, then it's just too painful to use.
> But there seems to be no good way to output spoken text without doing round-trips to a server and generate the audio there
As people have been pointing out, we've had mediocre TTS since the 80s. If it was a real benefit people would be using even the inadequate version.
I tried to use it...
Its python venv has grown to 6 GBytes in size. The demo sentence
> "This high quality TTS model works without a GPU"
works, it takes 3s to render the audio. Audio sounds like a voice in a tin can.
I tried to have a news article read aloud and failed with
> [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Expand node. Name:'/bert/Expand' > Status Message: invalid expand shape
If you are interested in TTS, you should explore alternatives
https://github.com/KittenML/KittenTTS
This is the model and Github page, this blog post looks very much AI generated.
I ask because their models are pretty small. Some sound awesome and there is no depdendency hell like I'm seeing here.
Example: https://rhasspy.github.io/piper-samples/#en_US-ryan-high