Hey! I'm the author of this thing. Let me know if you have any questions about it.
FYI, the generated C code is now part of the dev branch of Soundpipe here: http://pbat.ch/proj/soundpipe.html, my music DSP library, and it has also made it's way into the develop branch of AudioKit: http://audiokit.io.
Also, check out the original implementation Pink Trombone: https://dood.al/pinktrombone/. It's the perfect interface for this kind of model.
I would completely expect yours to be better in all ways (mine was basically a line for line transcription of all the audio and stripping out the UI stuff, where as you actually went to the effort of understanding the thing being ported).
I found your code last night, and it really freaked me out that there would be another person in the world who would want to make a C port of pink trombone and call it Voc as well.
Voc is pretty much a line-for-line port of PT as well, but I removed some bits like the simplex noise. I also wrote some small utilities to go with Voc, like small plotting programs and some plugins for my audio language for both the whole source-filter model and just the filter.
Still sinking my teeth into the literature. Voice synthesis has a very rich history!
What do you mean oldish version? Looks like it's from 2017 , or from an old version of pink trombone. Do you have the latest code in source control anywhere online?
Very cool (soundpipe). Since I was a wee lad, I've been thinking about colonizing all major open source DAW/sequencer/audio editor programs with a single processing core, to the extent which would be acceptable. I think a lot more could be done if these individual projects (like Audacity, Hydrogen, Lmms, etc.) were just basically "shells" on the audio processing pipeline.
I'm interested in doing the reverse actually - take a voice recording, reverse-map it to the parameters of the model, then use those parameters and a different physical vocal tract to change your voice!
This would have a large audience of voice and speech training professionals if it were available "ready to use" with no coding or compiling knowledge required. I think it's awesome!
Interesting use of Knuth's literate programming concept. I remember reading about it and the source code of TeX in PDF format a long time ago, but now that I have a chance to read another piece of code written in it, I'm finding it less readable because of the proportional fonts than if it were equally-commented but in a more conventional monospace programming font.
> I'm finding it less readable because of the proportional fonts than if it were equally-commented but in a more conventional monospace programming font.
My initial motivations for using literate programming with Voc was to take advantage of TeX's math mode to express what was happening numerically in code, as well as to have the ability to use BibTex inside the code.
My goal was really to build vocalizations, and not necessarily to produce speech. This engine is a bit more low level than that. It could be possible to build a speech engine on top of Voc though... next steps perhaps?
The font issue should be fixable with some configuration. I personally do literate programming in org mode and generate HTML files from it for my colleagues and myself. With it, code blocks output to HTML use a fixed width font. I haven't checked the tex output (low priority, the HTML output is good enough for me).
Hey yeah! I actually came across that when I first set out to make my project.
I was going to actually fork off your project, but decided it would be cleaner/faster to do it off the original code since I wanted to write it in ANSI C.
Slightly off-topic sorry, but why ANSI C and not C99 (or C11)? Is it common to come across systems where C99 isn't supported, or are there reasons why you prefer C89 to C99?
There aren't any real reasons I have for choosing C89 over C99. Both tend to be very portable, which is very nice if you aren't sure what operating system you are running on (if any, in many situations). I still write many programs using the "-std=c99" flag, but I never find myself in dire need of the extensions, basically honorary ANSI C. For projects like this that just do numerical processing, C89 C really isn't that much more of a hassle.
It was very encouraging for me to see some ports to C/C++ already in progress. At the time, it was definitely an overwhelming notion. That chunk of JS code looked impenetrable to me.
Nice, are you planning on releasing a Eurorack form-factor kit for your platform? I'd imagine this can be a good alternative to other digital modules like nw2s::b and Rebel's OWL.
Hello, yes we are going to do a limited edition run, it is actually a collaboration with Rebel Tech! From our latest Kickstarter update:
"The Bela Modular breaks out the audio, analog and digital I/Os to jacks, and handles voltage scaling for Eurorack-compatible CV levels, providing a total of 2 audio in, 2 audio out, 8 analog in, 8 analog out, 4 digital in, 4 digital out and 4 LEDs over two modules, 12HP and 10HP wide.
[...]
We are planning to do a small production run of Bela Modular units later this year. Please contact us[0] directly if you think you would like one of these, or stay tuned here and on the forum.[1]"
Would this be a good target for deep learning? It's low level in some sense, but still nicely parametric. It strikes me that this could be a good synth for some neural nets to learn how to play.
Yes, speech recognition and speech generation would be easier to implement if you used neural networks that were trained on these vocal cord inputs rather than audio samples. In either case, you'd need to solve the inverse problem to generate vocal cord parameters given an audio sample. This seems difficult but I'd imagine some commercial software packages do it to some extent.
I would imagine a neural network that fed parameter values to Voc would be far faster (real-time?) than something like WaveNet which needs to sample the output thousands of times a second.
It seems like it would be, right? There are 44 tract-diameters you can modify to shape the vocal tract, and these can be used to generated specific vowel formants. I can imagine you can build a system using deep learning that can find the best parameters to match a steady state periodic pitch. It's a bit how some speech codecs work, like LPC10.
Maybe, maybe not. LPC10 is a 8kHz speech codec optimized for low-bandwith signals. The Kelley-Lochbaum is a full-blown physical model of the tract.
What you put into the filter is important. The LF glottal pulse model used here is a pretty good excitation signal... aspiration noise REALLY makes a difference. It would still sound artificial, but it definitely wouldn't sound metallic.
Not directly, no. IIRC, programs like Espeak and Festival use formant synthesis, which would require explicit formant values. Voc models the tract itself... the main parameters are diameters in the vocal tract (which implicitly produce vowel sounds).
It may be possible to go the other way around and analytically derive parameters for Voc that match target formant frequencies. Not sure though...
There are other ways though e.g. if features like formants produced by the model can be differentiated with respect to vocal tract parameters, the latter could be estimated based on real data maybe
There are numerous programs that translate spoken words to mouth/jaw movements -- usually for animation (to match the 3D /2D model with the right movements for what it's supposed to say).
Yes and no. Perceptually, you don't really need to model everything to get convincing speech sounds. Most of the realism actually comes from performance, and not the mathematical model.
In a way, lips and mouth are accounted for here, but in a more abstract away. The KL model approximates the vocal tract as a series of cylindrical tubes with varying diameters. Segments of the tubes actually correspond to things like the tongue and mouth somewhat. In this model there is a really neat tongue control that manipulates these segments. It's quite expressive!
This model is a 1d waveguide, so it doesn't account for things like the curvature of the tract. More modern vocal modelling techniques include implementing a 2-dimensional waveguide, which does allow for this control.
FYI, the generated C code is now part of the dev branch of Soundpipe here: http://pbat.ch/proj/soundpipe.html, my music DSP library, and it has also made it's way into the develop branch of AudioKit: http://audiokit.io.
Also, check out the original implementation Pink Trombone: https://dood.al/pinktrombone/. It's the perfect interface for this kind of model.