Audible Genius Podcast
Being a musician today can mean so many different things. In the Audible Genius Podcast you’ll hear the stories of the people who live, work, and play in today’s vast music landscape. And through their experience see just how big the world of music truly is.
Audible Genius Podcast
Jeff Miller's Revolutionary Take on Generative Music for Relaxation and Sleep
What if the key to relaxation and creative flow is just a piece of tailored music? Join us as we sit with composer and designer Jeff Miller to explore this intriguing intersection of music and technology. Jeff enlightens us about his groundbreaking work with the Sync Project and the leading-edge Unwind Music app. This app cleverly utilized the principles of generative music to encourage relaxation and better sleep. Our conversation examines how the app matches music to the listener's mood and gradually adjusts it to achieve a desired state, an approach grounded in the ISO Principles of music therapy.
We'll also uncover how Jeff and his colleagues ingeniously used data and natural language processing to create a chatbot for Slack. This unique tool was designed to generate personalized playlists, helping to regulate mood and enhance creative flow. We also examine the meticulous process behind building the Unwind music app, spotlighting the significant roles of a diverse team, including music legends like Peter Gabriel.
Tune in as we conclude our fascinating conversation with Jeff, delving into the power of music in liminal spaces - those fleeting moments between doing and being. Here, Jeff shares how the principles applied in his work can be used to create a relaxing sleep podcast, drawing from his own experiences with meditation to guide listeners into tranquillity. With Jeff's valuable insights and our stimulating talk, you're set to discover the science and artistry behind using music as a powerful tool for relaxation and well-being. So, are you ready to tune in to relaxation? Let's get started!
Learn more about Jeff and his endeavors at www.jmcreative.com.
Listen to us on all our platforms.
www.audiblegenius.com/podcast
Our courses
Building Blocks: www.audiblegenius.com/buildingblocks
Syntorial: www.syntorial.com
Our Social Networks
Facebook: www.facebook.com/AudibleGenius
Instagram: www.instagram.com/audiblegenius/
TikTok: www.tiktok.com/@audiblegenius
Welcome to the Audible Genius podcast, where we take a behind the scenes look at being a musician. I'm Joe Hanley, and today I'm talking to Jeff Miller, a composer and designer who's worked extensively with technology that uses music to help people sleep. We'll be talking about how to create music that connects with a listener and then guides their state of mind down a particular path. This is a fascinatingly specific journey into composing technology in the human psyche. Let's get started All right. So, jeff, you've done a number of things in your career that I'm really interested in, and one of the first ones I want to talk about was your work with the SYNC project, where you made the Unwind Music app, which, simply put, used music to help people relax, fall asleep that kind of thing relieve stress. Tell me how did it work?
Speaker 2:Well, it begins with the goal of trying to figure out how to use music effectively to affect people's physiological state, and so there's a whole lot of thinking that went into it. So we can touch on any aspect of that that you like. But mechanically speaking, it was built around this concept of generative music, which I know you have some familiarity with that and I'm sure with some people in your audience do as well. But I'll just give a quick overview. Generative music was popularized by Brian Eno, and what it describes is music that's sort of ever different and changing, and it's performed by a system. So in 1978, brian Eno did this with tape loops, which is very analog. He had tape looping around chair legs and all kinds of crazy stuff, a bunch of very mechanical machines doing this work. So we wanted to do a digital version of that. It's very common in gaming generative music. But the other sort of piece of it that we wanted to add was to make it adaptive, and what that means is making it possible for the volume, the rhythm, the sample content and basically a bunch of other parameters to change in response to specific events or inputs, and in our case we were looking to use sensors, either through a phone or sensors that you would wear so that we could perform generative music but also respond to what was going on in your body, and so that's the sort of that's the palette that we're working with.
Speaker 2:When we created Unwind, which was a really interesting sort of combination of learning about how all these different musical parameters might affect somebody, and we did a bunch of research around that which I can talk about, and then building a system that would almost in a sense emulate the work that a music therapist would do when working with a client, which is rooted in this idea of ISO principle, which you and I talked about not long ago, but I can go into more detail around that as well.
Speaker 2:But in terms of how it worked, we had a system that was able to ingest a bunch of samples, as opposed to more of a synthesizer or MIDI type of performance. We used sample-based music. I composed a bunch of music to work with the system, and then we wrote a rules-based sort of system that would allow us to define specific levels of energy within a composition and then affect through these inputs of sensors or actually even through just a user input on the phone. We would understand, sort of where somebody was and then perform the music over time to take them where we wanted them to go.
Speaker 1:Okay, so let me start at the core of that. What is the ISO principle, that music therapy principle that was all built on?
Speaker 2:Sure, yeah, the ISO principle is defined as it's a technique by which music is matched to the mood of a client. This is the way music therapists work. Actually, it's at the core of music therapy. So you want to match the mood of your client and then gradually over time, modulate to affect the desired mood state. So you can imagine an example where somebody comes into the music therapist's session in a very agitated state. They're not going to play something peaceful and mellow to try to chill them out, they're going to actually try to meet them where they are with something maybe that's a little more active or maybe has a melodic content, that they, through their sort of lens of music, might meet the client mood better. And then they're going to sort of again, using their craft, modulate over time to help shift that client into a different headspace, which is just a magical if you think about it. Yeah, that is why music therapy is a thing, because there's a lot of evidence and science behind this stuff. It's fascinating, absolutely fascinating.
Speaker 1:It really is. So, since they're a human talking to a human, they could just gauge using their instinct, and then they have maybe one or a handful of instruments that they can pick and choose to start playing for them, and then they can just modify it as they go.
Speaker 2:Yeah, that's one very hands-on kind of way where you're performing music in real time, and often music therapists will have instruments that the clients can have if they're capable that they can use usually simple things, rhythmic instruments or just basic instruments that anybody could play. Sometimes therapists will use pre-recorded music as well. It really just depends on the client and on the approach. But they're still using their innate musical craft to gauge what the client's state is, and so that's what we wanted to do with Unwind. We wanted to create in this case we did it as an iPhone app we wanted to create a situation where you could put this thing in the hands of anyone and get enough information about them to understand where they are in the moment and then use generative music to meet them there and then, over time, take them somewhere. In our case, we were mostly targeting relaxation and sleep.
Speaker 1:So I love this process where you take something that's currently being done in real life human beings, human to human and then try to create this automated app version of that experience.
Speaker 2:Well, that's kind of blowing up right now, isn't it? I can't open my feed without reading something about AI, and so this was more of a machine learning model. It started with a very logical set of rules and responses, but it was intended to move in the direction where it would get better with time, and, yeah, it is a fascinating thing. I think people are of it's a polarizing idea in a lot of ways. I will say that we never set out to replace music therapists. Our main goal was to see could we create something that would be a helpful adjunct to other types of therapies in-person therapies, or even an adjunct to medication in some cases, because, as the SYNC project was kicking off as a startup, opioid crisis was in the news every day. It still should be, and it's the kind of thing where the ingredients that we were using to create this system could have been really meaningful adjunct to other types of therapies.
Speaker 1:Yeah, and I think that's one of the big things that instead of replacing a therapist, you're giving them an additional tool. In particular, I think it's right because they can do it at home. The therapist can't go home with you, so it's like a supplement to their work with you.
Speaker 2:Yeah, and we see this more and more as I've worked with a variety of healthcare professionals over the years this idea of we have a protocol to serve people in-person, but boy, we really don't want to leave them hanging when they're on their own. People need more access, and so that's always been the lens that I've looked at this kind of technology through. If you can make increased access and support the goal, then there really is nothing polarizing about it. There's always going to be a need or at least as far as I can see, there's always going to be a need for human contact and human expertise in therapeutic environments. But people are capable of self-directing and we do it all the time. This was the other insight that we had going into this work is that all of us use music and movies and TV for mood regulation. We do it without even thinking about it, yeah, yeah, and so, if you think about it through that lens, all media has potential therapeutic value. So what can you do to help people help themselves? I like to think of it that way.
Speaker 1:Yeah, it sounds like by creating this very singular experience in this app, you're curating it for them. Think about more of a wide environment, like social media, that can take you any direction of that direction a good direction. But this is more narrow, with one specific goal in mind.
Speaker 2:Yeah, well, I think the other application that we created, the Sync Music Bot, was a little bit more about curation, because that was relying on Spotify streaming platforms to bring music into functional audio playlists, which is a little different. The stuff we did with Unwind was intentionally original compositions, novel music, and the idea there is that when you're familiar with a piece of music, you can anticipate where it's going, and this concept of anticipation can be counter to letting go in the moment and allowing yourself to drift off. So it's a challenge, right, because, on the one hand, you really have to meet people's preferences. Some people do not want to hear acoustic guitars, some people do not want to hear electronic music, some people must hear acoustic guitars in order to relax. It's a really interesting thing.
Speaker 2:So you have to provide a range, I think, of sort of styles and genres, but by presenting music that is completely novel to that person and with generative it never plays the same way twice You're reducing this sort of thing that we tend to do, naturally, which is to pay attention to the music when we learn it. We learn it quickly. Most of us can name a familiar song within like three seconds of hearing it, and so if you can reduce that a little bit through novel music, then you have that much more of a chance to let your body and your mind respond to those aspects of the music that are designed to help you unwind.
Speaker 1:That's interesting. So if they know the song, like you said, they know where they're going. It almost sort of activates more like the conscious front of their mind, which is the opposite direction you're trying to take them in.
Speaker 2:In that case, yeah, yeah, particularly when you're aiming for sleep, you want to be really mindful of that. It's interesting. This is all stuff that I've held on to over the years since SYNC Project, which we started in 2016. But these sort of ingredients of mindfulness, isoprincipal, and then entrainment is the other sort of concept that I carry around with me, which is entrainment describes the way that our bodies gradually sync with external rhythms. So this is also again a lot of directional evidence and studies have been done around this. But, for example, slowing your breathing rate is shown to help reduce stress and your breathing rate actually is constantly adapting to external audio signals and other signals as well.
Speaker 2:So in music for rest, I use this concept to inform compositional guidelines in tempo and instrumental and rhythmic density and melodic motifs and repetition, stuff like that. So with original music developed for generative and adaptive playback, you can keep all this stuff in mind, whereas if you just pull a track off a Spotify, you've got to work with whatever that composition is. That's what the composer created, it's not going to change. But playlisting is interesting because you get a flow between tracks that can have a very powerful effect, just a different, different approach.
Speaker 1:Yeah, so that was a Slack bot, right, like he wrote, like an automated bot that would generate playlists for people in Slack for certain purposes.
Speaker 2:Yeah, so that was called. We call it a sync music bot not a terribly clever name, but the notion. There again, we had a pretty specific goal, which was to try to work with a set of known musical parameters and recommendation edges. In this case, we integrated with Spotify and we had the director of research at the time on our advisory board from Spotify, so we were able to get really under the hood of how their recommendation engine works and then, using that model, work within a classification system that was actually, for the most part, driven by users. So if you think about play listing in Spotify, there are millions of playlists out there that users have labeled relaxation or chill out or running. That was a big one Focus. And so what you have is a massive user base of people giving you this data saying this is what I listen to when I want to focus, this is what I want to listen to when I want to relax, this is what I want to listen to when I'm coding, this is what I want to listen to when I'm working in Adobe or drawing or whatever. There's all this information, all this data out there.
Speaker 2:So we took that body of data and the insight that people do this, naturally, and we created a chat bot that would get even more data from people in terms of what they were trying to do and what kinds of music, which artists they associated with doing that thing.
Speaker 2:So we decided that a great place to do it would be in Slack, where people are working all day long and where a lot of people are looking to. They keep that open while they're doing their work, and for many of us, work is especially creative people. Work is about getting into a flow state. There's something we have in common with people who do a lot of coding. So we created a chat bot dialogue that is really a plug-in for Slack and that would interact with both teams and individuals, and through that you could tell the bot what it is you were trying to do. Are you trying to code, are you trying to relax, are you trying to focus, do you want to have a nap? And then you'd give it a couple of indications of your preference and it would build for you essentially a functional playlist that would show up in Slack and you could play it right there, whether it was for a team or for an individual.
Speaker 1:Wow, and was it like the algorithm that you guys wrote for it? Was it just kind of a matter of feeding it all those playlists that already existed in Spotify? And then it kind of started to be able to think on its own, in the sense that it could take all the input from that user and what he wanted and pick and choose songs.
Speaker 2:I can't take any credit for any of that because I'm not an engineer or a data scientist, but those folks had it working intelligently out of the gate. So we weren't just pulling playlists from Spotify, we were adding our own sort of secret sauce to it, which was rooted in some of those early data insights we were able to get our hands on, but was really more about the dialogue we were having with you in Slack, and it was a very natural feeling kind of dialogue. And so the moment you gave us this idea of an example, input would be I want to code to Miles Davis and Deathpunk and you could give just those two sort of insights and that would tell us what your range of sort of preference was, and we would, along with that then, knowing what we knew about the recommendation engine under the hood and all of the parameters within that available to us, we would sculpt a functional playlist for you that met your preferences as well as took you on a sort of a journey.
Speaker 1:So you could speak to the bot in plain English, like you could just say, yeah, this is what I want to do, and it would be able to parse that and figure out what to do.
Speaker 2:Yeah, that was our first sort of foray into approaching natural language processing stuff, which, again at that time this was around 2017, I guess Now chatbots were kind of a big deal. So, as a startup, we were just looking for ways we could approach our mission, which, at its core, was using music to help people in positive ways. In more prescriptive ways, whether it was an iOS app or a chatbot didn't matter to us. What mattered was can we get deeper on this, can we find more data, Can we get more truth around this and can we present it to people in a way that feels good and honest and authentic?
Speaker 1:That's cool, is that bot still active.
Speaker 2:Unfortunately, neither of those neither on the Unwind app or the bot are still out there, the reason being that Sync project was acquired in 2018. And so along with that was our technology, and it's really up to the people who acquired us whether or not that stuff is ever presented again.
Speaker 1:But well, now going back to the Unwind music app. So you had this idea you wanted to take the ISO principles of music therapy. You wanted to turn it into an app form composing music, generative music, all that. So how do you get I want to talk about the music part, but how do you get to that, like, how do you get to the point from the idea to the OK? Here's how we should structure this music.
Speaker 2:Well, it's funny because in my case I really had to lean into my experience as a design professional. Everything OK with her.
Speaker 1:Yeah, yeah, I trapped my coffee. It has a lid, thank God. Glad to hear that.
Speaker 2:Yeah, so I have, along with this lifelong pursuit of music, I've got a significant career as a design lead in Boston and I just relied on that experience mostly to figure out how to get from A to B. It's not intuitive. You join a startup like a Sync project with a weird mission, sort of like a health adjacent mission. Everybody in the company is a musician also and I had to think about all of the practices that I would use to design an app or a website or any kind of piece of multimedia or communication, and it's really just foundational user experience design principles that come into play there. You have to first define who your audience is, what their pain is, what you're trying to do, and you have to speak about it in very human terms. So for me it was actually super exciting because when we joined Sync project it was about 10 of us and we had this crazy idea for Unwind.
Speaker 2:One of the people on staff was a student music therapist. We had a data scientist, a couple of data scientists, a couple engineers, full stack engineer, ios engineer. We had our founder, who was mainly a musician and sort of an entrepreneur and a lead technologist. So here we are in a room with this idea that we all have a lot of passion around. For me it was a and I'm sorry I should also say at our advisory board. Not only did we have the director of research for Spotify, but we had people like Peter Gabriel and St Vincent and Esapeca Solomon, and it's just an incredible cast of people available to us. I didn't get to spend as much time with those folks as I would have liked to, but for sure I can have one memorable meeting with Peter Gabriel showing up at our offices in downtown Boston, and that was an incredible moment because I'm able to do that thing that you do as a designer, where you talk about the audience you're trying to serve, you tell a story about what you want them to experience.
Speaker 2:You kind of go deep on what the potential components might be of that system and what that sort of interface for that might feel like, and then you get feedback and you have to be able to. As a designer, I think the obligation is to sort of, regardless of the discipline of the person sitting across from you, your obligation is to take their reaction to what you're describing and incorporate it. So it's very much a multidisciplinary, cross-functional approach to this puzzle of how do you create something that's ostensibly going to augment or, in the moment anyway, serve as sort of a proxy for a therapist, and there are some techniques in design that lend themselves very well to this, and if you have an engineer in the room, they're going to ask specific questions and then you just try to answer them together.
Speaker 1:That's fascinating. So is that what say, like Peter Gabriel and Saint Vincent Esapeca? Is that how you work with them, where you would just talk to them like telling them here's the thing we're making and here's our idea of how to make it? What do you think, just like here, with their experience, would just prompt them to say what their opinion would be of it?
Speaker 2:Yeah, and in those cases these are folks who are working in an advisory way. They didn't need to get their hands dirty with the technical stuff, although a lot of willingness to talk there, and certainly someone like Peter Gabriel. I mean, I'll be honest, and there's a picture of me on my website with my arm around him and the whole team is there, and I'm just kind of blown out in the moment as I'm there Because I mean, my God, if anybody understands emotions and music and it's him. But yeah, mostly it was just sort of like hey, here's what we think we're doing, mr Gabriel, and he's like oh, that's interesting. Here are some questions I have and here are some thoughts and with him I won't quote him because I would hate to risk such a thing, but my takeaway from it is that his concerns were primarily very human and very much like he was very interested in bringing it back to what is going to resonate with a human being, which just tells you what kind of person he is, I think.
Speaker 1:Yeah, yeah, it explains why his music is so powerful resonates with people.
Speaker 2:I guess that's where his mind is.
Speaker 1:OK. So then you go through these different processes. You get the strong idea of how you want to create this music. So now, like you said, you're using samples, you were composing some music and this is all being incorporated into a generative engine. So how does it work exactly? Is there, say, a library of samples, or like chunks of music you've composed, that it's piecing together in real time? Is that?
Speaker 2:Yeah Well, to answer that question, I might have to back up into some conceptual stuff a little. So, from a compositional perspective, I'm just working in, I use Reason a lot, I use Pro Tools, just like you pick your day DAW, I might be working with acoustic instruments, I might be working with electronic instruments in this case I was working with all of them but it's really all about intentionality as well as expression. So you know, I would structure these compositions so that they could modulate. So you have to make some assumptions that are somewhat universal when you're doing that. So when I talk about ISO Principle and meeting people where they are, I had to have a mental model in place where people are listening to this stuff on purpose, because we've said that we can help you go from a space where maybe you're struggling to fall asleep which might be as a result of stress or a racing mind and we're going to help you come to a place where you can rest. So I have to make some assumptions up front that you maybe are a little bit more awake or a little bit more agitated, or for whatever reason. You've chosen to do this listening, and so that's my starting point. From a compositional perspective, I'm coming at it from the place of. Let's assume that not everybody is showing up to this in a relaxed way, ready to fall asleep. Some people are going to need a little more distraction, a little more engagement, and so here's the basic recipe that I keep in my mind, which is especially for music.
Speaker 2:For rest, there's this notion of moving from a high density of layers to a minimal density of layers. Moving from a high beat salience to zero or very low and steady beat salience, moving from easy pattern recognition to no pattern recognition or chord changes to no chord changes, active melodic motion or broad melodic range to sparse melody or melodic range. So a lot of times I would compose things in either a ramp. You could actually look at my Pro Tools or my Reason File and you would see just the layers going down to nothing, from like 10 to zero. You could see the MIDI data very dense over here and almost just a bunch of big long drones over here. Sometimes I would compose them as more like a triangle, going from zero to a pyramid, like from zero to a peak and kind of back down again, and from all of that, from all those different layers, it's a lot of stuff that I think would be familiar to anyone who works with a DAW. You've got a choice you can export stems, which is what I think a lot of generative game engines use.
Speaker 2:Or what we did is we got a little more atomic than that. I would get down to sort of I've got these instrument lanes and within these instrument lanes I've got a variety of motifs that are living in there and I'm going to slice those up, and so we had a beat model on the other end where we could pull those samples in and tell them sort of when and where to show up. But we would use this concept of generative music and adaptability to say, well, if we think you're sort of somebody who's at a, let's say, there's a low, medium, high range of arousal state, if we think you're highly aroused, we might start off with more energy and then, over 10 or 15 minutes, start to wind that down. If we think that you've sort of shown up in more of a medium arousal state, we'll start you there and bring you down.
Speaker 2:So it's an interesting thing when you think about meeting people where they are. The more aroused you are and by aroused I mean the more sort of awake you are the less sleepy you are is another way to say the same thing. Or, just in general, are you showing up with sort of a need for help I'm sorry, let me go back on this your mood or your arousal state. You could think of it as how you perceive your state of wakefulness or your stress level, or your awareness of stress, or even mental or physical energy. If you think of that as a continuum from low to medium to high, and then you try to match that with this idea of pattern recognition or active chord changes or active melodic motion that's the craft of it is thinking compositionally, how to take the music that you're making and map it to that continuum and then slice it up and tell the system when to play what.
Speaker 1:Now a quick word for our listeners. Jeff is talking about composing music in the DAW, which is essentially music recording software. This is how so much music is created nowadays, not just for sleep purposes, but for any reason you can imagine. If this is something you want to do, I highly recommend our super effective course, Building Blocks. It'll take you step by step through the process of writing the building blocks of beats, like drum patterns, bass lines, chord progressions and melodies, and it all takes place in an actual online DAW where you create music as you go and for our loyal podcast listeners, you can get 20% off Building Blocks this month using coupon code SLEEP. Check it out at audiblegeniuscom.
Speaker 1:Yeah, that fascinates me and so many love us. One thing I loved was how you intentionally create music that's active. Whether it's like you said, the beats will salient or the melodies dense. The chords are changing a lot on purpose just to get their attention Kind of the way you have to get like if I need my kid to stop screaming around the house, I think it is attention first and then I have to calm them down, and it's that same idea, I think that's the amazing part of it, because, like any of us, I could go on to Spotify or YouTube right now and I can find something mellow that I think will pull me in the right direction if what I want is to relax.
Speaker 2:But it's amazing to think that there's technology that can actually sort of steer your mood over time, make it easier for you to acclimate. This is one of the reasons I started the podcast that I have too is because a lot of times if you go and look for content that's going to help you sleep, you'll find a lot of stuff that just immediately starts in that very sleepy space. You'll find a sleep story or something, and it's like you're supposed to go from 60 miles an hour, like I've been doing things all day. I'm very busy, my head is just hit the pillow and suddenly you're saying in a very sleepy voice follow me into story land. I can't do that, I'm not ready yet.
Speaker 2:Or, like I personally, I have a meditation practice. I'm very comfortable with mindfulness practice, but when I get into bed in that moment I'm not necessarily ready to start doing a deep breathing exercise. In fact, in a lot of ways, meditation is an invitation to be awake. It's about present moment awareness. It's not so much about drifting off. It's all the things that are intuitive that would help you relax. But at the same time, I think people do appreciate if you can meet them where they are and then move towards a space as opposed to just dropping them right into the water as it were.
Speaker 1:So you compose all these tracks of different levels of activity, I guess we could say, and all those different elements you mentioned, and you feed them to the engine and are you really having to tell the engine, okay, this is active, this is less active, like almost kind of rate the activity of each musical chunk, in a sense?
Speaker 2:Yeah, that's where we started, and again, I can't take credit for how the system ultimately performed and ultimately it did perform quite well but I started off with something that is composed in a static environment, and then we wrote what I think of as sort of rule sets around. These are the ingredients that are allowed in a high energy mode, these are the ingredients that are allowed in a medium energy mode, these are the ingredients allowed in a low energy mode, and that might also include some instructions around tempo. Some of that might have been baked into the compositions themselves or the samples themselves, but the idea and the way that it was constructed was that we would first feed it through sort of like this logic-based, rules-based mechanism and then the machine learning would take over.
Speaker 1:Got it. And when you say these are the ingredients, how granular did that get Like? Were you telling the thing like, hey, when the melody is playing this many notes or that kind of thing, or is it more like these clips are?
Speaker 2:active, just for sanity's sake. I had a limited number of instrument lanes. I think probably the busiest one was like 12. And within any instrument lane I might have had like 10 or 20 different clips and so. But they were designed compositionally in such a way that I knew which clips had more active melodic pieces. So let's say it's a guitar lane, I knew which one had the sort of busier guitar licks, I knew which one had the more sort of spacey guitar licks, and I could sort of target those not only just as clips but in terms of like, repetition or frequency or variety. Even so, when I'm composing this stuff, this is the kinds of things I'm thinking about Like is this a melodic motif that I will want to have repeat frequently? How many variations of this motif do I want? And then it's easy enough after that, because of the geniuses I was working with, to sort of tell the system like this is what I was thinking, that's cool, yeah.
Speaker 2:That's very cool. I mean, there's no GUI for it at that level. This was me sitting next to a very wise and experienced data scientist and a couple of engineers who, thank God, we all could talk about music together. We had that common language. So, as we're staring at these lines of code, we're also talking about things like melody and harmony and intended rhythm and stuff. We could do that. If I hadn't been working with musicians, I would have been up a creek.
Speaker 1:Oh yeah.
Speaker 2:Can you imagine, yeah?
Speaker 1:I would imagine they would need to have both a code brain and a music brain, so they knew how to connect them. Yeah, and then you guided them and how to connect them right. Is that kind of how it went?
Speaker 2:Absolutely, and it was through the design methodology, too, that we not only were able to leverage that common language but apply it to building this incredible piece of technology. But it's interesting In my experience, I've worked with a great many engineers and designers and technology folks who are also musicians. It's just, maybe it's a Boston thing, I don't know. I see it everywhere, though.
Speaker 1:Actually a friend of mine who he worked in sales for a long, long time. He's like you know what I want? To become a coder. Yeah, he lived in New York and he went to this really good coding school called Flatiron School Only like 2% of people get in. And after he got in and the owner said to him he's like we look for people with music experience. We just find there's something like I don't know some about their brains. They could just connect the dots like easier or faster and it's no more malleable in some way.
Speaker 2:Yeah, I mean I think there's there's directional evidence that linking neuroplasticity and creativity and music together. It's a I don't know. I don't like to think about it too much like it's a superpower. I like to think that actually anybody could be musical, anybody, given the time, like any other craft, they could find their way to it. I just know that it has helped me. I don't think I would have had a successful design career if I hadn't first spent my 10,000 hours as a musician. I didn't go to school for design, but nonetheless I've been working in that field for over 25 years with fairly massive teams and some very well-known established brands. It's just, and all of that is on the back of me being a musician and a band leader.
Speaker 1:That's very cool Now, with all these different instrument lanes, with all the different clips in them, were they all interchangeable? Could you say, combine a clip from instrument lane one with any clip from instrument lane three? Could they all be swapped in that sense?
Speaker 2:Yeah, it was very much an exercise in intentional arranging, okay. So again, getting back to this idea of intentionality, we had to create a system where we could, from a very literal, rules-based approach, say if we're targeting a medium state it was actually a little more granular than that. I'm making it simpler just for the discussion. If we've started you off at level three and we're working towards level one, well, when we get to two, this is what we think the arrangement should be. And then, within that system, we also had probability.
Speaker 2:So it wasn't just a matter of what's the probability of an individual clip in an individual lane, it was what's the probability of these two lanes interacting with each other? What's the probability of these two lanes interacting with each other? Because ultimately the machine has to have some freedom to say I'm going to render this performance now and I have choices, I have probability on my side. So, yeah, it's a trip. It's really wild to approach your music just to be sitting down with a blank canvas in front of you, like you always do, and then these are the things you're thinking about. It's no longer static, it's not going to be played the same way every time, and so suddenly that shifts how you think about your intentionality.
Speaker 1:Yeah, I bet, with that different end goal in mind, how much of it was just for you when you're composing this? How much for you was it just instinct, like just feel, versus how much thought this might be active enough or this might not be that kind of thing.
Speaker 2:Well, yeah, I think that's the craft. That's what a music therapist does. I'm not a music therapist, but I am a student of jazz. It's an improvisational sort of instinct that kicks in. And here's the thing it's like there was never any intention to write the one piece of music that was going to work for everybody forever. It was sort of a known thing. We're going to have to compose dozens of pieces. We're going to have to. Ultimately, what we were trying to do was build a platform that many artists could contribute to and over time, people would use the system and it would get smarter. And, more importantly, I think it would get smarter in service of an individual, because your individual needs and preferences are not static either. They change over time.
Speaker 2:For me, it was very much instinct-based, and I think I was along with thinking of things like rhythmic density or how many layers do I have going? One of the other factors that was really top of mind for me in these compositions was this idea of valence, which is another term that I wasn't even aware of until I started digging under the hood of Spotify to see how their recommendation that didn't work, but valence describes the musical positiveness that a track conveys, so tracks with a high valence sound more happy and cheerful, euphoric, and tracks with a low valence sound more negative, sad, depressed, angry. So that was really mostly what I was focused on with these compositions and I gave myself sort of like a baseline set of tempos. I didn't really want to do anything too up-tempo most of the time. It is intended to be relaxing music. Ultimately, dramatic shifts aren't good to promote sleep.
Speaker 2:But with my music I'm typically thinking modally. So I like to talk about the Lydian mode because basically it's a major scale which most people associate with happy music or resolved sounding things. But of course you've got that raised forth which to me introduces tension and uncertainty, and it's what I think of as sort of like a Lydian. I think of that as sort of a liminal mode, and so I personally find it's very beautiful, and since the goal is to meet people where they are, I don't shy away from Lydian mode or something more minor, sounding like Dorian, which also has that sort of liminality built into it with the natural six.
Speaker 2:I'm just careful about how much tension I allow for within that and I'm mindful about modulating in a direction that reduces harmonic tension over time. Interestingly too, it's shown that music with a low valence is actually shown to promote rest in many people. So this is, I think, fascinating. Maybe this is why we love to listen to dark side of the moon. It's not always about happy rainbows and sunshine. When you're trying to relax, it's very often something rooted in just minor pentatonic is going to have that physiological effect of chilling people out. It's remarkable.
Speaker 1:So with those unexpected scale degrees, like the Lydians raised forth, everyone's kind of expecting that regular fourth, even if they know theory or not. They just heard the major scale a million times. So that raised fourth is just like huh. Is that sort of the one way you're kind of trying to fight and not fight but remove anticipation? Those little unexpected things that kind of keep just sort of rocking their boat a little bit?
Speaker 2:I think you're making my point in that that is the choice we have as musicians. Right, we have a lot of choices available to us, even Western music. We've got 12 notes that can be combined in any kind of way. But that's where Kraft comes in and it's where I think this is the value of really developing yourself as an artist and developing not only your sort of empathy engine, which is, I think, super important, but your ability to intentionally go for things. I think about. This is just what great songwriters do you listen to? Peter Gabriel's so, or Michael Hedges' Aerial Boundaries, or any soundtrack by Hans Zimmer or Michael Giacchino, whether it's vocal or instrumental.
Speaker 2:There's an incredible amount of intentionality in that writing, in the moment. What is the scene that is happening? What is the picture I'm trying to create? Hedges did it with just an acoustic guitar, but it's remarkable. He has a song called, because it's there and you're just completely transported, and I think that is the power of music is to transport people, just like storytelling. So those are the choices Will I?
Speaker 2:Will I use in this composition, will I use the tension of that sharp four to grab your head in the moment and take it out of the rumination that you might be engaged in. Sure, yeah, I might do that. I also might place it in a different register as part of a melody, just to add a sense of wonder, because it can also do that. Right, you take a major 7 chord with that note floating on top of it. I mean, it is magic. So it's a really interesting context, is everything. But this is why, you know, when I think of this kind of work, I think it's very artist driven, it's very art driven and it's sort of. But once you've decided, you know, as an artist, you're going to, you know, especially if it was in a context like this where you're, you know, perhaps being presented as something that might have health and wellness benefits. You know, just be very intentional, try your hardest to be open to the idea of the context of the person you're writing for. They've just stopped their busy day. You know, for the first time in their day they've stopped doing and they're trying to move from that doing and thinking space to just a space of being, and that those transitions are very special moments, they're almost sacred. I had, you know, I think and I used this word earlier and it kind of flew by but this, this, that moment between, you know, trying to sleep and actually being asleep.
Speaker 2:You can think of this as a liminal state, which is that word liminal comes from Latin word liman, which means threshold, and those are the spaces where change is the only thing that is actually happening. You know, it's the transition between what is and what will be. And you know we can think of those as physical spaces and we often talk about mental states as spaces or places. But, like in the physical world, they're things like, you know, stairways or hallways or doorways. Emotionally it's kind of like life transitions or milestones, or you know, things that create uncertainty.
Speaker 2:These liminal spaces can feel uncertain. If you've ever walked down like a long, empty hallway or an empty, dark stairwell, you know. You know what I'm saying. And we don't we don't like uncertainty and a lot of stress sort of manifests as a result of uncertainty. And so you get to this emotional liminality when you are laying down for the first time and everything's quiet for the first time in your day. And you know there's also evidence that liminal states are linked to creativity and so our minds can get into this kind of a hyper-reality where we're trying to make sense of the uncertainty in our lives, and sometimes that manifests as art, like songs and books and poetry, but it can also manifest as you lying in bed, staring at your ceiling for seven hours. You know this transition from the waking or doing to the resting and being, rumination can kick in. And so if you're an artist and you're approaching this kind of idea, that's what you're trying to honor, I think, is that liminal space.
Speaker 2:That's how I think of it anyway.
Speaker 1:Yeah, no, I never heard that word before, but I love it because I myself meditate. I do a form of meditation called TM and it's pretty much all about that getting you taking you to that and passing you through it. And so when I sit there and I feel it, you know, back in the day it would have felt like, ah, it was almost unnerving that feeling because you're really letting go, like you're making yourself vulnerable and you're just like giving into your physiology, letting it do whatever it wants to do. But it's once you like, really lean into it. It is amazing, yeah absolutely.
Speaker 1:Yeah, and I actually changed the subject a bit your podcast Sleep Fader, which is a podcast all about helping people relax. And I was listening to an episode last week and I started to feel that and I didn't even have the intention of relaxing or sleeping I was like I'm going to listen to this podcast, I'm interviewing Jeff next week, I want to check this out. I'm sitting at my desk working and I'm just listening to it and all of a sudden, I didn't take long because you started, you talked a little bit and then you played this music. You're just like, hey, listen to this music for a bit, and I'm just doing that.
Speaker 1:And all of a sudden I was like, oh, like I started to feel that liminal that I get with meditation Wonderful, it just did it to me. I wasn't even, I didn't have the goal or anything like that. It just did it to me. And I noticed in that music what you were talking about. You were paying attention to how the tracks were coming and going. I think you started with one or two things and you added some, and then again it started to take those tracks away.
Speaker 2:Well, I'm very happy to hear that and thank you for checking it out. I mean, it is, the podcast is a manifestation of all this work that I've done over the last seven years. It's not knowing where that work may take me next. A podcast is a wonderful vehicle to sort of bring all this experience together in a place where I can really be super intentional in any way that I want. And so, for me, it comes back to those core ingredients.
Speaker 2:I'm thinking about mindfulness and the effect it's had on my life and my ability to cope with stress, and the fact that, by definition, mindfulness is awareness that arises through paying attention on purpose. So, what am I going to give you to pay attention to on purpose? Well, I'm going to give you, in this case, I'm going to give you music and I'm going to give you storytelling. These are the things that I think are have been shown for eons to help people move from one state of mind to another. But what I'm going to do is think about things to like ISO principle and entrainment, and I'm going to think about all this stuff we've been talking about in terms of choices we make with music and sound, and I'm going to apply this recipe to a format that anybody can access from anywhere and hopefully, before they get to the end of the story, they're out.
Speaker 1:Yeah, I started so, yeah, it was your introduction music story and then it was like a calming sound at the end I think in this case it was waves, you know, like ocean sounds. And I found myself halfway through the story. I was paying attention to the story and then, when I'm realizing it again, this reminded me of meditation. I wasn't, I was just like off and just like so relaxed, just sort of sitting back in my chair, and it was a couple of minutes ago. I was like, oh wait, I'm not, I'm not listening to the story anymore. Is that kind of the goal? Like you want them at some point to find themselves just off in their own world, in a sense?
Speaker 2:Absolutely. It's funny because I've always wanted to be a novelist as much as I wanted to be Eddie Van Halen or Michael Hedges. I wanted to be Stephen King. I just I guess I'm an escapist of some sort, but the storytelling piece of it you know, as applications like Headspace and Calm have been bringing sleep stories into the mainstream, my observation is that a lot of these were self-contained, kind of one-offs, and again they wanted to bring you very quickly into that sleepy space. The sleepy sort of fake sleepy voice starts immediately. And this is not to disparage either of those applications. I actually use both and I think there's amazing benefits to using them.
Speaker 2:But for me personally, I was very interested in this idea of you know, could I use a podcast format to help people sleep, to practice telling a story that was more serialized, which is people love to binge serialized stories. So I wanted to give people something to look forward to and could I write it in such a way where it really doesn't matter if you finish the story or not? If you listen to episode one and you don't make it to the end, can you listen to episode two and still feel comfortable Like you haven't missed anything. So that's been an interesting exercise. I don't know if I'm succeeding on that front yet. I'm waiting for people to tell me. Some people have told me they will. Actually, you know, they'll listen to the first couple of parts of the podcast and then they won't make it to the end of the story and so the next night they'll go back like halfway and then listen through to the next episode. I think these are the things I can't control and I'm just. You know, I'm so grateful anytime I get any kind of feedback from folks in terms of this.
Speaker 2:But it is structured very intentionally as a three act thing. Act one is I've got like the same ambient environment every time. I call it the studio lounge. This is just me talking in almost like a very natural, non-sleepy way. It's conversational. And then, within that that first act, I will always play a piece of music that I've created and I'll talk. I'll sort of try to give you some present, aware, present, moment, moment, moment, awareness, cultivating mindfulness cues. That's usually about five or six minutes. And then act two is going into the actual story itself, which is, you know, it's original. I write it all. It's episodic, so you know it plays out over, over different episodes and within the story.
Speaker 2:I try to include not mindfulness practices, but I think of them more as sort of mindful moments where you know you're in a sense invited to inhabit what the character is experiencing. And I write these very purposefully to give your mind an opportunity not only to follow a story but to observe and appreciate and maybe within that find some sense of gratitude. You know I try to explore themes that people can relate to, and there's always in within the story musical interludes too, which is it's a common practice. You'll hear it in some of the best public radio podcasts. It's a great storytelling device, right for transitional moments to use music. I'm being very deliberate about the music, as we've been discussing. So you know from Act One, where I'll start with something that's meant to engage and sort of, you know, pull your mind away from your busy day and any rumination you might be engaged with. I'll use some of those same ingredients in Act Two, but likely I'm going to reduce some of those layers. I'm going to reduce some of that beat salience that we talked about. I'm going to reduce them a lot of cringe.
Speaker 2:And then Act Three, so after the story's done, this is usually about 20 minutes or 25 minutes in.
Speaker 2:It's just down to ambient environment. So wherever the story leaves you, whether it's by a waterfall or in a rainy city or wherever you are, that's where you're going to stay. And my voice, of course, is modulating over time so that by the time you get to the end of the story I've probably slowed down my pace a little bit. You might not even notice when I stopped talking, and the reason I have a big long usually five or 10 minutes of ambient sound at the end is because you don't want to make any abrupt changes with audio, whether it's in somebody's ears or a speaker next to them. This is something I learned both at Sync Project and when I worked with Bose on their sleep products. The notion is a good, steady signal. Something like rain is a great example, because it's kind of like white noise. You just let that go without a lot of modulation, without a lot of dynamics, and then just give it a nice long fade out. I've reduced the chance of waking you up.
Speaker 2:Oh, right yeah okay, because it's very easy to create a waking event, like if I let two or three minutes go by and then suddenly started talking again, I can guarantee it's going to wake you up. And that happens to me all the time when I'm listening to podcasts myself. If I forget to hit that little button in the lower right hand corner that says at the end of this episode, stop It'll start up the next one. And the next thing, you know I'm listening to ads or something. Oh, okay, and that's the other thing too. Sleepfader has no ads. Like, maybe someday I'll be lucky enough to have sponsors, but this thing is brand new and, as you know, the business of podcasting is a. You know it can go in many directions. For me, this has so far been purely an exercise in taking what I've learned over the last seven years, trying to put it together in a package that I feel will be effective to really help people relax, and it's just a great creative project for me.
Speaker 1:That's cool. So the music during the story is the interludes. That's almost more of like a subliminal reduction, yeah, like, because the focus is more on the story, but the music is progressing from the intro activity to the most calm, just wave sounds or something like that.
Speaker 2:Yeah, yeah, and it's. You know, those same principles that we talked about from music in terms of you know the perceived energy, you know which could be a combination of you know layers or just a dynamic ingredients or just any kind of active motion within a track. You know I applied that same kind of thinking to the sound design too. So if you listen to the podcast, you know in that lounge area I'm talking about, you know there's some. There's some birds chirping. Sometimes there's like very quiet wind chimes playing. In the background you might hear some ocean sounds.
Speaker 2:It's generally speaking. I'm trying to create a somewhat noticeable environment because, again, I'm trying to like pull you out of whatever headspace might be distracting you from sleep into something else and within the storytelling you're going to hear very little distracting audio because I'm trying to modulate in that direction of relaxation. So, whether it's with the musical sounds or with the ambient sounds, I'm very deliberately creating like a ramp down. That's the best way I can meet a general audience of people, sort of where they are, and pull them in the direction of rest.
Speaker 1:Because you don't have, you don't exactly know where the listener's state is at that point. But with Unwind App you're using biometrics to actually measure where they were like, like, say, their pulse, like was that big one? Like their heart rate.
Speaker 2:Yeah, we would use in our prototype actually, which was a it was actually a web app. We use nothing more than the accelerometer and the phone, so and you can see some of this on my website as well but you would be able to just hold the phone in your hand and we would get your heart rate from that and use that as your starting point for what we delivered and the Unwind itself. We actually had a couple of user inputs too, where they were just simple sliders. You could say how tired are you? Zero to 10?, how stressed are you? Zero to 10, that type of thing, and we could use that as an indicator as well.
Speaker 2:I've always felt that the sensor piece and paying attention to heart rate variability or respiration are very powerful, I think, ways to inform any kind of audio based therapeutic. But there's no substitute when you're really trying to make a machine smarter. There's no substitute for getting a dialogue going with the person. Just tell me how you feel and then, ideally later on, you'd ask them how did that work out for you? Really, because I think you need both these things until you've got a super smart machine. It's artificial intelligence and machine learning. They don't happen overnight. You really do have to feed them data to make them smart, and in the case where you're talking about individual preference, it becomes even more important. So my hope is that, as these technologies are continued to be developed and these types of therapeutics are more widely adopted, is that the folks making them consider the importance of that dialogue.
Speaker 2:People want it anyway. We discovered that in research that we did as well, which is that you give people too many choices that can be paralyzing. Even the idea of just even though, like I said earlier, we all use music for mood regulation without even thinking about it all the time the moment you sort of change the context and put it in an app or in a machine next to your bed or something, it's new. It's still new this idea of hey, here's a product or a service and if you listen to it on purpose, it's going to help you. People want the dialogue. They want to actually feel like you're attending to them and recognizing them for who they are. It feels good, it creates comfort.
Speaker 1:Yeah, so the dialogue has two purposes then. One, it's also kind of feeding data, because the biometrics alone aren't going to cut it. They're part of the story, but they need to actually know what you're thinking, and together it does a good job of generating the right kind of music. But it's also just the user experience, to make them consciously feel like they're being heard. Yeah, in a sense.
Speaker 2:Yeah, and it's about. It's a big part of mindfulness and possibly TM too, but it's a. This idea of intentionality comes back. You're setting an intention and so you can take that moment of interacting with an app or a machine sitting on the side of your bed. That can be a mindful moment that enhances the experience. It's just you're stopped doing the other stuff you were doing and now you're doing this. I think this is the reason why vinyl never went away. Really, I think that we ultimately love the intentionality of it. You can't just hit a button and let it play in the background. You have to open up the dust jacket and pull the thing out and be careful with it and put it on the turntable, and you've got to make sure that you're not just dropping the needle randomly on the thing. It's a mindful act.
Speaker 1:It makes us feel good, that's a good analogy, because not now, I know what you're. I kind of know what you're communicating, because there is that feeling that when you had to actually set the music up, even set the mechanical process in place, that you're part of it, all of a sudden it's not just this thing being passively sent at you from your phone. It's a thing you are involved in.
Speaker 2:Yeah, if I walk across my studio here I have to, and I want to listen to music. I might know what I want to hear before I approach my record collection, but a lot of the time there's a little bit of flipping through the spines, even though I've been looking at them for years and I have a bunch of new ones in the mix too. It's like it's that tactile moment of touching the records, looking at the spines, thinking about you're actually thinking about how it might make you feel. You're thinking how could this affect me?
Speaker 2:In the moment, as a designer and when I think about these kinds of experiences, I love the notion of engaging that part of a person. It's going to take them out of whatever they're worrying about, whatever stresses they have, and it's going to sort of almost force them to think about how they feel and how they want to feel. If you can do that, the sensors can make things more comfortable. If you're trying to create something, it's really going to change the way someone's breathing, for example. You want to make sure that you're not doing something that's too fast or too slow. That could be really uncomfortable. The sensors can help with that but ultimately giving the person an opportunity to engage with the moment is what it's all about. Isn't that what music is? Absolutely.
Speaker 1:Well, Jeff, this has been really fascinating. There's other things that you've done. What is your website again?
Speaker 2:The Sleepvader website is sleepvadercom. That is where you can find all the information about the podcast and connect to that. I hope folks will give it a try. Anybody who does, please know that. I will welcome any feedback that you've got. My personal website is jmcreativecom. That's just my professional site where I highlight some stories about my work as a designer. In this case, I've focused quite a bit of it on my recent work over the last seven years or so.
Speaker 1:Yeah Well, it's all very fascinating. I recommend your podcast because it really unwinding me. That's unwinding, you're sleeping. They helped me calm down. You just found yourself a really fascinating niche in the music technology world. It's just fascinating to hear about.
Speaker 2:Well, thanks so much. I'm really grateful for a chance to talk about it with you. The tools that you've produced as well have been super helpful in my life. Without Centorial, I would know nothing about all the knobs and buttons sitting next to me. Now I know quite a bit and it helps me do a lot of the sound design that happens in Sleepvader. Super grateful for your work as well.
Speaker 1:Yeah, my pleasure. Thanks so much, jeff. Thank you. Thanks for listening to the Audible Genius podcast.
Speaker 1:Now, as you listen to these musician stories, you may find yourself wanting to make your own music or maybe you already can, but you feel the need to brush up on fundamentals, fill in some gaps. Well, I've got some super effective and engaging courses that help aspiring digital musicians find their voice and create music they love. And these courses are more than just a series of videos. They have interactive challenges in a music software environment where you actually create music as you go and get real experience. The first course I recommend is Building Blocks, where you'll learn beat composition and music theory in an online music studio. Check it out at AudibleGeniuscom. We also have Centorial, an award-winning course on synthesis, where you'll learn how to create your own sounds with a synthesizer. Check that out at Centorialcom, and both of these courses are designed by yours truly and the team here at Audible Genius. So if you've ever had a desire to make your own music, I highly encourage you to check them out. Thanks again for listening and I'll see you on the next episode.
