Audio signal visualizer. IMHO a useful thing for audio control. Or not?

[IgorA100]

I have a potential (alas, free, but a good friend) customer who needs audio monitoring and preferably visual.
And I had an idea to use ZM and "attach" a visualizer to ZM on the Montage & Watch page.
I liked the visualizer: https://github.com/hvianna/audioMotion-analyzer
It would be possible to add a parameter in the camera settings, for example, "Visualization". The value of the parameter is "Video", "Audio", "Video + Audio".
By default, the value = "Video". In this case, the behavior of ZM will be the same as now.
If you select "Audio", then instead of the video stream, the audio analyzer will be displayed.
If you select "Video + Audio", then the video stream will be displayed, and under it the audio analyzer.

@connortechnology what do you think about this idea?

[IgorA100]

[connortechnology]

I love it.

My inly thought I'd it will mess with layout in montage if some have it and some don't. But whatever. I'm sure you will work it out

[IgorA100]

1. I will try to make sure that both the standard layout and the custom layout are not corrupted when using "Video + Audio"
2. When using only "Video" the layout will not be corrupted 100%. When using only "Audio" the layout should not be corrupted 99.99%.
   I will try :)

[IgorA100]

I wonder if it's hard to implement audio detection, similar to video?
I think that would be great!

[connortechnology]

I think it should be pretty easy. Volume should be easy, then can look at certain sound detection like glass breaking, It is a whole world untapped. I want to do bird.net identification.

[IgorA100]

I want to do bird.net identification.

Yeah, I saw your Slack messages. Do you want to recognize birds by song or by picture?

[connortechnology]

Song. Most of my outdoor cameras have amazing mics actually. Can hear anything going on in the neighbourhood. Tons of bird song.

[IgorA100]

Audio stream not working when using ffmpeg for live viewing on Watch page?
I didn't pay attention to this before because I mostly used RTSPToWeb. Now there is no sound when using ffmpeg

[connortechnology]

rtsp2web is the only way to get the audio stream on live view. Otherwise all you get is mjpeg. We will fix that in 1.39.

[IgorA100]

Audio visualization only works with MSE type so far. It doesn't want to work with WebRTC. With HLS I can't get any sound at all yet. When using AAC there is no broadcast of either sound or image...

[IgorA100]

Hm Isaac, another thought occurred to me :)
Is it difficult to implement recording of an audio stream without video? For example, if only audio control is needed and so as not to take up extra space in the storage.

[connortechnology]

not difficult

[IgorA100]

Great!

[IgorA100]

@connortechnology I need to decide on a name for the new option so that I don't have to redo anything later.
Now I did it like this:

Your opinion?

[connortechnology]

What does it mean to have visualisation of video? I don't understand the setting. Seems to me it should just be audio visualisation: on/off

[IgorA100]

What does it mean to have visualisation of video?

:))))))) Yes, I don't like it either....

on/off is not the best option, because I want to make it more universal.

1. Only video, as it works now. (OFF)
2. Only audio, without video. That is, there may be a group of cameras where it is not necessary to use video at all, but only audio control is needed. (...???...)
3. Video + audio (ON)

[connortechnology]

The word Visualisation does not apply to video. We Visualize audio, but we view video. So There should be a line for Video Player: None, Janus, RTSP2Web, etc. Then Audio Visualisation... and I guess theoretically we might have different options there, so maybe use a drop down with None, option1, option2 etc.

[IgorA100]

The word Visualisation does not apply to video. We Visualize audio, but we view video. So There should be a line for Video Player: None, Janus, RTSP2Web, etc.

Ok, I agree.

1. FFmpeg
2. RTSP2Web
3. go2rtc - I think you are implementing this, ?
4. Janus
5. None

Also, probably, it is necessary to add an additional field to the DB to store this option?

Then Audio Visualisation... and I guess theoretically we might have different options there, so maybe use a drop down with None, option1, option2 etc.

I didn't plan it that way. When you enable audio visualisation, an additional set of settings will be available, in which you can configure different modes.
I use this visualiser: https://github.com/hvianna/audioMotion-analyzer

[connortechnology]

And in the end, this is just the default when we go to view the video, we should be able to toggle any of these while viewing...

[IgorA100]

Yes, of course.
But, it depends on what you mean by "toggle any of these"
Switch the Video Player type or audio visualization?
If you mean to switch the Video Player type, then this is a separate job, not related to audio visualization.
The ability to "disable/enable/default" the audio analyzer for all monitors will be provided on the Montage & Watch page.

[IgorA100]

I suggest dividing it into two stages:

1. Add a checkbox to enable audio visualization (add EnableAudioVisualization to the DB). Additional audio visualization options will also have to be stored in the DB, right?
2. Implement switching of Video Player including the "None" option
   Ok?

[connortechnology]

yup, soudns good to me.