Multi Tone Piano Transcription
Automatic music transcription (AMT) is a fundamental problem in Musical Information Retrieval, which aims to automatically convert played music into a symbolic notation. Piano is a typical multi pitch instrument and has a wide range of tonal characteristics. Therefore, piano-specific AMT is an important and challenging task for which various audio-based methods have been proposed. However, the performance of those methods seems to have reached a limit, and a promising direction for improving them could be to incorporate visual information.
In this paper, we propose a novel visual-based piano music transcription method that combines the advantages of both audio and computer vision. The method performs note onset estimation from the pianist’s hands movements in the piano keyboard captured by a camera. This visual information is then used to enhance AMT by replacing the audio signal with a computed one and extracting the piano music from the video. The proposed system achieves a high F-measure over 90% in AMT with a small number of training examples, and its accuracies are comparable to those obtained with the state-of-the-art audio only systems.
The AMT is performed using a deep learning-based model that constructs a multi-layer network to realize end-to-end training. In order to obtain the visual information as input, the model uses a special spectral envelope matched filter on multiple frequency bands. Then the detection results are stitched together to form a single video. This approach allows for a number of applications. The most obvious use case is to transcribe silent piano videos, which have potential for both entertainment and education.
Multi Tone Piano Transcription Based on Visualization
A key challenge in piano transcription is to accurately capture articulation and vibrato of the fingers. A visual representation of the piano timbre is an efficient tool to address this issue. However, the accuracy of capturing such features is limited by the quality of the piano recording.
Scientific computing visualization is a technique that can transform piles of disorganized data into intuitive geometric images through the use of different computer methods. This can make it easier to understand and analyze the data. The purpose of this study is to develop a method that can improve the quality of piano recordings by combining scientific visualization technology with knowledge recognition algorithms.
This work is based on a recurrent neural network (RNN)-based model. It is a type of model that learns to predict a given sequence of inputs by repeatedly recursively applying the RNN to its own outputs. It is able to model correlations between stream-level data, which is a more accurate way of representing music than frame-based models. In addition, the model can also be trained to regress on the chord progressions, which is important in the piano context. The results from this work show that the proposed method outperforms the GBS and thresholding post-processing in terms of recall and precision, while achieving a higher F-measure on all metrics. Moreover, the proposed algorithm has the ability to predict notes that do not exist in the original data, which can alleviate the accumulated prediction errors in GBS that result in bad transcription performances.