In our technologically advanced times, when machine translation becomes qualitatively more successful and quantitatively more productive, translation technologies gain more and more trust not only of the people in need of translation, but of interpreters and translators as well. Taking into account the numerous advantages of producing translations without human boundaries, MT is becoming a reliable solution to the challenges of a constantly rising demand in quick and cheap translations. This paper focuses on the assessment of Speech-to-Speech translation apps, which bring together all of the state-of-the-art translation technology, namely, Automatic Speech Recognition, Machine Translation and Text-to-Speech Synthesis. The aim of this paper is to encompass all measurable components of a successful translation and thereby asses the overall quality of translations of conversational language produced by the S2S translation app ILA. This multi-layered research consist of a quality assessment, light-post editing, an automated translation metrics and a comparison of machine and human produced translations. Moreover, the translation output is assessed with respect to the language pair and the situation in order to gain perspective on how they affect the translation output. The given results indicate a high quality of translations produced by the S2S translation app, showing the large potential of speech technologies. Additionally, this research is setting ground for further research in this field.